Title: SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Out-of-distribution detection

CLIP

low-rank approximation

trustworthy AI
Abstract: Out-of-distribution (OOD) detection is crucial for the safe deployment of neural networks. Existing CLIP-based approaches perform OOD detection by devising novel scoring functions or sophisticated fine-tuning methods. In this work, we propose SeTAR, a novel, training-free OOD detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection via post-hoc modification of the model's weight matrices using a simple greedy search algorithm. Based on SeTAR, we further propose SeTAR+FT, a fine-tuning extension optimizing model performance for OOD detection tasks. Extensive evaluations on ImageNet1K and Pascal-VOC benchmarks show SeTAR's superior performance, reducing the relatively false positive rate by up to 18.95\% and 36.80\% compared to zero-shot and fine-tuning baselines. Ablation studies further validate our approach's effectiveness, robustness, and generalizability across different model backbones. Our work offers a scalable, efficient solution for OOD detection, setting a new state-of-the-art in this area.

SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation

Abstract

Out-of-distribution (OOD) detection is crucial for the safe deployment of neural networks. Existing CLIP-based approaches perform OOD detection by devising novel scoring functions or sophisticated fine-tuning methods. In this work, we propose SeTAR, a novel, training-free OOD detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection via post-hoc modification of the model's weight matrices using a simple greedy search algorithm. Based on SeTAR, we further propose SeTAR+FT, a fine-tuning extension optimizing model performance for OOD detection tasks. Extensive evaluations on ImageNet1K and Pascal-VOC benchmarks show SeTAR's superior performance, reducing the relatively false positive rate by up to 18.95% and 36.80% compared to zero-shot and fine-tuning baselines. Ablation studies further validate SeTAR's effectiveness, robustness, and generalizability across different model backbones. Our work offers a scalable, efficient solution for OOD detection, setting a new state-of-the-art in this area.

1 Introduction

The task of out-of-distribution (OOD) detection (Hendrycks & Gimpel, 2017; Ming et al., 2022) aims to identify whether input data comes from an unknown distribution. It has garnered significant attention in the machine learning community recently (Hendrycks et al., 2020; Xu et al., 2021; Miyai et al., 2023a). While machine learning models are trained with supervised in-distribution (ID) data, they often struggle to generalize to OOD data encountered in real-world applications (Emmott et al., 2016) like autonomous vehicles and healthcare. These OOD samples pose challenges as they are not represented in the training data. Consequently, OOD detection plays a crucial role in developing reliable and trustworthy machine-learning models suitable for real-world deployment (Bai et al., 2023). It allows models to filter out and reject these awkward inputs effectively, and enables the use of curated and labeled OOD samples to further train for a more robust model in the wild.

Previous research has primarily focused on detecting OOD instances in either visual (DeVries & Taylor, 2018; Liang et al., 2018; Hendrycks et al., 2022) or textual data (Hu & Khan, 2021; Zheng et al., 2020; Zhou et al., 2021). Recently, significant progress has been made in multimodal tasks like multimodal retrieval (Li et al., 2023; Caesar et al., 2018) and image classification (Yu et al., 2022), thanks to vision-and-language pretrained (VLP) models like CLIP (Radford et al., 2021). More recent

studies have explored OOD detection with CLIP, grouped into zero-shot methods (Fort et al., 2021; Ming et al., 2022; Miyai et al., 2023b) and finetuning-based methods (Ming & Li, 2023; Tao et al., 2023; Miyai et al., 2023a). However, the zero-shot methods suffer from suboptimal performance due to potential domain gaps with ID downstream data. On the other hand, finetuning-based methods carry the risk of deconstructing the intricate representations learned by CLIP which requires a meticulously designed training strategy. Sparsification-based approaches (Sun et al., 2021; Djurisic et al., 2023) have demonstrated potential in OOD detection within CNNs, leveraging the assumption that ID and OOD samples produce distinct activation patterns. Nevertheless, their effectiveness diminishes in large-scale pre-trained models such as CLIP, where activation differences become more subtle, thereby limiting their applicability primarily to models fine-tuned on downstream ID-domain datasets.

In this work, we propose SeTAR, a training-free and effective OOD detection method by selective low-rank approximations. Low-rank approximation is to approximate a given matrix by finding a lower-rank matrix that closely resembles the original matrix. Previous research has demonstrated that using low-rank approximation matrices can achieve comparable performance to full parameters in various scenarios, as observed in tasks such as large language model (LLM) fine-tuning (Hu et al., 2022) and model pruning (Hajimolahoseini et al., 2021). These approaches typically preserve the same rank across different low-rank approximation matrices. In our work, we demonstrate that it is possible to significantly enhance the performance of OOD detection by selectively manipulating the weight matrices in the CLIP model, including the choice of weight matrices and the ratio of singular vectors to be reduced. Specifically, we propose a simple top-to-bottom and image-to-text greedy search algorithm to manipulate W up in the CLIP model. Our method applies to various model backbones and does not require any additional training or new parameters. Building upon SeTAR , we further demonstrate its effectiveness for fine-tuning initialization, referred to as SeTAR+FT.

We conduct extensive evaluations and achieve state-of-the-art performance on common OOD detection benchmarks for CLIP, including the ImageNet1K and Pascal-VOC benchmarks. Compared to vanilla MCM and GL-MCM, SeTAR with the CLIP backbone reduces relatively FPR95 by 9.5% and 12.0% on average across two benchmarks, respectively. When further integrate fine-tuning into SeTAR, SeTAR+FT outperforms the state-of-the-art fine-tuning baselines LoCoOp (Miyai et al., 2023a) and LoRA (Hu et al., 2022). Moreover, we perform a comprehensive ablation study and analysis to verify and understand SeTAR. In summary, our key results and contributions:

We propose SeTAR, a simple yet effective OOD detection method based on selective lowrank approximation. It is training-free as it only performs post-hoc modification to weight matrices. SeTAR applies to a variety of scoring functions and model backbones. It can be readily integrated with existing zero-shot OOD detection methods.

We further extend SeTAR to SeTAR+FT, which demonstrates the effectiveness of SeTAR in improving the performance of finetuning-based OOD detection methods and achieving new state-of-the-art results.

We extensively evaluate SeTAR and SeTAR+FT across a diverse set of OOD detection tasks. It consistently outperforms baseline methods and establishes new state-of-the-art results on CLIP-based OOD detection benchmarks. On ImageNet1K, SeTAR achieves an AUROC of 91.32% with CLIP backbone and GL-MCM score. The score further increases to 92.31% when combined with the finetuning-based detection method.

We perform comprehensive ablation studies and empirical analyses to verify and understand SeTAR. We hope that this work will shed light on future explorations on in-depth understanding of the SeTAR method. 3

2 Preliminaries

CLIP Architecture The CLIP model (Radford et al., 2021) comprises an image encoder E v ( · ) and a text encoder E t ( · ) , aligned via contrastive learning on web-scale image-text pairs. We focus on CLIP-ViT, where the image encoder is a Vision Transformer (ViT). Each ViT layer includes a multihead self-attention sublayer and a feed-forward sublayer. In the self-attention module, the hidden state is projected into different spaces using learnable parameters W q , W k , W v . The outputs are

LayerNorm

Multi-head

Attention

LayerNorm

+

Patch/Token

Embedding

FeedForward

+

𝑁

×

Up Projection

W

୳୮

W

ୢ୭୵୬

Down Projection

Activation Function

=

(a)

(b)

(c)

(d)

W

୳୮

U

Σ

V

୘

=

W

෡

୳୮

U

′

Σ′

V

ᇱ୘

Σ

Σ′

Low-rank

Approximation

concatenated and projected back with another linear matrix W o . The feed-forward module projects the hidden state into a wider space using W up and then back with W down after a non-linear activation (Figure 1). Given the similarity between the image and text encoder layers, we adopt consistent notations for the linear matrices in both. Each encoder also includes a linear projector W p to map their representations into a shared space for contrastive learning.

Zero-shot OOD Detection with CLIP Zero-shot OOD detection aims to separate ID and OOD data without an ID training dataset. Given the CLIP, the ID classes are defined by the classification task of interest, which differs from the classes used in CLIP pretraining. Accordingly, OOD is defined as classes not belonging to any ID class, making the OOD detector a binary classifier. MCM (Ming et al., 2022) and GL-MLM (Miyai et al., 2023b) are two zero-shot CLIP-based OOD detection methods. Formally, let x be the test image and T in = { y c } K c =1 be the set of text prompts containing M ID class labels (e.g., "a photo of a [CLASS]"). The image is segmented into l image patches x = ( x 1 , ..., x l ) . Following CLIP, we add [cls] before the image patches and use the output of [cls] from the visual projector W p as the global image feature ( h v ∈ R d ). The outputs of other patches are projected by the visual projector as the local image features ( p v = ( p v 1 , ..., p v l ) ∈ R l × d ). For the text prompt y c ∈ T in, we add an additional [eos] after the text tokens and use the output feature of [eos] from the textual projector W p as the concept feature of ID class c ( h t c ∈ R d ).

The label-wise image-concept matching (IWIC) score measures how well a test image x aligns with a concept y c , using either global or local features. The global IWIC score s G c ( · ) is the cosine similarity between the global image feature h v and the concept feature h t c : s G c ( x ) = cos _ sim( h v , h t c ) . The local IWIC score s L c ( · ) is the max-pooled cosine similarity between image patch features p v i and the concept feature h t c : s L c ( x ) = max i cos _ sim( p v i , h t c ) . The MCM and GL-MCM scores are defined as:

where τ and τ ′ are the temperature hyperparameters. MCM only uses global image features, while GL-MCM additionally considers local image features. For ID data, both MCM and GL-MCM scores will be matched to one of the concept features with a high score; and vice versa. As a result, our OOD detection function can be formulated as:

where S ( x ) is either the MCM or GL-MCM score, λ is the threshold value. By convention, G ( x ) = 1 represents the ID class and G ( x ) = 0 indicates the OOD class. The λ is chosen so that a high fraction of ID data (e.g., 95%) is above the threshold. We follow previous work (Miyai et al., 2023a) to use either MCM or GL-MCM score for OOD detection in this work.

3 Method

We introduce SeTAR, a training-free and effective technique for improving OOD detection performance (see Figure 1). Our key idea is to perform post-hoc modification to CLIP weight matrices by selectively replacing them with their low-rank approximations. It is complementary to existing CLIP-based zero-shot OOD detection methods and could be further extended to finetuning-based methods, which we term as SeTAR+FT.

3.1 OODDetection with Selective Low-Rank Approximation

Low-Rank Approximation Given a linear matrix W ∈ R m × n , its Singular Value Decomposition (SVD) is denoted as W = U Σ V ⊤ , where U = [ u 1 , u 2 , · · · , u m ] ∈ R m × m , V = [ v 1 , v 2 , · · · , v n ] ∈ R n × n , and Σ ∈ R m × n is a matrix whose entries are all zero except for the singular values of W . These singular values appear in decreasing order on the diagonal (i.e. σ ↓ i ( W ) ). The SVD of W can be reformulated as in Equation 4. Given a hyperparameter r ∈ N + , a rankr approximation of W is matrix ̂ W that minimizes ∥ W -̂ W ∥ 2 and satisfies rank ( ̂ W ) ≤ r . The optimal solution of this problem ̂ W is provided by Eckart-Young-Mirsky theorem (Low-Rank Approximation, 2024) using Singular Value Decomposition (see Equation 5).

In this work, we will use the term minor singular components to refer to entries in the SVD corresponding to small singular values. These components are removed in low-rank approximation. The term of principle singular components is used to refer to entries in the SVD corresponding to large singular values. These components are kept in a low-rank approximation of the matrix.

OODDetection with Selective Low-Rank Approximation SVD-based weight pruning, particularly in noise-prone layers, can substantially reduce a network's sensitivity to minor perturbations, leading to enhanced stability and robustness (Yao et al., 2024). This stability is crucial for OOD detection, as it ensures the model's reliable performance across a wide range of inputs. Building on this, we propose a method to improve OOD detection by selectively applying low-rank approximation to weight matrices. By decomposing a weight matrix W into its singular values and vectors, we can identify and retain the principle singular components that significantly contribute to the model's performance. This approach ensures that the essential features of W are preserved while discarding the less critical minor singular components. Given a weight matrix W in CLIP (e.g., W up or W k ), we replace the matrix with its low-rank approximation part ̂ W as described in Equation 5 (see Figure 1). Given the rank reduction ratio Θ , the rank of ̂ W is determined by r ( ̂ W ) = round ((1 -Θ) · r ( W )) . This selective low-rank approximation leverages the compact representation provided by SVD to enhance the model's ability to detect OOD instances effectively without requiring additional training. We demonstrate our method's ability to improve OOD detection (Table 1) while maintaining ID classification performance (Table 7) in Section 4.2 and Section 4.5.

HyperParameter Search Algorithm Due to the presence of many weight matrices in CLIP, each consisting of hundreds of singular values, conducting a complete search over all combinations of low-rank approximation weight matrices is impractical. Therefore, we propose a greedy search algorithm to determine the rank reduction ratio for each weight matrix. Among all linear weight matrices in each encoder layer, we focus on W up as it is most effective according to our preliminary experiment. For simplicity, we assume both image and text encoders have N encoder layers. As shown in Algorithm 1, we search by first enumerating all N vision encoder layers sequentially from top to bottom and then all N text encoder layers in the same way. This search order is concisely denoted as searching from 2 N to the first layer in CLIP. We compare different search algorithms in Section 4.4. The rank reduction ratio for each layer is the objective in SeTAR which is searched among the candidate list Θ = { Θ 0 , Θ 1 , · · · , Θ J } according to the loss on the validation set. We employ the LoCoOp loss (Equation 12) proposed in (Miyai et al., 2023a) as our loss

function. This loss requires only ID images. It contains an ID loss for ID image classification and an OOD loss to push away pseudo OOD features from the ID class text embeddings where the pseudo OOD features are from ID-irrelevant nuisances (Equation 10) (e.g., backgrounds) in CLIP's local features. We refer the readers to Miyai et al. (2023a) or Appendix B for more details.

For Θ j ∈ Θ , we remove Θ j (in percent) singular values along with their corresponding singular vectors to obtain the approximated matrix ̂ W up (Equation 5). It is worth noting that the rank reduction raio candidate list includes Θ 0 = 0 , indicating that the weight matrix has the chance to remain unmodified.

Algorithm 1 The hyperparameter search in SeTAR.

With the searched rank reduction ratio, the weight matrix W up in each CLIP layer is replaced and updated with its approximation. The SeTAR can be easily applied to different ViT backbones (Table 8), by replacing the model weight matrices with their low-rank approximations in a similar approach. Then SeTAR detects the OOD data samples following MCM (Equation 1), GL-MCM (Equation 2) or other scoring-based OOD detection method with the approximated model. We provide an example procedure of the greedy search in Listing 1 for better understanding.

3.2 OODDetection with SeTAR-enhanced Low-rank Adaptation

SeTAR can be further combined with LoRA (Hu et al., 2022) as a novel low-rank adaptation method for OOD detection, which we refer to as SeTAR+FT . Specifically, we first apply SeTAR to the pre-trained CLIP model to obtain the reserved rank r for each weight matrix W . Then we have

where ̂ W is the low-rank approximation of W found by SeTAR , with A and B being the minor singular components. During finetuning, we keep ̂ W frozen and only update the low-rank matrix A and B . In this way, we retain the principle singular components in the original weight matrix and only update the minor singular components.Unlike LoRA, which evenly distributes the finetuning rank budget across all layers, SeTAR+FT adjusts the rank for each layer, resulting in more effective and efficient fine-tuning (Table 2 and Figure 6). More details are provided in Section 4.3.

4 Experiments

4.1 Experimental Setup

Dataset Following previous work (Ming et al., 2022; Miyai et al., 2023b), we use two real-world datasets created from ImageNet1K (Deng et al., 2009) and Pascal-VOC (Everingham et al., 2009) as the ID datasets. For OOD datasets, we follow Ming et al. (2022) to preprocess iNaturalist, SUN, Places and Texture, and follow Miyai et al. (2023b) to preprocess ImageNet22K and COCO data. For finetune experiments, we follow Miyai et al. (2023a) to use ImageNet1K as the ID dataset. The detailed description and statistics of the datasets are provided in Appendix C.

Settings Following existing studies (Ming et al., 2022; Miyai et al., 2023b,a), we use CLIP ViTB/16 4 (Radford et al., 2021) as our backbone. Both image and text encoders have 12 layers. More results with different backbones are in Section 4.4. The rank reduction ratio candidates range from 0 to 40% in 5% intervals. We use a temperature of 1 5 , unless stated otherwise. In all experiments, we use one CLIP text prompt: "a photo of a [CLASS],", where [CLASS] is the ID class name. We set hyperparameters λ (Equation 12) and top-K (Equation 10) according to the specific ID datasets and backbones. Detailed settings are in Table 12, with a sensitivity analysis in Section 4.4. For SeTAR+FT and LoRA experiments, the learning rate and epoch number are set to 1 e -2 and 5 for all experiments. The LoRA rank r is set to match the trainable parameters of SeTAR+FT. Detailed settings are in Table 13. We report results from three runs with seeds 3, 4, 5 6 . All experiments are conducted on a single NVIDIA RTX 4090 GPU. The time cost for low-rank approximation with CLIP-base on the ImageNet1K validation set is about 20 minutes.

Metrics We use the following metrics for evaluation. (1) the false positive rate (FPR95) for out-ofdistribution (OOD) samples at a fixed true positive rate (TPR) of 95% for in-distribution samples, with lower values targeting better performance; and (2) the area under the receiver operating characteristic curve (AUROC) for OOD samples, with higher values indicating better performance.

Baselines We evaluate SeTAR against MCM (Ming et al., 2022) and GL-MCM (Miyai et al., 2023b), state-of-the-art zero-shot OOD detection methods on CLIP. We also compare SeTAR+FT with fine-tuning baselines NPOS (Tao et al., 2023), CoOp (Zhou et al., 2022), LoCoOp (Miyai et al., 2023a), and LoRA (Hu et al., 2022). More details are in Appendix D.

4.2 Training-free Results

The training-free OOD detection performances are summarized in Table 1. Compared with zero-shot baselines, a salient observation is that on both MCM and GL-MCM, using SeTAR outperforms the vanilla method by a large margin across all OOD detection tasks. For example, using Pascal-VOC as ID, SeTAR yields a relatively average reduction of 12.84% FPR95 on MCM and 18.95% FPR95 on GL-MCM. Considering that SeTAR is generally applicable and training-free, these results are very encouraging. Comparing SeTAR with scoring function MCM and GL-MCM, SeTAR+GL-MCM performs better on all OOD detection tasks. However, the superiority of GL-MCM score over MCMappears to be contingent upon the choice of the model backbone. As evidenced in Table 8, SeTAR+MCM demonstrates superior performance with a relatively average FPR95 reduction of 8.30% compared to SeTAR+GL-MCM with CLIP-large as the backbone on ImageNet1K.

4.3 Fine-tuning Results

In this section, we compare SeTAR+FT with fine-tuning baselines, including NPOS (Tao et al., 2023), CoOp (Zhou et al., 2022), LoCoOp (Miyai et al., 2023a) and LoRA (Hu et al., 2022). LoCoOp is the state-of-the-art prompt-learning OOD detection method on CLIP. LoRA is a representative parameter-efficient fine-tuning method. Following previous work (Tao et al., 2023; Zhou et al., 2022; Miyai et al., 2023a), we report the results on the the ImageNet1K benchmark in Table 2. We observe that SeTAR+FT outperforms all baselines on both MCM and GL-MCM scoring functions. For example, with CLIP-base as the backbone, SeTAR+FT achieves a relatively average FPR95 reduction of 3.97% and 6.67% compared to LoCoOp and LoRA. Moreover, when scaled up to CLIP-large, SeTAR+FT outperforms LoCoOp and LoRA by relatively 17.92% and 12.45% FPR95 on the same benchmark. Similar results are observed on Swin Transformer (Liu et al., 2021), where SeTAR+FT outperforms LoRA by relatively 17.36% and 36.80% FPR95 on MSP and Energy scoring functions, respectively. The larger improvement on Swin Transformer may stem from its reliance on ImageNet training, making it prone to overfitting and weaker at OOD detection. Our method mitigates these issues, enhancing Swin's generalization to OOD instances. These results demonstrate the effectiveness and scalability of SeTAR+FT in improving the OOD detection performance.

Furthermore, as shown in Figure 6, SeTAR+FT demonstrates faster convergence and lower loss than LoRA, especially in OOD loss, indicating that SeTAR+FT is more effective in adapting the pre-trained weights to the OOD detection task.

4.4 Ablation Study

In this section, we conduct ablation studies with CLIP-base to understand our design choices.

Image v.s. Text modality Table 3 shows an ablation study on the modality involved in SeTAR.

As shown, the vision modality outperforms the text modality, indicating the vision modality is more dominant in enhancing the model's performance. When considering the vision modality alone and the combined vision+text modality, the latter either outperforms or achieves comparable average results to the former. Consequently, we make modifications to both the vision and text modalities in SeTAR to enhance overall performance.

Different Weight Types In this part, we present empirical evidence for modifying W up. We first compare the performance of SeTAR with different types of weight matrix in each Transformer layer, including W q , W k , W v , W o , W up and W down. As shown in Figure 2 and Figure 3 of Appendx F, the X -axis denotes the number of weight matrixes (layers) that we have searched, while the Y -axis is the average AUROC and FPR95. The results show that W up

consistently outperforms other weight matrices in terms of both AUROC and FPR95. In addition to weight matrics in each transformer layer, CLIP has one projection matrix W p on top of each encoder,

which serves to project image/text representations into a shared space. In Table 4, we compare the performance of SeTAR with and without modifying W p . We search W p first right before searching the image/text encoder. The results show that frozen W p brings a relatively reduction of 4 . 20% FPR95. Consequently, we keep W p frozen in SeTAR.

Different Search Algorithms At each step of the greedy search, SeTAR traverses the subsequent W up in a predefined order and searches over different thresholds. We compare our method with two alternatives: modality-interleaved greedy search (MIS) and layer-exhaustive search (LES). MIS searches the image and text layers in an interleaved manner, while LES simultaneously searches over both layers and thresholds at each step. SeTAR-S, has linear complexity with respect to the number of model layers, similar to MIS, whereas LES has quadratic complexity.

Table 5 presents the comparison results. SeTARS demonstrates better overall performance than MIS. Notably, MIS encounters limitations when the image and text towers have different layer counts (e.g., CLIP-large with 24 image layers and 12 text layers). Therefore, we choose SeTAR-S for better generalization. Compared to LES, SeTAR-S performs better in terms of both FPR95 and AUROC, as LES's locally optimal algorithm may not achieve a global optimal solution. These results validate the superiority of our top-to-bottom layer search strategy.

Different Prune Strategies Inspired from SVD, SeTAR modify the model weights by pruning the minor singular components, and retains the principle components that contribute the most to the model's performance. To validate this design, we compare SeTAR with two alternatives: principal component pruning and random pruning pruning. Principal component takes the opposite approach, retaining minor components and pruning major ones. Random pruning, on the other hand, prunes weights randomly. As shown in Table 6, principle pruning suffers from a significant performance drop compared to SeTAR , while random pruning performs slightly better than principle pruning. These results demonstrate the effectiveness of SeTAR 's design choice in pruning the minor components.

Sensitivity Analysis on λ and top-K In this section, we present the sensitivity analysis of the hyperparameters λ (Figure 4) and top-K (Figure 5). As observed in Figure 4, the average AUROC remains stable at lower values and slightly decreases as λ increases for both SeTAR+MCM and SeTAR+GL-MCM. Notably, the optimal setting of λ may vary depending on the model backbone, with our experiments indicating that CLIP-large may require a larger λ than CLIP-base. Despite this variation, the λ parameter demonstrates strong transferability across datasets for the same backbone. Swapping the optimal λ between ImageNet1K and Pascal-VOC has a minimal performance impact, consistently outperforming the vanilla method. With the VOC-optimized λ on ImageNet1K, CLIPbase achieves an FPR95 of 40.91 and AUROC of 91.02, and CLIP-large reaches 46.73 FPR95 and 91.81 AUROC. Conversely, using the ImageNet1K-optimized λ on Pascal-VOC, CLIP-base achieves 33.18 FPR95 and 93.65 AUROC, while CLIP-large attains 44.39 FPR95 and 92.3 AUROC.

Top-K controls the number of OOD regions considered in LoCoOp loss: higher values include more OOD regions, with top-K equal to the number of ID classes covering all OOD regions, and top-K set to 0 focusing solely on ID loss. The optimal top-K depends on the number of ID categories, making it non-transferable across datasets. However, SeTAR remains robust to top-K variations, as shown in Figure 5, except at extreme values (0 or the maximum number of classes). We recommend setting top-K to around 30% of the total categories, such as 300 for ImageNet1K and 4 for Pascal-VOC. For the Swin-base model, top-K at 300 on ImageNet1K yields an FPR95 of 56.82 and AUROC of 85.68 with MSP, and an FPR95 of 52.56 and AUROC of 84.51 with Energy.

4.5 Analyses

Can SeTAR Improve Image Classification?

To evaluate the impact of SeTAR and SeTAR+FT on classification accuracy, we present our results on ID dataset ImageNet1K and OOD datasets SUN, Places and Texture in Table 7 7 . SeTAR effectively maintains the average accuracy, with minor variations observed across different datasets. Among the fine-tuned baselines, LoCoOp exhibits a 1% decrease in accuracy compared to Vanilla CLIP, whereas LoRA

shows an improvement of 0.94%. Notably, SeTAR+FT surpasses both baselines, improving the average accuracy by 1.45% compared to Vanilla CLIP. These results highlight the efficacy of SeTAR and SeTAR+FT in improving OOD detection without compromising classification accuracy.

SeTAR is Effective on Different Architectures and Score Functions We expand on Table 1 with results on ViT and CNN backbones and various score functions. For ViT-based models, we evaluate OOD detection using CLIP-large 8 and Swin Transformer 9 (Liu et al., 2021), alongside CLIP-base. The Swin Transformer (Liu et al., 2022) is trained on ImageNet1K. Since it lacks a text encoder, we apply SeTAR to the image ViT only. For Swin Transformer, we use two common scoring functions: MSP (Hendrycks & Gimpel, 2017), which leverages softmax confidence, and the Energy score (Liu et al., 2020), with T = 0 . 1 for OOD detection. We also integrate CLIP-base with the NegLabel score function (Jiang et al., 2024), which uses large-scale negative labels. As shown in Table 8, SeTAR consistently outperforms baselines across all backbones and scoring functions, significantly reducing FPR95 by relatively 20.61% with the Energy score on Swin Transformer. These results demonstrate SeTAR 's effectiveness in improving OOD detection for unimodal image encoders, with further confirmation from SeTAR+FT results (Table 2) across different model backbones.

We further explore SeTAR's potential on CNN architecture, and compare it with methods such as Softmax, Energy (Wu et al., 2023), ReAct (Sun et al., 2021), DICE (Sun & Li, 2022), and ASH (Djurisic et al., 2023) on ResNet50 10 . Since ResNet lacks local features for OOD loss, we conduct experiments using only ID loss. We apply low-rank approximation to the in- and outfeature dimensions of the convolutional layers,

combined with ASH for search. As shown in Table 9, SeTAR establishes new state-of-the-art results on ResNet, demonstrating its effectiveness across both ViT and CNN architectures.

Near-OOD Results To further evaluate SeTAR's performance on diverse OOD tasks, we test it on a more challenging near-OOD setting using ImageNet1K as the ID dataset and SSBHard (Vaze et al., 2022) as the OOD dataset. As shown in Table 10, SeTAR and SeTAR+FT outperform the baselines, demonstrating superior performance in near-OOD scenarios.

5 Related Work

Out-of-Distribution Detection Previous work explores OOD detection with unimodal (DeVries & Taylor, 2018; Hendrycks & Gimpel, 2017; Hu & Khan, 2021; Zheng et al., 2020; Zhou et al., 2021) and multimodal (Fort et al., 2021; Ming et al., 2022; Tao et al., 2023; Miyai et al., 2023a) models. Numerous methodologies (Lee et al., 2018; Huang et al., 2021; Sun et al., 2022; Wang et al., 2022; Wu et al., 2023) have been developed to tackle OOD detection in computer vision. Existing CLIP-based OOD detection methods include zero-shot (Fort et al., 2021; Ming et al., 2022; Miyai et al., 2023b; Dai et al., 2023; Wang et al., 2023; Jiang et al., 2024) and fine-tuning (Ming & Li, 2023; Tao et al., 2023; Miyai et al., 2023a). Zero-shot methods like MCM (Ming et al., 2022) and GL-MCM (Miyai et al., 2023b) don't require in-distribution training data but may perform suboptimally due to domain gaps. Other approaches integrate external knowledge. For example, CLIPN (Wang et al., 2023) pre-trains a novel NO-encoder on the CC-3M dataset (Sharma et al., 2018) to empower CLIP's "no" logic for zero-shot evaluation. NegLabel (Jiang et al., 2024) demonstrates better performance than CLIPN by introducing large-scale negative labels for enhanced label scoring. Fine-tuning methods (Ming & Li, 2023; Tao et al., 2023; Miyai et al., 2023a) improve OOD detection by adapting to in-distribution data but risk damaging the pretraining representations, needing careful training strategies. CNN-based OOD detection methods, including ReAct (Sun et al., 2021), ASH (Djurisic et al., 2023), DICE (Sun & Li, 2022), CIDER (Ming et al., 2023), PALM (Lu et al., 2024), and Hopfield Boosting (Hofmann et al., 2024), have also demonstrated strong results. However, methods like ReAct and ASH rely on the assumption that ID and OOD images produce distinct activations in models trained on ID data. This assumption does not hold in large-scale pre-trained models like CLIP, where activations for ID and OOD images are not significantly different, limiting the effectiveness of such approaches in enhancing CLIP's zero-shot OOD detection capabilities. SeTAR, in contrast, offers high compatibility with various scoring functions (e.g., MCM, GL-MCM, MSP, Energy), multiple model backbones (e.g., CLIP, Swin, ResNet), and advanced OOD techniques such as NegLabel. Designed to be both lightweight and efficient, SeTAR addresses the demand for resource-efficient solutions in OOD detection.

Low-rank Approximations of Weight Matrices Neural networks trained with overparameterization often exhibit low-rank properties (Oymak et al., 2019). These properties are utilized in both model training (Povey et al., 2018; Hu et al., 2022) and post-hoc processing (Hajimolahoseini et al., 2021; Sharma et al., 2023). In training, some works (Sainath et al., 2013; Zhang et al., 2014; Zhao et al., 2016) impose low-rank constraints, while LoRA (Hu et al., 2022) adapts pretrained LLMs to downstream tasks using trainable low-rank matrices. For post-hoc processing, pruning methods (Yu et al., 2017; Hajimolahoseini et al., 2021) reduce weight matrix ranks by retaining top-K components from SVD. While pruning preserves model behavior, performance declines with increased intervention. LASER (Sharma et al., 2023) focuses on pruning individual layers to enhance factual answering capabilities. It utilizes a simple greedy search strategy on a validation set, which is not applicable for OOD detection due to the absence of a validation set. In contrast, our approach introduces a selective rank reduction strategy specifically tailored for OOD detection. We systematically analyze and compare different greedy search techniques, evaluating their effectiveness across various layers and model backbones.

6 Conclusion

We propose SeTAR , a simple and effective OOD detection method using post-hoc low-rank approximation on weight matrices W up with a top-down, image-to-text greedy search. SeTAR offers several advantages: (1) training-free, (2) scalable to unimodal and multimodal models, and (3) complementary to existing OOD scoring functions. Building on SeTAR , we introduce SeTAR-FT, a finetuning method that adapts the model to in-distribution data for improved OOD detection. We evaluate SeTAR and SeTAR-FT on large-scale benchmarks, including ImageNet1K and Pascal-VOC. Results show that both achieve state-of-the-art OOD detection performance. We hope our work inspires further research and contributes to more robust and reliable models.

References

Bai, H., Canal, G., Du, X., Kwon, J., Nowak, R. D., and Li, Y. Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. In International Conference on Machine Learning , 2023.

Caesar, H., Uijlings, J., and Ferrari, V . Coco-stuff: Thing and stuff classes in context. In CVPR , 2018.

Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR , 2014.

Dai, Y., Lang, H., Zeng, K., Huang, F., and Li, Y. Exploring large language models for multi-modal out-of-distribution detection. ArXiv , abs/2310.08027, 2023. URL https://api. semanticscholar.org/CorpusID:263909127 .

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

DeVries, T. and Taylor, G. W. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint:1802.04865 , 2018.

Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely simple activation shaping for outof-distribution detection. In The Eleventh International Cosun2021reactnference on Learning Representations , 2023. URL https://openreview.net/forum?id=ndYXTEL6cZz .

Emmott, A., Das, S., Dietterich, T., Fern, A., and Wong, W.-K. A meta-analysis of the anomaly detection problem. arXiv preprint:1503.01158 , 2016.

Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. IJCV , 88:303-308, 2009.

Fort, S., Ren, J., and Lakshminarayanan, B. Exploring the limits of out-of-distribution detection. In NeurIPS , 2021.

Hajimolahoseini, H., Rezagholizadeh, M., Partovinia, V., Tahaei, M. S., Awad, O. M., and Liu, Y. Compressing pre-trained language models using progressive low rank decomposition. In NeurIPS , 2021.

Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , 2017.

Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., and Song, D. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 2744-2751, Online, July 2020.

Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-distribution detection for real-world settings. In ICML , 2022.

Hofmann, C., Schmid, S., Lehner, B., Klotz, D., and Hochreiter, S. Energy-based hopfield boosting for out-of-distribution detection, 2024. URL https://arxiv.org/abs/2405.08766 .

Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR , 2022.

Hu, Y. and Khan, L. Uncertainty-aware reliable text classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pp. 628-636, New York, NY, USA, 2021.

Huang, R., Geng, A., and Li, Y. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS , 2021.

Jiang, X., Liu, F., Fang, Z., Chen, H., Liu, T., Zheng, F., and Han, B. Negative label guided OOD detection with pretrained vision-language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=xUO1HXz4an .

Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS , 2018.

Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR , 2018.

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV , 2014.

Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based out-of-distribution detection. In NeurIPS , 2020.

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021.

Liu, Z., Hu, H., Lin, Y ., Yao, Z., Xie, Z., Wei, Y ., Ning, J., Cao, Y ., Zhang, Z., Dong, L., Wei, F., and Guo, B. Swin transformer v2: Scaling up capacity and resolution, 2022.

Low-Rank Approximation. Low-rank approximation - Wikipedia, the free encyclopedia, January 2024. https://en.wikipedia.org/w/index.php?title=Low-rank_approximation& oldid=1196167027 .

Lu, H., Gong, D., Wang, S., Xue, J., Yao, L., and Moore, K. Learning with mixture of prototypes for out-of-distribution detection. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=uNkKaD3MCs .

Ming, Y. and Li, Y. How does fine-tuning impact out-of-distribution detection for visionlanguage models? International Journal of Computer Vision , 132(2):596-609, September 2023. ISSN 1573-1405. doi: 10.1007/s11263-023-01895-7. URL http://dx.doi.org/10.1007/ s11263-023-01895-7 .

Ming, Y., Cai, Z., Gu, J., Sun, Y., Li, W., and Li, Y . Delving into out-of-distribution detection with vision-language representations. In NeurIPS , 2022.

Ming, Y., Sun, Y., Dia, O., and Li, Y . How to exploit hyperspherical embeddings for out-of-distribution detection? In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=aEFaE0W5pAd .

Miyai, A., Yu, Q., Irie, G., and Aizawa, K. Locoop: Few-shot out-of-distribution detection via prompt learning. In Thirty-Seventh Conference on Neural Information Processing Systems , 2023a.

Miyai, A., Yu, Q., Irie, G., and Aizawa, K. Zero-shot in-distribution detection in multi-object settings using vision-language foundation models. arXiv preprint arXiv:2304.04521 , 2023b.

Oymak, S., Fabian, Z., Li, M., and Soltanolkotabi, M. Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian, 2019.

Povey, D., Cheng, G., Wang, Y., Li, K., Xu, H., Yarmohammadi, M. A., and Khudanpur, S. Semiorthogonal low-rank matrix factorization for deep neural networks. In Interspeech , 2018. URL https://api.semanticscholar.org/CorpusID:4949673 .

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. IJCV , 115(3):211-252, 2015.

Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing , pp. 6655-6659. IEEE, 2013.

Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL , 2018.

Sharma, P., Ash, J. T., and Misra, D. The truth is in there: Improving reasoning in language models with layer-selective rank reduction, 2023.

Sun, Y. and Li, Y. Dice: Leveraging sparsification for out-of-distribution detection. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , pp. 691-708, Berlin, Heidelberg, 2022. Springer-Verlag. ISBN 978-3-031-20052-6. doi: 10.1007/978-3-031-20053-3_40. URL https://doi.org/10.1007/978-3-031-20053-3_ 40 .

Sun, Y., Guo, C., and Li, Y . React: Out-of-distribution detection with rectified activations. In NeurIPS , 2021.

Sun, Y., Ming, Y., Zhu, X., and Li, Y. Out-of-distribution detection with deep nearest neighbors. In ICML , 2022.

Tao, L., Du, X., Zhu, X., and Li, Y. Non-parametric outlier synthesis. In ICLR , 2023.

Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In CVPR , 2018.

Vaze, S., Han, K., Vedaldi, A., and Zisserman, A. Open-set recognition: A good closed-set classifier is all you need. In ICLR , 2022.

Wang, H., Liu, W., Bocchieri, A., and Li, Y. Can multi-label classification networks know what they don't know? In NeurIPS , 2021.

Wang, H., Li, Z., Feng, L., and Zhang, W. Vim: Out-of-distribution with virtual-logit matching. In CVPR , 2022.

Wang, H., Li, Y., Yao, H., and Li, X. Clipn for zero-shot ood detection: Teaching clip to say no. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 1802-1812, 2023. URL https://api.semanticscholar.org/CorpusID:261076240 .

Wu, Q., Chen, Y., Yang, C., and Yan, J. Energy-based out-of-distribution detection for graph neural networks. ArXiv , abs/2302.02914, 2023. URL https://api.semanticscholar.org/ CorpusID:256615740 .

Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR , 2010.

Xu, K., Ren, T., Zhang, S., Feng, Y., and Xiong, C. Unsupervised out-of-domain detection via pre-trained transformers. In ACL , 2021.

Yao, X., Hu, X., Yang, S., and Liu, Y. Enhancing in-context learning performance with just svd-based weight pruning: A theoretical perspective, 2024. URL https://arxiv.org/abs/2406.03768 .

Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. CoCa: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Yu, X., Liu, T., Wang, X., and Tao, D. On compressing deep models by low rank and sparse decomposition. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 67-76, 2017. URL https://api.semanticscholar.org/CorpusID:24553488 .

Zhang, Y., Chuangsuwanich, E., and Glass, J. R. Extracting deep neural network bottleneck features using low-rank matrix factorization. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 185-189, 2014. URL https://api.semanticscholar. org/CorpusID:1791734 .

Zhao, Y., Li, J., and Gong, Y . Low-rank plus diagonal adaptation for deep neural networks. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5005-5009, 2016. URL https://api.semanticscholar.org/CorpusID:10506309 .

Zheng, Y., Chen, G., and Huang, M. Out-of-domain detection for natural language understanding in dialog systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28: 1198-1209, 2020.

Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recognition. TPAMI , 40(6):1452-1464, 2017.

Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. IJCV , 2022.

Zhou, W., Liu, F., and Chen, M. Contrastive out-of-distribution detection for pretrained transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 1100-1111, Online and Punta Cana, Dominican Republic, November 2021.

A Impact Statements

Limitation While we demonstrate the effectiveness of our method on OOD detection, we acknowledge that our work has several limitations. First, despite we show the robustness of our method to hyperparameters, the optimal hyperparameters may vary across different model backbones. Future work is needed to explore the autonomous selection of hyperparameters. Second, we design SeTAR+FT in a simple and straightforward manner, which may not be the most efficient or effective way to adapt the model to the ID downstream data. More sophisticated strategies for model adaptation are worth exploring in future research. Third, we only conduct experiments to detect visual OOD inputs and ignore inputs in other modalities such as textual, audio and video. This is primarily because our model is based on CLIP. Exploring the development of OOD detectors across diverse modalities remains an active research topic for future investigation.

Ethical Considerations Our study addresses the challenge of OOD detection through low-rank approximation, which is particularly relevant for ensuring the reliability and trustworthiness of vision-and-language pre-trained models. Future investigations on fairness, privacy and transparency neural-based models should be encouraged to mitigate the existing data biases and safety problems for a responsible, helpful and trustworthy AI system in diverse real-world applications.

Future Societal Consequences Our proposed SeTAR achieves impressive OOD detection performance, which is beneficial to various real-world machine learning applications, such as healthcare and autonomous vehicles. The identification of anomalies or unexpected data points is crucial for decision-making and risk management with AI models. A better OOD detector facilitates the development of trustworthy machine-learning models that can reject unknown data inputs and help alleviate the hallucination problem. Moreover, better OOD detectors like SeTAR can help to select and label the unfamiliar data samples to further train a stronger model in the wild.

B Loss Function

To improve the model's OOD detection ability, it is crucial to define a loss function that pushes OOD samples far from ID samples while keeping ID samples close to each other. However, since OOD samples are unavailable during development, we address this issue by using the LoCoOp loss (Miyai et al., 2023a) for both SeTAR and SeTAR+FT. The main idea is to create pseudo OOD features with ID-irrelevant nuisances (e.g., backgrounds) in CLIP's local features.

Specifically, we divide the image into patches, represented by the set of all patch indices I = { 0 , 1 , 2 , . . . , H × W -1 } , where H and W denote the height and width of the patch features. Next, we compute the cosine similarity between the image patch features p v i and the text features h t c of the image label. The classification prediction probabilities for each patch i are then given by:

For a given image patch related to an ID category, the corresponding ID label should be among its top-K predictions. Conversely, for patches unrelated to the ID label, such as background regions, the ID label should be excluded from the top-K predictions. Based on this intuition, the indices of ID-irrelevant regions within an image are defined by Equation 10, where rank ( p i ( y = y | x )) denotes the rank of the true class y among all ID classes, and K is the hyperparameter.

After identifying out-of-distribution (OOD) regions, it is expected that their image features will differ significantly from the ID text embeddings. To enhance this distinction, entropy maximization is employed to increase the entropy of p j ( y | x ) , where p j denotes the classification prediction probabilities for region j ∈ J . The entropy maximization is formally defined as follows:

Here, H ( · ) represents the entropy function. The overall loss function combines the ID loss (crossentropy loss for ID predictions) with the OOD loss. Here λ is the hyperparameter that regulates the proportion of the OOD loss.

C Data

We use two real-world datasets created from ImageNet1K (Deng et al., 2009) and Pascal-VOC (Everingham et al., 2009) as the ID dataset. We use ImageNet-1K validation set as the ID test set following Ming et al. (2022), and preprocess Pascal-VOC following Miyai et al. (2023b). we build two ID validation sets for low-rank approximation. The ID validation set of ImageNet1K is collected by sampling one image for each label from the ImageNet1K training set. For Pascal-VOC, For Pascal-VOC, We randomly sample 10% images as the ID validation set and leave the rest as the ID test set.

For OOD datasets, we follow Ming et al. (2022) to preprocess iNaturalist, SUN, Places and Texture, and follow Miyai et al. (2023b) to preprocess ImageNet22K and COCO data. We only evaluate the OOD datasets that have no overlapping categories as the ID dataset.

We provide more details about the datasets used in our experiments, in terms of data sources, preprocessing, and the statistics for each dataset, as shown in Table 11 and below.

ImageNet1K We use the ImageNet-1000 (ILSVRC2012) (Deng et al., 2009) dataset for ID validation and testing. The original dataset contains 1.2 million training images and 50,000 validation images from 1000 classes, and is widely used for image classification. We follow Ming et al. (2022) to construct the ImageNet1K ID test set from the validation set. Additionally, we curate an ImageNet1K ID validation set from the training set by randomly selecting one image for each label.

Pascal-VOC The Pascal VOC (Visual Object Classes) (Everingham et al., 2009) dataset is a benchmark dataset widely used in computer vision, featuring annotated images across multiple object categories. We use the Pascal-VOC subset collected by Miyai et al. (2023b) as the ID dataset, each image has single-class ID objects and one or more OOD objects. The ID validation and test set are split by 1:9 for each class, resulting in 94 and 906 images, respectively.

iNaturalist iNaturalist (Van Horn et al., 2018) is a biodiversity dataset containing millions of labeled images of plants, animals, and insects. Ming et al. (2022) construct a subset with 10,000 images by de-duplicating concepts overlapped with ID datasets.

Places Places (Zhou et al., 2017) is a scene-centric database with 205 scene categories and 2.5 million images. We use the SUN subset collected by Ming et al. (2022) as the OOD test set, which contains 10,000 images that are not overlapped with the ID classes.

SUN SUN (Scene UNderstanding) (Xiao et al., 2010) is a comprehensive collection of labeled images representing a diverse range of indoor and outdoor scenes. We use the SUN subset collected by Ming et al. (2022) as the OOD test set, which contains 10,000 images that are not overlapped with the ID classes.

Texture The Texture dataset (DTD) (Cimpoi et al., 2014) comprises 5640 images categorized into 47 terms inspired by human perception, aimed at replicating human-like texture recognition in machines. Again, we use the subset collected by Ming et al. (2022) as the OOD test set.

ImageNet22K The ImageNet-22K dataset (Russakovsky et al., 2015), formerly known as ImageNet21K, addresses the underestimation of its additional value compared to the standard ImageNet-1K

pretraining, aiming to provide high-quality pretraining for a broader range of models. We use the filtered subset collected by Wang et al. (2021) as the OOD test set for MC-COCO and Pascal-VOC ID test sets.

COCO Miyai et al. (2023b) curated an MS-COCO OOD test set (COCO for short) with 1,000 images that are not overlapped with the Pascal-VOC ID classes, which we use as OOD testing data for Pascal-VOC ID test set.

D Fine-tune Baselines

We compare SeTAR+FT with 4 finetuning-based baselines. These baselines include:

NPOS. NPOS (Tao et al., 2023) generates virtual anomalies in low-probability regions of ID data without relying on distribution assumptions, enhancing discrimination during training.

CoOp. CoOp (Zhou et al., 2022) optimizes prompts for vision-language models with learnable context vectors for efficient few-shot learning.

LoCoOp. LoCoOp (Miyai et al., 2023a) improves upon CoOp by leveraging CLIP's local features to better distinguish between ID and OOD samples, achieving higher detection accuracy with less training data. We follow the official code 11 to prepare and fine-tune the LoCoOp with CLIP-base and CLIP-large. Follow Miyai et al. (2023a), the top-K, λ , learning rate and epoch num are set to 200, 0.25, 0.002 and 50. Temperature is set to 1 and the text prompt is initiated with ' X X X X X X X X X X X X X X X X [CLASS] ', where [CLASS] is the ID class name. We average the results from 3 seeds finetuned with 1-shot ImageNet1K valid data.

LoRA. LoRA (Hu et al., 2022) is a low-rank adaptation method that injects trainable lowrank decomposition matrices into the pre-trained model to adapt to downstream tasks. We apply low-rank adaptation to the same weight type as SeTAR+FT, the rank of each layer is set to match the trainable parameters of SeTAR. Details settings can be found in Table 13.

E HyperParameters Settings

The hyperparameters for SeTAR are shown in Table 12. And the hyperparameters for SeTAR+FT and LoRA are shown in Table 13.

F More Detailed Experiment Results

In this section, we present additional detailed results from the main paper. This includes the detailed results of fine-tuned baselines on the ImageNet1K benchmark in Table 14; detailed ablation results on modality, W p , λ , and top-K in Table 15, Table 16, Table 19, and Table 21; and detailed results of SeTAR with different search algorithms, prune strategies and backbones in Table 18, Table 20, Table 17 and Table 22.

0

5

10

15

20

25

Num. of visited layers

90.0

90.2

90.4

90.6

90.8

91.0

91.2

Avg. AUROC

0

5

10

15

20

25

Num. of visited layers

40

41

42

43

44

45

46

Avg. FPR95

Wup

Wdown

Wq

Wk

Wv

Wout

baseline

(a) MCM score

0

5

10

15

20

25

Num. of visited layers

90.0

90.5

91.0

91.5

Avg. AUROC

0

5

10

15

20

25

Num. of visited layers

33

34

35

36

37

38

39

Avg. FPR95

Wup

Wdown

Wq

Wk

Wv

Wout

baseline

(b) GL-MCM score

0

5

10

15

20

25

Num. of visited layers

92.0

92.5

93.0

93.5

Avg. AUROC

0

5

10

15

20

25

Num. of visited layers

32

34

36

38

40

42

Avg. FPR95

Wup

Wdown

Wq

Wk

Wv

Wout

baseline

(a) MCM score

0

5

10

15

20

25

Num. of visited layers

93.25

93.50

93.75

94.00

94.25

94.50

94.75

95.00

Avg. AUROC

0

5

10

15

20

25

Num. of visited layers

24

26

28

30

32

Avg. FPR95

Wup

Wdown

Wq

Wk

Wv

Wout

baseline

(b) GL-MCM score

0.00

0.05

0.10

0.15

0.20

0.25

0.30

The Value of

90.6

90.8

91.0

91.2

Avg. AUROC

0.00

0.05

0.10

0.15

0.20

0.25

0.30

The Value of

34

36

38

40

42

44

Avg. FPR95

SeTAR+GL-MCM

(a) ImageNet1K

0.00

0.05

0.10

0.15

0.20

0.25

0.30

The Value of

93.50

93.75

94.00

94.25

94.50

94.75

Avg. AUROC

0.00

0.05

0.10

0.15

0.20

0.25

0.30

The Value of

24

26

28

30

32

34

36

Avg. FPR95

SeTAR+GL-MCM

(b) Pascal-VOC

0

200

400

600

800

1000

The Value of

K

90.9

91.0

91.1

91.2

91.3

Avg. AUROC

0

200

400

600

800

1000

The Value of

K

34

36

38

40

Avg. FPR95

SeTAR+GL-MCM

(a) ImageNet1K

0

2

4

6

8

10

12

14

The Value of

K

93.6

93.8

94.0

94.2

94.4

94.6

94.8

Avg. AUROC

0

2

4

6

8

10

12

14

The Value of

K

24

26

28

30

32

Avg. FPR95

SeTAR+GL-MCM

(b) Pascal-VOC

0

5

10

15

Epoch

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

loss_total

SeTAR+FT

LoRA

(a) LoCoOp Loss

0

5

10

15

Epoch

0.2

0.4

0.6

0.8

1.0

1.2

1.4

loss_id

SeTAR+FT

LoRA

(b) ID Loss

0

5

10

15

Epoch

5.6

5.4

5.2

5.0

4.8

4.6

4.4

loss_ood

SeTAR+FT

LoRA

(c) OOD Loss

V-11

V-10

V-9

V-8

V-7

V-6

V-5

V-4

V-3

V-2

V-1

T-11

T-10

T-9

T-8

T-7

T-6

T-5

T-4

T-3

T-2

T-1

T-0

Modality-LayerIndex

IN1K

VOC

0.15

0.15

0.00

0.00

0.00

0.00

0.00

0.00

0.05

0.00

0.00

0.15

0.00

0.00

0.00

0.00

0.00

0.25

0.10

0.20

0.05

0.05

0.00

0.20

0.10

0.35

0.40

0.05

0.25

0.35

0.10

0.00

0.00

0.00

0.25

0.35

0.00

0.25

0.05

0.10

0.20

0.05

0.05

0.00

0.10

0.00

0.25

0.00

(a) CLIP-base

V-23

V-22

V-21

V-20

V-19

V-18

V-17

V-16

V-15

V-14

V-12

V-11

V-10

V-9

V-8

V-7

V-6

V-5

V-4

V-3

V-2

V-1

T-11

T-10

T-9

T-8

T-7

T-6

T-5

T-4

T-3

T-2

T-1

T-0

Modality-LayerIndex

IN1K

VOC

0.00

0.00

0.00

0.05

0.15

0.10

0.00

0.00

0.05

0.10

0.20

0.00

0.15

0.00

0.10

0.00

0.00

0.05

0.00

0.00

0.00

0.15

0.00

0.40

0.35

0.00

0.00

0.00

0.10

0.05

0.35

0.00

0.20

0.00

0.05

0.20

0.40

0.35

0.00

0.20

0.00

0.30

0.25

0.10

0.05

0.25

0.05

0.30

0.05

0.10

0.15

0.05

0.20

0.15

0.30

0.10

0.10

0.20

0.10

0.35

0.00

0.05

0.05

0.40

0.00

0.15

0.00

0.30

0.40

0.05

0.05

0.40

(b) CLIP-large

3-1

3-0

2-17

2-16

2-15

2-14

2-13

2-12

2-11

2-10

2-9

2-8

2-7

2-6

2-5

2-4

2-3

2-2

2-1

2-0

1-1

1-0

0-1

0-0

StageIndex-BlockIndex

IN1K

0.40

0.40

0.40

0.40

0.40

0.00

0.00

0.00

0.00

0.10

0.00

0.00

0.00

0.25

0.00

0.00

0.00

0.00

0.00

0.00

0.40

0.30

0.00

0.10

(c) Swin-base

tower_type

weight_type

layer_num

best_ratio

total_loss*

id_loss

ood_loss

val_acc

ood_patch_percent

step

0

visual

W_up

11

0.15

0.647777

1.093326

-4.455494

71.399998

38.906631

1

visual

W_up

10

0.15

0.644654

1.083629

-4.389751

71.799998

39.293876

2

visual

W_up

9

0.00

0.644654

1.083629

-4.389751

71.799998

39.293876

3

visual

W_up

8

0.00

0.644654

1.083629

-4.389751

71.799998

39.293876

4

visual

W_up

7

0.00

0.644654

1.083629

-4.389751

71.799998

39.293876

5

visual

W_up

6

0.00

0.644654

1.083629

-4.389751

71.799998

39.293876

6

visual

W_up

5

0.00

0.644654

1.083629

-4.389751

71.799998

39.293876

7

visual

W_up

4

0.00

0.644654

1.083629

-4.389751

71.799998

39.293876

8

visual

W_up

3

0.05

0.640844

1.079729

-4.388844

71.999998

39.209695

9

visual

W_up

2

0.00

0.640844

1.079729

-4.388844

71.999998

39.209695

10

visual

W_up

1

0.00

0.640844

1.079729

-4.388844

71.999998

39.209695

11

visual

W_up

0

0.15

0.640132

1.079109

-4.389775

72.199998

39.156123

12

text

W_up

11

0.00

0.640132

1.079109

-4.389775

72.199998

39.156123

13

text

W_up

10

0.00

0.640132

1.079109

-4.389775

72.199998

39.156123

14

text

W_up

9

0.00

0.640132

1.079109

-4.389775

72.199998

39.156123

15

text

W_up

8

0.00

0.640132

1.079109

-4.389775

72.199998

39.156123

16

text

W_up

7

0.00

0.640132

1.079109

-4.389775

72.199998

39.156123

17

text

W_up

6

0.25

0.630751

1.075123

-4.443716

71.600001

38.808673

18

text

W_up

5

0.10

0.630514

1.078703

-4.481889

71.599997

38.246428

19

text

W_up

4

0.20

0.622065

1.075958

-4.538932

72.000001

38.452552

20

text

W_up

3

0.05

0.620440

1.079326

-4.588857

71.999997

38.649488

21

text

W_up

2

0.05

0.618521

1.076858

-4.583368

71.600001

38.444899

22

text

W_up

1

0.00

0.618521

1.076858

-4.583368

71.600001

38.444899

23

text

W_up

0

0.20

0.615174

1.069851

-4.546776

72.499997

38.642345

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have carefully crafted the abstract and introduction to accurately reflect the contributions and scope of the paper. Specifically, we propose a novel training-free method, SeTAR with a finetuning extension SeTAR+FT, and demonstrate its effectiveness for OOD detection tasks.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations of the proposed method are discussed in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper includes detailed experimental and hyperparameters settings in Section 4.1 and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code are available at https://github.com/X1AOX1A/SeTAR .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed experimental settings in Section 4.1 and Appendix E. We give the details of our design choices in Section 4.4 and datasets in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the results with standard deviation from runs of 3 seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the details of the compute resources in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We provide the ethical considerations in Appendix A.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide the broader impacts in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not directly provide pre-trained models or scraped datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We include the citation and URL of models and datasets used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes SeTAR for CLIP-based OOD detection. SeTAR is based on low-rank approximation. It determines the optimal rank for each weight block with a greedy hyperparameter search using validation samples. While SeTAR itself is training-free, the paper also proposes SeTAR+FT which incorporates LoRA as a training extension. Experiments demonstrate that SeTAR outperforms existing methods on CLIP-specific benchmarks.\n\nSTRENGTHS:\nThe paper is written in good quality, and the proposed method sounds reasonable.\n\nWEAKNESSES:\n1. My main concern is that the performance improvement of SeTAR (SeTAR+FT) seems limited compared with MCM and GL-MCM (LoCoOp). In Table 1, for example, all improvements of SeTAR in terms of AUROC are within 1%. The only case where SeTAR leads to noticeable/significant improvements is when the model is the image-only Swin-T (Table 2). Would the authors comment on why this is the case?\n\n2. I'm confused by some of the numbers discussed in the main text. In line 220-222, it says `For example, using Pascal VOC as ID, SeTAR yields an average reduction of 12.84% FPR95 on MCM and 18.95% FPR95 on GL-MCM`. How is the \"12.84\" and \"18.95\" computed? According to Table 1, on Pascal VOC, the average FPR95 of MCM and SeTAR is 38.88 and 32.46, respectively. The average FPR95 of GL-MCM and SeTAR is 31.12 and 23.86, respectively. A similar confusion of mine is regarding line 238-240 `when scaled up to CLIP-large, SeTAR+FT outperforms LoCoOp and LoRA by 17.92% and 12.45% FPR95 on the same benchmark`, which doesn't seem to match with the numbers reported in Table 2. Am I missing anything?\n\n3. All considered OOD datasets are far-OOD w.r.t. the ID set. Near-OOD detection has long been recognized as a more challenging and realistic problem [1, 2]. I would be very interested to see how well SeTAR performs on ImageNet v.s. NINCO or SSB [2].\n\nI'm willing to adjust my score if my comments are addressed or clarified.\n\n[1] Detecting semantic anomalies\n\n[2] OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

comments: SUMMARY:
The paper introduces SeTAR, a novel training-free out-of-distribution (OOD) detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection by post-hoc modifying the model’s weight matrices using a greedy search algorithm. The paper also extends this method to SeTAR+FT, a fine-tuning approach to further optimize OOD detection performance.

STRENGTHS:
1.	The paper proposes a unique training-free method for enhancing OOD detection, which is novel.
2.	The paper includes thorough ablation studies and sensitivity analyses, which help in understanding the robustness and generalizability of the proposed approach.
3.	The authors provide extensive empirical evaluations on multiple benchmarks, showing the effectiveness of SeTAR and SeTAR+FT.

WEAKNESSES:
-	The performance gains compared to baselines are relatively small. Given that these results are achieved using a greedy search strategy, the potential of SeTAR may be limited.
-	There is a lack of theoretical analysis explaining why SeTAR is effective. Understanding the underlying principles is crucial for advancing the method and its applications.
-	The experiments indicate that the optimal hyperparameters (λ and K) vary significantly across different backbones. In practical OOD detection scenarios, it is challenging to obtain OOD samples for hyperparameter tuning. Therefore, SeTAR needs to be strengthened in this aspect to ensure robust performance without extensive hyperparameter tuning.
P.S. The layout of Figure 1 is somewhat cluttered.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm for OOD detection along with CLIP models.\nIt observes that pruning based on SVD decomposition on CLIP models can improve the OOD detection performance.\nA greedy search algorithm is developed for searching pruning rations of each weight in CLIP models.\n\nExperiments on regular settings demonstrate obvious performance when compared with the Vanilla GL_MCM method.\n\nSTRENGTHS:\n(1) The paper is written clearly and easy to follow.     \n(2) The method is simple but effective.             \n(3) The ablation studies are sound.\n\nWEAKNESSES:\n(1) It seems that the proposed method is general enough to apply to regular models not just CLIP models.    \n     Will it also work on CNN-based ResNets and ViT models trained on datasets, like ImageNet and CIFAR?        \n      The experimental results focus on CLIP-based models in Table 1.\n     More comparisons with previous methods using CNN-based models should be included.     \n     \n(2) SVD decomposition pruning has strong connections with sparsification-based methods [1,2,3]. \n     Comparisons with this kind of method should be included.\n\n[1] Dice: leveraging sparsification for out-of-distribution detection.     \n[2] Extremely simple activation shaping for out-of-distribution detection.       \n[3] ReAct: out-of-distribution detection with rectified activations.                   \n\n(3) The proposed greedy search algorithm is efficient. However, it can not be guaranteed to be converged.\n     What's the performance variance for 2 independent runs of the search algorithm?\n\nQUESTIONS:\n(1) The main results in the paper concentrate on CLIP models.\n      Results on other CNN-based models and comparisons with previous methods should be included in the paper.\n\n(2) The proposed greedy search algorithm can not be guaranteed to be converged.\n      Thus, a discussion on the stability of the proposed algorithm is required.\n\nLIMITATIONS:\nA section of Impact Statements including limitations is included in the paper", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents SETAR, a novel method designed to enhance out-of-distribution (OOD) detection without requiring additional training. The proposed method leverages rank reduction techniques applied to the model weights, specifically targeting the minor singular components, while retaining the principal components that significantly contribute to the model’s performance. SETAR is evaluated across various model backbones, including CLIP-base, CLIP-large, and Swin-base, and demonstrates notable improvements in OOD detection tasks. The paper also provides comprehensive experiments and ablation studies to validate the effectiveness and efficiency of SETAR.\n\nSTRENGTHS:\n1.\tNovelty and Innovation: The introduction of a training-free method for improving OOD detection is a significant contribution. By focusing on rank reduction of model weights, the method offers a fresh perspective compared to traditional training-intensive approaches.\n\t2.\tComprehensive Evaluation: The paper provides extensive experimental results across multiple datasets and model backbones, showcasing the robustness and generalizability of SETAR.\n\t3.\tEffective Performance: The method achieves substantial improvements in OOD detection metrics, such as FPR95 and AUROC, demonstrating its practical utility. The significant performance boost on Swin-base compared to CLIP-base and CLIP-large is particularly notable, likely due to the inherent design differences and stronger zero-shot performance of CLIP models.\n\t4.\tDetailed Analysis: The inclusion of ablation studies and sensitivity analyses helps in understanding the impact of various components and hyperparameters, offering valuable insights into the method’s functioning.\n\nWEAKNESSES:\nI hope the authors could address my following questions.\n\nQUESTIONS:\n1. Table 2: Could you provide more insights on why the performance boost on Swin-Base is significantly more pronounced compared to CLIP-Base and CLIP-Large? Is this disparity related to CLIP’s stronger zero-shot performance?\n\n2. Figure 2: For ImageNet-1K, it appears that decomposing the vision encoder alone is sufficient for OOD detection, with the combination of both vision and text encoders yielding only minor improvements. Could you explain this in more detail?\n\n3. The proposed method is training-free, but the greedy search algorithm used for determining the rank reduction ratio list and performing rank reduction does require computation time. Can you provide details on the time required for the greedy search and rank reduction for SeTAR, as well as the overall computation time of SeTAR+FT, compared to previous fine-tuning methods?\n\n4. The proposed method seems to be an application of LASER, the only contribution is the adaptation of LASER to CLIP model, and benefits OOD detection. Can the authors justify this? The paper is technically solid with comprehensive experiments, but I think the paper is not novel enough given the similarities compared to LASER.\n\nLIMITATIONS:\nLimitation discussed in Appendix", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}