SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation

Authors: yixia  li
boya  xiong
guanhua  chen
yun  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:07:37

Abstract: Out-of-distribution (OOD) detection is crucial for the safe deployment of neural networks. Existing CLIP-based approaches perform OOD detection by devising novel scoring functions or sophisticated fine-tuning methods. In this work, we propose SeTAR, a novel, training-free OOD detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection via post-hoc modification of the model's weight matrices using a simple greedy search algorithm. Based on SeTAR, we further propose SeTAR+FT, a fine-tuning extension optimizing model performance for OOD detection tasks. Extensive evaluations on ImageNet1K and Pascal-VOC benchmarks show SeTAR's superior performance, reducing the relatively false positive rate by up to 18.95\% and 36.80\% compared to zero-shot and fine-tuning baselines. Ablation studies further validate our approach's effectiveness, robustness, and generalizability across different model backbones. Our work offers a scalable, efficient solution for OOD detection, setting a new state-of-the-art in this area.

Main paper: 
 introduction: 
 The task of out-of-distribution (OOD) detection (Hendrycks & Gimpel, 2017; Ming et al., 2022) aims to identify whether input data comes from an unknown distribution. It has garnered significant attention in the machine learning community recently (Hendrycks et al., 2020; Xu et al., 2021; Miyai et al., 2023a) . While machine learning models are trained with supervised in-distribution (ID) data, they often struggle to generalize to OOD data encountered in real-world applications (Emmott et al., 2016) like autonomous vehicles and healthcare. These OOD samples pose challenges as they are not represented in the training data. Consequently, OOD detection plays a crucial role in developing reliable and trustworthy machine-learning models suitable for real-world deployment (Bai et al., 2023) . It allows models to filter out and reject these awkward inputs effectively, and enables the use of curated and labeled OOD samples to further train for a more robust model in the wild.Previous research has primarily focused on detecting OOD instances in either visual (DeVries & Taylor, 2018; Liang et al., 2018; Hendrycks et al., 2022) or textual data (Hu & Khan, 2021; Zheng et al., 2020; Zhou et al., 2021) . Recently, significant progress has been made in multimodal tasks like multimodal retrieval (Li et al., 2023; Caesar et al., 2018) and image classification (Yu et al., 2022) , thanks to vision-and-language pretrained (VLP) models like CLIP (Radford et al., 2021) . More recent The details of the feed-forward sublayer. (c) For each encoder layer, we replace the W up weight matrix with its low-rank approximation W up . (d) The illustration of Σ before and after low-rank approximation. More details are in Section 3.1. concatenated and projected back with another linear matrix W o . The feed-forward module projects the hidden state into a wider space using W up and then back with W down after a non-linear activation (Figure 1 ). Given the similarity between the image and text encoder layers, we adopt consistent notations for the linear matrices in both. Each encoder also includes a linear projector W p to map their representations into a shared space for contrastive learning.Zero-shot OOD Detection with CLIP Zero-shot OOD detection aims to separate ID and OOD data without an ID training dataset. Given the CLIP, the ID classes are defined by the classification task of interest, which differs from the classes used in CLIP pretraining. Accordingly, OOD is defined as classes not belonging to any ID class, making the OOD detector a binary classifier. MCM (Ming et al., 2022) and GL-MLM (Miyai et al., 2023b) are two zero-shot CLIP-based OOD detection methods. Formally, let x be the test image and T in = {y c } K c=1 be the set of text prompts containing M ID class labels (e.g., "a photo of a [CLASS]"). The image is segmented into l image patches x = (x 1 , ..., x l ). Following CLIP, we add [cls] before the image patches and use the output of [cls] from the visual projector W p as the global image feature (h v ∈ R d ). The outputs of other patches are projected by the visual projector as the local image features (p v = (p v 1 , ..., p v l ) ∈ R l×d ). For the text prompt y c ∈ T in , we add an additional [eos] after the text tokens and use the output feature of [eos] from the textual projector W p as the concept feature of ID class c (h t c ∈ R d ). The label-wise image-concept matching (IWIC) score measures how well a test image x aligns with a concept y c , using either global or local features. The global IWIC score s G c (•) is the cosine similarity between the global image feature h v and the concept feature h t c : s G c (x) = cos_sim(h v , h t c ). The local IWIC score s L c (•) is the max-pooled cosine similarity between image patch features p v i and the concept feature h t c : s L c (x) = max i cos_sim(p v i , h t c ). The MCM and GL-MCM scores are defined as:EQUATIONEQUATIONwhere τ and τ ′ are the temperature hyperparameters. MCM only uses global image features, while GL-MCM additionally considers local image features. For ID data, both MCM and GL-MCM scores will be matched to one of the concept features with a high score; and vice versa. As a result, our OOD detection function can be formulated as:EQUATIONwhere S (x) is either the MCM or GL-MCM score, λ is the threshold value. By convention, G(x) = 1 represents the ID class and G(x) = 0 indicates the OOD class. The λ is chosen so that a high fraction of ID data (e.g., 95%) is above the threshold. We follow previous work (Miyai et al., 2023a) to use either MCM or GL-MCM score for OOD detection in this work. 
method: 
 We introduce SeTAR, a training-free and effective technique for improving OOD detection performance (see Figure 1 ). Our key idea is to perform post-hoc modification to CLIP weight matrices by selectively replacing them with their low-rank approximations. It is complementary to existing CLIP-based zero-shot OOD detection methods and could be further extended to finetuning-based methods, which we term as SeTAR+FT.Low-Rank Approximation Given a linear matrix W ∈ R m×n , its Singular Value Decomposition (SVD) is denoted asW = U ΣV ⊤ , where U = [u 1 , u 2 , • • • , u m ] ∈ R m×m , V = [v 1 , v 2 , • • • , v n ] ∈ R n×n, and Σ ∈ R m×n is a matrix whose entries are all zero except for the singular values of W . These singular values appear in decreasing order on the diagonal (i.e. σ ↓ i (W )). The SVD of W can be reformulated as in Equation 4. Given a hyperparameter r ∈ N + , a rank-r approximation of W is matrix W that minimizes ∥W -W ∥ 2 and satisfies rank( W ) ≤ r. The optimal solution of this problem W is provided by Eckart-Young-Mirsky theorem (Low-Rank Approximation, 2024) using Singular Value Decomposition (see Equation 5).EQUATIONEQUATIONIn this work, we will use the term minor singular components to refer to entries in the SVD corresponding to small singular values. These components are removed in low-rank approximation.The term of principle singular components is used to refer to entries in the SVD corresponding to large singular values. These components are kept in a low-rank approximation of the matrix.OOD Detection with Selective Low-Rank Approximation SVD-based weight pruning, particularly in noise-prone layers, can substantially reduce a network's sensitivity to minor perturbations, leading to enhanced stability and robustness (Yao et al., 2024) . This stability is crucial for OOD detection, as it ensures the model's reliable performance across a wide range of inputs. Building on this, we propose a method to improve OOD detection by selectively applying low-rank approximation to weight matrices. By decomposing a weight matrix W into its singular values and vectors, we can identify and retain the principle singular components that significantly contribute to the model's performance. This approach ensures that the essential features of W are preserved while discarding the less critical minor singular components. Given a weight matrix W in CLIP (e.g., W up or W k ), we replace the matrix with its low-rank approximation part W as described in Equation 5 (see Figure 1 ). Given the rank reduction ratio Θ, the rank of W is determined by r( W ) = round((1 -Θ) • r(W )).This selective low-rank approximation leverages the compact representation provided by SVD to enhance the model's ability to detect OOD instances effectively without requiring additional training. We demonstrate our method's ability to improve OOD detection (Table 1 ) while maintaining ID classification performance (Table 7 ) in Section 4.2 and Section 4.5.HyperParameter Search Algorithm Due to the presence of many weight matrices in CLIP, each consisting of hundreds of singular values, conducting a complete search over all combinations of low-rank approximation weight matrices is impractical. Therefore, we propose a greedy search algorithm to determine the rank reduction ratio for each weight matrix. Among all linear weight matrices in each encoder layer, we focus on W up as it is most effective according to our preliminary experiment. For simplicity, we assume both image and text encoders have N encoder layers. As shown in Algorithm 1, we search by first enumerating all N vision encoder layers sequentially from top to bottom and then all N text encoder layers in the same way. This search order is concisely denoted as searching from 2N to the first layer in CLIP. We compare different search algorithms in Section 4.4. The rank reduction ratio for each layer is the objective in SeTAR which is searched among the candidate list Θ = {Θ 0 , Θ 1 , • • • , Θ J } according to the loss on the validation set. We employ the LoCoOp loss (Equation 12) proposed in (Miyai et al., 2023a) as our loss function. This loss requires only ID images. It contains an ID loss for ID image classification and an OOD loss to push away pseudo OOD features from the ID class text embeddings where the pseudo OOD features are from ID-irrelevant nuisances (Equation 10) (e.g., backgrounds) in CLIP's local features. We refer the readers to Miyai et al. (2023a) or Appendix B for more details. l ← 2N to 1 do W * = W l up T * [l] = 0 for counter j ← 1 to J do r = round((1 -Θ[j]) • rank(W l up )) W = r i=1 σ ↓ i uiv ⊤ i Calcluate loss L l j on D by replacing W l up with W if L l j < L * then W * = W; T * [l] = Θ[j]; L * = L l j ; end end W l up := W * end return T *For Θ j ∈ Θ, we remove Θ j (in percent) singular values along with their corresponding singular vectors to obtain the approximated matrix W up (Equation 5). It is worth noting that the rank reduction raio candidate list includes Θ 0 = 0, indicating that the weight matrix has the chance to remain unmodified.With the searched rank reduction ratio, the weight matrix W up in each CLIP layer is replaced and updated with its approximation. The SeTAR can be easily applied to different ViT backbones (Table 8 ), by replacing the model weight matrices with their low-rank approximations in a similar approach. Then SeTAR detects the OOD data samples following MCM (Equation 1), GL-MCM (Equation 2) or other scoring-based OOD detection method with the approximated model. We provide an example procedure of the greedy search in Listing 1 for better understanding.SeTAR can be further combined with LoRA (Hu et al., 2022) as a novel low-rank adaptation method for OOD detection, which we refer to as SeTAR+FT. Specifically, we first apply SeTAR to the pre-trained CLIP model to obtain the reserved rank r for each weight matrix W . Then we haveEQUATION)where W is the low-rank approximation of W found by SeTAR , with A and B being the minor singular components. During finetuning, we keep W frozen and only update the low-rank matrix A and B. In this way, we retain the principle singular components in the original weight matrix and only update the minor singular components.Unlike LoRA, which evenly distributes the finetuning rank budget across all layers, SeTAR+FT adjusts the rank for each layer, resulting in more effective and efficient fine-tuning (Table 2 and Figure 6 ). More details are provided in Section 4.3. 
experimental setup: 
 Dataset Following previous work (Ming et al., 2022; Miyai et al., 2023b) , we use two real-world datasets created from ImageNet1K (Deng et al., 2009) and Pascal-VOC (Everingham et al., 2009) as the ID datasets. For OOD datasets, we follow Ming et al. (2022) to preprocess iNaturalist, SUN, Places and Texture, and follow Miyai et al. (2023b) to preprocess ImageNet22K and COCO data.For finetune experiments, we follow Miyai et al. (2023a) to use ImageNet1K as the ID dataset. The detailed description and statistics of the datasets are provided in Appendix C. 12) and top-K (Equation 10) according to the specific ID datasets and backbones. Detailed settings are in Table 12 , with a sensitivity analysis in Section 4.4. For SeTAR+FT and LoRA experiments, the learning rate and epoch number are set to 1e -2 and 5 for all experiments. The LoRA rank r is set to match the trainable parameters of SeTAR+FT. Detailed settings are in Table 13 . We report results from three runs with seeds 3, 4, 56 . All experiments are conducted on a single NVIDIA RTX 4090 GPU. The time cost for low-rank approximation with CLIP-base on the ImageNet1K validation set is about 20 minutes.Metrics We use the following metrics for evaluation. (1) the false positive rate (FPR95) for out-ofdistribution (OOD) samples at a fixed true positive rate (TPR) of 95% for in-distribution samples, with lower values targeting better performance; and (2) the area under the receiver operating characteristic curve (AUROC) for OOD samples, with higher values indicating better performance.Baselines We evaluate SeTAR against MCM (Ming et al., 2022) and GL-MCM (Miyai et al., 2023b) , state-of-the-art zero-shot OOD detection methods on CLIP. We also compare SeTAR+FT with fine-tuning baselines NPOS (Tao et al., 2023) , CoOp (Zhou et al., 2022) , LoCoOp (Miyai et al., 2023a) , and LoRA (Hu et al., 2022) . More details are in Appendix D.The training-free OOD detection performances are summarized in Table 1 . Compared with zero-shot baselines, a salient observation is that on both MCM and GL-MCM, using SeTAR outperforms the vanilla method by a large margin across all OOD detection tasks. For example, using Pascal-VOC as ID, SeTAR yields a relatively average reduction of 12.84% FPR95 on MCM and 18.95% FPR95 on GL-MCM. Considering that SeTAR is generally applicable and training-free, these results are very encouraging. Comparing SeTAR with scoring function MCM and GL-MCM, SeTAR+GL-MCM performs better on all OOD detection tasks. However, the superiority of GL-MCM score over MCM appears to be contingent upon the choice of the model backbone. As evidenced in Table 8 , SeTAR+MCM demonstrates superior performance with a relatively average FPR95 reduction of 8.30% compared to SeTAR+GL-MCM with CLIP-large as the backbone on ImageNet1K.In this section, we compare SeTAR+FT with fine-tuning baselines, including NPOS (Tao et al., 2023) , CoOp (Zhou et al., 2022) , LoCoOp (Miyai et al., 2023a) and LoRA (Hu et al., 2022) . LoCoOp is the state-of-the-art prompt-learning OOD detection method on CLIP. LoRA is a representative parameter-efficient fine-tuning method. Following previous work (Tao et al., 2023; Zhou et al., 2022; Miyai et al., 2023a) , we report the results on the the ImageNet1K benchmark in Table 2 . We observe that SeTAR+FT outperforms all baselines on both MCM and GL-MCM scoring functions. For example, with CLIP-base as the backbone, SeTAR+FT achieves a relatively average FPR95 reduction of 3.97% and 6.67% compared to LoCoOp and LoRA. Moreover, when scaled up to CLIP-large, SeTAR+FT outperforms LoCoOp and LoRA by (Liu et al., 2021) , where SeTAR+FT outperforms LoRA by relatively 17.36% and 36.80% FPR95 on MSP and Energy scoring functions, respectively. The larger improvement on Swin Transformer may stem from its reliance on ImageNet training, making it prone to overfitting and weaker at OOD detection. Our method mitigates these issues, enhancing Swin's generalization to OOD instances. These results demonstrate the effectiveness and scalability of SeTAR+FT in improving the OOD detection performance.Furthermore, as shown in Figure 6 , SeTAR+FT demonstrates faster convergence and lower loss than LoRA, especially in OOD loss, indicating that SeTAR+FT is more effective in adapting the pre-trained weights to the OOD detection task.In this section, we conduct ablation studies with CLIP-base to understand our design choices. Image v.s. Text modality Table 3 shows an ablation study on the modality involved in SeTAR.Table 3 : Ablation study on modality. As shown, the vision modality outperforms the text modality, indicating the vision modality is more dominant in enhancing the model's performance. When considering the vision modality alone and the combined vision+text modality, the latter either outperforms or achieves comparable average results to the former. Consequently, we make modifications to both the vision and text modalities in SeTAR to enhance overall performance. Different Search Algorithms At each step of the greedy search, SeTAR traverses the subsequent W up in a predefined order and searches over different thresholds. We compare our method with two alternatives: modality-interleaved greedy search (MIS) and layer-exhaustive search (LES). MIS searches the image and text layers in an interleaved manner, while LES simultaneously searches over both layers and thresholds at each step. SeTAR-S, has linear complexity with respect to the number of model layers, similar to MIS, whereas LES has quadratic complexity. Table 5 presents the comparison results. SeTAR-S demonstrates better overall performance than MIS. Notably, MIS encounters limitations when the image and text towers have different layer counts (e.g., CLIP-large with 24 image layers and 12 text layers). Therefore, we choose SeTAR-S for better generalization. Compared to LES, SeTAR-S performs better in terms of both FPR95 and AUROC, as LES's locally optimal algorithm may not achieve a global optimal solution. These results validate the superiority of our top-to-bottom layer search strategy.Different Prune Strategies Inspired from SVD, SeTAR modify the model weights by pruning the minor singular components, and retains the principle components that contribute the most to the model's performance. To validate this design, we compare SeTAR with two alternatives: principal component pruning and random pruning pruning. Principal component takes the opposite approach, retaining minor components and pruning major ones. Random pruning, on the other hand, prunes weights randomly. As shown in Table 6 , principle pruning suffers from a significant performance drop compared to Se-TAR , while random pruning performs slightly better than principle pruning. These results demonstrate the effectiveness of SeTAR 's design choice in pruning the minor components.Sensitivity Analysis on λ and top-K In this section, we present the sensitivity analysis of the hyperparameters λ (Figure 4 ) and top-K (Figure 5 ). As observed in Figure 4 , the average AUROC remains stable at lower values and slightly decreases as λ increases for both SeTAR+MCM and SeTAR+GL-MCM. Notably, the optimal setting of λ may vary depending on the model backbone, with our experiments indicating that CLIP-large may require a larger λ than CLIP-base. Despite this variation, the λ parameter demonstrates strong transferability across datasets for the same backbone. Swapping the optimal λ between ImageNet1K and Pascal-VOC has a minimal performance impact, consistently outperforming the vanilla method. With the VOC-optimized λ on ImageNet1K, CLIPbase achieves an FPR95 of 40.91 and AUROC of 91.02, and CLIP-large reaches 46.73 FPR95 and 91.81 AUROC. Conversely, using the ImageNet1K-optimized λ on Pascal-VOC, CLIP-base achieves 33.18 FPR95 and 93.65 AUROC, while CLIP-large attains 44.39 FPR95 and 92.3 AUROC.Top-K controls the number of OOD regions considered in LoCoOp loss: higher values include more OOD regions, with top-K equal to the number of ID classes covering all OOD regions, and top-K set to 0 focusing solely on ID loss. The optimal top-K depends on the number of ID categories, making it non-transferable across datasets. However, SeTAR remains robust to top-K variations, as shown in Figure 5 , except at extreme values (0 or the maximum number of classes). We recommend setting top-K to around 30% of the total categories, such as 300 for ImageNet1K and 4 for Pascal-VOC. For the Swin-base model, top-K at 300 on ImageNet1K yields an FPR95 of 56.82 and AUROC of 85.68 with MSP, and an FPR95 of 52.56 and AUROC of 84.51 with Energy. & Taylor, 2018; Hendrycks & Gimpel, 2017; Hu & Khan, 2021; Zheng et al., 2020; Zhou et al., 2021) and multimodal (Fort et al., 2021; Ming et al., 2022; Tao et al., 2023; Miyai et al., 2023a) models. Numerous methodologies (Lee et al., 2018; Huang et al., 2021; Sun et al., 2022; Wang et al., 2022; Wu et al., 2023) have been developed to tackle OOD detection in computer vision.Existing CLIP-based OOD detection methods include zero-shot (Fort et al., 2021; Ming et al., 2022; Miyai et al., 2023b; Dai et al., 2023; Wang et al., 2023; Jiang et al., 2024) and fine-tuning (Ming & Li, 2023; Tao et al., 2023; Miyai et al., 2023a) . Zero-shot methods like MCM (Ming et al., 2022) and GL-MCM (Miyai et al., 2023b) don't require in-distribution training data but may perform suboptimally due to domain gaps. Other approaches integrate external knowledge. For example, CLIPN (Wang et al., 2023) pre-trains a novel NO-encoder on the CC-3M dataset (Sharma et al., 2018) to empower CLIP's "no" logic for zero-shot evaluation. NegLabel (Jiang et al., 2024) demonstrates better performance than CLIPN by introducing large-scale negative labels for enhanced label scoring. Fine-tuning methods (Ming & Li, 2023; Tao et al., 2023; Miyai et al., 2023a) improve OOD detection by adapting to in-distribution data but risk damaging the pretraining representations, needing careful training strategies. CNN-based OOD detection methods, including ReAct (Sun et al., 2021) , ASH (Djurisic et al., 2023) , DICE (Sun & Li, 2022) , CIDER (Ming et al., 2023) , PALM (Lu et al., 2024) , and Hopfield Boosting (Hofmann et al., 2024) , have also demonstrated strong results. However, methods like ReAct and ASH rely on the assumption that ID and OOD images produce distinct activations in models trained on ID data. This assumption does not hold in large-scale pre-trained models like CLIP, where activations for ID and OOD images are not significantly different, limiting the effectiveness of such approaches in enhancing CLIP's zero-shot OOD detection capabilities. SeTAR, in contrast, offers high compatibility with various scoring functions (e.g., MCM, GL-MCM, MSP, Energy), multiple model backbones (e.g., CLIP, Swin, ResNet), and advanced OOD techniques such as NegLabel. Designed to be both lightweight and efficient, SeTAR addresses the demand for resource-efficient solutions in OOD detection.Low-rank Approximations of Weight Matrices Neural networks trained with overparameterization often exhibit low-rank properties (Oymak et al., 2019) . These properties are utilized in both model training (Povey et al., 2018; Hu et al., 2022) and post-hoc processing (Hajimolahoseini et al., 2021; Sharma et al., 2023) . In training, some works (Sainath et al., 2013; Zhang et al., 2014; Zhao et al., 2016) impose low-rank constraints, while LoRA (Hu et al., 2022) adapts pretrained LLMs to downstream tasks using trainable low-rank matrices. For post-hoc processing, pruning methods (Yu et al., 2017; Hajimolahoseini et al., 2021) reduce weight matrix ranks by retaining top-K components from SVD. While pruning preserves model behavior, performance declines with increased intervention. LASER (Sharma et al., 2023) focuses on pruning individual layers to enhance factual answering capabilities. It utilizes a simple greedy search strategy on a validation set, which is not applicable for OOD detection due to the absence of a validation set. In contrast, our approach introduces a selective rank reduction strategy specifically tailored for OOD detection. We systematically analyze and compare different greedy search techniques, evaluating their effectiveness across various layers and model backbones. 
conclusion: 
 We propose SeTAR , a simple and effective OOD detection method using post-hoc low-rank approximation on weight matrices W up with a top-down, image-to-text greedy search. SeTAR offers several advantages: (1) training-free, (2) scalable to unimodal and multimodal models, and (3) complementary to existing OOD scoring functions. Building on SeTAR , we introduce SeTAR-FT, a finetuning method that adapts the model to in-distribution data for improved OOD detection. We evaluate SeTAR and SeTAR-FT on large-scale benchmarks, including ImageNet1K and Pascal-VOC.Results show that both achieve state-of-the-art OOD detection performance. We hope our work inspires further research and contributes to more robust and reliable models.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide detailed experimental settings in Section 4.1 and Appendix E. We give the details of our design choices in Section 4.4 and datasets in Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report the results with standard deviation from runs of 3 seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not directly provide pre-trained models or scraped datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We include the citation and URL of models and datasets used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Limitation While we demonstrate the effectiveness of our method on OOD detection, we acknowledge that our work has several limitations. First, despite we show the robustness of our method to hyperparameters, the optimal hyperparameters may vary across different model backbones. Future work is needed to explore the autonomous selection of hyperparameters. Second, we design Se-TAR+FT in a simple and straightforward manner, which may not be the most efficient or effective way to adapt the model to the ID downstream data. More sophisticated strategies for model adaptation are worth exploring in future research. Third, we only conduct experiments to detect visual OOD inputs and ignore inputs in other modalities such as textual, audio and video. This is primarily because our model is based on CLIP. Exploring the development of OOD detectors across diverse modalities remains an active research topic for future investigation.Ethical Considerations Our study addresses the challenge of OOD detection through low-rank approximation, which is particularly relevant for ensuring the reliability and trustworthiness of vision-and-language pre-trained models. Future investigations on fairness, privacy and transparency neural-based models should be encouraged to mitigate the existing data biases and safety problems for a responsible, helpful and trustworthy AI system in diverse real-world applications.Future Societal Consequences Our proposed SeTAR achieves impressive OOD detection performance, which is beneficial to various real-world machine learning applications, such as healthcare and autonomous vehicles. The identification of anomalies or unexpected data points is crucial for decision-making and risk management with AI models. A better OOD detector facilitates the development of trustworthy machine-learning models that can reject unknown data inputs and help alleviate the hallucination problem. Moreover, better OOD detectors like SeTAR can help to select and label the unfamiliar data samples to further train a stronger model in the wild.To improve the model's OOD detection ability, it is crucial to define a loss function that pushes OOD samples far from ID samples while keeping ID samples close to each other. However, since OOD samples are unavailable during development, we address this issue by using the LoCoOp loss (Miyai et al., 2023a) for both SeTAR and SeTAR+FT. The main idea is to create pseudo OOD features with ID-irrelevant nuisances (e.g., backgrounds) in CLIP's local features.Specifically, we divide the image into patches, represented by the set of all patch indices I = {0, 1, 2, . . . , H × W -1}, where H and W denote the height and width of the patch features. Next, we compute the cosine similarity between the image patch features p v i and the text features h t c of the image label. The classification prediction probabilities for each patch i are then given by:EQUATIONFor a given image patch related to an ID category, the corresponding ID label should be among its top-K predictions. Conversely, for patches unrelated to the ID label, such as background regions, the ID label should be excluded from the top-K predictions. Based on this intuition, the indices of ID-irrelevant regions within an image are defined by Equation 10, where rank(p i (y = y|x)) denotes the rank of the true class y among all ID classes, and K is the hyperparameter.J = {i | rank(p i (y = y|x)) > K} (10)After identifying out-of-distribution (OOD) regions, it is expected that their image features will differ significantly from the ID text embeddings. To enhance this distinction, entropy maximization is employed to increase the entropy of p j (y|x), where p j denotes the classification prediction probabilities for region j ∈ J. The entropy maximization is formally defined as follows:EQUATIONHere, H(•) represents the entropy function. The overall loss function combines the ID loss (crossentropy loss for ID predictions) with the OOD loss. Here λ is the hyperparameter that regulates the proportion of the OOD loss.L = L id + λL ood (12)C Data Texture The Texture dataset (DTD) (Cimpoi et al., 2014) comprises 5640 images categorized into 47 terms inspired by human perception, aimed at replicating human-like texture recognition in machines. Again, we use the subset collected by Ming et al. (2022) as the OOD test set.The ImageNet-22K dataset (Russakovsky et al., 2015) , formerly known as ImageNet-21K, addresses the underestimation of its additional value compared to the standard ImageNet-1Kpretraining, aiming to provide high-quality pretraining for a broader range of models. We use the filtered subset collected by Wang et al. (2021) We compare SeTAR+FT with 4 finetuning-based baselines. These baselines include:• NPOS. NPOS (Tao et al., 2023) generates virtual anomalies in low-probability regions of ID data without relying on distribution assumptions, enhancing discrimination during training.• CoOp. CoOp (Zhou et al., 2022) optimizes prompts for vision-language models with learnable context vectors for efficient few-shot learning.• LoCoOp. LoCoOp (Miyai et al., 2023a) improves upon CoOp by leveraging CLIP's local features to better distinguish between ID and OOD samples, achieving higher detection accuracy with less training data. We follow the official code11 to prepare and fine-tune the LoCoOp with CLIP-base and CLIP-large. Follow Miyai et al. (2023a) , the top-K, λ, learning rate and epoch num are set to 200, 0.25, 0.002 and 50. Temperature is set to 1 and the text prompt is initiated with "X X X X X X X X X X X X X X X X [CLASS]", where [CLASS] is the ID class name. We average the results from 3 seeds finetuned with 1-shot ImageNet1K valid data.• LoRA. LoRA (Hu et al., 2022) is a low-rank adaptation method that injects trainable lowrank decomposition matrices into the pre-trained model to adapt to downstream tasks. We apply low-rank adaptation to the same weight type as SeTAR+FT, the rank of each layer is set to match the trainable parameters of SeTAR. Details settings can be found in Table 13 .The hyperparameters for SeTAR are shown in Table 12 . And the hyperparameters for SeTAR+FT and LoRA are shown in Table 13 . In this section, we present additional detailed results from the main paper. This includes the detailed results of fine-tuned baselines on the ImageNet1K benchmark in 20 , Table 17 and Table 22 . V-11 V-10 V-9 V-8 V-7 V-6 V-5 V-4 V-3 V-2 V-1 V-0 T-11 T-10 T-9 T-8 T-7 T-6 T-5 T-4 T-3 T-2 T-1 T-0 Modality-LayerIndex IN1K VOC0.15 0.15 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.00 0.00 0.15 0.00 0.00 0.00 0.00 0.00 0.25 0.10 0.20 0.05 0.05 0.00 0.20 0.10 0.35 0.40 0.05 0.25 0.35 0.10 0.00 0.00 0.00 0.25 0.35 0.00 0.25 0.05 0.10 0.20 0.05 0.05 0.00 0.10 0.00 0.25 0.00 (a) CLIP-baseV-23 V-22 V-21 V-20 V-19 V-18 V-17 V-16 V-15 V-14 V-13 V-12 V-11 V-10 V-9 V-8 V-7 V-6 V-5 V-4 V-3 V-2 V-1 V-0 T-11 T-10 T-9 T-8 T-7 T-6 T-5 T-4 T-3 T-2 T-1 T-0 Modality-LayerIndex IN1K VOC0.00 0.00 0.00 0.05 0.15 0.10 0.00 0.00 0.05 0.10 0.20 0.00 0.15 0.00 0.10 0.00 0.00 0.05 0.00 0.00 0.00 0.15 0.00 0.40 0.35 0.00 0.00 0.00 0.10 0.05 0.35 0.00 0.20 0.00 0.05 0.20 0.40 0.35 0.00 0.20 0.00 0.30 0.25 0.10 0.05 0.25 0.05 0.30 0.05 0.10 0.15 0.05 0.20 0.15 0.30 0.10 0.10 0.20 0.10 0.35 0.00 0.05 0.05 0.40 0.00 0.15 0.00 0.30 0.40 0.05 0.05 0.40EQUATION0.40 0.40 0.40 0.40 0.40 0.00 0.00 0.00 0.00 0.10 0.00 0.00 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.30 0.00 0.10 (c) Swin-base • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Code are available at https://github.com/X1AOX1A/SeTAR.https://huggingface.co/openai/clip-vit-base-patch16Temperature is set to 1.0 for the scaled CLIP logits, equivalent to the unscaled CLIP logits with a temperature of 100. We adopt the unscaled setting in our implementation.For SeTAR , the results are the same under different random seeds as it does not require training.We do not report classification accuracy on iNaturalist as we failed to match the labels for the OOD test set.https://huggingface.co/openai/clip-vit-large-patch14https://huggingface.co/microsoft/swinv2-base-patch4-window16-256https://download.pytorch.org/models/resnet50-19c8e357.pthhttps://github.com/AtsuMiyai/LoCoOp 


Decision: All three reviewers agree that this work is of interest and value to the NeurIPS community and that it should be accepted. The AC agrees.

Authors are requested to revise their manuscript based on the reviewer feedback, incorporating the various discussions and additional results from the author-reviewer exchanges.

Reviews: review: paper_summary: paper_summary: The paper presents SETAR, a novel method designed to enhance out-of-distribution (OOD) detection without requiring additional training. The proposed method leverages rank reduction techniques applied to the model weights, specifically targeting the minor singular components, while retaining the principal components that significantly contribute to the model’s performance. SETAR is evaluated across various model backbones, including CLIP-base, CLIP-large, and Swin-base, and demonstrates notable improvements in OOD detection tasks. The paper also provides comprehensive experiments and ablation studies to validate the effectiveness and efficiency of SETAR.
strength_weakness: strength_weakness: I hope the authors could address my following questions.
questions: questions: 1. Table 2: Could you provide more insights on why the performance boost on Swin-Base is significantly more pronounced compared to CLIP-Base and CLIP-Large? Is this disparity related to CLIP’s stronger zero-shot performance?

2. Figure 2: For ImageNet-1K, it appears that decomposing the vision encoder alone is sufficient for OOD detection, with the combination of both vision and text encoders yielding only minor improvements. Could you explain this in more detail?

3. The proposed method is training-free, but the greedy search algorithm used for determining the rank reduction ratio list and performing rank reduction does require computation time. Can you provide details on the time required for the greedy search and rank reduction for SeTAR, as well as the overall computation time of SeTAR+FT, compared to previous fine-tuning methods?

4. The proposed method seems to be an application of LASER, the only contribution is the adaptation of LASER to CLIP model, and benefits OOD detection. Can the authors justify this? The paper is technically solid with comprehensive experiments, but I think the paper is not novel enough given the similarities compared to LASER.
limitations: limitations: Limitation discussed in Appendix
paper_summary: paper_summary: This paper proposes an algorithm for OOD detection along with CLIP models.
It observes that pruning based on SVD decomposition on CLIP models can improve the OOD detection performance.
A greedy search algorithm is developed for searching pruning rations of each weight in CLIP models.

Experiments on regular settings demonstrate obvious performance when compared with the Vanilla GL_MCM method.
strength_weakness: strength_weakness: (1) It seems that the proposed method is general enough to apply to regular models not just CLIP models.    
     Will it also work on CNN-based ResNets and ViT models trained on datasets, like ImageNet and CIFAR?        
      The experimental results focus on CLIP-based models in Table 1.
     More comparisons with previous methods using CNN-based models should be included.     
     
(2) SVD decomposition pruning has strong connections with sparsification-based methods [1,2,3]. 
     Comparisons with this kind of method should be included.

[1] Dice: leveraging sparsification for out-of-distribution detection.     
[2] Extremely simple activation shaping for out-of-distribution detection.       
[3] ReAct: out-of-distribution detection with rectified activations.                   

(3) The proposed greedy search algorithm is efficient. However, it can not be guaranteed to be converged.
     What's the performance variance for 2 independent runs of the search algorithm?
questions: questions: (1) The main results in the paper concentrate on CLIP models.
      Results on other CNN-based models and comparisons with previous methods should be included in the paper.

(2) The proposed greedy search algorithm can not be guaranteed to be converged.
      Thus, a discussion on the stability of the proposed algorithm is required.
limitations: limitations: A section of Impact Statements including limitations is included in the paper
paper_summary: paper_summary: The paper introduces SeTAR, a novel training-free out-of-distribution (OOD) detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection by post-hoc modifying the model’s weight matrices using a greedy search algorithm. The paper also extends this method to SeTAR+FT, a fine-tuning approach to further optimize OOD detection performance.
strength_weakness: strength_weakness: -	The performance gains compared to baselines are relatively small. Given that these results are achieved using a greedy search strategy, the potential of SeTAR may be limited.
-	There is a lack of theoretical analysis explaining why SeTAR is effective. Understanding the underlying principles is crucial for advancing the method and its applications.
-	The experiments indicate that the optimal hyperparameters (λ and K) vary significantly across different backbones. In practical OOD detection scenarios, it is challenging to obtain OOD samples for hyperparameter tuning. Therefore, SeTAR needs to be strengthened in this aspect to ensure robust performance without extensive hyperparameter tuning.
P.S. The layout of Figure 1 is somewhat cluttered.
questions: questions: Please refer to weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This work proposes SeTAR for CLIP-based OOD detection. SeTAR is based on low-rank approximation. It determines the optimal rank for each weight block with a greedy hyperparameter search using validation samples. While SeTAR itself is training-free, the paper also proposes SeTAR+FT which incorporates LoRA as a training extension. Experiments demonstrate that SeTAR outperforms existing methods on CLIP-specific benchmarks.
strength_weakness: strength_weakness: 1. My main concern is that the performance improvement of SeTAR (SeTAR+FT) seems limited compared with MCM and GL-MCM (LoCoOp). In Table 1, for example, all improvements of SeTAR in terms of AUROC are within 1%. The only case where SeTAR leads to noticeable/significant improvements is when the model is the image-only Swin-T (Table 2). Would the authors comment on why this is the case?

2. I'm confused by some of the numbers discussed in the main text. In line 220-222, it says `For example, using Pascal VOC as ID, SeTAR yields an average reduction of 12.84% FPR95 on MCM and 18.95% FPR95 on GL-MCM`. How is the "12.84" and "18.95" computed? According to Table 1, on Pascal VOC, the average FPR95 of MCM and SeTAR is 38.88 and 32.46, respectively. The average FPR95 of GL-MCM and SeTAR is 31.12 and 23.86, respectively. A similar confusion of mine is regarding line 238-240 `when scaled up to CLIP-large, SeTAR+FT outperforms LoCoOp and LoRA by 17.92% and 12.45% FPR95 on the same benchmark`, which doesn't seem to match with the numbers reported in Table 2. Am I missing anything?

3. All considered OOD datasets are far-OOD w.r.t. the ID set. Near-OOD detection has long been recognized as a more challenging and realistic problem [1, 2]. I would be very interested to see how well SeTAR performs on ImageNet v.s. NINCO or SSB [2].

I'm willing to adjust my score if my comments are addressed or clarified.

[1] Detecting semantic anomalies

[2] OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection
questions: questions: See weaknesses
limitations: limitations: Yes

Comments: title: Appreciation for Feedback and Request for Score Review
General Responses
Thank You for Your Consideration and Feedback
SeTAR’s Distinct Advantages and Compatibility
Anticipating Further Remarks
Gratitude for Your Valuable Feedback and Support
Grateful for Your Continued Feedback and Consideration
Clarification and Innovations in SeTAR for OOD Detection
Results on CNNs (Continued)
Comparisons with CLIPN and NegLabel
Results on CNNs
Thanks for the responses from the authors
Reply to further questions
Further questions
Thanks for the Feedback and Revised Score
Thanks for the rebuttal
comment: Dear Reviewer SyYP,

Thank you for your recognition of our paper. We appreciate your comment that “the paper deserves to be accepted for its applications on CLIP models.” Your feedback has been incredibly valuable to us.

As the rebuttal period is nearing its end, we have provided a quick summary of our responses and updates. We kindly ask you to consider these in your final scoring.

Thank you once again for your valuable review.

Best regards,

The Authors
**Dear AC and Reviewers,**

We thank all reviewers for their thorough comments and constructive feedback. We greatly appreciate that the reviewers found our paper to be well-written (SyYP, hoZJ), novel (aAbP, 6fvA), effective (SyYP, aAbP), with solid and comprehensive experiments and analysis (aAbP, 6fvA), and sufficient for acceptance (SyYP, aAbP). We would like to summary some key points in the rebuttal:

**1. Effectiveness of Our Method:**
SeTAR consistently demonstrates superior performance across various OOD detection tasks, surpassing existing zero-shot methods and sparsification techniques. This includes its effectiveness when applied to different score functions and backbones, such as CLIP, Swin, and ResNet. Notably, SeTAR+FT outperforms the current state-of-the-art in efficient fine-tuning baselines, offering significant improvements in both AUC and FPR metrics. In scenarios involving label expasion, SeTAR not only matches but exceeds the performance of the baseline NegLabel, establishing a new state-of-the-art. Furthermore, our method exhibits robust performance in both Far-OOD and Near-OOD scenarios, making it a versatile and powerful tool for diverse OOD detection challenges.

**2. Compatibility:**
A key strength of SeTAR is its high compatibility with various score functions (MCM, GL-MCM, MSP, Energy), multiple model backbones (CLIP, Swin, ResNet), and advanced OOD methods like CLIPN and NegLabel. This flexibility distinguishes SeTAR from other CLIP-based methods such as CLIPN, NegLabel, and LoCoOp, as well as from CNN-based sparsification methods. The ability to integrate seamlessly with different approaches underscores SeTAR's potential to enhance and complement existing methods, rather than being in conflict with them. This compatibility allows our method to be effectively applied in a wide range of OOD detection settings, further solidifying its practical value.

**3. Efficiency:**
SeTAR is designed to be both lightweight and efficient, addressing the need for a resource-effective solution in OOD detection. Specifically tailored for zero-shot scenarios, SeTAR requires no additional training, making it highly practical for real-world applications where computational resources may be limited. Moreover, SeTAR+FT leverages our approach alongside existing parameter-efficient fine-tuning methods, enabling highly efficient training while still achieving state-of-the-art performance. This efficiency is particularly evident when compared to methods that rely heavily on extensive training or large amounts of external data, such as CLIPN or NegLabel, which require significantly more resources.

We are committed to refining our paper to ensure it is as clear and accurate as possible. We will make sure to address all the comments and suggestions provided by the reviewers and will expand the discussion of related works on OOD detection in computer vision to strengthen the context and positioning of our contributions. We sincerely appreciate the reviewers' insights and support throughout this process.

Sincerely,

All authors
Thank you very much for your thoughtful consideration and for raising the rating. We appreciate your constructive feedback throughout the review process. Your insights have been invaluable in helping us refine our work, and we are grateful for your support.
Thank you for your clarification. Overall, I believe this paper is a borderline. I will raise the rating to 5 and be open to the AC's judgment.
# SeTAR’s Distinct Advantages and Compatibility

We are pleased to learn that our previous rebuttal successfully addressed many of your concerns, and we sincerely appreciate your thoughtful feedback throughout this process. However, we would like to take this opportunity to further clarify some key aspects regarding the performance of SeTAR, particularly in comparison to existing methods like CLIPN and NegLabel.

# 1. **New Perspective and Orthogonality:**
   
- Our method enhances OOD performance from a novel angle and is designed to be **highly compatible with a wide range of OOD methods**. SeTAR can be integrated with different score functions (MCM, GL-MCM, MSP, Energy), various backbones (CLIP, Swin, ResNet), and advanced OOD methods like CLIPN and NegLabel. This flexibility underscores that our approach is not in conflict with existing OOD methods but rather offers a complementary solution that can be layered with them. **By focusing on improving the intrinsic capability of models**, SeTAR is positioned to continuously enhance the effectiveness of other potential OOD methods in the future.

- In contract, both CLIPN and NegLabel enhance OOD detection by leveraging negative text labels, which **inherently limits their applicability to vision-language models like CLIP**. This approach is not directly applicable to pure vision models such as Swin and ResNet, which do not incorporate a text encoder. In contrast, our method is not constrained by such limitations and can be effectively applied to both vision-language models and pure vision models, offering a broader applicability across different model architectures.

# 2. **Lightweight and Efficient Approach:**

- SeTAR and SeTAR+FT are designed as **lightweight and efficient methods**. SeTAR specifically addresses zero-shot scenarios, while SeTAR+FT combines our approach with existing parameter-efficient fine-tuning methods to enable efficient training. In contrast, other methods often rely on extensive training or large amounts of external data.
- For instance, CLIPN involves training an additional NO-encoder using the CC-3M dataset, which is **3000 times** larger in data volume, **25 times** greater in trainable parameters, and requires over **300 thousand times** training time, compared to SeTAR+FT. Similarly, NegLabel constructs negative labels information from **70,000 label candidates**.
- In contrast, SeTAR operates independently of external training data such as CC-3M or additional labels, and requires no further training, underscoring its efficiency and practicality.

# 3. **Advancing the State of the Art:**

- Even when applied to a high-performing method like NegLabel, SeTAR further enhances model performance. Specifically, we were able to **reduce the average FPR by an additional 9% relative to NegLabel’s already impressive 25.40 average FPR**, achieving a new state-of-the-art with a 23.09 average FPR. This demonstrates SeTAR’s potential to push the boundaries of current OOD detection performance to new levels.

We hope these clarifications further illustrate the value and uniqueness of SeTAR. We are grateful for your consideration and are committed to refining our work based on your insights.
Thank you for your response. I have read the other comments as well as all your rebuttals. The contribution of this paper is mainly focused on OOD detection tasks based on CLIP. However, SeTAR faces certain challenges when compared to CLIPN and NegLabel. I still have concerns about the relatively limited performance of SeTAR. I believe this paper is a borderline (4 or 5). I may raise the rating to 5 after thorough discussion in the next phase.
Dear Reviewer 6fvA

As the discussion period is coming to a close, we would appreciate it if you could let us know whether our recent rebuttal has addressed some of your concerns or questions. We are more than happy to address any further issues you may have. Engaging in this discussion will greatly help us in refining and improving our paper.

We look forward to your response or acknowledgment once you have read our message, as your support is very important to us.

Best regards,

The Authors
We sincerely appreciate the reviewer’s thoughtful comments and the decision to raise the score. We will ensure that the discussion on related works in OOD detection, particularly the references provided, is expanded and integrated into the revised version of our paper. This will further strengthen the context and positioning of our contributions. Thank you again for your valuable feedback and support throughout the review process.
Thank you for your additional clarifications. Based on your responses to all the reviewers, I think the paper’s contributions are sufficient for acceptance. I recommend that the authors expand the discussion of related works on OOD Detection in Computer Vision [1,2,3] in the revised version to further strengthen the paper. I will raise my score to 6.

[1] How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?

[2] Learning with Mixture of Prototypes for Out-of-Distribution Detection

[3] Energy-based Hopfield Boosting for Out-of-Distribution Detection
We sincerely appreciate the reviewer’s continued engagement and thoughtful feedback. We will include the CNN comparison results in the main paper as suggested.

We hope that the additional comparisons enhance the overall evaluation of our work and would be grateful if the reviewer could kindly reconsider the rating in light of these updates.
Thanks for your responses to address my concerns.

I strongly recommend including the comparisons with CNNs in the main paper. Although the improvements seem not obvious with CNNs, I still think the paper deserves to be accepted for its applications on CLIP models.
We sincerely appreciate the reviewer’s thoughtful feedback and the opportunity to clarify and expand on certain aspects of our work.

# 1. Clarification on LASER

We appreciate the reviewer’s reminder regarding Section 5.1 of the LASER paper. We also took note of this section during our review. What we intended to highlight in our original rebuttal is that, while LASER employs a single greedy search strategy, our work delves deeper into the nuances of conducting greedy search effectively for OOD detection (as detailed in Section 4.4). Specifically, LASER’s approach focuses on composing reductions across layers without exploring the broader landscape of greedy search strategies. In contrast, we systematically analyze and compare different greedy search techniques, evaluating their effectiveness across various layers and backbones. This detailed exploration allows our method to be more finely tuned for the specific challenges of OOD detection, thereby providing a more robust and versatile solution.

# 2. Innovation in Post-Hoc Sparsification Methods

Post-hoc sparsification methods are widely utilized to enhance CNN-based OOD detection, with well-known examples including ReAct[1], ASH[2], and Dice[3]. These methods typically operate in the weight or activation space, aiming to improve OOD detection by modifying the model’s internal structures. However, a significant limitation of methods like ASH and ReAct is that they are designed for models that have been trained on in-domain (ID) data, such as ResNet50, where distinct activations for ID and OOD samples are expected. These methods fail to generalize to zero-shot OOD detection models like CLIP, where the model has not been fine-tuned on any ID data, and the activations for ID and OOD samples share similar distributions.

Our approach addresses these limitations by introducing two key innovations: first, we operate within the space derived from SVD decomposition, which allows us to capture the most informative components of the model while discarding noise. Second, our method is specifically designed to function as a post-hoc approach compatible with CLIP’s zero-shot OOD detection capabilities. This dual innovation not only enables our method to bypass the limitations of traditional sparsification techniques but also allows it to enhance OOD detection in models that are not specifically trained on ID data. We will ensure that these distinctions and innovations are clearly articulated in the revised version of our paper.

---

**References**:

- [1] ReAct: out-of-distribution detection with rectified activations. 
- [2] Extremely simple activation shaping for out-of-distribution detection.
- [3] Dice: leveraging sparsification for out-of-distribution detection.
# 2. Effectiveness of Our Method Across CLIP, Swin, and CNN Architectures

1. **Broad Effectiveness Across Architectures:** Our method has consistently proven effective across a range of architectures. As detailed in Table 1 and Table 7 of our paper, **SeTAR outperforms vanilla methods on both CLIP and Swin models**. Furthermore, even with a basic setup on ResNet, **SeTAR surpasses current state-of-the-art sparsification-based methods**. The comparison with Swin in our previous response underscores the limitations of previous sparsification-based approaches, which struggle with models like Swin. In contrast, **SeTAR achieves state-of-the-art performance across all major vision architectures**, demonstrating its versatility and generalizability.

2. **Significance for ViT-based Models:** ViT-based models, like CLIP, are receiving increasing attention in research due to their scalability and strong performance [5][6][7][8][9]. Our method’s superior results on ViT models highlight its potential for advancing OOD detection in these architectures, making it particularly relevant as ViT models become more widely adopted.

3. **Limitations of Sparsification-Based CNN Methods:** Sparsification-based CNN methods like ReAct and ASH cannot be applied as a post-hoc method to CLIP-based zero-shot OOD detection model. Both methods rely on the assumption that ID and OOD images produce distinct activations in models trained specifically on ID data, such as ResNet50. However, in large-scale pretrained models like CLIP, the activations for ID and OOD images are not significantly different. Consequently, methods like ReAct and ASH, which are limited to models trained on downstream ID-domain tasks, constrain their effectiveness in enhancing CLIP’s zero-shot OOD detection capabilities. In contrast, our method can be applied as a post-hoc method to enhance CLIP's zero-shot OOD detection capabilities.

4. **Contribution of Specialized Methods:** Despite their limitations, specialized methods play a crucial role in their respective domains. For instance, sparsification-based CNN methods significantly enhance OOD detection in CNN models, even though they may not perform well on CLIP models. Similarly, methods like GL-MCM [10] and LoCoOp [11], which utilize CLIP’s local features, substantially improve MCM scores and the performance of fine-tuned models. Although these methods are specialized, they contribute meaningfully to the ongoing development and advancement of the field.

---

**References:**

- [1] ELRT: Towards Efficient Low-Rank Training for Compact Neural Networks  
- [2] Learning Low-rank Deep-Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification  
- [3] Training CNNs with Low-Rank Filters for Efficient Image Classification  
- [4] Convolutional Neural Networks with Low-Rank Regularization  
- [5] Multimodal Learning with Transformers: A Survey  
- [6] Self-Supervised Multimodal Learning: A Survey  
- [7] The Llama 3 Herd of Models  
- [8] Scaling Vision Transformers to 22 Billion Parameters  
- [9] How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites  
- [10] Zero-Shot In-Distribution Detection in Multi-Object Settings Using Vision-Language Foundation Models  
- [11] LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning
We apologize for the late reply and appreciate your patience.

# 1. Comparisons with CLIPN

- CLIPN [1] is pre-trained on the **CC-3M** dataset for **10 epochs**, involves training an additional NO-encoder for over 64 million parameters 
- On ViT-B/16, CLIPN-C achieves an FPR95 of 38.59 and an AUROC of 86.35, while CLIPN-A achieves an FPR95 of 31.10 and an AUROC of 93.10.
- In contrast, our SeTAR method does not involve any training. When combined with fine-tuning (SeTAR+FT), it operates in a 1-shot setting, utilizing only **1,000 images** for training, with just **1.6% of the parameters** being trainable over **5 epochs**.
- For comparison, SeTAR achieves an FPR95 of 33.12 and an AUROC of 91.32 without any training. With minimal fine-tuning (SeTAR+FT), it achieves an FPR95 of 32.19 and an AUROC of 92.31 with only 1,000 training samples.

Given the significant differences in the amount of training data, the number of trainable parameters, and the computational resources required, we believe that a direct comparison between CLIPN and our method is not entirely fair. Despite the minimal computational resources involved, SeTAR and SeTAR+FT achieve performance levels close to those of CLIPN, demonstrating the efficiency of our approach.

# 2. Comparisons with NegLabel

- **Difference from NegLabel:** Our method differs from NegLabel [2] in its focus. SeTAR primarily aims to enhance the model's intrinsic performance through SVD pruning, without incorporating any additional knowledge or inputs. In contrast, NegLabel improves OOD detection by constructing large scale virtual negative labels from the data perspective.
- **Compatibility with NegLabel:** Since SeTAR and NegLabel have different focuses, they are not mutually exclusive. As mentioned in our paper, SeTAR is highly compatible with various score functions and can also work alongside data augmentation methods like NegLabel. For instance, by simply merging negative labels into the label space for searching and testing, SeTAR can further surpass NegLabel's performance.

    |ViT-B/16 |iNaturalist FPR↓|iNaturalist AUC↑|SUN FPR↓|SUN AUC↑|Places FPR↓|Places AUC↑|Texture FPR↓|Texture AUC↑|Average FPR↓|Average AUC↑|
    |:---|-:|-:|-:|----:|-:|-:|-:|-:|-:|-:|
    |CLIPN\*|23.94 |95.27|26.17|93.93|33.45|92.28|40.83|90.93|31.10|93.10|
    |NegLabel\* |1.91 |99.49|20.53|95.49|35.59|91.64|43.56|90.22|25.40|94.21|
    |SeTAR|0.15 |99.54|19.06|95.84|30.63|92.22|42.54|90.30|**23.09**|**94.48**|

    > \* cited from [2]
    
In summary, SeTAR demonstrates strong performance through its structural enhancements, particularly with SVD pruning, which significantly improves model robustness and OOD detection without requiring additional training. Furthermore, when combined with data augmentation methods like NegLabel, SeTAR's effectiveness is further amplified, showing even greater improvements in OOD detection metrics. 

---

**References**:

- [1] CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No
- [2] Negative Label Guided OOD Detection with Pretrained Vision-Language Models
We sincerely appreciate the reviewer’s insightful comments. Here are our detailed responses:

# 1. Results on ResNet50

1. **Setup:** We conducted experiments using only the ID loss, applying low-rank approximation on the in-feature and out-feature dimensions of the convolutional layers, combined with ASH for search. The results are as follows:

    | ResNet50 |iNaturalist FPR↓ |iNaturalist AUC↑ |SUN FPR↓ |SUN AUC↑ |Places FPR↓ |Places AUC↑ |Texture FPR↓ |Texture AUC↑ |Average FPR↓ |Average AUC↑ |
    |:--|---:|---:|---:|---:|-:|-:|--:|--:|--:|--:|
    | Softmax \* |54.99|87.74 |70.83 |80.86 |73.99 |79.76 | 68.00 | 79.61 | 66.95 | 81.99 |
    | Energy \* |55.72|89.95 |59.26 |85.89 |64.92 |82.86 | 53.72 | 85.99 | 58.41 | 86.17 |
    | ReAct \* |20.38|96.22 |24.20 |94.20 |33.85 |91.58 | 47.30 | 89.80 | 31.43 | 92.95 |
    | DICE \*|25.63|94.49 |35.15 |90.83 |46.49 |87.48 | 31.72 | 90.30 | 34.75 | 90.77 |
    | ASH-P \* |44.57|92.51 |52.88 |88.35 |61.79 |61.79 | 42.06 | 89.70 | 50.32 | 89.04 |
    | ASH-B \* |14.21|97.32 |22.08 |95.10 |33.45 |92.31 | 21.17 | 95.50 | 22.73 | 95.06 |
    | ASH-S \* |11.49|97.87 |27.98 |94.02 |39.78 |90.98 | 11.93 |97.60| 22.80 | 95.12 |
    | SeTAR|10.08|98.11|27.68|94.15|39.22|91.24| 12.54 | 97.51 | **22.38** | **95.25**|
    
    > \* cite from "Extremely simple activation shaping for out-of-distribution detection."
    
2. **Not Highly Dependent on OOD Loss (Eq. 11):** Even under this simple setup, SeTAR outperforms ASH's best performance. This demonstrates that our method can effectively enhance the model's ability to distinguish between ID and OOD samples, even when relying solely on the ID loss. It’s important to note that this result was obtained under a basic setup, and due to time constraints, we did not further tune or explore other detailed configurations specific to CNNs, such as:

   - **Low-Rank Settings for Convolutional Layers:** The optimal low-rank structure and dimensions for convolutional layers have not been thoroughly researched. For example, ELRT [1] proposes low-rank approximation directly in the high-order tensor format, while other methods [2][3][4] conduct and maintain the 4-D convolutional layer in the format of a low-rank 2-D matrix.

   - **Pseudo-OOD Feature Extraction in CNNs:** As mentioned in our previous response, there are methods to construct OOD features within CNNs as well, which could further improve our model's performance.
Thanks for the responses from the authors.

Q1. I think it is a crucial weakness that the method heavily depends on the loss (Eq. 11).
       Most previous research conducts experiments on CNNs. 
       Although the authors provide comparisons with previous methods using the Swin backbone, the results seem to be a little bit wired, especially for ASH-S. I still recommend the authors could compare their method and previous works on CNNs for fair comparisons.
We sincerely appreciate the thoughtful feedback provided by the reviewer. Here are our responses:

# 1. Alternative Loss Function for Eq. 11

- Our loss function (Eq. 11) leverages local features because it requires pseudo-OOD features, which are inherently present in ViTs and can also be constructed in CNNs. In CNNs, alternative methods for OOD feature construction are available, such as the approach proposed in [NPOS](https://arxiv.org/pdf/2303.02966)[4], where boundary ID embeddings are selected based on non-parametric k-NN distances, and outliers are synthesized by sampling from a multivariate Gaussian distribution centered around these boundary embeddings. Additionally, [CLIP-OS](https://arxiv.org/pdf/2404.00323)[5] suggests using CLIP for outlier synthesis, which can similarly be adapted for constructing OOD features in CNNs.

- However, due to the complexity of implementation and time constraints, we were unable to conduct the corresponding experiments. Nevertheless, our method is not limited to CLIP models; by utilizing different OOD feature construction methods, our approach can be readily adapted to CNN models as well.

# 2. Comparisons on Swin Backbone

- We conducted experiments using the Swin backbone with ReAct [3], DICE [1], and ASH [2] methods. Specifically:

    1. **Codebase:** For ASH and ReAct, we implemented Swin-transformer based on the [official ASH repository](https://github.com/andrijazz/ash/blob/main/config/vit_config.yml), strictly following the original settings and applying sparsification before the final linear layer. Since ASH does not provide an implementation for DICE, we based our implementation on the [DICE official repository](https://github.com/deeplearning-wisc/dice/tree/master/models).
      
    2. **Hyperparameter Search:** Following ASH, we experimented with various parameter combinations and reported the best results: DICE pruning thresholds included {10%, 15%, 70%}, ReAct clipping thresholds {1.0, 1.5, 1.33}, and ASH-S pruning thresholds included {60%, 65%, 90%, 95%}.
      
    3. **Datasets:** We used ImageNet1K as the ID dataset.

- The results are shown below. It is evident that compared to sparsification-based methods, SeTAR exhibits the best performance. ReAct shows improvements over the baseline in both scoring functions, while DICE shows improvement only with MSP; ASH-S performs poorly across the board.

    | SwinV2-Base| Score| iNaturalist FPR↓ | iNaturalist AUC↑ | SUN FPR↓ | SUN AUC↑ | Places FPR↓ | Places AUC↑ | Texture FPR↓ | Texture AUC↑ | Average FPR↓ | Average AUC↑ |
    |:-|:-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|
    | Vanilla* | MSP|44.78 |89.89 |63.12 |82.81 | 67.07 | 81.45 |62.04 |82.33 |59.25 |84.12 |
    | React* | MSP|42.98 |90.39 |61.34 |83.89 | 65.11 | 82.64 |61.22 |81.37 |57.66 |84.57 |
    | DICE*| MSP|43.02 |89.03 |62.22 |78.31 | 65.82 | 79.35 |57.75 |81.48 |57.20 |82.04 |
    | ASH-S* | MSP|53.21 |78.72 |73.71 |66.56 | 79.75 | 60.75 |60.27 |75.52 |66.73 |70.39 |
    | SeTAR (Ours) | MSP| **41.44** |**91.08** |**60.05** |**85.04** | **64.31** | **83.70** |**58.39** |**83.26** |**56.05** |**85.77** |
    | Vanilla* | Energy |57.52 |81.60 |71.98 |72.93 | 76.90 | 68.90 |53.65 |80.96 |65.01 |76.10 |
    | React* | Energy |41.78 |88.34 |60.98 |79.19 | 68.07 | 75.56 |53.72|80.39|56.14|80.87|
    | DICE*| Energy |64.45 |74.91|83.04|62.67|95.18|47.05|94.65|33.45|84.33|58.18|
    | ASH-S* | Energy |99.67 |19.26 |99.29 |22.91 | 99.51 | 21.49 |98.51 |33.45 |99.25 |24.28 |
    | SeTAR (Ours) | Energy |**41.71** |**89.42** |**56.53** |**83.29** | **62.84** | **80.20** |**45.37** |**84.76** |**51.61** |**84.42** |
    
    > \* denotes our rerun
    
These results demonstrate that SeTAR significantly outperforms sparsification-based methods, highlighting the effectiveness of our approach.

# 3. Verification of ASH-S Implementation

- Due to the poor performance of ASH-S, we verified the results using the official code. Although the paper does not report ViT results, related scripts are available in their codebase. We tested the official [ViT script](https://github.com/andrijazz/ash/blob/main/config/vit_config.yml), and the results are shown below, confirming that ASH-S continues to perform poorly on ViT.
    
    | ViT-B/16 | Score| iNaturalist FPR↓ | iNaturalist AUC↑ | SUN FPR↓ | SUN AUC↑ | Places FPR↓ | Places AUC↑ | Texture FPR↓ | Texture AUC↑ | Average FPR↓ | Average AUC↑ |
    |:-|:-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:|
    |Vanilla*| Energy |64.08 |79.24|72.77|70.25|74.30|68.44|58.46|79.30|67.40|74.31|
    |ASH-S*| Energy |99.98 |7.28 |99.64|17.82|99.59|19.72|98.09|27.31|99.32|18.03|
    
    > \* denotes our rerun

**References**:

- [1] Dice: leveraging sparsification for out-of-distribution detection.  
- [2] Extremely simple activation shaping for out-of-distribution detection.  
- [3] ReAct: out-of-distribution detection with rectified activations.
- [4] Non-Parametric Outlier Synthesis.
- [5] CLIP-driven Outliers Synthesis for few-shot OOD detection.
Thank you for the detailed reply.

I find the explanation about the "limited room for AUC improvement" somewhat inadequate. Additionally, I noticed that your primary comparisons are with MCM and its variant GL-MCM. I suggest comparing SeTAR with more recent baselines, such as “CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No” (ICCV2023) and “Negative Label Guided OOD Detection with Pretrained Vision-Language Models” (ICLR2024), to more convincingly demonstrate its performance advantages.

Your responses to the theoretical analysis and hyperparameter tuning concerns have addressed some of my doubts. Thank you for the additional clarifications.
Thank you to the authors for providing comprehensive clarifications. Most of my concerns have been addressed. I would like to note that the contribution of greedy rank search across multiple layers has already been proposed in the LASER paper (Sec. 5.1, “Composing reductions across layers”). Given the technical soundness, extensive experiments, and the innovative application of rank reduction in the domain of OOD detection using a Vision-Language model, I will maintain my original score.
Hi, thanks for the responses from the authors. I still have some confusion on the paper.
(1) Is there any alternative loss function for Eq. 11? 
      Although the proposed method is general, its capability is heavily limited by this loss.

(2) Could the authors provide comparisons with sparsification-based methods using Swin or ViT backbones?
Thank you very much for your constructive feedback and for raising the score. We are grateful for your detailed review and are glad that our responses addressed your concerns. We will ensure that the manuscript clearly indicates that the performance improvements are relative. Additionally, we will incorporate the near-OOD results into the main text as suggested. Your insights have been invaluable in enhancing our work, and we sincerely appreciate your support and encouragement.
My concerns are addressed well and I'd like to raise my score to 6. For point 2, please make it clear in the manuscript that the performance improvement is relative (by default I think people assume/expect absolute improvement being discussed). For point 3, I strongly recommend including the near-OOD results in the main text so that later works can continue to work the meaningful & challenging near-OOD detection problem (rather than keep working only on far-OOD by following previous works).