SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation
Authors: Yixia Li
Boya Xiong
Guanhua Chen
Yun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Out-of-distribution detection
CLIP
low-rank approximation
trustworthy AI
Primary area: safety_in_machine_learning
TL;DR: SeTAR, a training-free method that improves out-of-distribution detection by post-hoc modification of weight matrices, with extended fine-tuning version SeTAR+FT.

Abstract: Out-of-distribution (OOD) detection is crucial for the safe deployment of neural networks. Existing CLIP-based approaches perform OOD detection by devising novel scoring functions or sophisticated fine-tuning methods. In this work, we propose SeTAR, a novel, training-free OOD detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection via post-hoc modification of the model's weight matrices using a simple greedy search algorithm. Based on SeTAR, we further propose SeTAR+FT, a fine-tuning extension optimizing model performance for OOD detection tasks. Extensive evaluations on ImageNet1K and Pascal-VOC benchmarks show SeTAR's superior performance, reducing the relatively false positive rate by up to 18.95\% and 36.80\% compared to zero-shot and fine-tuning baselines. Ablation studies further validate our approach's effectiveness, robustness, and generalizability across different model backbones. Our work offers a scalable, efficient solution for OOD detection, setting a new state-of-the-art in this area.

SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation
Yixia Li Boya Xiong" Guanhua Chen't Yun Chen
Southern University of Science and Technology
Shanghaii University of Finance and Economics
MoF Key Laboratory of Interdisciplinary Research of Computation and Economics,
Shanghai University of Finance and Economics liyixia@me.. com, xiongboyau163. sufe.............. sufe. edu. cn
chengh3@sustech.c edu. en, yunchen@sufe.. manchen@sufe.edu..cr
Abstract
nut-of-distribution (OOD) detection is crucial for the safe deployment of neural networks. Existing CLIP-based approaches perform OOD detection by devising novel scoring functions sophisticated fine-tuning methods. In this work, pro- pose SeTAR, a novel, training-free OOD detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection via post-hoc modification of the model's ' weight matrices using simple greedy search algorithm. Based on SeTAR, we further propose SeTAR+FT, a fine-tuning extension optimizing model performance for OOD detection tasks. Extensive evaluations on ImageNet and Pascal-VOC benchmarks show SeTAR's 's superior performance, reducing the relatively false positive rate by up to 18.95% and 36. 36.80% compared to zero-shot and fine-tuning baselines. Ablation studies further validate SeTAR'ss effectiveness, robustness, and generalizability across different model backbones. Our work offers a scalable, efficient solution for OOD detection, setting new state-of- sate-ooftthe-ar in this area.
Introduction
The task of out-of-distnn of-distribution (OOD) detection (Hendrycks & Gimpel 2017 Ming et al. 2022) aims identify whether input data comes from an unknown distribution. It has garnered significant attention in the machine learning community recently (Hendrycks et al. 2020 Xu et al. 2021 Miyai a (023a). While machine learning models are trained with supervised in-tistribttion (ID) data, they often struggle to generalize to OOD data encountered in real-world applications (Emmott et al. 2016) like autonomous vehicles and healthcare. These OOD samples pose challenges as they are not represented in the training data. Consequently, OOD detection plays crucial role in developing reliable and trustworthy machine-learning models suitable for real-world deployment (Bai al. 2023). It allows models to filter out and reject these awkward inputs effectively, and enables the use of curated and labeled OOD samples to further train for a more robust model in the wild.
Previous research has primarily focused on detecting OOD instances in either visual (De Vries & Taylor 2018 |iing et al. 2018 Hendrycks et al.) 2022 or textual data (Hu & Khan 2021 Zheng et al. 2020 Zhou et al. (021). Recently, significant progress has been made in multimodal tasks like
multimodal retrieval (L.i et al. 2023) Caesar al.) (2088) and image classification (Yu al.) (022).
thanks to vision-and-languaa pretrained (VLP) models like CLIP (Radford al.) 2021). More recent
"Equal Contribution.
Coorrspponding Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

studies have explored OOD detection with CLIP, grouped into zero-shot methods (Fort et al. 2021
Ming al. [022 Miyai et al. 2023b) and finetuning- methods (Ming & Lij 2023] Tao et al. 2023] MMyai et al. 2023a). However, the zero-shot methods suffer from suboptimal performance due to potential domain gaps with IID downstream data. On the other hand, finetuning-
methods carry
the risk of deconstructing the intricate representations learned by CLIP which requires a meticulously designed training strategy. Sparsification-bast have demonstrated potential in OOD detection within CNNs, leveraging the assumption that ID and OOD samples produce distinct activation patterns. Nevertheless, their effectiveness diminishes in large-scale pre-trained models such as CLIP, where activation differences become more subtle, thereby limiting their applicability primarily to models tine-tunedde on downstream ID-domain datasets.
approaches (Sun et al., 2021 Djurisic et al. 2023)
In this work, we propose SeTAR, training-free fareingg-feeeaae and effective OOD detection method by selective low-rank approximations. Low-rank approximation is to approximate given matrix by finding a lower-rank matrix that closely resembles the original matrix. Previous research has demonstrated that using low-rank approximation matrices can achieve comparable performance to full parameters in various scenarios, as observed in tasks such large language model (LLM) fine-tuning (Hu et al., 2022) and model pruning (Hajioolhoosinii et al. 2021). These approaches typically preserve the same rank across different low-rank approximation matrices. In our work, we demonstrate that it is possible to significantly enhance the performance of OOD detection by selectively manipulating the weight matrices in the CLIP model, including the choice of weight matrices and the ratio of singular vectors to be reduced. Specifically, we propose a simple top-to-bottoon and image-to textext greedy search algorithm to manipulate W up in the CLIP model. Our method applies to various model backbones and does not require any additional training or new parameters. Building upon SeTARR, we further demonstrate its effectiveness for fine-tuning initialization, referred to STTAR+FT.
We conduct extensive evaluations and achieve sfate-of-the-arr performance on common OOD detec- tion benchmarks for CLIP, including the ImageNet K and Pascal- benchmarks. Compared to vanilla MCM and GL-MCM, SeTAR with the CLIP backbone reduces relatively FPR95 by 9.5% and 12.0% on average across two benchmarks, respectively. When further integrate fine-tuning into SeTAR, SeTAR+FT outperforms the sfate-of-the-an fine-tuning baselines LoCoOp MMiyai et al. 2023ab and LoRA (Hu et al.) 2022). Moreover, perform comprchensive ablation study and analysis to verify and understand SETAR. In summary, our key results and contributions:
1. We propose SeTAR, a simple yet effective OOD detection method based on selective low- rank approximation. It is training- free as it only performs post-hoc modification weight matrices. SeTAR applies to a variety of scoring functions and model backbones. It can be readily integrated with existing zero-shot OOD detection methods.
2. We further extend SeTAR SeTAR+FT, which demonstrates the effectiveness of SeTAR in improving the performance of finetuning- OOD detection methods and achieving new state-of-the-art results.
3. We extensively evaluate SeTAR and SeTAR+FT across a diverse set of OOD detection tasks. It consistently outperforms baseline methods and establishes new statc-of-the the-the-rtt art results on CLiP-based OOD detection benchmarks. On ImageNetIK, ScTAR achieves an AUROC of 91.32% with CLIP backbone and GL-MCM score. The score further increases to 92.31% %% when combined with the finetuning- based detection method.
4. We perform comprehensive ablation studies and empirical analyses to verify and under- stand SeTARR We hope that this work will shed light on future explorations on in-depth understanding of the SeTAR method
2 Preliminaries
CLIP Architecture The CLIP model (Radford al., 2021) comprises image encoder E" and a text encoder (.).. aligned via contrastive learning on web-seale image-text pairs. We focus on CLIP-ViT, where the image encoder is a Vision Transformer (ViT). Each ViT layer includes multihead self-attention sublayer and a feed-forward sublayer. In the self-attention module, the hidden state projected into different spaces using learnable parameters Wq. Wk. W.. The outputs are
3 Code available https:

---

N
Wup
vr
FeodForward
Waown Down Projection
Low-nank
Appreximation
LayerNorm
Wup
Activation F F'ucction
Mulit-head Atiention
(c)
Up Projection
Wap
LayerNorm
Parch Tokcon
Emboolding
Σ
Σ
(a)
(b)
(d)
The overview of SeTAR. (a) The structure of the CLIP image and text encoder. (b) The details of the feed-forward sublayer. (c) For each encoder layer, replace the W up weight
Figure 1:
matrix with its low-rank approximation W Wup- (d) The illustration of E before and after low-rank
approximation. More details are in Section 3.1
concatenated and projected back with another linear matrix W.. The feed-forward module projects the hidden state into wider space using W up and then back with W after non-linear actívation (Figure T.. Given the similarity between the image and text encoder layers, we adopt consistent notations for the linear matrices in both. Each encoder also includes linear projector W, map their representations into a shared space for contrastive learning.
Zero-shot OOD Detection with CLIP Zero-shot OOD detection aims to separate ID and OOD data without an ID training dataset. Given the CLIP, the ID classes are defined by the classification task of interest, which differs from the classes used in CLIP pretraining. Accordingly, OOD is defined as classes not belonging to any ID class, making the OOD detector a binary classifier. MCM (Ming ett al., (022) and GL-MLM (Miyai et al. 2023b) are two zero-shot CLIP-based OOD detection methods. Formally, let x be the test image and Tin = {yc)<= be the set text prompts containing M ID class labels (e.g.. "a photo of [CLASS]"). The image segmented into / image patches x = (x1,..., 21). Following CLIP, we add [cls] before the image patches and use the output [cls] from the visual projector Wp as the global image feature (h" E RR). The outputs of other patches are projected by the visual projector as the local image features (p" = (pi: pt) E TR(xd), For the text prompt ye T we add an additional [eos] after the text tokens and use the output feature of [cos] from the textual projector W, as the concept feature of ID class e f E IRd).
1
The label-wise image-concept matching (IWIC) score measures how well test image x aligns with a concept ye, using either global or local features. The global IWIC score s. (.) is the cosine similarity between the global image feature h" and the concept feature ht: o (x) cos pos_simmh"' ,()) The local IWIC score s/((-) is the max- max-pooled cosine similarity between image patch features p. and the concept feature h;: s. (x) max, cos cos__oosss____ sim(p", ht). The MCM and GL-MCM scores defined as:
e (x0/T
SMCM (x) EKe (x)//T
(1)
(x1/7
SCL-MCM (x) = SMCM (x) max
(2)
ΣK ext(x)/7"
where and r' are the temperature hyperparameters. MCM only uses global image features, while GL-MCM additionally considers local image features. For ID data, both MCM and GL-MCM scores will be matched of the concept features with high score; and vice versa. As result, OOD detection function can be formulated as:
{
S (x) > S
G (x)
<
where S (x) either the MCM or GL-MCM score, is the threshold value. By convention, G(x) represents the ID class and G(x) = 0 indicates the OOD class. The À is chosen that a high fraction of ID data (e.g., 95%) ) above the threshold. We follow previous work (Miyai et al.) 2023a) to use either MCM GL-MCM score for OOD detection in this work.

---

3 Method
We introduce ScTAR, training-free and effective technique for improving OOD detection perfor- mance (see Figure Our key idea is to perform post-hoc modification to CLIP weight matrices by selectively replacing them with their low-rank approximations. It complementary to existing CLIP-based zero-shot OOD detection methods and could be further extended to finetuning- based methods, which we term as SeTAR+FT.
3.1 OOD Detection with Selective Low-Rank Approximation
Low-Rank Approximation Given a linear matrix W E [Rm its Singular Value Decomposition (SVD) is denoted as W = UEVT, where U [ U2,." '1mm IR m V = [U1, 22; V. E Raxx, and Σ E _ is matrix whose entries are all zero except for the singular values of W. These singular values appear in decreasing order on the diagonal (i.e. of (W)). The SVD of W can be reformulated in Equation 4 Given a hyperparameter E N+ rank-r approximation of W is matrix W that minimizes | - W and satisfies rank W S r. The optimal solution of this problem W is provided by Fckart- Youuarr-Yuug---- Mouung-Missy theorem Low-Rank Approximation 2024) using Singular Value Decomposition (see Equation (5)
min(m.x)
W
(W),,
(4)
W Σ (W)ar
(5)
In this work, we will use the term minor singular components to refer to entries in the SVD corresponding to small singular values. These components are removed in low-rank approximation. The term of principle singular components is used to refer to entries in the SVD corresponding to large singular values. These components are kept in low-rank approximation of the matrix.
OOD Detection with Selective Low-Rank Approximation SVD-based weight pruning, partieu- larly in noise-prone layers, can substantially reduce network's 's sensitivity to minor perturbations, leading to enhanced stability and robustness (Yao et al.) (024). This stability is crucial for OOD detection, as ensures the model's reliable performance across wide range of inputs. Building on this, we propose method to improve OOD detection by selectively applying low-rank approximation to weight matrices. By decomposing weight matrix W into its singular values and vectors, we can identify and retain the principle singular components that significantly contribute to the model's performance. This approach ensures that the essential features of W are preserved while discarding the less critical minor singular components. Given weight matrix W in CLIP (e.g.. W up or Wi), we
replace the matrix with its low-rank approximation part as described in Equation 5 (see Figure 1.. Given the rank reduction ratio θ, the rank of W determined by r r(W) round (1 3) rr()). This selective low-rank approximation leverages the compact representation provided by SVD to enhance the model's ability to detect OOD instances effectively without requiring additional training.
We demonstrate our method's 's ability improve OOD detection (Table while maintaining ID
classification performance (Table in Section and Section 4.5
HyperParameter Search Algorithm Due the presence of many weight matrices in CLIP, each consisting of hundreds of singular values, conducting a complete search over all combinations of low-rank approximation weight matrices is impractical. Therefore, we propose a greedy search algorithm to determine the rank reduction ratio for each weight matrix. Among all linear weight matrices in each encoder layer, we focus on Wup as it is most effective according our preliminary experiment. For simplicity, we assume both image and text encoders have N encoder layers. As shown in Algorithm [ we search by first enumerating all N vision encoder layers sequentially from top to bottom and then all N text encoder layers in the same way. This search order is concisely denoted as searching from 2N to the first layer in CLIP. We compare different search algorithms in Section 4.4 The rank reduction ratio for each layer is the objective in SeTAR which is searched among the candidate list e = (On, OI.... ..)) according to the loss on the validation
set. We employ the LoCoOp loss (Equation 12 proposed in (Miyai et al. 2023a as our loss

---

function. This loss requires only ID images. It contains an ID loss for ID image classification and an OOD loss to push away pseudo OOD features from the ID class text embeddings where the pseudo OOD features are from II>irrelevant nuisances (Equation T0) (e.g.. backgrounds) in CLIP's local features. We refer the readers to Miyai et al. (2023) or Appendix B for more details.
For 3 @ we remove , (in percent) sin- gular values along with their corresponding sin- gular vectors to obtain the approximated ma- trix Wup (Equation 5). It is worth noting that the rank reduction raio candidate list includes Oo 0, indicating that the weight matrix has the chance to remain unmodified.
Algorithm 1 The hyperparameter search in SeTAR.
Data: Valid set D. Input: Leygh 2.N, rank reduction ratio candidates @ with length J, loss Lo D WITHOUT Se- TAR.
Result: Rank reduction ratio // Current best loss
with length 2NV.
L Lo
for LayerNumm + 2N to 1 do
With the searched rank reduction ratio, the weight matrix Wup in each CLIP layer is re- placed and updated with its approximation. The SeTAR can be easily applied to different ViT backbones (Table [). by replacing the model weight matrices with their low-rank approxima- tions in similar approach. Then SeTAR de- tects the OOD data samples following MCM (Equation M. GL-MCM (Equation (2) or other -oorigg-based OOD detection method with the approximated model. We provide an example procedure of the greedy search in Listing for better understanding.
W WW.. [ -
for counter j + 1 to do
round +[]]) rank(W
W - otu..
Calcluate loss C2 on D by replacing W with
W
if C L" then
W = W.T
0[1 - C,j;
end
end
W ==
end return T
3.2 OOD Detection with SeTAR-enhanced Low-rank Adaptation
SeTAR can be further combined with LoRA (Hu et al. 2022) as a novel low-rank adaptation method for OOD detection, which we refer to F T. Specifically, we first apply SeTAR to the pre-trained CLIP model to obtain the reserved rank r for each weight matrix W. Then we have
W W + BBX
(6)
min(m.n)
B
Σ
(W))
(7)
min(m.n)
A
Σ
(WW)
(8)
where W is the low-rank approximation of W found by SETAR, with A and B being the minor
singular components. During finetuning, we keep W frozen and only update the low-rank matrix A and B. In this way, retain the principle singular components in the original weight matrix and only update the minor singular components. rank budget across all layers. ScTAR+FT adjusts the rank for each layer, resulting in more effective and efficient fine-tuning (Table 2 and Figure 6). More details are provided in Section 4.3
LoRA, which evenly distributes the finetuning
4 Experiments
4.1 Experimental Setup
Dataset Following previous work (Ming al.; 2022 Miyai et 2023b), we use two real-worke datasets created from ImageNetI (Deng et al.
2009) and Pascal- VOC (Everingham et al., (209) as the datasets. For OOD datasets, we follow Ming et al. (2022) to preprocess iNaturalist, SUN, Places and Texture, and follow Miyai et al. (2023b) to preprocess ImageNer22K and COCO data. For finetune experiments, we follow Miyai et al. (2023a) to use ImageNet K as the ID dataset. The detailed description and statistics of the datasets are provided in Appendix C
5

---

Table 1: Training-free results of (PR95(FPR) and AUROC(AUC) compared to zero-shot base- lines CLIP-base. Bold values represent the highest performance. is cited from Miyai et al. (003b)) where o represents the absence of reporting in the paper. denotes the result of our re-run. denotes the OOD dataset has overlapping categories with the ID dataset. We do not report standard deviations since no training is involved.
iNaturalist
ImageNet22K
COCO
Average
FPR.
ALCT
FPRE
AUCT
EPR|
AUCT
FPR.
AUCT
FPR, AUCT
PPR
AUCT
FPR.
AUCT
ImageNNtII
MCM Score
Vanilla MCM
30,91
94.61
37.59
92.57
41.69
89.77
57.77
86.11
12.74
90.77
Vanilla MCM
32.07
94.43
38.65
92.37
43.73
90.03
57.89
86.13
43.09
90.74
SETAR
26.92
35.57
92.79
90.16
55.83
86.58
91.05
GL-MCM Score
Vanilla GL-MMMM
83.63
35.47
Vanilla GL-MCM"
15.34
96.62
30.65
93.01
37.76
90.07
57.41
83,73
35.29
90.86
ScTAR
13.36
96.92
28.17
93.36
36,80
90.40
54.17
84.59
33.12
91.32
Pascal-VOC
MCM Score
Vanilla MCM
8.20
98.23
28,
94.68
51.70
91.45
51.40
90.94
54.50
89.02
38.88
92.86
Vanilla MCM
7.24
27.91
92.45
50.60
53.70
37.
ScTAR
4.59
98.77
24.91
95.15
28.46
93.21
40.44
93.58
48.25
92.08
48.10
89.70
32.46
93.74
GL-MCM Score
Vanilla GL-MCM
4.20
Vanilla GL-MCM
4.33
98.81
22.94
94.63
26.20
93.11
41.61
92. 88
37,88
93.17
43.20
90.71
29. 44
93.88
SeTAR
3.66
98.96
21.93
94,81
25.04
93.62
20.35
96.36
31.47
94.31
40.70
91.19
23.86
94.87
Settings Following existing studies (Ming al.
CLIP ViT- et al. 2021) as our backbone. Both image and text encoders have 12 layers. More
2022
Miyai et al.
2023b a),
results with different backbones are in Section 4.4 The rank reduction ratio candidates range from 0 to 40% in 5% intervals. We use a temperature of 15 unless stated otherwise. In all experiments, CLIIP text prompt: "a photo of [CLASST.". where [CLASS] is the ID class name. We set hyperparameters À (Equation [2) and top-K (Equation I0) according to the specific ID datasets and backbones. Detailed settings are in Table 12, with a sensitivity analysis in Section 4.4 For SeTAR+FT and LoRA experiments, the learning rate and epoch number are set to le 2 and 5 for all experiments. The LoRA rank is set match the trainable parameters of SeTAR+F.. Detailed settings are in Table 13 We report results from three runs with seeds 3, 4, $ All experiments are conducted on a single NVIDIA RTX 4090 GPU. The time cost for low-rank approximation with CLIP-base on the ImageNet1 K validation set is about 20 minutes.
Metries We use the following metrics for evaluation. (1) the false positive rate (FPR95) for out-of- distribution (OOD) samples at fixed true positive rate (TPR) of 95% for in-distribution samples, with lower values targeting better performance; and (2) the area under the receiver operating characteristic curve (AUROC) for OOD samples, with higher values indicating better performance.
Baselines We evaluate SeTAR against MCM (Ming et al. 2022) 2) and GL-MCM (Miyai et al.
2023b), state-of-the-ard zero-shot OOD detection methods on CLI.P.
We also compare FeTAR+FI
with fine-tuning baselines NPOS (Tao et al., 2023), CoOp (Zhou et al.
2022). LoCoOp (Miyai et al.)
2023a), and LoRA (Hu al. 2022).
More details are in Appendix D
4.2 Training-Fres Results
The training-frrt OOD detection performances are summarized in Table Compared with zero-shot baselines, salient observation is that both MCM and GL-MCM, using SeTAR outperforms the vanilla method by a large margin across all OOD detection tasks. For example, using Pascal-VOC as ID, SeTAR yields a relatively average reduction of 12.84% FPR95 on MCM and 18.95% FPR95 on GL-MCM. Considering that SeTAR is generally applicable and training-fre free, these results are very encouraging. Comparing SeTAR with scoring function MCM and GL-MCM, SeTAR+GL-MCM performs better on all OOD detection tasks. However, the superiority of GL-MCM score over MCM appears be contingent upon the choice of the model backbone. As evidenced in Table g S demonstrates superior performance with a relatively average FPR95 reduction of 8.30% compared to ScTAR+GL-MCM with laree as the backbone on ImageNet
https://hugginge http:/gginnnacc..coo///a///////e co/openai/clip-vte basp--se----------- patch16
TTemperature to f.0 scaled CLIP logits, equivalent the unscaled CLIP logits with temperature of 100. We adopt the unscaled setting in our implementation.
For SeTAR, the results are the same under different random seeds it does require training.

---

4.3 Fine-tuning Results
In this section, we compare FTT with fine-tuning baselines, including NPOS ( (Tao al. 2023), CoOp (Zhou et al.) (022). LoCoOp (Miyai et al.) 2023a) and LoRA (Hu et al. 2022). LoCoOp the state-of-the-an art prompt-learning OOD detection method on CLIP. LoRA is
fine-tuning method. Following previous work (Tao et al. 2023; Zhou et al.) 2022; Miyai et al.] 2023a), we report the results on the the ImageNetIK benchmark in Table 2 We observe that SeTAR+FT outperforms all baselines on both MCM and GL-MMCM scoring functions. For example, with CLIIP-bse as the backbone, FFTAR+FT achieves relatively average FPR95 reduction of 3.97% and 6.67% compared to LoCoOp and LoRA. Moreover, when scaled up to Carp- FeTAR+FT outperforms LoCoOp and LoRA by relatively 17.92% and 12.45% FPR95 on the same benchmark. Similar results are observed on Swin Transformer (Liu et al. 2021), where SeTAR+FT outperforms LoRA by relatively 17.36% and 36.80% FPR95 MSP and En- ergy scoring functions, respectively. The larger improvement Swin Transformer may stem from its reliance on ImageNet training, mak- ing prone to overfitting and weaker at OOD detection. Our method mitigates these issues, en- hancing Swin's 's generalization to OOD instances. These results demonstrate the effectiveness and scalability of TTA++FF in improving the OOD detection performance.
representative parameter-efficien
Table 2: Fine-tuning results ImageNet benchmark, Bold values indicate the highest per- formance. is cited from Tao et al. (2023). denotes our re-run results, + indicates the standard
deviation from 3 runs.
CLiP-base
MCM Score
GL-MCM Score
FPR95_
AUROCT
FPR951
AUROCT
NPOST
42.20
90.43
36.86
90.37
CaOp'
44.81
90,03
36.58
90.25
LoCoOp'
40.17
91.53
33.52
92.14
LoCoOp
39.7624.00
91,22+4.02
34, 1421664
91.73-0.17
LoRA"
41 41.6720.1
90.8520.01
34.36g011
90.88_0.0r
SeTAR+FT
38.77 10.22
11.55jn0.0
_2.19_am
92.31
CLIP-large
MCM Score
GL-MCM Score
FPR95,
AUROCT
FPR951
AUROCT
LoCoOp LORA"
40.74.a..0
46. 46.74g118
89.32_n.s0
Furthermore, shown in Figure 6 SETAR+FT demonstrates faster convergence and lower loss than LoRA, especially in OOD loss, indicating that SeTAR+FT is more effective in adapting the pre-trained weights the OOD detection task.
38.62
91.6610.02
43.39
89.76
FcTAR+FT
34.75
92.8640.15
37.05
91.83_0.12
MSP Score
Energy Score
Swin-base
FPRO5,
AUROCT
FPR951
AUROCT
LORA"
$7.02
80.49.0.01
62. 17.172000
72.80_.0.00
SeTAR+FT
47.12go.12
10.40
1o37
88.01_n.51
4.4 Ablation Study
In this section, we conduct ablation studies with CLIP-hase to understand our design choices.
Image v.s. Text modality Table 3 shows an ablation study on the modality involved in SeTAR.
As shown, the vision modality outperforms the text modality, indicating the vision modality is more dominant in enhancing the model's perfor- mance. When considering the vision modality alone and the combined vision+text modality, the latter either outperforms or achieves com-- parable average results to the former. Conse- quently, we make modifications to both the vi- sion and text modalities in SeTAR to enhance overall performance.
Table 3: Ablation study on modality.
Vision
Text
Vision+Text
Score
FPR.
AUC1
FPR.
AUC1
FPR.
AUC1
ImageNetIK
MCM
40.27
91.24
42.78
90.50
40.24
91.05
GL-MCM
32.97
91.60
35.82
90.55
33.12
91.32
Pascal- VOC
MCM
33.19
93,45
33.47
93.42
32.46
93.74
GL-MCM
24.88
94.51
24.59
94.52
23.86
94.87
Different Weight Types In this part, we present empirical evidence for modifying W up-
Table 4: Comparison results of SeTAR with and without considering projection matrix Wp-
We first compare the performance of ScTAR with different types of weight matrix in each Transformer layer, including W.. Wx. W,. W.. W and W downn As shown in Figure and Fig- ure [ of Appendx F the X-xxis denotes the number of weight matrixes (layers) that we have searched, while the -axii is the average AU- ROC and FPR95. The results show that W... up consistently outperforms other weight matrices in terms of both AUROC and FPR95. In addition to weight matrics in each transformer layer, CLIP has one projection matrix Wp p on top of each encoder,
Vanilla
SeTAR
W, ScTAR w/o W,
Score
EPR
AUCT
FPRJ
AUCT
FPR
AUCT
ImageNet1K
MCM
43.09
90.74
41.79
90.74
40.24
91.05
GL-MCM
35.29
90.86
34.30
91.24
33.12
91.32
Pascal-VOC
MCM
37.24
92,98
35.94
93.32
32.46
93,74
GL-MCM
29.44
93.88
23.34
94.82
23.86
94.87
7

---

which serves to project image/text representations into a shared space. In Table 4 we compare the performance of SeTAR with and without modifying Wp. We search W, first right before searching the image/text encoder. The results show that frozen Wp brings a relatively reduction of 4.20% FPR95. Consequently, we keep Wp frozen in SeTAR.
Different Search Algorithms each step of the greedy search, SeTAR traverses the subsequent Wup in a predefined order and searches over different thresholds. We compare our method with two alternatives:
greedy search (MIS) and layer-exhaustive search (LES), MIS searches the image and text layers in an interleaved manner, while LES simultaneously searches over both layers and thresholds at each step. SeTARRS, has linear complexity with re- spect to the number of model layers, similar to MIS, whereas LES has quadratic complexity. Jrreetss the comparison results. SeTAR- demonstrates better overall performance than MIS. Notably, MIS encounters limitations when the image and text towers have different layer counts (e.g., CLIP-large large with 24 image lay- ers and 12 text layers). Therefore, we choose SeTAR-S for better generalization. Compared to LES, SeTARRS performs better in terms of both FPR95 and AUROC, LES's locally opti- mal algorithm may not achieve a global optimal solution. These results validate the superiority of our top-to-bottom layer search strategy.
Table 5: Results for different search algo- rithms. Here LES, MIS and SeTAR-S stand for layer-exhaustive search, modality-interleaw interleaave greedy search, and the search algorithm of SeTAR.
Score
LES
MIS
S-TAR-S
FPR
AUCT
EPR.
AUCT
FPR.
AUC1
ImageNetIK MCM
41.99
90.78
40.55
91.00
40.24
91.05
GL-MCM
33,90
91.08
33.36
91.29
33.12
91.32
Pascal-VOC
MCM
35.11
93.60
33.93
93.58
32.46
93.74
GL-MCM
24.48
94.57
22.87
94,84
23,86
94.87
Different Prune Strategies Inspired from SVD, SeTAR modify the model weights by prun- ing the minor singular components, and retains the principle components that contribute the most to the model's performance. To validate this design, we compare SeTAR with two alter-
natives: principal component pruning and random pruning pruning. Principal component takes
the opposite approach, retaining minor compo- nents and pruning major ones. Random pruning, on the other hand, prunes weights randomly. As shown in Table principle pruning suffers from a significant performance drop compared Se- TAR, while random pruning performs slightly better than principle pruning. These results demonstrate the effectiveness of SeTTAR 's de- sign choice in pruning the minor components.
Table 6: Results for different pruning strategies.
Principle
Random
Minor
Score
FPR
AUC1
FPR.
AUC1
FPR.
AUC1
ImageNetIK
MCM
43.09
90.74
43.09
90.74
40.24
91.05
GL-MCM
35,29
90.86
35.29
90.86
33.12
91.32
VOC
MCM
38.20
92.44
33.57
93.09
32.46
93.74
GLMMC
25.36
93.67
26.20
94.66
23.86
94.87
Sensitivity Analysis on À and top-K In this section, we present the sensitivity analysis of the hyperparameters À (Figure 4 and top-K (Figure 5). As observed in Figure 4 the average AUROC remains stable at lower values and slightly decreases as À increases for both ScTAR+MCM and SeTAR+GL-MCM, Notably, the optimal setting of may vary depending on the model backbone, with our experiments indicating that CLIP-large may require a larger À than CLIP-base Despite this variation, the X parameter demonstrates strong transferability across datasets for the same backbone. Swapping the optimal À between ImageNet1K K and Pascal-VOC has a minimal performance impact, consistently outperforming the vanilla method. With the VOC-optimized on ImageNet1 CLIP- base achieves an FPR95 of 40.91 and AUROC of 91.02, and CI.re-large reaches 46.73 FPR95 and on Pascal-VOC, CLIP-basc achieves
91.81 AUROC. Conversely, using the 33.18 FPR95 and 93.65 AUROC, while Caipe attains 44.39 FPR95 and 92.3 AUROC.
Top-K controls the number of OOD regions considered in LoCoOp loss: higher values include more OOD regions, with top-K equal to the number of ID classes covering all OOD regions, and top- K set to focusing solely on ID loss. The optimal top-K depends on the number of ID categories, making it non- non-transferable across datasets. However, SeT remains robust to top-K variations, as shown in F except at extreme values (0 or the maximum number of classes). We recommend setting top-K K to around 30% of the total categories, such as 300 for ImageNet1 K and 4 for Pascal-VOC. For the Swin-base model, top-K at 300 on ImageNet K yields an FPR95 of 52.82 and AUROC of 85.68 with MSP, and FPR95 of 52.56 and AUROC of 84.51 with Energy.
8

---

4.5 Analyses
Can SeTAR Improve Image Classification? To evaluate the impact of SeTAR and Se- TAR+FT on classification accuracy, we present our results on ID dataset I K and OOD datasets SUN, Places and Texture Table 7 SeTAR effectively maintains the average ac- curacy, with minor variations observed across different datasets. Among the fine-tuned base- lines, LoCoOp exhibits a 1% decrease in accu- racy compared to Vanilla CLIP, whereas LoRA shows an improvement of 0.94%. Notably, SeTAR+FT surpasses both baselines, improving the average accuracy by 1.45% compared Vanilla CLIP. These results highlight the efficacy of SeTAR and Se TAR+FT in improving OOD detection without compromising classification accuracy.
Table 7: Image classification results with differ- ent methods. We use ImageNet (INIK) as ID dataset. denotes the results of our re-run. The results averaged over 3 runs.
Method
INIK
SUN
Places
Texture
Average
Vanilla CLIP
64,07
75.77
45.65
43.60
57.27
LoCoOp'
64.93
75.89
46,47
37.79
56.27
LoRA"
65.43
76.86
46.58
43.98
58.21
ScTAR
63.97
75.50
45.81
43.76
$7.26
SeTAR+FT
67.02
77.94
46.64
43.28
58.72
We expand on Table I with results on ViT and CNN backbones and various score functions. For ViT-based models, we
SeTAR Effective on Different Architectures and Score Functions
evaluate OOD detection using CLIP-large and Swin Transformer (Lu et al. 2021), alongside 2022) is trained on ImageNerIK. K. Since it lacks
[
CLIP-base. The Swin Transformer (Liu et al.
text encoder, we apply SeTAR to the image ViT
only, For Swin Transformer, we use two common 2017), which leverages softmax confidence, and
scoring functions: MSP (Hendrycks & Gimpel
the Energy score (Liu al. 2000)). with 7
0.1
for OOD detection. We also integrate CL.IP-base
with the NegLabel score function (Jiang al.) 2024), which uses large-scale negative labels. As shown in T SeTAR consistently outper- forms baselines across all backbones and scor- ing functions, significantly redueing FPR95 by relatively 20.61% % with the Energy score on Swin Transformer. These results demonstrate SeTAR 's effectiveness in improving OOD detection for unimodal image encoders, with further confir- mation from SETARRFT results (Table across different model backbones.
Table 8: Results for different ViT backbones.
Vanilla Method
SeTAR
Backbone
Score
FPR. AUC1
FPR.
AUCT
ImageNetIK
CLIP-buse
NegLabel
25.40
94.21
23.09
94.48
Care-
MCM
37.19
91.73
36.26
91.92
CLIP-large
GL-MCM
40.65
89.98
39.54
90.22
Swin-buse
MSP
59.25
84.12
56.05
85.77
Swin-base
Energy
65.01
76.10
51.61
84.42
Pascal-VOC
CLIP-large
MCM
52.21
91.68
42,57
92.91
CLiP-large
GL-MCM
43.96
92.45
31.12
94.00
We further explore SeTAR's potential on CNN architecture, and compare it with methods such
Table 9: Results on ResNet50. We use Ima-
K as the ID dataset. t is cited from Djurisic
as Softmax, Energy (Wu et al.] (023), Re-
et al. (0033)
Act (Sun et al. 2021). DICE (Sun & Li 2022). and ASH Djurisic et al. 2023) on ResNet5("
Method
FPR]
AUC
Method
FPR_
AUC1
Since ResNet lacks local features for OOD loss,
Softmax
66.95
81.99
ASH-Pi
50.32
89.04
we conduct experiments using only ID loss. We apply low-rank approximation the in- and out- feature dimensions of the convolutional layers, combined with ASH for search. As shown in Table 9 SeTAR establishes new state-of-the-art results on ResNet, demonstrating its effectiveness across both ViT and CNN architectures.
Energy!
58.41
86.17
ASH-B
22.73
95.06
ReAct
31.43
92.95
ASH-S*
22,80
95.12
DICE
34.75
90.77
SETAR
22.38
95.25
Near-OOD Results To further evaluate Se- TAR's 's performance on diverse OOD tasks, we test it on a more challenging near-OOD setting using ImageNet1K K as the ID dataset and SSB- Hard (Vaze et al. 2022) as the OOD dataset. As shown in Table 10 TeR and SeTAR+FT out- perform the baselines. demonstrating superior performance in ncar-OOD scenarios.
Table 10: Near-OOD results on CLIP-base.
MCM Score
GL. MLM Score
Method
Category
FPR.
AUC1
EPR]
AUC1
Vanilla
Training-Fre
89.28
63.88
85.62
67.63
ScTAR
Training-Free
88.29
64.20
84,03
68.29
LoCoOp
Training-Free
89.72
63.45
86.79
65.93
LoRA
Finetuning
88.52
65.38
84,39
68.85
FeTAR
Finetuning
87.16
68.13
84.72
70.42
We do not report classification accuracy on iNaturalist as we failed to match the labels for the OOD test set.
httppp//uuugggggggeecccc//orrrer patch- window16-
http:://dommooaa.pytrrcc..ooo http:///ooooolooocccccco///////e 19c83333 pth

---

5 Related Work
Dut-of-Distribution Detection Previous work explores OOD detection with unimodal (DeVries
& Taylor 2018 Hendryckk & Gimpel, 2017; Hu & Khan, 2021 Zheng et al. 2020;; Zhou ct al.
2021) and multimodal (Fort et al. 2021 Ming et al. 2022] Tao et al. 2023] Miyai et al. 2023a)
models. Numerous methodologies (Lee et al. 2018 8 Huang et al. 2021 Sun et al.) 2022 Wang
ct al. 2022; Wu et al., 2023) have been developed
tackle OOD detection in computer vision.
Existing CLIP-based OOD detection methods include zero-shot (Fort et al. 2021 Ming et al. 2022
Miyai et al., 2023b Dai et al. 2023; Wang et al. 2023 Jiang et al. 2024) and fine-tuning (Ming &
Li 2023 Tao et al.) |023] Miyai et al. 2023a). Zero-shot methods like MCM (Ming et al.) 2022)
and GL-MCM (Miyai et al. 2023b) don't 't require in-distribution training data but may perform
suboptimally due to domain gaps. Other approaches integrate external knowledge. For example.
CLIPN (Wang et al. 2023) pre-trains a novel NO-encoder on the CC-3M dataset (Sharma et al. 2018) to empower CLIP's s "no" logic for zero-shot evaluation. NegLabel (Jiang et al. 2024) demonstrates better performance than CLIPN by introducing large-scale negative labels for enhanced label scoring. Fine-tuning methods (Ming & Li 2023: Tao et al.] 2023] Miyai et al. (023a) improve OOD detection by adapting to in-distribution data but risk damaging the pretraining representations, needing careful
training strategies. CNN-based OOD detection methods, including ReAct (Sun et al. 2021 ASH
(Djurisic et al. 2023), DICE (Sun & Li, (022), CIDER (Ming et al. 2023). PALM (Lu et al. 2024).
and Hopfield Boosting (Hofmann et al. (2024), have also demonstrated strong results. However, methods like ReAct and ASH rely on the assumption that ID and OOD images produce distinct activations in models trained on ID data. This assumption does not hold in large-scale pre-trained models like CLIP, where activations for ID and OOD images are not significantly different, limiting the effectiveness of such approaches in enhancing CLIP's zero-shot OOD detection capabilities. SeTAR, in contrast, offers high compatibility with various scoring functions (e.g.. MCM, GL-MCM, MSP, Energy), multiple model backbones (e.g.. CLIP, Swin, ResNet), and advanced OOD techniques such as NegLabel. Designed to be both lightweight and efficient, SeTAR addresses the demand for resource-efficient solutions in OOD detection.
Low-rank Approximations of Weight Matrices Neural networks trained with over- parameterization often exhibit low-rank properties (Oymak al., 2019). These properties are utilized in both model training (Povey et al. 2018: Hu et al. 2022) and post-hoc processing (Haji-
molahoseini et al. 2021, Sharma et al. 2023). In training, some works (Sainath et al., 2013 Zhang et al. 2014:2200 Zhao et al. 2016) impose low-rank constraints, while LoRA (Hu et al. 2022 adapts
pretrained LLMs to downstream tasks using trainable low-rank matrices. For post-hoe processing,
al. 2021 reduce weight matrix ranks by retain- ing top-K components from SVD. While pruning preserves model behavior, performance declines with increased intervention. LASER (Sharma et al., 2023) focuses on pruming individual layers to enhance factual answering capabilitics. utilizes a simple greedy search strategy on a validation set, which is not applicable for OOD detection due to the absence of validation set. In contrast, our approach introduces a selective rank reduction strategy specifically tailored for OOD detection. We systematically analyze and compare different greedy search techniques, evaluating their effectiveness across various layers and model backbones.
pruning methods (Yu et al.
2017, Hajimolahoseini
Conclusion
We propose SeTAR, a simple and effective OOD detection method using post-hoe low-rank ap- proximation on weight matrices Wup with top-down, image-to-text greedy search. SeTAR offers several advantages: (1) training-fres (2) scalable to unimodal and multimodal models, and (3) complementary to existing OOD scoring functions. Building on SeTAR, we introduce SeTAR-FT. a finetuning method that adapts the model to in-distribution data for improved OOD detection. We evaluate SeTAR and FeT on large-scale benchmarks, including ImageNet1K K and PaSCal-VOC. Results show that both achieve tfate-of-the-an OOD detection performance. We hope our work inspires further research and contributes to more robust and reliable models.
10

---

Acknowledgements
This project was supported by National Natural Science Foundation of China (No. 62306132, No. 62106138). We thank the anonymous reviewers for their insightful feedbacks this work.
References
Bai, H., Canal, G., Du, X., Kwon, J., Nowak, R. D., and Li, Y. Feed two birds with one scone: Exploiting wild data for both out-of-distributie generalization and detection. In International Conference Machine Learning, 2023.
Caesar, H., Uijlings, J., and Ferrari, V. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR, 2014.
Dai, Y., Lang, H., Zeng, K., Huang, F., and Li, Y. Exploring large language models for multi-modas out-of-distribus detection. ArXiv, abs/2310.08027, 2023. URL https://app semanticscholar org/ Dtg//orppsID::233901122 2639091277
Deng, J., Dong, W. Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
DeVries, and Taylor, G. W. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint: 04865,, 2018.
Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely simple activation shaping for out- of-distribution detection. In The Eleventh International Representations, 2023.
on Learning
Emmott, A., Das, S., Dietterich, T., Femm A., and Wong, W.-K meta-analysis of the anomaly detection problem. arXiv 500print: 2016.
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. IJC,, 88:303-308, 2009.
Fort, S., Ren, J., and Lakshminarayanan, B. Exploring the limits out-of-dissrribuioo detection. In NeurlPS, 2021.
Hajimolahoseini, H., Rezagholizadeh, M., Partovinia, V., Tahaci, M. S.. Awad, O. M., and Liu, Compressing pre-trained language models using progressive low rank decomposition. In NeurIPS, 2021.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of- -itto--distribttion examples in neural networks. In ICLR, 2017.
Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., and Song, D. Pretrained trans- formers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2744-2751, Online, July 2020.
Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-distriibttio detection for real-world settings. In ICMI. 2022.
Hofmann, C., Schmid, S., Lehner, B., Klotz, D., and Hochreiter, S. Energy Easegy-based hopfield boosting wttptttss///xxxi..otoo abs/ 2ttp://0000000666...
for out-of-distributioo detection, 2024.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu. Z., Li. Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Hu, Y. and Khan, L. Uncertainty-awan reliable text classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp. 628-636, New York, NY, USA, 2021.

---

Huang, R., Geng, A., and Li, Y. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS, 2021.
Jiang, X., Liu, F., Fang, Z.,, Chen, H., Liu, T., Zheng, F., and Han, B. Negative label guided OOD detection with pretrained vision-language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:
eview. forommmidddxxd0H-2222222
Lee, K., Lee, K., Lee, H., and Shin, J. simple unified framework for detecting -ittr--distribution samples and adversarial attacks. In NeurlPS, 2018.
Li, J., Li. D.. Savarese, S., and Hoi, S. BLIP-2: bootstrapping language-iman pre-training with frozen image encoders and large language models. Proceedings of the 40th International Conference on Machine Learning, 2023.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of -iftrrf-itiiitiionn image detection in neural networks. ICLR, 2018.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. Microsoft coco: Common objects in context. In ECCV, 2014.
Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based distriibittiiioo detection. In NeurlPS, 2020.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei. Y.. Zhang. Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Liu,2 Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning. J., Cao, Y., Zhang, Z., Dong, L.,, Wei, F., and Guo, B. Swin transformer v2: Scaling up capacity and resolution, 2022.
Low-Rank Approximation. Low-rank approximation Wikipedia, the free encyclopcdiaa, January 2024. https://en. wikipedia org/w/ w/
brggw/index. inddxiiddxxx///e http:://wwwo......ttoaooaoooooo. pepp:tttlee.oovrraanapppooooiiid
11616-1191167077
Lu. H., Gong, D.. Wang, S., Xue, J., Yao, L., and Moore, K. Learning with mixture of prototypes for out-of-distribution detection. In The Twelfth International Conference on Learning Representations, 2024. URI /ttp////ppeeeiee/////e/e/ee net/forumm http://oouumiiddulllaaaaaa La-NNKK
Ming, Y. and Li, Y. How does fine-tuning impact out-of-distribution detection for vision- language models? International Journal of Computer Vision, 132(2):59--009, September 2023. ISSN 1573-1405. doi: 10. 10.1007/s1 126007031111------------. URI. http:://dx..oi..rrg/tbtb 1007/ s11263-023-01895
Ming. Y., Cai, Z., Gu, J., Sun, Y. Li, W., and Li, Y. Delving into out-of-distribution detection with vision-language representations. In NeurlPS, 2022.
Ming, Y., Sun, Y., Dia, O., Li, How to exploit hyperspherical embeddings for detection? In The Eleventh International Conference on Learning Representations, 2023. URL net/ orum setes//formmidd=af-eeee d=aEFaE055pPdd
Miyai, A., Yu, Q., Irie, G., and Aizawa, K. Locoop: Few-shot out-of-distribution detection via prompt learning. In Sheert-Seventh Conference on Neural Information Processing Systems, 2023a.
Miyai, A., Yu, Q. Irie, G., and Aizawa, K. Zero-shot in-distribation detection in multi-object settings using vasion-language foundation models. arXiv preprint arXiv:2304.04521, 2023b.
Oymak, S., Fabian, Z., Li, M., and Soltanolkotabi, M. Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian, 2019,
Povey, D., Cheng, G., Wang. Y., Li, K., Xu, H., Yarmohammadi, M. A., and Khudanpur, S. Semi- orthogonal low-rank matrix factorization for deep neural networks. In Interspeech, 2018. URL htttps://ant www.setimccontooaaeo
wwtgCCorpusDDDDD 4944967
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P. Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
12

---

Russakovsky, O., Deng, J., Su, H.. Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., al. Imagenet large scale visual recognition challenge. IJCV, 115(3):11--55. 2015.
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 JEEE international conference on acoustics, speech and signal processing, pp.
IEEE, 2013.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACI., 2018.
Sharma, P., Ash, T., and Misra, D. The truth is in there: Improving reasoning in language models with seyer-seeectvve rank reduction, 2023.
Sun, and Li, Y. Dice: Leveraging sparsification for out-of-distributiio detection. Computer Vi- sion ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 691-708, Berlin, Heidelberg. 2022. Springer- Verlag. ISBN 978-3-031-20052--1 doi:
10. 11007/9778-3-031-2005----- -0. 40
URL https://doli org 1007 978 031 -100-220000 20053-3
Sun, Y., Guo, C., and Li, Y. React: Out-of-distribution detection with rectified activations. In NeurIPS, 2021.
Sun. Y.. Ming, Y., Zhu, X., and Li, Y. Out-of-distrribtion detection with deep nearest neighbors. In ICML, 2022.
Tao, L., Du, X., Zhu, X., and Li, Y. Non-parametric outlier synthesis. In ICLR, 2023.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In CVPR, 2018.
Vaze, S., Han, K., Vedaldi, A., and Zisserman, A. Open-set recognition: A good closed-set classifier is all you need. ICLR, 2022.
Wang. H., Liu, W., Bocchieri, A., and Li, Y. Can multi-label classification networks know what they don't know? In NeurlPS, 2021.
Wang. H., Li, Z., Feng, L.., and Zhang, W. Vim: Out-of-distribution with virtual-logit matching. In CVPR, 2022.
Wang, H., Li, Y., Yao, H., and X. Clipn for zero-shot ood detection: Teaching clip to say no. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 1802-1812, 2023. URL https://api. swmanticscholsse.e..
org/ http/CoopuID::2117755550
Wu, Q. Chen, Y., Yang, C., and Yan, J. Energy-based out-of-distributtion detection for graph neural networks. ArXiv, abs/2302.02914, 2023. URL https://api...... http://pp...emmmiiiiccccccll.eee UtL..ttpps//ppp....mmmt..ccooool .ogg
CorpusID:25661577
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Xu, K., Ren, T., Zhang, S., Feng, Y., and Xiong, C. Unsupervised out-of-domain detection via pre-trained transformers. In ACL, 2021.
Yao, X., Hu, X., Yang, S., and Liu, Y. Enhancing in-context learning performance with just svd-based weight pruning: A theoretical perspective, 2024. URL htt.httts:///ratt //tp:v/.//xiv..v// org/abs// abs/ absp////....2///////// 2406 03768
Yu, J., Wang. Z., Vasudevan, V., Yeung. L., Seyedhosseini, M., and Wu, Y. CoCa: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.00 01917, 2022.
Yu, X., Liu, T., Wang. X., and Tao, D. On compressing deep models by low rank and sparse decomposition. 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 67-76, 2017. URL https::/ //api. semanticscholar. Org/CorpusID::45555 24553448
13

---

Zhang, Y., Chuangsuwanich, E., and Glass, J. R. Extracting deep neural network bottleneck features using low-rank matrix factorization. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 185-189, 2014. URL htttps://appp semanticscholarr org/ Ltg//orpusID... 1791734
Zhao, Y., Li,,, and Gong, Y. Low-rank plus diagonal adaptation for deep neural networks, 2016 JEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5005-5009, 2016. URL httpttt::// / api semanticscholar: org/ : 10506309
Zheng, Y., Chen, G., and Huang, M. Out-of-domain detection for natural language understanding in dialog systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 2020.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recognition. TPAMI, 40(6):1452- 1464, 2017.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. IJCV, 2022.
Zhou, Liu, F., and Chen, M. Contrastive out-of-distribution detection for pretrained transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1100-1111, Online and Punta Cana, Dominican Republic, November 2021.
14

---

A Impact Statements
Limitation While demonstrate the effectiveness of our method OOD detection, acknowl- edge that our work has several limitations. First, despite we show the robustness of our method to hyperparameters, the optimal hyperparameters may vary across different model backbones. Future work is needed to explore the autonomous selection of hyperparameters. Second, we design Se- TAR+FT in simple and straightforward manner, which may not be the most efficient effective way to adapt the model to the ID downstream data. More sophisticated strategies for model adaptation are worth exploring in future research. Third, we only conduct experiments to detect visual OOD inputs and ignore inputs in other modalities such as textual, audio and video. This is primarily because our model is based on CLIP. Exploring the development of OOD detectors across diverse modalities remains an active research topic for future investigation.
Ethical Considerations Our study addresses the challenge of OOD detection through low-rank approximation, which is particularly relevant for ensuring the reliability and trustworthiness of vision-and-language pre-trained models. Future investigations neural-basee models should be encouraged to mitigate the existing data biases and safety problems for responsible, helpful and trustworthy Al system in diverse real-world applications.
fairness, privacy and transparency
Future Societal Consequences Our proposed ScTAR achieves impressive OOD detection perfor- mance, which is beneficial to various real-world machine learning applications, such as healthcare and autonomous vehicles. The identification of anomalies or unexpected data points is crucial for decision-making and risk management with AI models. A better OOD detector facilitates the development of trustworthy machine-learning models that can reject unknown data inputs and help alleviate the hallucination problem. Moreover, better OOD detectors like SeTTAR can help to select and label the unfamiliar data samples to further train stronger model the wild.
B Loss Function
To improve the model's OOD detection ability, is crucial to define a loss function that pushes OOD samples far from ID samples while keeping ID samples close to each other. However, since OOD samples are unavailable during development, we address this issue by using the LoCoOp loss (Miyai et al. 2023a) for both SeTAR and SeTAR+FT. The main idea is to create pseudo OOD features with ID-irrelevant nuisances (e.g., backgrounds) in CLIP's local features.
Specifically, we divide the image into patches, represented by the set of all patch indices 1 = (0, 1, 2,....., x W 11, where H and W denote the height and width of the patch features. Next, compute the cosine similarity between the image patch features p and the text features hy of the image label. The classification prediction probabilities for each patch i are then given by:
exp(cos_sim(p): (tt()os__smmmpp,t)))))) N1)/T')
( m|x)
K
exp(cos sim(p; ht)/T)
For given image patch related to an ID category, the corresponding ID label should be among its top-K predictions. Conversely, for patches unrelated to the ID label, such as background regions, the ID label should be excluded from the top-K predictions. Based on this intuition, the indices of
ID-irrelevant regions within an image are defined by Equation 10 where rank(p:(y = y[x)) denotes the rank of the true class y among all ID classes, and K is the hyperparameter.
{il rank(p: = y > K}
(10)
After identifying nut-of-distribure (OOD) regions, it is expected that their image features will differ significantly from the ID text embeddings. To enhance this distinction, entropy maximization employed to increase the entropy of p; x). where Pj denotes the classification prediction probabilities for region J. The entropy maximization is formally defined as follows:
Lood = -H (p;)
(11)
Here, (.) represents the entropy function. The overall loss function combines the loss (cross- entropy loss for ID predictions) with the OOD loss. Here À is the hyperparameter that regulates the proportion of the OOD loss.
L = Lia + XLood
(12)
15

---

C Data
Table 11: The statistics of the dataset used in this paper. 'ID' and "OOD' denote in-distribution and out-of -it-r---iitibttinn, respectively.
Data
Type
Valid Size
Test Size
I (Deng
2009
ID
1,000
50,000
Pascal-VOO VOC
Everingham
2009
ID
94
906
iNaturalist
Van Horn al.
2018
0OD
0
10,000
SUN
X1a0 el al. 2010
00D
0
10,000
Places
Zhou ct a.. 2017
00D
0
10,000
Texture
Impor el al. 2014
00D
5,640
ImageNe122K
Kussakovsky al
2015
0OD
18,335
COCO
(Lnn al. 2014
00D
1,000
We use two real-world datasets created from ImageNet1K (Deng et al., 2009) and Pascal- VOC (Everingham et al. 2009) as the ID dataset. We use ImageNet- validation set as the ID test set following Ming et al. (2022). and preprocess Pascal- following Miyai et al. (2023b). we build two ID validation sets for low-rank approximation. The ID validation set of ImageNetIK collected by sampling one image for each label from the ImageNetIK training set. For Pascal-VOC, For Pascal-VOC, We randomly sample 10% images as the ID validation set and leave the rest as the ID test set.
For OOD datasets, we follow Ming et al. (2022) to preprocess iNaturalist, SUN, Places and Texture, and follow Miyai et al. (2023b) to preprocess ImageNet22K and COCO data. We only evaluate the OOD datasets that have no overlapping categories as the ID dataset.
We provide more details about the datasets used in experiments, in terms of data sources, preprocessing, and the statistics for each dataset, as shown in Table and below.
ImageNetIK We the ImageNet- (ILSVRC2012) (Deng et al. 2009) dataset for ID vali- dation and testing. The original dataset contains 1.2 million training images and 50,000 validation images from 1000 classes, and widely used for image classification. We follow Ming al. (2022) to construct the ImageNetIK ID test set from the validation set. Additionally, we curate an ImageNet ID validation set from the training set by randomly selecting one image for each label.
The Pascal VOC (Visual Object Classes) (Everingham et al. 2009) dataset is benchmark dataset widely used in computer vision, featuring annotated images across multiple object categories. We use the Pascal-VOC subset collected by Miyai et al.) (2023)) as the ID dataset, each image has single-class ID objects and one or more OOD objects. The ID validation and test set are split by 1:9 for each class, resulting in 94 and 906 images, respectively.
PoS
iNaturalist iNaturalist (Van Horn et al. 2018) is a biodiversity dataset containing millions of labeled images of plants, animals, and insects. MMig al. (2022) construct subset with 10,000 images by de-duplicating concepts overlapped with ID datasets.
Places Places (Zhou et al. 2017) is scene-centric database with 205 scene categories and 2.5 million images. We use the SUN subset collected by Ming et al. (2022) as the OOD test set, which contains 10,000 images that are not overlapped with the ID classes.
SUN SUN (Scene UNderstanding) (Xiao et al. 2010) is a comprehensive collection of labeled images representing diverse range of indoor and outdoor scenes. We use the SUN subset collected by Ming et al. (2022) as the OOD test set, which contains 10,000 images that are not overlapped with the ID classes.
Texture The Texture dataset (DTD) (Cimpoi et al., 2014) comprises 5640 images categorized into 47 terms inspired by human perception, aimed replicating human-like texture recognition in machines. Again, we use the subset collected by Ming et al. (2022) as the OOD test set.
ImageNet22K The ImageNet-22K dataset (Russakovsky al. 2015), formerly known as ImageNet- 2 21K, addresses the underestimation of its additional value compared to the standard ImageNet-IK
16

---

pretraining, aiming to provide filtered subset collected by Wang al. (2021) as the OOD test set for MC-COCO and Pascal-VO ID test sets.
pretraining for broader range of models. We use the
COCO Miyai et al. (2023b) curated an MS-COCO OOD test set (COCO for short) with 1,000 images that are not overlapped with the Pascal-VOL ID classes, which use OOD testing data for Pascal- VOC ID test set.
D Fine-tune Baselines
We compare SeTAR+FT with 4 finetuning- baselines. These baselines include:
2023) generates virtual anomalies in low-probability regions of ID data without relying on distribution assumptions, enhancing discrimination during training.
NPOS. NPOS (Tao et al.
2022) optimizes prompts for vision-language models with learnable context vectors for efficient few-shot learning.
CoOp. CoOp (Zhou et al.
LoCoOp. LoCoOp (Miyai et al. 2023a) improves upon CoOp by leveraging CLIP's local features to better distinguish between ID and OOD samples, achieving higher detection accuracy with less training data. We follow the official codq 11| to prepare and fine-tune the LoCoOp with CLIP-base and CLIP-large Follow Miyai et al. (2023a), the top-K, X, learning rate and epoch num are set to 200, 0.25, 0.002 and 50. Temperature is set to and the text prompt is initiated with "X X X X X X X X X X x X X X [CLASS]", where [CLASS] is the ID class name. We average the results from 3 seeds finetuned with 1-shot ImageNet valid data.
LoRA. LoRA (Hu et al. 2022) is a low-rank adaptation method that injects trainable low- rank decomposition matrices into the pre-trained model to adapt to downstream tasks. We apply low-rank adaptation the same weight type as SeTAR+FT. the rank of each layer set to match the trainable parameters of SeTTAR. Details settings can be found in Table 13
E HyperParameters Settings
The hyperparameters for ScTAR are shown in Table 12 And the hyperparameters for SeTAR+FT and LoRA are shown in Table 13
Table 12: Hyperparameters for SeTAR. Temperature is set to except for Swin-base with Energy score, where is set 0.1.
Backbone
Dataset
top-K
ImageNetIK K
0.10
300
CLIP-base
Pascal-VOC
0.05
4
ImageNer1 K 0.50
300
CLiP-large
Pascal-VOC
0.30
Swin-base
ImageNerIK
0.01
700
Table 13: Hyperparameters for SeTAR+FT and LoRA on ImageNetIK, Temperature set to except for Swin-base with Energy score, which is set to 0.1.
Backbone
top-K K
LR
Epoch
Rank for LoRA
Alpha for LoRA
CLIP-base
0.10
300
0.01
32
16
Carp-large
0.50
300
0.01
16
Swin-base
0.01
700
0.01
112
16
17

---

More Detailed Experiment Results
In this section, we present additional detailed results from the main paper. This includes the detailed results of fine-tuned baselines on the ImageNet1K benchmark in Table 14 detailed ablation results on modality, W p. X, and top-K in Table T5] Table 16 Table 19 and Table 2T and detailed results of SeTAR with different search algorithms, prune strategies and backbones in Table [8] Table 20
Table 17 and Table 22
Table 14: Detail results of FPR95(FPR) and AUROC(AUC) compared with fine-tuned baselines
on ImageNet1K benchmark. is cited from Tao et al. (2023). denotes the results of our re-run.
iNaturalist
SUN
Places
Texture
Average
Method
FPR
AUC1
FPR.
AUC1
FPR
AUC1
FPR
AUC1
FPR
AUC1
CLIP-base
MCM Score
NPOS
19.59
95.68
48.26
89.70
49.82
88.77
51.12
87.58
42.20
90.43
CoOp"
43.38
91.26
38.53
91.95
46.68
89.09
50.64
87.83
44.81
90.03
LoCoOp!
38.49
92.49
33.27
93.67
39.23
91.07
49.25
89.13
40.17
91.53
LoCoOp"
31.33
93.64
33.68
93.37
42.31
90.10
51.72
87.75
39.76
91.22
LoRA"
30.50
94.51
35.08
92.87
43.20
90.03
57.91
85.97
41.67
90.85
ScTAR+FT
32.95
93.41
30.26
93.81
38.56
91.24
53.32
87.72
38.77
91.55
GL-MCM Score
NPOSS
18.70
95.36
38.99
90.33
41.86
89.36
47.89
86.44
36.86
90.37
CoOp"
21.30
95.27
31.66
92.16
40.44
89.31
52.93
84.25
36.58
90.25
LoCoOpt
24.61
94.89
25.62
94.59
34.00
92.12
49.86
87.49
33.52
92.14
LoCoOp
18.97
95.90
27.33
94.31
37.29
90.75
52.98
85.95
34.14
91.73
LoRA"
15.16
96.48
27.99
93.48
36.74
90.30
57.56
83.24
34.36
90.88
SETAR+FT
21.62
95.43
23.38
94.89
32.60
91.93
51.18
87.01
32.19
92.31
CLIP-Jarge
MCM Score
LoCoOp
41.84
91.77
35.28
92.78
41.52
90.01
44.33
89.96
40.74
91.13
LoRA"
34.65
93.65
29.78
94.21
36.65
91.59
53.40
87.18
38,62
91.66
SeTAR+FT
22.41
95.83
40.07
91.98
45.19
90.13
31.37
93.48
34.75
92.86
GL-MCM Score
LoCoOp"
51.56
89,45
37.85
92.43
43.86
89.33
53.72
86.05
46.74
89.32
LoRA"
41.00
91.96
31.69
93.85
39.65
90.79
61.22
82.46
43,39
89.76
SETAR+FT 36.56
91.93
34.81
93.08
41.08
90.66
35.74
91.66
37.05
91.83
Swin-base
MSP Score
LoRA"
43.14
87.02
62.66
78.04
67.95
74.90
54.34
81.99
57.02
80.49
SeTAR+FT
29.10
94.38
52.39
86.75
57.67
85.80
49.31
84.28
47.12
87.80
Energy Score
LoRA"
62.49
71.48
65.05
71.47
75.00
63.24
46.13
85.02
62.17
72.80
ScTAR+FT
29.76
91.56
42.76
87.06
51.73
82.85
32.90
90.56
39.29
88.01
18

---

Table 15: Detail results of ablation study on modality. We CLIP-B/16 backbone.
Method
iNaturalist
SUN
Places
Texture
ImageNer23K
COCO
Average
FPR
AUC1
FPR.
AUC
FPR.
AUC
FPR.
AUCT
FPR
AUCT
FPR|
AUCI
AUCT
ImageNNtIK
MCM Score
Visual
29.69
94.58
35.15
92.99
41.25
90.45
55,00
86.92
40.27
91.24
Text
30.21
94.33
38.39
92.27
44.48
58.05
85.64
42.78
Visual+Tor Text
26.92
94.67
35.57
92.79
42.64
90.16
55.83
86.58
40.24
91.05
GL-MCM Score
Visual
13.81
96.93
27
93.67
32.97
Text
15.44
96.54
30.77
92.78
38.95
89.71
58.14
83.17
35.82
90.55
Visual-Text
13.36
Pascal-VOC VOC
MCM Score
Visual
4.13
98.63
26.31
94.58
30.44
92.58
42.48
93.20
45.19
92.36
50.60
89.36
33.19
93.45
Text
7.29
98.06
26.33
94.68
30.25
92.65
44.55
92.25
44.38
92.40
48.00
90.45
33:47
93.42
Visual+
4.59
96.71
24.91
95.15
28.46
93.21
40.44
93.58
48.25
92.08
48.10
89.70
32.46
93.74
GL-MCM Score
Visual
3,90
98.8
22.40
94.27
26.22
31.40
Text
3.55
99.01
21.26
94.48
24.87
92.96
30.89
94.07
29.86
94.49
37.10
92.09
24.59
94.52
Visual+ Text
3,66
98.96
21.93
94.81
25.04
93.62
20.35
96.36
31.47
94.31
40.70
91.19
23,86
94.87
Table 16: Detail results of SETAR with and without considering projection matrix Wp. We use CLIP-B/16 as backbone.
Method
INaturalist
SUN
Places
Texture
ImageNer22K
COCO
Averuge
FPR.
AUCT
FPR]
AUC
HPK]
AUC*
HPR]
AUCT
EPR]
AUCT
HPR
AUCT
HPR.
AUCT
Net1k
MCM Score
Vanilla MCM
32.07
94.43
38.65
92.37
43.73
90.03
57.89
86.13
43.09
90.74
SeTAR wlo Wp
26.92
94.67
35.57
92.79
42.64
90.16
55.83
$6.58
40.24
91.05
GL-MCM Score
Vanilla GL-MCM
SeTAR W
19.08
95.69
26.52
93.93
35.18
91.01
56.42
$4.34
34,30
91.24
SETAR wio W,
13.36
96.92
28.17
93.36
36,80
90.40
54.17
84.59
33.12
91.32
Pascal-VOC
MCM Score
Vanilla MCM
7.24
98.23
27.91
94.55
32.40
92.45
51 61
91.89
50.660
01.42
53.70
89:30
37.24
92.98
35.94
SeTAR wlo W.
4.59
98.71
24.91
95.15
28.45
93.21
40.44
93.58
48.25
92.08
48.10
89.70
32.46
93.74
GL-MCM Score
Vanilla GL-MCM
SeTAR W. 3,20
98.93
20.73
94.77
23.91
93.53
22.06
95,89
30.65
94.38
39.50
91.41
23.34
94.82
SETAR wio W.
3.66
98.96
21,93
94.81
25.04
93.62
20.35
96.36
31,47
94.31
40.70
91.19
23,86
94.87
19

---

Table 17: Detail results for S with different backbones. is cited from Jiang al. 2024
denotes the result of our re-run.
Method
INaturalist
SUN
Places
Texture
ImageNet22K
COCO
Average
FPRJ
AUC1
FPRE
AUCT
EPR]
AUCT
FPR.
AUCT
FPR.
AUCT
FPR
AUC1
FPR
AUCT
ImageNetIK
Vanilla MCM
32.07
94.43
38.65
92.37
13.73
90.03
$7.89
56.13
43.09
90.74
SrTAR+MCM
26.92
94.67
35.57
92.79
42.64
90.16
55.83
86.58
40.24
91.05
90.07
SeTAR+GL-MMM
13.36
96.92
28.17
93.36
36.80
90.40
54.17
84.59
33.
91.32
Vanilla NegLabel?
1.91
99.69
20.53
95.49
35.59
91.64
43.56
90.22
25.40
94.21
SeTAR
CLIP-targe
Vanilla MCM
28.17
94.97
29.18
94.12
33.66
92.37
57.73
85.46
37.19
91.73
S:TAR-MCM
Vanilla GL-MCM-
29.58
94.43
32.54
93.35
37.18
91.43
63.28
80.71
40.65
89.98
DeTARIGI-MMC
30.96
94.04
28.72
94.08
34.58
91.89
63.90
80.89
39.4
901.2
Swin Wwinsoroffremrr
Vanilla
59 19.25
84.12
ScTAR-MSP
41.44
91.08
60.05
85.04
64.31
83.70
$8.39
83 26
$6.05
85.77
Vanilla Energy"
S7.52
81.60
71,98
72.93
76.90
68,90
53.65
80.96
65.01
76.10
Faergy
41.71
83.29
80.20
Pascal-VOC
CLIP-buse
Vanilla
SeTAR 1MM
4.59
98.71
24.91
95.15
28.46
93.21
40.44
93.58
48.25
92.08
48.10
89.70
32:46
93.74
Vanilla GL-MCM
4,33
98.81
22.94
94.63
26 26.20
93.11
61
92.88
57.88
93.17
43.70
90.71
29.44
93. 88
ScTAR+GL-MCM
CLIP-barge
Vanilla MCM
42,90
94.69
44.27
93.28
41.48
91.57
61.33
89
63.37
91.20
59.90
89.40
52.21
91.68
SETAR MCM
26.05
96.23
94.20
92.45
50.32
42.57
Vanilla GL-MCM"
23.29
96.17
40.76
93.49
41.23
91.69
54.98
89
$3.19
92.67
50.30
91.09
43.96
92.45
ScTARGGL-MUM
9.62
97.51
27:75
94.75
28.85
92.99
41.77
92.40
39.45 49
93.98
39. 99.30
92.38
31.12
94.00
Table 18: Detail results for different search algorithms. Here LES stands for laycr-exhaustive greedy search, MIS stands for -onallttyinterleave greedy search, and SeTAR-S stands for the search
algorithm SeTAR, which searches vision and text layers sequentially. We use CLIP-B/16 as backbone.
Method
INaturalist
SUN
Places
Texture
ImageNer22K
COCO
Average
EPRI
AUCT
FPR
AUCT
FPR
AUCT
EPRL
AUCT
EPR
AUCT
FPRJ
AUCT
FPR
AUCT
ImageNettK MCM Score
LES
30.25
94.26
36.42
92.79
42.97
90.15
58.33
85.89
41.99
90,78
28.63
94.46
35.41
92.80
42.37
90.17
55.78
86.59
40.55
91.00
ScTAR-S
26.92
94.67
35.57
92.79
42.64
90.16
55.83
86.58
40.24
91.05
LES
14.43
96.61
27.81
93.49
36.16
90.51
$7.20
83.72
33.90
91.08
14.14
96.76
28.
33.36
91.29
ScTAR-S
13.36
96.92
28.1
93.36
36.80
90.40
$4,17
84,59
33.12
91.32
Pascal- VOC
MCM Score
5:20
MIS
5.82
98.49
25.52
95.04
30.10
92.98
43.95
93.06
50.00
92.06
48.20
80.35 844
33.93
93.58
ScTAR-S
4.59
98.71
24.91
95.15
28.46
93.21
40.44
93.58
48.25
92.08
48.10
89.70
32.46
93.74
GL-MCM Score
LES
3.89
98.87
21.56
94.56
24.70
93.32
23.35
80
32.99
93.82
40.40
91.03
24.48
94.57
MIS
3.53
98.95
20.87
94.77
24.30
93.47
19.91
96.24
29.59
94.40
39.00
91.21
22.87
94.84
ScTAR-S
3.66
98.96
21.93
94,81
25.04
93.62
20.35
96.36
31.47
94.31
40.70
91.19
23.86
94.87
20

---

Table 19: Detail results of ablation study on A. We use CLIP-B/16 backbone.
iNaturalist
SUN
Places
Texture
ImageNer22K
COCO
Average
FPR.
AUC
FPR.
AUCT
FPR]
AUC1
FPR.
AUC1
FPR.
AUC
FPR.
AUC
FPR.
AUC1
I MCM
Score
0.01
28.31
94.60
36.83
92.74
43.01
90.10
55.85
86.58
41.00
91.00
94.75
92.70
90.15
55.57
91.02
0.10
26.92
94.67
35.57
92.79
42.64
90.16
55.83
86.58
40.24
91.05
34,29
93.66
92.85
90.24
58
42.65
90.69
0.20
34,89
93.62
35.59
92.88
41.95
90.28
58.19
86.11
42,66
90.72
35.88
93.42
92.76
90.18
58.
90.55
0.30
37.72
93.26
36.27
92.64
42.35
90.10
58.46
86.03
43.70
90,50
GL-MCM Score
0.01
13,98
96.76
29.20
93.17
37.56
90.09
54.10
84.47
33.71
91.12
0.05
13.90
96.79
28.84
93.24
37.25
90.32
54.20
84.33
33.55
91.17
13.36
93.36
90.40
54.17
33.12
91.32
0.15
16.85
96.12
26.99
93.72
35.14
90.74
56.79
83.90
33.94
91.12
93.70
0,25
18.30
95.87
27.55
93.64
36.06
90.58
$8.28
83.70
35.05
90.95
93.63
VOC MCM Score
5.58
98.43
25.14
94.94
93.01
40.41
93.35
47.85
92.12
60
89.37 37
93.54
0.05
4.59
98.71
24.91
95.15
28.46
93.21
40.44
93.58
48.25
92.08
48. 40
89.70
32.46
93.74
95.06
48.69
93.65
0.15
5,97
98.53
26.50
95.07
30.88
93.05
46.22
92.94
50.99
92.07
49.80
89 80
35.06
93.58
98.53
95.08
93.06
45.43
93.00
50.68
92.16
49.40
89
93.62
0.25
6.41
98.43
26. 19
94,99
31.24
92.89
47.36
92.72
50.41
92.13
50.20
89.74
35.30
93.48
98.
26.98
94.80
92.65
48.67
50.53
51.10
89.77
36.04
93.37
GL-MCM Score
0.01
4.42
98.83
22.72
94.73
25.93
93.51
22.07
96.22
32,62
94.27
43,50
90.91
25.21
94.74
0.05
3,66
98.96
21.93
94.81
25.04
93.62
20.35
96.36
31.47
94.31
40.70
91.19
23.86
94.87
0.10
3.79
98 98.94
21.40
94.76
25.05
93.49
20.74
96.29
30.42
94.48
40.00
91.20
23.57
94,86
3,50
20.83
94.84
93.55
20.57
29.84
38.50
91.25
22.93
94.87
0,20
3.50
98.94
20.72
94.74
24.13
93.48
19.95
96.28
29.22
94.46
38.60
91.19
22.69
94.85
4.14
21.54
94.85
93.54
23.37
32.18
40.30
24.48
94.90
0,30
4.15
98.90
21.40
94.63
25.16
93.33
23.01
96.03
31.02
94.44
38.90
91.40
23.94
94.79
Table 20: Detail results on different pruning strategies. We use CLIP-B/16 as backbone.
iNaturalist
SUN
ImageNe
COCO
Average
Method
FPR
AUC
FPR)
AUCI
FPR1
AUCE
FPR.
AUC1
FPR.
AUCE
FPR
AUCI
FPR.
AUC
ImageNet1K
MCM Score
Principle 32.07
94.43
38.65
92.37
43.73
90.03
57.89
86.13
43.09
90.74
Random
32.07
94.43
38.65
92.37
43.73
90.03
57.89
86.13
43.09
90,74
Minor
35.57
92.79
42.64
90.16
40.24
GL-MCM Score
Principle
30.65
37.76
35.29
Random
32.07
94.43
38.63
92.37
43.73
90.03
$7.89
86.13
43.09
90.74
Pascal-VOC
Principle
9.91
98.01
29.24
93.91
32.89
92.30
$4.43
90.30
53.53
91.07
49.20
89.07
38.20
92.44
Random
7.24
27.45
64.60
32.52
92.43
52.97
35.57
93.09
Minor
4,59
98.71
24.91
95.15
28. 46
93.21
40.44
93.58
48.25
92.08
48. 10
89,70
32,46
93.74
GL-MCM Score
Principle
3.10
98.62
20.07
94.41
22.
93.38
38.53
92.19
31.61
93.07
36.50
90.34
25.36
93.67
Rundom
3.47
98.99
20.04
95.46
24.07
93.95
31.76
94,86
35.71
93.67
42.17
91.04
26.20
94.66
Minor
3.66
98.96
21.93
94.81
25.04
93.62
20.35
96.36
31.47
94.31
40.70
91.19
23.86
94.87
21

---

Table 21: Detail results of ablation study on top-K. We use CLIP-B/16 backhone.
INaturalist
SUN
Places
Texture
ImageNaR22K
COCO
Average
K
FPR.
AUCT
FPRE
AUCT
FPR,
AUCT
FPR
AUC1
FPR
AUCT
FPR
AUCT
FPR4
AUCT
ImageNetIK
MCM Score
100
26.92
94.67
35.57
92.79
42.64
90.16
55.83
86.58
40.24
91.05
91.05
300
26.92
94.67
35.57
92.79
42.64
90.16
55,83
86.58
40.24
91.05
400
26.92
94.67
35.57
92.79
42.64
90.16
55,83
86.58
40.24
91.05
500
26.92
94.67
35.57
92.79
42.64
90.16
55,83
86.58
40.24
91.05
600
26.50
94,70
36.22
92.66
43.04
90.10
55.82
86.46
40.39
90.98
700
26.92
94.67
35.57
92.79
42.64
90.16
55,83
86.58
40.24
91.05
800
26.92
94.67
35.57
92.79
42.64
90.16
55,83
86.58
40.24
91.05
900
29.38
94.38
36,02
92.75
42.47
90.24
55,20
86.77
40.77
91.03
1000
30.63
94.17
36.24
92.93
42.58
90.24
56.84
86.34
41.57
90.92
GL-MCM Score
0
14,02
96. .0
28.32
93:40
36.91
90.52
54.68
84.32
33.48
91.26
100
13.36
96.92
28.17
93.36
36.80
90.40
54.17
84.59
33.12
91.32
200
13.36
96.92
28.17
93.36
36.80
90.40
54.17
84.59
33.12
91.32
36.80
91.32
400
13.36
96.92
28.17
93.36
36.80
90.40
54,17
84.59
33.12
91.32
500
96.92
28.17
36.80
33.12
91.32
600
14.02
96.80
28.32
93.40
36.91
90.52
54,68
84.32
33.48
91.26
700
96.92
28.17
93.36
36.80
90.40
54.17
84.59
33.12
91.32
800
13,36
96.92
28.17
93.36
36.80
90.40
54.17
84.59
33.12
91.32
900
14.71
96.63
28.64
93.31
36.56
90.41
54.04
84.78
33.49
91.28
1000
15.82
96.42
28.61
93.46
37.20
90.40
54.75
84.35
34.10
91.16
VOC MCM Score
D
5,58
98.43
25.14
94.94
29.13
93.01
40.41
93.35
47.85
92.12
49.9 60
89.37
32.95
93,54
24.91
4 4,59
98.71
24.91
95.15
28.46
93.21
40.44
93. 58
48.25
92.08
48. 10
89.70
32.46
93.74
89.37
8 5.27
98.45
24.26
94.98
28.31
93.06
39.61
93.31
46.99
92.11
48. 18.10
89.38
32.09
93.55
5,58
98.43
25.14
94.94
29.13
93.01
40.41
93.35
47.85
92.12
49.60
89.37
32.95
93.54
4,59
98.71
24.91
95.15
28.46
93.21
40.44
93.58
48.25
92.08
48.10
89.70
32.46
93.74
98.43
25.14
94.94
29.13
93.01
40.41
93.35
47.85
92.12
49.60
89.37
32.95
93.54
GL-MCM Score
0
4.42
98.83
22.72
94.73
25.93
93.51
22.07
96.22
32.62
94.27
43.50
90.91
25.21
94.74
2
3.66
98.96
21.93
25.04
93.62
20.35
96.36
31,47
40.70
91.19
23,86
94.87
4
3,66
98. 96
21.93
94.81
25.04
93.62
20.35
96.36
31.47
94.31
40.70
91.19
23.86
94.87
98.83
22.72
25.93
93.51
22.07
96.22
32.62
94.27
43,50
90.91
25.21
94,74
8
4.47
98.84
22.76
94.79
25.99
93.56
22.39
96.19
32.85
94.27
43.30
90.95
25.29
94.76
96.22
43.50
12
3.66
98.96
21.93
94.81
25.04
93.62
20.35
96.36
31.47
94.31
40.70
91.19
23.86
94.87
Table 22: Detail results of ResNet50, We use ImageNetIK as the ID dataset. i cited from Djurisic
al. (2023).
iNaturalist FPR
SUN
Places
Texture
Average
Method
AUC1
FPR.
AUC1
FPR.
AUC
FPRJ
AUC
FPR
AUC
Softmax
54.99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81.99
Energy
55.72
89.95
59.26
85.89
64.92
82.86
53.72
85.99
58.41
86.17
ReAct
20.38
96.22
24.20
94.20
33.85
91.58
47.30
89.80
31.43
92.95
DICE
25.63
94.49
35.15
90.83
46.49
87.48
31.72
90.30
34.75
90.77
ASH-P
44.57
92.51
52.88
88.35
61.79
61.79
42.06
89.70
50.32
89.04
ASH-Bt
14.21
97.32
22.08
95.10
33.45
92.31
21.17
95.50
22.73
95.06
ASH-S*
11.49
97.87
27.98
94.02
39.78
90.98
11.93
97.60
22.80
95.12
SeTAR
10.08
98.11
27.68
94.15
39.22
91.24
12.54
97.51
22.38
95.25
22

---

91.2
Rio
91.0
AUROCC
$0.8
20.0
60.4
Aug.
90.2
90.0
W-W-WWW
(a) MCM score
Wip
Mose
37
91.0
Ave.
90.5
Aup
34-
90.0
Iaseine
(b) GL-MCM score
Figure 2: Average AUROC/FPR95 of different weight types on ImageNet1K benchmark. We use CLIP-B/16 B/16 as a backbone.
Wie
Wooe
Arr SPHOS
30
42.5
Was
92.0
baseline
(a) MCM score
32
W.a
94.25
$1.00
93.75
Ras
91.50
baseline
10
vfviited
(b) GL-MCM score
Figure 3: Average AUROC/FPR95 of different weight types Pascal-VOC benchmark. We CLIP-B/16 as a backbone.
23

---

91.2
Avg. A.-OL
91.0
Arg. FRRDS
Se SATR-AOC SABARGLLME
30
20.8
20.6
34
0.00
0.15
0.25
0.10
9.20
The Vauu of
The Value of
(a) ImageNet1
30
34
24.50
32
FIROS
Akg.
$4.00
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.00
0.05
0.30
0.15
0.20
0.25
0.30
The Value oFA
The Valian of
(b) Pascal-VOC
Figure 4: Ablation studies on À on different ID datasets. We use CLIP-B/16 as backbone.
91.3
91.2
Av2. FPADS
91.1
Avp
90.9
400
100
1002
200
400
1000
The Value of
The Value of
(a) ImageNet|
94.8
32
24.0
94.4
Awy. FAUS
64.2
20
Aug.
33.6
1.0
12
6
20
12
14
The Value of
The Vallan of
(b) Rascal-VOC
Figure 5: Ablation studies on top-K on different ID datasets. We use CLIP-B/16 as backbone.
24

---

66-
64-
622
-41
(a) LoCoOp Loss
(b) ID Loss
(c) OOD Loss
Figure 6: Loss plots of SETAR+FT v.s. LoRA on ImageNetIK. We use CLIP-B/16 as a backbone. F demonstrates faster convergence across all losses, especially in the OOD loss. For reference, with MCM score, SeTAR+FT achieves an average FPR of 38.77 at epoch 5. While LoRA achieves average FPR of 42.88, 39.92 and 42.23 epoch 1, 5 and 15, respectively.
1N0K
0.00
0.80
2.50
0.20
0.35
5.40
0.23
0.00
0.00
V11
V.30
V.9
V7
V6
V5
44
V3
v.0
T31
T10
T-
T-3
74
T.5
73
T2
Modality Layenndee
(a) CLIP-base
8.0
9.0
033
042
033
vio
wis
Via
V17
Vaa
vGa
vén
05 vi5
via
Voz
V2
(b) CLIP-large
5.43
0,40
0.40
3.00
0.00
0.00
0.00
0.10
0.00
0.00
3.00
3.25
0.00
0.00
0.00
0.00
0.40
0.30
33
30
2-17
2:16
2-15
2-34
2-13
232
2-11
2-10
2-4
2/7
2:5
2-5
244
22
24
2/0
14
14
511
0.
Stagetndex Blackiddx
(c) Swin-base
Figure 7: Visualization of SeTAR rank reduction ratio distribution on different ID datasets with different backbones. IN1K, VOC stand for ImageNet and Pascal-VOCC And V, T stand for visual modality and text modality of the CLIP model.
25

---

10401_12pe
best,roc10
loiol_
id,less
sod.1ess
va1.260
col.petoh.perress
0.647777
1.29325
-4.455494
21-322225
0.644654
1.383622
-4.589753
71-792225
203523366
1.083625
5.999751
293876
664654
1.683629
-4. 56976661
293876
644654
-6
799999
293876
0.00
044654
1.C83620
-4.389752
11.799925
203576
0.05
646844
1.079725
4.389544
71 292225
23 208505
K_9P
0.00
640844
1.679739
-4.300044
21.229020
35. 208595
20
0.00
6464444
1-679720
-4.380544
21-222230
35 205595
22
0
0.
G. C40132
1-679159
-4.389775
72 122122220
39 156123
0.00
640132
1-679199
369775
72 199998
39 156128
0.00
640132
1.079109
34988775
0.
640132
1.072102
-4.38975
156123
0.0p
640132
1.079100
-4.389775
K.Up
64666322
1.079109
0.622665
1.675858
638932
00.000001
452552
0.626440
1.670326
4.888857
999997
646488
0.618521
L.670858
44.00368
020021
444899
K.
0.00
0.618522
1.076858
-4.585565
71 600601
25 444859
text
0.20
0.618174
1.069853
-4.54975
72 499997
38. 642345
Listing I: Example procedure of ScTAA on ImageNet with CLIP-base. We search the visual and text tower from top to bottom. each step, we select the best ratio that minimizes the loss.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have carefully crafted the abstract and introduction to accurately reflect the contributions and scope of the paper. Specifically, we propose novel waining-free method, SeTTRR with a finetuning extension SeTAR+FT, and demonstrate its effectiveness for OOD detection tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the proposed method are discussed in Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper includes detailed experimental and hyperparameters settings in Section 4.T and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code are available at https://github.. htaptuau..com/XAAAXAA//SSeeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iis.cc/ppile wwtpcccpppppppppeeeeeeeeeeeeeeee htt.:///gg..eee/ooeSSuuuiiiisss /Cc////oddiiisssseeeeeee sww.iissisiiiiiiiico for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide detailed experimental settings in Section 4.1 and Appendix We give the details of our design choices in Section 4.4 and datasets in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the results with standard deviation from runs of seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details of the compute resources in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:://// httth:://t/unttt http://www....ppp..a.///aaaaaaaa
Answer: [Yes]
Justification:
We provide the ethical considerations in Appendix
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: provide the broader impacts Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not directly provide pre-trained models or scraped datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include the citation and URL of models and datasets used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "z4c4Yinjmp", "reviewer": "Reviewer_aAbP", "summary": "The paper presents SETAR, a novel method designed to enhance out-of-distribution (OOD) detection without requiring additional training. The proposed method leverages rank reduction techniques applied to the model weights, specifically targeting the minor singular components, while retaining the principal components that significantly contribute to the model’s performance. SETAR is evaluated across various model backbones, including CLIP-base, CLIP-large, and Swin-base, and demonstrates notable improvements in OOD detection tasks. The paper also provides comprehensive experiments and ablation studies to validate the effectiveness and efficiency of SETAR.", "questions": "1. Table 2: Could you provide more insights on why the performance boost on Swin-Base is significantly more pronounced compared to CLIP-Base and CLIP-Large? Is this disparity related to CLIP’s stronger zero-shot performance?\n\n2. Figure 2: For ImageNet-1K, it appears that decomposing the vision encoder alone is sufficient for OOD detection, with the combination of both vision and text encoders yielding only minor improvements. Could you explain this in more detail?\n\n3. The proposed method is training-free, but the greedy search algorithm used for determining the rank reduction ratio list and performing rank reduction does require computation time. Can you provide details on the time required for the greedy search and rank reduction for SeTAR, as well as the overall computation time of SeTAR+FT, compared to previous fine-tuning methods?\n\n4. The proposed method seems to be an application of LASER, the only contribution is the adaptation of LASER to CLIP model, and benefits OOD detection. Can the authors justify this? The paper is technically solid with comprehensive experiments, but I think the paper is not novel enough given the similarities compared to LASER.", "limitations": "Limitation discussed in Appendix", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tNovelty and Innovation: The introduction of a training-free method for improving OOD detection is a significant contribution. By focusing on rank reduction of model weights, the method offers a fresh perspective compared to traditional training-intensive approaches.\n\t2.\tComprehensive Evaluation: The paper provides extensive experimental results across multiple datasets and model backbones, showcasing the robustness and generalizability of SETAR.\n\t3.\tEffective Performance: The method achieves substantial improvements in OOD detection metrics, such as FPR95 and AUROC, demonstrating its practical utility. The significant performance boost on Swin-base compared to CLIP-base and CLIP-large is particularly notable, likely due to the inherent design differences and stronger zero-shot performance of CLIP models.\n\t4.\tDetailed Analysis: The inclusion of ablation studies and sensitivity analyses helps in understanding the impact of various components and hyperparameters, offering valuable insights into the method’s functioning.", "weaknesses": "I hope the authors could address my following questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Performance Disparity on Swin-Base\n\n- The performance boost on Swin-Base is more pronounced because Swin is trained directly on ImageNet, lacking a text-encoder and thus requiring training solely on IN1K. This can lead to overfitting on ID data and poor recognition of OOD images, providing more room for improvement. Our method helps alleviate this issue, improving Swin's generalization to OOD samples.\n\n- In contrast, CLIP models are pretrained on large image-text datasets, which provides robust representation capabilities for both ID and OOD images. Consequently, CLIP’s baseline OOD performance is already strong, leaving less room for further improvement compared to Swin-Base. Therefore, the pronounced performance boost seen with Swin-Base is due to its initial lower performance and higher potential for enhancement through our method.\n\n# 2. Modality Difference for Performance\n\n1. **Significance of Vision Modality:** As shown in Figure 2, the experiments demonstrate that the vision encoder is more critical for OOD detection tasks compared to the text encoder. This is intuitive since, in image-based OOD detection, the vision encoder is essential for extracting features and identifying OOD patches. The text encoder, on the other hand, shows limited improvement in performance for this specific task, indicating that the vision modality holds greater importance.\n\n2. **Combined Modality for Optimal Performance:** While decomposing the vision encoder alone provides substantial improvements, incorporating the text encoder, albeit with minor gains, ensures we maximize the model’s capabilities. By leveraging both vision and text modalities, we achieve better overall performance, which is why we opted for the Vision+Text modality approach in our method.\n\n# 3. Computation Time Concerns for SeTAR and SeTAR+FT\n\nWe appreciate the reviewer's concern regarding the computation time required for our proposed SeTAR method. Here is the comparison for CLIP-base on ImageNet1K:\n\n1. **SeTAR and Fine-Tuning Times:**\n   - SeTAR requires approximately 14 minutes to complete the Vision+Text greedy search and rank reduction for 1K images. If we only apply the search to the Vision modality, the total time is about 7 minutes, which also achieves competitive performance as noted in the previous point.\n   - SeTAR+FT takes a total of 14 minutes and 11 seconds, consisting of two stages. The first stage is SeTAR for low-rank searching, which takes about 14 minutes. The second stage is LoRA tuning, which takes around 11 seconds for 5 epochs on the same 1,000 images, primarily due to the small model size and limited development set.\n   - For comparison, LoCoOp fine-tuning takes about **16 minutes** for 50 epochs.\n\n2. **Detailed Time Analysis for SeTAR:**\n   - **Greedy Search Loss Calculation:** This accounts for 47.67% of the total time.\n   - **Dataloader:** Takes up 11.24% of the time. This delay occurs because the CPU cannot match the GPU speed for smaller models and few samples; for larger models, this delay can be negligible.\n   - **SVD Pruning and Reloading:** To maintain code clarity and compatibility, each step involves reloading and applying SVD pruning, which takes 20% of the total time. We plan to optimize this by loading the model once at the beginning and pre-computing SVD in parallel to avoid redundant calculations. This optimization could reduce the time taken by these steps to about 1/24 of its current value.\n   - With these optimizations, we estimate the overall time for SeTAR could be reduced to about half of the current duration, approximately **7 minutes** for Vision+Text and **3.5 minutes** for Vision-Only searching.\n   \nThese points highlight the efficiency of SeTAR and SeTAR+FT in both low-rank searching and fine-tuning compared to existing methods.\n\n# 4. Novelty Concerns Compared to LASER\n\nWe appreciate the reviewer's feedback and the opportunity to clarify the unique contributions of SeTAR compared to LASER:\n\n1. **Distinct Approach of SeTAR:**\n   - **Beyond Simple Application:** SeTAR is not just an adaptation of LASER to the CLIP model. LASER primarily focuses on pruning individual layers to enhance factual answering capabilities and does not extensively explore different greedy pruning strategies. Additionally, LASER relies on a validation set for selection, which is not suitable for OOD detection.\n   - **Greedy Pruning Algorithm and OOD Detection:** SeTAR focuses on designing a greedy pruning algorithm tailored for OOD detection. To address the challenge of unavailable OOD validation sets, we extract OOD information from ID images to guide the algorithm. We also conducted a sensitivity analysis of different parameters. Furthermore, SeTAR includes a comprehensive analysis comparing various modalities, search algorithms, pruning strategies, and backbones, enhancing our understanding of low-rank pruning beyond what LASER provides.\n\n2. **Innovations in SeTAR+FT:**\n   - **Dynamic Rank Adjustment:** In addition to the training-free search algorithm, we explored the potential of using SeTAR for fine-tuning. Traditional LoRA distributes the rank evenly across all layers, leading to inefficiencies and performance losses.\n   - **Effective and Efficient Fine-Tuning:** By combining SeTAR with LoRA, SeTAR first identifies the impact of different layers on performance and dynamically adjusts the rank accordingly. This approach initializes different LoRA weights more effectively, tailored to the specific ID dataset, resulting in a more effective and efficient fine-tuning process.\n\nThese points illustrate that SeTAR offers significant advancements over LASER, both in methodology and application, particularly for OOD detection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qaJFbv9jFV", "reviewer": "Reviewer_SyYP", "summary": "This paper proposes an algorithm for OOD detection along with CLIP models.\nIt observes that pruning based on SVD decomposition on CLIP models can improve the OOD detection performance.\nA greedy search algorithm is developed for searching pruning rations of each weight in CLIP models.\n\nExperiments on regular settings demonstrate obvious performance when compared with the Vanilla GL_MCM method.", "questions": "(1) The main results in the paper concentrate on CLIP models.\n      Results on other CNN-based models and comparisons with previous methods should be included in the paper.\n\n(2) The proposed greedy search algorithm can not be guaranteed to be converged.\n      Thus, a discussion on the stability of the proposed algorithm is required.", "limitations": "A section of Impact Statements including limitations is included in the paper", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is written clearly and easy to follow.     \n(2) The method is simple but effective.             \n(3) The ablation studies are sound.", "weaknesses": "(1) It seems that the proposed method is general enough to apply to regular models not just CLIP models.    \n     Will it also work on CNN-based ResNets and ViT models trained on datasets, like ImageNet and CIFAR?        \n      The experimental results focus on CLIP-based models in Table 1.\n     More comparisons with previous methods using CNN-based models should be included.     \n     \n(2) SVD decomposition pruning has strong connections with sparsification-based methods [1,2,3]. \n     Comparisons with this kind of method should be included.\n\n[1] Dice: leveraging sparsification for out-of-distribution detection.     \n[2] Extremely simple activation shaping for out-of-distribution detection.       \n[3] ReAct: out-of-distribution detection with rectified activations.                   \n\n(3) The proposed greedy search algorithm is efficient. However, it can not be guaranteed to be converged.\n     What's the performance variance for 2 independent runs of the search algorithm?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Applicability to CNN-Models\n\n1. **Will it work on CNN-based ResNets?**\n   - No. Our method is not applicable to pure CNN models like traditional ResNet50. The loss function (Eq. 12) in our approach includes an OOD loss (Eq. 11), which relies on local features from the attention layer. Since pure CNN models lack self-attention layers, our method cannot be directly used or compared with CNN-based models.\n   - However, in CLIP-ResNet models, a self-attention layer is added to the last layer of the ResNet tower. Therefore, we can conduct experiments on CLIP-ResNet models.\n   - Specifically, we used the CLIP-ResNet50x4 model as the backbone with ImageNet1K as the ID dataset. By applying SVD pruning on the conv1 layer of each vision layer and  W_{up} on each text layer, we obtained the following results, demonstrating that our method is also applicable to CLIP-ResNet models (\\* stands for our re-run).\n\n        | CLIP-ResNet50x4  | iNaturalist FPR↓ | iNaturalist AUC↑ | SUN FPR↓ | SUN AUC↑ | Places FPR↓ | Places AUC↑ | Texture FPR↓ | Texture AUC↑ | Average FPR↓ | Average AUC↑ |\n        |:----------------|-------------------:|-------------------:|-----------:|-----------:|--------------:|--------------:|---------------:|---------------:|---------------:|---------------:|\n        | Vanilla MCM\\*    | 44.03 | 91.58 | 35.18 | 92.83 | 44.38 | 89.38 | 57.29 | 85.99 | 45.22 | 89.95 |\n        | SeTAR+MCM        | 41.29 | 92.12 | 35.44 | 92.76 | 43.01 | 89.85 | 54.82 | 86.89 | **43.64** | **90.40** |\n        | Vanilla GL-MCM\\* | 32.17 | 93.09 | 46.64 | 89.27 | 51.85 | 85.86 | 44.47 | 86.49 | 43.78 | 88.68 |\n        | SeTAR+GL-MCM     | 30.15 | 93.73 | 45.01 | 89.58 | 49.82 | 86.68 | 42.32 | 87.30 | **41.83** | **89.32** |\n\n2. **Will it work on ViT models trained on datasets like ImageNet and CIFAR?**\n   - Yes. The Swin-Transformer is a ViT model that does not include a text-encoder but only an image-encoder. As shown in Table2 and Table 7, we used the Swin-base model trained on ImageNet1K as the backbone and observed significant improvements over the original model.\n\n3. **More Comparisons with Previous Methods Using CNN-based Models:**\n   - As mentioned, our method is not applicable to CNN-based models. More relevant comparisons are addressed in the response to question 2.\n\nThese points clarify that while our method is not suitable for pure CNN models, it is effective for models incorporating attention mechanisms, such as CLIP-ResNet and ViT models like Swin-Transformer.\n\n# 2. Comparisons with Sparsification-Based CNN Methods\n\nWe appreciate the reviewer's suggestion and have carefully reviewed the referenced papers. These methods are based on CNN-ResNet models and the ImageNet1K dataset. However, there is a significant difference between these models and the CLIP-ResNet:\n\n1. **Training Differences:**\n   - CNN-ResNet models, due to the lack of a text-encoder, are fine-tuned directly on ImageNet1K.\n   - In contrast, CLIP-ResNet models are not trained directly on ImageNet1K. Therefore, directly comparing the results of CLIP-ResNet with CNN-ResNet models is not meaningful, since ID-domain training would largely boost the model performance.\n\n2. **Lack of Suitable CLIP-ResNet Models:**\n   - We attempted to find CLIP-ResNet models that were fine-tuned on ImageNet1K for a fair comparison but were unable to locate such models.\n\n3. **Truly Training-Free Nature of Our Method:**\n   - Our method is **literally training-free**, as it does not require any training on the ID dataset. In contrast, the backbones used in the sparsification-based methods require training on the ID dataset.\n\nThese points highlight the challenges in making direct comparisons with sparsification-based methods and underscore the unique, training-free advantage of our approach.\n\n# 3. Convergence and Performance Variance Concerns\n\nThe SVD algorithm used in our method is quite stable. We have compared the results using different random seeds (3, 4, and 5), and the SVD results are consistent across these different seeds. Therefore, SeTAR does not involve performance variance due to the deterministic nature of the SVD algorithm used in our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVxtSpHhH1", "reviewer": "Reviewer_6fvA", "summary": "The paper introduces SeTAR, a novel training-free out-of-distribution (OOD) detection method that leverages selective low-rank approximation of weight matrices in vision-language and vision-only models. SeTAR enhances OOD detection by post-hoc modifying the model’s weight matrices using a greedy search algorithm. The paper also extends this method to SeTAR+FT, a fine-tuning approach to further optimize OOD detection performance.", "questions": "Please refer to weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper proposes a unique training-free method for enhancing OOD detection, which is novel.\n2.\tThe paper includes thorough ablation studies and sensitivity analyses, which help in understanding the robustness and generalizability of the proposed approach.\n3.\tThe authors provide extensive empirical evaluations on multiple benchmarks, showing the effectiveness of SeTAR and SeTAR+FT.", "weaknesses": "-\tThe performance gains compared to baselines are relatively small. Given that these results are achieved using a greedy search strategy, the potential of SeTAR may be limited.\n-\tThere is a lack of theoretical analysis explaining why SeTAR is effective. Understanding the underlying principles is crucial for advancing the method and its applications.\n-\tThe experiments indicate that the optimal hyperparameters (λ and K) vary significantly across different backbones. In practical OOD detection scenarios, it is challenging to obtain OOD samples for hyperparameter tuning. Therefore, SeTAR needs to be strengthened in this aspect to ensure robust performance without extensive hyperparameter tuning.\nP.S. The layout of Figure 1 is somewhat cluttered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Performance Gains\n\n1. **Limited Room for AUC Improvement:**\n   - The baseline AUC scores are above 90, leaving limited room for significant improvement. Despite this, our method still achieves AUC improvements, demonstrating its effectiveness even in a high-performance context.\n\n2. **Significant FPR Improvement:**\n   - For the False Positive Rate (FPR), our method shows notable improvements. These reductions in FPR highlight the practical benefits of our approach in reducing false positives, enhancing the reliability of OOD detection.\n\n3. **Pronounced Performance on Swin-Base:**\n   - As shown in Table 2 and Table 7, the performance boost on Swin-Base is significant. For example, using SeTAR with the Energy score on ImageNet1K, the reductions in FPR and improvements in AUC are over 20% and 9.8% compared to the baseline. SeTAR+FT further reduces FPR and improves AUC by more than 36% and 20% compared to the baseline.\n\nThese points emphasize the substantial performance gains achieved by our method, particularly on Swin-Base, highlighting its effectiveness in various contexts.\n\n# 2. Theoretical Analysis of SeTAR's Effectiveness\n\nTo address this, we draw on theoretical principles from recent work on SVD-based weight pruning, particularly from the study titled \"[Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective](https://arxiv.org/pdf/2406.03768)\".\n\nSeTAR employs selective low-rank approximation through SVD-based pruning, which aligns with established theoretical frameworks that explain how such techniques can enhance model performance:\n\n1. **Gradient Stability and Generalization:**\n   - The theoretical analysis of SVD-based weight pruning shows that pruning can stabilize the gradient updates in neural networks. By pruning weights, particularly in layers where noise might be higher, we reduce the sensitivity of the network to small perturbations, leading to more stable and robust performance. This stability is crucial for OOD detection tasks, as it allows the model to maintain its performance across varying inputs.\n\n2. **Matrix Condition Number and Noise Reduction:**\n   - The effectiveness of weight pruning can also be understood through the concept of matrix condition numbers. High condition numbers indicate ill-conditioned problems prone to significant errors due to small perturbations. Pruning minor singular values reduces the condition number, stabilizing the model and enhancing its robustness to noise. This is crucial for tasks like OOD detection, where stability and robustness are key.\n\n3. **Retention of Principal Components:**\n   - By retaining the principal components (i.e., the components corresponding to the largest singular values), SeTAR ensures that the most critical information is preserved while reducing noise. This principle, rooted in the Eckart-Young-Mirsky theorem, provides an optimal low-rank approximation that maintains essential features necessary for effective OOD detection.\n\n\n# 3. Hyperparameter Tuning\n\n1. **Robustness of Top-K Hyperparameter:**\n   - The optimal Top-K parameter is related to the number of ID categories and cannot be directly transferred between different ID datasets. However, as shown in Figure 6, this parameter is quite robust. We generally recommend setting it to 30% of the total number of categories. For example, ImageNet1K (300/1000) and Pascal-VOC (4/14). For the Swin-base model, setting Top-K to 300 also yields good performance.\n       \n    | Backbone | Score | Vanilla Method FPR↓ | Vanilla Method AUC↑ | SeTAR (TopK700) FPR↓ | SeTAR (TopK700) AUC↑ | SeTAR (TopK300) FPR↓ | SeTAR (TopK300) AUC↑ |\n    |:-----|:--------|-------:|-------:|--------:|--------:|--------:|--------:|\n    | Swin-base| MSP | 59.25| 84.12| **56.05** | **85.77** | 56.82 | 85.68 |\n    | Swin-base| Energy| 65.01| 76.10| **51.61** | **84.42** | 52.56 | 84.51 |\n\n2. **Transferability of λ Hyperparameter:**\n   - The λ parameter shows transferability across different datasets for the same backbone. As shown in Table 9, the λ for CLIP-base ranges between 0.05 and 0.1, while for CLIP-large it ranges between 0.3 and 0.5.\n\n3. **Hyperparameter Transferability Across Datasets:**\n   - The optimal hyperparameters exhibit transferability across different datasets. For instance, the optimal hyperparameters for CLIP-Base on ImageNet1K and Pascal-VOC can be interchanged with minimal performance degradation, and both outperform the vanilla method.\n\n    | ImageNet1K | Score  | Vanilla Method FPR↓ | Vanilla Method AUC↑ | SeTAR Optimal hyperpara FPR↓ | SeTAR Optimal hyperpara AUC↑ | SeTAR with VOC optimal hyperpara FPR↓ | SeTAR with VOC optimal hyperpara AUC↑ |\n    |:--|:-------|---:|---:|---:|---:|---:|---:|\n    | CLIP-base  | MCM| 43.09| 90.74| **40.24**| **91.05**| 40.41 | 91.02 |\n    || GL-MCM | 35.29| 90.86| **33.12**| **91.32**| 33.55 | 91.17 |\n    | CLIP-large | MCM| 37.19| 91.73| **36.26**| **91.92**| 36.73 | 91.81 |\n    || GL-MCM | 40.65| 89.98| **39.54**| **90.22**| 39.18 | 90.10 |\n        \n    | Pascal-VOC | Score| Vanilla Method FPR↓ | Vanilla Method AUC↑ | SeTAR Optimal hyperpara FPR↓ | SeTAR Optimal hyperpara AUC↑ | SeTAR with ImageNet1K optimal hyp FPR↓ | SeTAR with ImageNet1K optimal hyp AUC↑ |\n    |:-----------|:-------|------:|------:|--:|--:|------------:|------------:|\n    | CLIP-base| MCM| 37.24| 92.98| **32.46** | **93.74** | 33.18|93.65|\n    || GL-MCM | 29.44| 93.88| **23.86** | **94.87**| 23.57| 94.86|\n    | CLIP-large | MCM| 52.21| 91.68| **42.57** | **92.91** | 44.39| 92.34|\n    || GL-MCM | 43.96 | 92.45 | **31.12** | **94.00**| 33.74 | 93.76|\n\n4. **Figure 1 Layout:**\n   - We appreciate the feedback on Figure 1 and have adjusted its layout for better clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S3MtPczIoS", "reviewer": "Reviewer_hoZJ", "summary": "This work proposes SeTAR for CLIP-based OOD detection. SeTAR is based on low-rank approximation. It determines the optimal rank for each weight block with a greedy hyperparameter search using validation samples. While SeTAR itself is training-free, the paper also proposes SeTAR+FT which incorporates LoRA as a training extension. Experiments demonstrate that SeTAR outperforms existing methods on CLIP-specific benchmarks.", "questions": "See weaknesses", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is written in good quality, and the proposed method sounds reasonable.", "weaknesses": "1. My main concern is that the performance improvement of SeTAR (SeTAR+FT) seems limited compared with MCM and GL-MCM (LoCoOp). In Table 1, for example, all improvements of SeTAR in terms of AUROC are within 1%. The only case where SeTAR leads to noticeable/significant improvements is when the model is the image-only Swin-T (Table 2). Would the authors comment on why this is the case?\n\n2. I'm confused by some of the numbers discussed in the main text. In line 220-222, it says `For example, using Pascal VOC as ID, SeTAR yields an average reduction of 12.84% FPR95 on MCM and 18.95% FPR95 on GL-MCM`. How is the \"12.84\" and \"18.95\" computed? According to Table 1, on Pascal VOC, the average FPR95 of MCM and SeTAR is 38.88 and 32.46, respectively. The average FPR95 of GL-MCM and SeTAR is 31.12 and 23.86, respectively. A similar confusion of mine is regarding line 238-240 `when scaled up to CLIP-large, SeTAR+FT outperforms LoCoOp and LoRA by 17.92% and 12.45% FPR95 on the same benchmark`, which doesn't seem to match with the numbers reported in Table 2. Am I missing anything?\n\n3. All considered OOD datasets are far-OOD w.r.t. the ID set. Near-OOD detection has long been recognized as a more challenging and realistic problem [1, 2]. I would be very interested to see how well SeTAR performs on ImageNet v.s. NINCO or SSB [2].\n\nI'm willing to adjust my score if my comments are addressed or clarified.\n\n[1] Detecting semantic anomalies\n\n[2] OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Performance Gains\n\n1. **Limited Room for AUC Improvement:**\n   - The baseline AUC scores are above 90, leaving limited room for significant improvement. Despite this, our method still achieves AUC improvements, demonstrating its effectiveness even in a high-performance context.\n\n2. **Significant FPR Improvement:**\n   - For the False Positive Rate (FPR), our method shows notable improvements. These reductions in FPR are significant and highlight the practical benefits of our approach in reducing false positives, enhancing the reliability of OOD detection.\n   \n 3. **Swin-Base Performance:**\n    - The performance boost on Swin-Base is more pronounced because Swin is trained directly on ImageNet, lacking a text-encoder and thus requiring training solely on IN1K. This can lead to overfitting on ID data and poor recognition of OOD images, providing more room for improvement. Our method helps alleviate this issue, improving Swin's generalization to OOD samples.\n\n    - In contrast, CLIP models are pretrained on large image-text datasets, which provides robust representation capabilities for both ID and OOD images. Consequently, CLIP’s baseline OOD performance is already strong, leaving less room for further improvement compared to Swin-Base. Therefore, the pronounced performance boost seen with Swin-Base is due to its initial lower performance and higher potential for enhancement through our method.\n\n# 2. Numerical Clarifications\n\n1. **Clarification on Pascal VOC FPR95 Reductions:**\n   - Mean MCM FPR95 reduction: (37.24 - 32.46) / 37.24 = 12.84%\n   - Mean GL-MCM FPR95 reduction: (29.44 - 23.86) / 29.44 = 18.95%\n\n2. **Clarification on CLIP-large FPR95 Improvements:**\n   - Mean LoCoOp FPR95 improvement: (40.74 + 46.74 - 34.75 - 37.05) / (40.74 + 46.74) = 17.92%\n   - Mean LoRA FPR95 improvement: (38.62 + 43.39 - 34.75 - 37.05) / (38.62 + 43.39) = 12.45%\n\nThese clarifications demonstrate the calculations behind the reported improvements, providing transparency and accuracy in our results.\n\n# 3. Near-OOD Results\n\n- We appreciate the suggestion and have added results for the CLIP-base backbone, with ImageNet1K as the ID dataset and SSB-Hard as the OOD dataset. SeTAR and SeTAR+FT show superior performance compared to the baselines.\n\n    | CLIP-base    | Category          |   MCM Score FPR↓ |   MCM Score AUC↑ |   GL-MCM Score FPR↓ |   GL-MCM Score AUC↑ |\n    |:-------------|:------------------|-----------------:|-----------------:|--------------------:|--------------------:|\n    | Vanilla      | Training-Free     |            89.28 |            63.88 |              85.62  |              67.63  |\n    | SeTAR        | Training-Free     |          **88.29** |          **64.20** |            **84.03** |            **68.29** |\n    | LoCoOp       | Finetuning (3run) |            89.72 |            63.45 |              86.79  |              65.93  |\n    | LoRA         | Finetuning (3run) |            88.52 |            65.38 |            **84.39** |              68.85  |\n    | SeTAR+FT     | Finetuning (3run) |          **87.16** |          **68.13** |              84.72  |            **70.42** |\n\nThese results highlight SeTAR's and SeTAR+FT's robust performance in challenging near-OOD scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All three reviewers agree that this work is of interest and value to the NeurIPS community and that it should be accepted. The AC agrees.

Authors are requested to revise their manuscript based on the reviewer feedback, incorporating the various discussions and additional results from the author-reviewer exchanges.
Decision: Accept (poster)