SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conduct ablation studies on  $K$  in Eq. (4).  $K$  is an important parameter to pick up OOD regions from the local features. We experimented with a 16-shot setup using  $K = 0$  (all the local regions are used for OOD), 100, 200, 300, 400, 500, 900, 980, and 1,000 (same as CoOp). In Fig. 3, we summarize the results of ablation studies. First, let us look at the leftmost point of the graph,  $K = 0$ , i.e., the result when all the local regions are treated as OOD. We can see that  $K = 0$  degrades the detection performance in all settings. This is because it treats all object-relevant regions as OOD. Then, going around  $K = 100$ , we can see that the performance improves dramatically because the number of such false negative OOD features is decreasing. LoCoOp maintains high performance after  $K = 200$ . In particular, even at  $K = 980$ , a large performance gap exists with  $K = 1,000$  (CoOp). At  $K = 980$ , the number of OOD regions per image is about one, indicating that LoCoOp is effective even when only a few local OOD features are used for training. In general, the value of  $K$  is not sensitive to the performance and is not difficult to select as long as it is not set to extreme values such as 0 or 1,000.
Comparison with CoCoOp. We compare our LoCoOp with CoCoOp [61]. CoCoOp is also representative work for prompt learning, which generates an input-conditional text token for each image. For comparison, we add per-image inference time (in milliseconds, averaged across test images) for evaluation metrics [45]. This is because OOD detection methods are applied before test-time input comes to the close-set classifier, so the short inference time is important. We use a single Nvidia A100 GPU for the measurements. We show the 16-shot results of CoCoOp and LoCoOp in Table 2. This result demonstrates that our LoCoOp outperforms CoCoOp in performance and inference speed. Especially, CoCoOp takes a much longer inference time than our LoCoOp. That is because it requires an independent forward pass of instance-specific prompts for each image through the text encoder. When inferring ImageNet OOD benchmarks (total number of images is 85,640), CoCoOp takes 3.55 hours, while LoCoOp with MCM takes only 3.7 minutes.
ID accuracy of LoCoOp. We examine the ID accuracy of LoCoOp. OOD detection is a preprocessing before input data comes into the ID classifier, and its purpose is to perform binary classification of ID or OOD [15]. However, it is also important to investigate the relationship between close-set ID accuracy and OOD detection performance [49]. In Table 3, we show the 16-shot ID accuracy of CoOp and LoCoOp on the ImageNet-1K validation set. This result shows that LoCoOp is slightly inferior to CoOp in ID accuracy. We hypothesize that excluding OOD nuisances that are correlated with ID objects will degrade the ID accuracy. For example, in some images of dogs, the presence of green grass in the background may help identify the image as a dog. Therefore, learning to remove the background information could make it difficult to rely on such background information to determine that the image is a dog. However, this study demonstrates that excluding such backgrounds improves
Table 4: Comparison results with CLIP-ResNet-50. We find that LoCoOp with GL-MCM (LoCoOpGL) is effective with ResNet-50.
Method	iNaturalist	SUN	Places	Texture	Average
FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑
MCM [30]	31.98	93.86	46.09	90.75	60.56	85.67	60.00	85.72	49.66	89.00
GL-MCM [33]	37.20	90.98	38.74	91.21	49.49	86.40	43.46	88.14	42.22	89.18
CoOpGL	33.68	92.59	40.0	90.77	48.14	86.8	31.27	91.69	38.27	90.46
LoCoOpGL (ours)	22.81	95.38	36.44	92.22	44.10	88.3	34.47	91.89	34.45	91.95
Table 5: Comparison with other methods for ID-irrelevant regions extraction.
Method	iNaturalist	SUN	Places	Texture	Average
FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑
Ent.	25.05	94.28	31.92	92.97	39.85	89.94	48.71	87.52	36.38	91.18
Prob.	16.19	96.55	23.66	94.76	32.62	91.83	42.12	89.96	28.65	93.27
Rank (ours)	16.05	96.86	23.44	95.07	32.87	91.98	42.28	90.19	28.66	93.52
the OOD detection performance. This study exhibits that a strong ID classifier is not always a robust OOD detector for prompt learning.
Effectiveness with CNN architectures. We examine the effectiveness of LoCoOp with CNN architectures. In the main experiments, we use a Vision Transformer architecture as a backbone following existing work [46, 30]. However, for the fields of OOD detection, CNN-based models also have been widely explored [17, 43]. Following this practice, we use ResNet-50 as a CNN backbone and examine the effectiveness of LoCoOp. Table 4 shows the 16-shot results with ResNet-50. We find that similar to the results with ViT, our LoCoOp is effective for ResNet-50.
Comparison with other methods for ID-irrelevant region extraction. We explore the effectiveness of several methods for ID-irrelevant region extraction. Although we use a Top- $K$  rank-based thresholding approach, two other approaches, entropy-based thresholding and probability-based thresholding, approaches are also candidates for the methods. Entropy-based thresholding methods use the threshold of the entropy of  $p_i$ , whereas probability-based thresholding methods use the threshold of the value of  $p_i(y = y^{\mathrm{in}}|\boldsymbol{x})$ . For entropy-based thresholding methods, we extract regions where the entropy of  $p_i$  is less than  $\frac{\log M}{2}$  because  $\frac{\log M}{2}$  is the half value of the entropy of  $M$ -dimensional probabilities, which is following existing studies [40, 55]. For probability-based thresholding methods, we extract regions where  $p_i(y = y^{\mathrm{in}}|\boldsymbol{x}^{\mathrm{in}})$  is less than  $1 / M$  simply because the ID dataset has  $M$  classes. We summarize the 16-shot OOD detection results in Table 5. For the entropy-based method, the performance is poor since it is challenging to determine the appropriate threshold. On the other hand, for the probability-based method, the performance is comparable to our rank-based method since we can determine the threshold intuitively. However, we consider that, from the user's perspective, the rank is easier to interpret and handle than the probability value. Therefore, we use rank-based thresholding in this paper.
Visualization of extracted OOD regions. In Fig.4, we show visualization results of extracted OOD regions. The performance of OOD extraction is key to our method. From this visualization result, we find that our Rank-based approach can accurately identify OOD regions.
5 Theoretical background
The effectiveness of separating foregrounds (ID objects) and backgrounds is theoretically guaranteed by existing research [31, 39]. Ming et al.[31] states that when the classifier only uses foreground features, the optimal decision boundary can be obtained. With conventional classifiers, it was difficult for the classifiers to use only foreground features. Contrary to this, CLIP has rich local visual-text alignments and can identify the foregrounds and backgrounds. Based on the theoretical analysis in [31], LoCoOp achieves high OOD detection performance.
ID: tench (fish)
ID: scarf
ID: French horn
ID: Appenzeller Sennenhund (dog)
Figure 4: Visualization of extracted OOD regions. We find that our approach can correctly extract ID-irrelevant regions.
ID: military aircraft
ID: guinea pig
Application to other light-weight tuning methods. In this study, we focus on text prompt learning methods [60, 61] because they are the most representative methods for tuning CLIP. These days, other light-weight tuning methods such as Tip-Adapter, which learns a lightweight residual feature adapter [57] and visual prompt methods [21, 2], which learn a visual prompt added to the input image [2] or inside the layer [21], have been proposed for image classifications. It is our future work to apply our LoCoOp to these methods [57, 21, 2].
Application to the model without the rich local visual-text alignment. This approach requires models with strong local visual-text alignment capabilities, such as the image encoder in CLIP. Consequently, applying this approach to models lacking such capabilities can prove challenging. Nonetheless, given the numerous methods currently emerging based on CLIP, our research is influential in this research field.
Extending to other visual recognition tasks. In this study, we focus on only classification tasks. However, prompt learning is widely used for other tasks, such as object detection [10] and segmentation [10]. We will solve other tasks as important future work. However, existing studies about OOD detection do not deal with both classification and other tasks simultaneously [8, 26, 7], and most studies in OOD detection address classification tasks [53]. Therefore, we believe that our findings in the classification task are influential.
Out-of-distribution detection. Various approaches have been developed to address OOD detection in computer vision [14, 23, 34-36, 15, 17, 19, 28, 45, 29, 26, 50, 53, 56, 32]. One of the common OOD detection approaches is to devise score functions such as probability-based method [15, 17, 19, 28, 44, 45], logit-based method [29, 17, 44], and feature-based method [26, 50]. Another approach is to leverage OOD data for training-time regularization [16, 54, 9, 52]. These OOD detection methods assume the use of backbones trained with single-modal data. In recent years, much attention has been paid to how multimodal pre-trained models, such as CLIP, can be used for out-of-distribution detection. Existing CLIP-based OOD detection methods have explored the zero-shot methods [12, 11, 30, 33] and fine-tuned method [46]. However, these methods have their own limitations regarding performance and training efficiency. To bridge the gap between fully supervised and zero-shot methods, developing a few-shot OOD detection method that utilizes a few ID training images for OOD detection is essential. Before CLIP emerged, Jeong et al. [20] addressed the few-shot OOD detection, using only a few ID training data for OOD detection, but their experiments have only been done on small data sets (e.g., Omniglot [25], CIFAR-FS [3]), showing the limitations of the performances of few-shot learning methods without CLIP. By leveraging the powerful representations of CLIP, few-shot OOD detection could provide a more efficient and effective solution.
Prompting for foundation models. Prompting is a heuristic way to directly apply foundation models to downstream tasks in a zero-shot manner. Prompt works as a part of the text input that instructs the model to perform accordingly on a specific task. However, such zero-shot generalization relies heavily on a well-designed prompt. Prompt tuning [27] proposes to learn the prompt from downstream data in the continual input embedding space, which is a parameter-efficient way of fine-tuning foundation models. Although prompt tuning was initially developed for language models, it has been applied to other domains, including vision-language models [60, 61, 42, 38, 13]. CoOp [60], which applies prompt tuning to CLIP, is a representative work that effectively improves CLIP's performance on corresponding downstream tasks. However, existing prompt learning methods focus on improving only close-set ID classification accuracy, and their performance for OOD detection remains unclear. In this work, we propose a novel method for prompt learning for OOD detection.
Local features of CLIP. Recent work [59] revealed that CLIP has local visual features aligned with textual concepts. For classification tasks, global features are used, which are created by pooling the feature map with a multi-headed self-attention layer [48]. However, these features are known to lose spatial information [41], so they are not suitable for tasks that require spatial information, such as semantic segmentation. To adopt CLIP to segmentation, Zhou et al. [59] proposed the CLIP's local visual features aligned with textual concepts, which can be obtained by projecting the value features of the last attention layer into the textual space. Existing studies used CLIP's local features for semantic segmentation [59, 62, 22], multi-label classification [42] and ID detection [33], which need to detect local objects in an image. Unlike previous studies, we use local features to pick up ID-irrelevant parts (e.g., backgrounds), which can be regarded as OOD features for OOD detection.
We present a novel vision-language prompt learning approach for few-shot OOD detection. To solve this problem, we present LoCoOp, which performs OOD regularization during training using the portions of CLIP local features as OOD images. CLIP's local features have a lot of ID-irrelevant information, and by learning to push them away from the ID class text embeddings, we can remove the nuisances in the ID class text embeddings, leading to a better separation of ID and OOD. Experiments on the large-scale ImageNet OOD detection benchmark demonstrate the advantages of our LoCoOp over zero-shot, fully supervised detection methods and existing prompt learning methods.
Broader Impact. This work proposed a prompt learning approach for few-shot OOD detection, improving the performance and efficiency of existing detection methods. Our work reduces the data-gathering effort for data-intensive applications, so it makes the technology more accessible to individuals and institutions that lack abundant resources. However, it is important to exercise caution in sensitive applications as our approach may make these systems vulnerable to misuse by individuals or organizations seeking to engage in fraudulent or criminal activities. Therefore, human oversight is necessary to ensure that our approach is used ethically and not relied upon solely for making critical decisions.
This work was partially supported by JST JPMJCR22U4, CSTI-SIP, and JSPS KAKENHI 21H03460.
[1] Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. In EMLP, 2021.
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022.
[3] Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In ICLR, 2019.
[4] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[26] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, 2018.
[27] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
[28] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
[29] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS, 2020.
[30] Yifei Ming, Ziyang Cai, Jiuming Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. In *NeurIPS*, 2022.
[31] Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of-distribution detection. In AAAI, 2022.
[32] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Can pre-trained networks detect familiar out-of-distribution data? arXiv preprint arXiv:2310.00847, 2023.
[33] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Zero-shot in-distribution detection in multi-object settings using vision-language foundation models. arXiv preprint arXiv:2304.04521, 2023.
[34] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In ICLR, 2019.
[35] Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV, 2018.
[36] Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In CVPR, 2019.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[38] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
[39] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In NeurIPS, 2019.
[40] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In NeurlPS, 2020.
[41] Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In ACL, 2022.
[42] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. In NeurIPS, 2022.
[43] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV, 2022.
[44] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurIPS, 2021.
[45] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML, 2022.
[46] Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR, 2023.
[47] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[49] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In ICLR, 2022.
[50] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In CVPR, 2022.
[51] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[52] Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In ICCV, 2021.
[53] Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. In NeurIPS Datasets and Benchmarks Track, 2022.
[54] Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximum classifier discrepancy. In ICCV, 2019.
[55] Qing Yu, Atsushi Hashimoto, and Yoshitaka Ushiku. Divergence optimization for noisy universal domain adaptation. In CVPR, 2021.
[56] Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.09301, 2023.
[57] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. In ECCV, 2022.
[58] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 40(6):1452-1464, 2017.
[59] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022.
[60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[61] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
[62] Ziqin Zhou, Bowen Zhang, Yinjie Lei, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.