SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Atsuyuki Miyai $^{1}$  Qing Yu $^{1,2}$  Go Irie $^{3}$  Kiyoharu Aizawa $^{1}$
$^{1}$ The University of Tokyo  $^{2}$ LY Corporation  $^{3}$ Tokyo University of Science
{miyai,yu,aizawa}@hal.t.u-tokyo.ac.jp goirie@ieee.org
We present a novel vision-language prompt learning approach for few-shot out-of-distribution (OOD) detection. Few-shot OOD detection aims to detect OOD images from classes that are unseen during training using only a few labeled in-distribution (ID) images. While prompt learning methods such as CoOp [60] have shown effectiveness and efficiency in few-shot ID classification, they still face limitations in OOD detection due to the potential presence of ID-irrelevant information in text embeddings. To address this issue, we introduce a new approach called Local regularized Context Optimization (LoCoOp), which performs OOD regularization that utilizes the portions of CLIP local features as OOD features during training. CLIP's local features have a lot of ID-irrelevant nuisances (e.g., backgrounds), and by learning to push them away from the ID class text embeddings, we can remove the nuisances in the ID class text embeddings and enhance the separation between ID and OOD. Experiments on the large-scale ImageNet OOD detection benchmarks demonstrate the superiority of our LoCoOp over zero-shot, fully supervised detection methods and prompt learning methods. Notably, even in a one-shot setting - just one label per class, LoCoOp outperforms existing zero-shot and fully supervised detection methods. The code is available via https://github.com/AtsuMiyai/LoCoOp.
Detecting out-of-distribution (OOD) samples is crucial for deploying machine learning models in real-world scenarios, where new classes of samples can naturally arise and require caution. Common approaches for OOD detection use single-modal supervised learning [15, 17, 19, 28, 45, 29, 26, 50, 1]. These supervised methods achieve good results but have limitations e.g., these methods require a lot of computational and annotation costs for training. These days, CLIP [37] achieves surprising performances for various downstream tasks, and the application of CLIP to OOD detection is beginning to attract increasing attention [11, 30, 33, 46].
Previous studies on CLIP-based OOD detection have focused on extreme settings. Some studies [12, 11, 30, 33] have explored zero-shot OOD detection methods that do not require any in-distribution (ID) training data, whereas other studies [46] have developed fully supervised methods that require the entire ID training data. These methods have comparable performances, but both approaches have their own limitations. On the one hand, the zero-shot methods [30, 33] do not require any training data, but they may encounter a domain gap with ID downstream data, which limits the performance of zero-shot methods. On the other hand, fully supervised methods [46] utilize the entire ID training data but may destroy the rich representations of CLIP by fine-tuning, which also limits the performance despite requiring enormous training costs. To overcome the limitations of fully supervised and zero-shot methods, it is essential to develop a few-shot OOD detection method that utilizes a few ID training images for OOD detection. By finding a balance between the zero-shot
Figure 1: Overview of the OOD regularization for our LoCoOp. Our approach first extracts ID-irreverent nuisances (OOD) from CLIP's local features. Then, it performs entropy maximization on a per-pixel basis for the extracted OOD features. This OOD regularization allows OOD nuisances and all ID text embeddings to be dissimilar, thus preventing the inclusion of nuisances in ID text embeddings and improving test-time detection performance.
and fine-tuned methods, few-shot OOD detection has the potential to provide a more efficient and effective solution for OOD detection.
In this study, we tackle CLIP-based few-shot OOD detection, where we utilize only a few ID images for training CLIP. The most common approach to train CLIP with a few ID training data is prompt learning [60, 61], which trains the prompt's context words while keeping the entire pre-trained parameters fixed. CoOp [60] is the representative work of prompt learning, but we observe that CoOp does not still work well for OOD detection. While CoOp is designed to bring the given ID image and its corresponding class text embedding closer together, it also has the unintended effect of bringing the text embeddings closer to the backgrounds or ID-irrelevant objects in the ID image. Consequently, the text embeddings may contain information that is not relevant to the ID image, resulting in incorrectly high confidence scores for OOD images.
To apply prompt learning to OOD detection, we propose a simple yet effective approach called Local regularized Context Optimization (LoCoOp). An overview of our LoCoOp approach is shown in Fig. 1. We observe that the local features of CLIP contain many ID-irrelevant nuisances, such as backgrounds. LoCoOp treats such ID-irrelevant nuisances as OOD and learns to push them away from the ID class text embeddings, which removes unnecessary information from the text embeddings of ID classes and prevents the model from producing high ID confidence scores for the OOD features. To achieve this, LoCoOp first identifies the ID-irrelevant regions using the segmentation classification prediction results [59] for each local region. Next, it performs entropy maximization of the classification probabilities for the extracted regions, which ensures the features in ID-irrelevant regions are dissimilar to any ID text embedding.
Our proposed method, LoCoOp, offers two significant advantages. Firstly, it can perform OOD regularization at a minimal cost. The standard approach to OOD regularization is OOD exposure [16, 54, 52], which utilizes external OOD samples for training. However, it can be quite cumbersome to collect a lot of OOD images. In addition, it can be computationally expensive as the number of OOD images and training time are proportional. On the other hand, LoCoOp uses only local features of CLIP as OOD. Therefore, even when 100 OOD features are used per ID image, the training time is only about  $1.4 \times$  longer and GPU memory consumption is only about  $1.1 \times$  higher than those of the baseline method without OOD regularization (i.e., CoOp). Secondly, LoCoOp is highly compatible with GL-MCM [33], a test-time OOD detection method that separates ID and OOD using both global and local features. Since LoCoOp learns local OOD features, it can accurately identify whether the given local region is ID or OOD. This enables LoCoOp to improve the OOD detection performance with GL-MCM, significantly outperforming other methods.
Through experiments on the large-scaled ImageNet OOD benchmarks [18], we observe that our LoCoOp significantly outperforms existing zero-shot, few-shot, and fully supervised OOD detection methods. Notably, even in a one-shot setting – just one label per class, LoCoOp outperforms existing
zero-shot and fully supervised detection methods, which is a remarkable finding. The contributions of our study are summarized as follows:
- We first tackle CLIP-based few-shot OOD detection, where we utilize only a few labeled ID images to train CLIP for OOD detection.
- We propose a novel prompt learning approach called LoCoOp, which leverages the portions of CLIP local features as OOD features for OOD regularization (see Fig. 1).
- LoCoOp brings substantial improvements over prior zero-shot, fully supervised, and prompt learning methods on the large-scale ImageNet OOD benchmarks. Notably, LoCoOp outperforms them with only one label per class (see Table 1).
For OOD detection with pre-trained models [12, 30, 24], the ID classes refer to the classes used in the downstream classification task, which are different from those of the upstream pre-training. Accordingly, OOD classes are those that do not belong to any of the ID classes of the downstream task. An OOD detector can be viewed as a binary classifier that identifies whether the image is an ID image or an OOD image. Formally, we assume that we have an ID dataset  $D^{\mathrm{in}}$  of  $(x^{\mathrm{in}}, y^{\mathrm{in}})$  pairs where  $x^{\mathrm{in}}$  denotes the input ID image, and  $y^{\mathrm{in}} \in Y^{\mathrm{in}} := 1, \dots, M$  denotes the ID class label. Let  $D^{\mathrm{out}}$  denote an OOD dataset of  $(x^{\mathrm{out}}, y^{\mathrm{out}})$  pairs where  $x^{\mathrm{out}}$  denotes the input OOD image, and  $y^{\mathrm{out}} \in Y^{\mathrm{out}} := M + 1, \dots, M + O$  denotes the OOD class label. Then, there are no overlaps of classes between ID and OOD. Formally,  $Y^{\mathrm{out}} \cap Y^{\mathrm{in}} = \emptyset$
We study the scenario where the model is fine-tuned only on the training set  $D_{\mathrm{train}}^{\mathrm{in}}$  without any access to OOD data. The test set contains  $D_{\mathrm{test}}^{\mathrm{in}}$  and  $D_{\mathrm{test}}^{\mathrm{out}}$  for evaluating OOD performance. Unlike existing studies using a large number of ID training samples [46] or no ID training samples [30], we use only a few samples (1, 2, 4, 8, and 16 samples per class) for  $D_{\mathrm{train}}^{\mathrm{in}}$ .
2.2 Reviews of CoOp
CoOp [60] is a pioneering method that utilizes vision-language pre-trained knowledge, such as CLIP [37], for downstream open-world visual recognition. While CLIP uses manually designed prompt templates, CoOp sets a portion of context words in the template as continuous learnable parameters that can be learned from few-shot data. Thus, the classification weights can be represented by the distance between the learned prompt and the visual feature.
Given an ID image  $\pmb{x}^{\mathrm{in}}$ , a global visual feature  $f^{\mathrm{in}} = f(\pmb{x}^{\mathrm{in}})$  is obtained by the visual encoder  $f$  of CLIP. Then, the textual prompt can be formulated as  $\pmb{t}_m = \{\omega_1, \omega_2, \dots, \omega_N, \pmb{c}_m\}$ , where  $\pmb{c}_m$  denotes the word embedding of the ID class name,  $\omega = \{\omega_n|_{n=1}^N\}$  are learnable vectors where each vector has the same dimension as the original word embedding and  $N$  denotes the length of context words. With prompt  $\pmb{t}_m$  as the input, the text encoder  $g$  outputs the textual feature as  $\pmb{g}_m = g(\pmb{t}_m)$ . The final prediction probability is computed by the matching score as follows:
$$
p (y = m \mid \boldsymbol {x} ^ {\text {i n}}) = \frac {\exp \left(\operatorname {s i m} \left(\boldsymbol {f} ^ {\text {i n}} , \boldsymbol {g} _ {m}\right) / \tau\right)}{\sum_ {m ^ {\prime} = 1} ^ {M} \exp \left(\operatorname {s i m} \left(\boldsymbol {f} ^ {\text {i n}}, \boldsymbol {g} _ {m ^ {\prime}}\right) / \tau\right)}, \tag {1}
$$
where  $\sin (\cdot ,\cdot)$  denotes cosine similarity, and  $\tau$  stands for the temperature of Softmax. The final loss  $\mathcal{L}_{\mathrm{coop}}$  is the cross-entropy loss with Eq. (1) and its ground truth label  $y^{\mathrm{in}}$ .
2.3 Proposed approach
In this section, we present our LoCoOp approach for few-shot OOD detection. An overview of our proposed method is shown in Fig. 1. Our methods consist of two components. The first is to extract ID-irrelevant regions from CLIP local features, and the second is to perform OOD regularization training with the extracted regions. It performs these two operations for each iteration.
In the following, we will first provide a brief overview of how to obtain local features from CLIP. Then, we will describe our two key components: the method to extract object-irrelevant regions from ID images and the OOD regularization loss used during training.
Preliminary. To obtain local features of CLIP, we project the visual feature  $\pmb{f}^{\prime}_{i}^{\mathrm{in}}$  of each region  $i$  from the feature map to the textual space [59, 42, 33] as follows:
$$
\boldsymbol {f} _ {i} ^ {\text {i n}} = \operatorname {P r o j} _ {v \rightarrow t} (v \left(\boldsymbol {f} _ {i} ^ {\prime} ^ {\text {i n}}\right)), \tag {2}
$$
where  $v$  denotes the value projection and  $\mathrm{Proj}_{v\to t}$  denotes the projection from the visual space into the textual space. These projections are inherent in CLIP and do not need to be trained. This local feature  $f_{i}^{\mathrm{in}}$  has a rich local visual and textual alignment [59].
Extraction of ID-irrelevant regions. We need to select the region indices of ID-irrelevant regions from a set of all region indices  $I = \{0,1,2,\dots,H\times W - 1\}$ , where  $H$  and  $W$  denote the height and width of the feature map. We handle this in a very simple and intuitive way. Similar to the segmentation task [59], we can obtain the classification prediction probabilities for each region  $i$  during training by computing the similarity between the image features  $f_{i}^{\mathrm{in}}$  of each region  $i$  and the text features of the ID classes. The formulation is as follows:
$$
p _ {i} (y = m \mid \boldsymbol {x} ^ {\text {i n}}) = \frac {\exp \left(\operatorname {s i m} \left(\boldsymbol {f} _ {i} ^ {\text {i n}} , \boldsymbol {g} _ {m}\right) / \tau\right)}{\sum_ {m ^ {\prime} = 1} ^ {M} \exp \left(\operatorname {s i m} \left(\boldsymbol {f} _ {i} ^ {\text {i n}} , \boldsymbol {g} _ {m ^ {\prime}}\right) / \tau\right)}. \tag {3}
$$
When a region  $i$  of  $\pmb{x}^{\mathrm{in}}$  corresponds to a part of an ID object, the ground truth class  $y^{\mathrm{in}}$  should be among the top-  $K$  predictions. In contrast, if a region  $i$  is ID-irrelevant, such as the background,  $y^{\mathrm{in}}$  should not appear among the top-  $K$  predictions because of the lack of semantic relationship between region  $i$  and the ground truth label  $y^{\mathrm{in}}$ . Based on this observation, we identify regions that do not include their ground truth class in the top-  $K$  predicted classes as ID-irrelevant regions  $J$ . The formulation is as follows:
$$
J = \{i \in I: \operatorname {r a n k} \left(p _ {i} \left(y = y ^ {\mathrm {i n}} \mid \boldsymbol {x} ^ {\mathrm {i n}}\right)\right) > K \}, \tag {4}
$$
where  $\mathrm{rank}(p_i(y = y^{\mathrm{in}}|\boldsymbol{x}^{\mathrm{in}}))$  denotes the rank of the GT class  $y^{\mathrm{in}}$  among all ID classes. Note that a set of regions  $J$  of an image  $\boldsymbol{x}^{\mathrm{in}}$  is updated during training since  $p_i(y = m\mid \boldsymbol{x}^{\mathrm{in}})$  is updated.
This method may appear to be a parameter-dependent method since it depends on the hyperparameter  $K$ . However, we argue that the optimal  $K$  is not difficult to search for. This is because, when applying OOD detection in the real world, it is assumed that the users understand what ID data is. For example, when ID is ImageNet-1K [5], users know the number of fine-grained classes or semantic relationships between classes in advance. Using prior knowledge, such as the number of fine-grained classes, helps determine the value of  $K$  (up to top-  $K$ , the prediction may be wrong).
OD regularization. With  $J$  (a set of region indices of ID-irrelevant regions), we perform OOD regularization, which removes unnecessary information from the text feature. The OOD features  $\pmb{f}_j^{\mathrm{in}}$  of each OOD region  $j\in J$  should be dissimilar to any ID text embedding. Therefore, we use entropy maximization for regularization, which is used for detecting unknown samples during training [40, 55]. Entropy maximization makes the entropy of  $p_j(y|\pmb{x}^{\mathrm{in}})$  larger and enables the OOD image features  $\pmb{f}_j^{\mathrm{in}}$  to be dissimilar to any ID text embedding. The loss function for this regularization is as follows:
$$
\mathcal {L} _ {\text {o o d}} = - H \left(p _ {j}\right), \tag {5}
$$
where  $H(\cdot)$  is the entropy function and  $p_j$  denotes the classification prediction probabilities for region  $j \in J$ .
Final Objective. The final objective is as follows:
$$
\mathcal {L} = \mathcal {L} _ {\text {c o o p}} + \lambda \mathcal {L} _ {\text {o o d}}, \tag {6}
$$
where  $\mathcal{L}_{\mathrm{coop}}$  is the loss for CoOp [60] and  $\lambda$  is a hyperparameter. Here,  $\mathcal{L}_{\mathrm{coop}}$  is applied for the entire image.
2.4 Test-time OOD detection
In testing, we use MCM score [30] and GL-MCM score [33]. With these functions, we classify test-time images into  $\pmb{x}^{\mathrm{in}}$  and  $\pmb{x}^{\mathrm{out}}$ . The details are as follows:
(a) FPR95
(b) AUROC
Figure 2: Few-shot OOD detection results with different numbers of ID labeled samples. We report average FPR95 and AUROC scores on four OOD datasets in Table 1. The lower value is better for FPR95, and the larger value is better for AUROC. We find that in all settings, our proposed LoCoOp with GL-MCM (red one) outperforms CoOp by a large margin.
MCM score. The MCM score [30] utilizes the softmax score of global image features and text features. Formally,  $S_{\mathrm{MCM}} = \max_m \frac{\exp(\sin(\pmb{f}, \pmb{g}_m) / \tau)}{\sum_{m' = 1}^M \exp(\sin(\pmb{f}, \pmb{g}_{m'}) / \tau)}$ , where  $\tau = 1$ . The rationale is that, for ID data, it will be matched to one of the prototype vectors with a high score and vice versa.
GL-MCM score. The GL-MCM score [33] utilizes the softmax score of both global and local image features and text features. The score function is  $S_{\mathrm{GL - MCM}} = S_{\mathrm{MCM}} + \max_{m,i}\frac{\exp(\sin(\pmb{f}_i,\pmb{g}_m) / \tau)}{\sum_{m' = 1}^{M}\exp(\sin(\pmb{f}_i,\pmb{g}_{m'}) / \tau)}$ . When OOD objects appear in the given ID image, the matching score with global features might be incorrectly low. Utilizing the matching score with local features can compensate for the low global matching score and produce correct ID confidence.
3.1 Experimental Detail
Datasets. We use the ImageNet-1K dataset [5] as the ID data. Therefore,  $M$  is set to 1,000. For OOD datasets, we adopt the same ones as in [18], including subsets of iNaturalist [47], SUN [51], Places [58], and TEXTURE [4]. For the few-shot training, we follow the few-shot evaluation protocol adopted in CLIP [37] and CoOp [60], using 1, 2, 4, 8, and 16 shots for training, respectively, and deploying models in the full test sets. The average results over three runs are reported for comparison.
Setup. Following existing studies [46, 30, 33], we use ViT-B/16 [6] as a backbone. Specifically, we use the publicly available CLIP-ViT-B/16 models (https://github.com/openai/CLIP). The resolution of CLIP's feature map is  $14 \times 14$  for CLIP-ViT-B/16. For  $K$ , we use 200 in all experiments. For  $\lambda$ , we use 0.25 in all experiments. The sensitivities of  $K$  and  $\lambda$  are described in the Analysis section 4 for  $K$  and in the supplementary material for  $\lambda$ . Other hyperparameters (e.g., training epoch=50, learning rate=0.002, batch size=32, and token lengths  $N = 16$ ) are the same as those of CoOp [60]. We use a single Nvidia A100 GPU for all experiments. Since LoCoOp takes approximately 1.4 times longer than CoOp, an additional experiment was conducted using 70 epochs for CoOp, but the results did not differ from those of 50 epochs. Therefore, we report all the results with 50 epochs.
Comparison Methods. To evaluate the effectiveness of our LoCoOp, we compare it with zero-shot detection methods, fully-supervised detection methods, and the baseline prompt learning method. For zero-shot OOD detection methods, we use MCM [30] and GL-MCM [33], which are state-of-the-art zero-shot detection methods. For fully-supervised detection methods, we follow the comparison methods [28, 50, 45, 46] in NPOS [46]. All the other baseline methods are fine-tuned using the same pre-trained model weights (i.e., CLIP-B/16). For NPOS, the MCM score function is used in the original paper. For a fair comparison, we reproduce NPOS and also report the results with the GL-MCM score. For prompt learning methods, we use CoOp [60], which is the representative work and the baseline for our LoCoOp. As for the comparison with CoCoOp [61], which is another representative work, we show the results in Analysis Section 4.
Table 1: Comparison results on ImageNet OOD benchmarks. We use ImageNet-1K as ID. We use CLIP-B/16 as a backbone. Bold values represent the highest performance.  $\dagger$  is cited from [46]. * is our reproduction. We find that LoCoOp with GL-MCM (LoCoOpGL) is the most effective method.
Method	iNaturalist	SUN	Places	Texture	Average
FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑
Zero-shot
MCM [30]*	30.94	94.61	37.67	92.56	44.76	89.76	57.91	86.10	42.82	90.76
GL-MCM [33]*	15.18	96.71	30.42	93.09	38.85	89.90	57.93	83.63	35.47	90.83
Fine-tuned
ODIN [28]†	30.22	94.65	54.04	87.17	55.06	85.54	51.67	87.85	47.75	88.80
ViM [50]†	32.19	93.16	54.01	87.19	60.67	83.75	53.94	87.18	50.20	87.82
KNN [45]†	29.17	94.52	35.62	92.67	39.61	91.02	64.35	85.67	42.19	90.97
NPOS_MCM [46]†	16.58	96.19	43.77	90.44	45.27	89.44	46.12	88.80	37.93	91.22
NPOS_MCM [46]*	19.59	95.68	48.26	89.70	49.82	88.77	51.12	87.58	42.20	90.43
NPOS_GL*	18.70	95.36	38.99	90.33	41.86	89.36	47.89	86.44	36.86	90.37
Prompt learning	one-shot (one label per class)
CoOpMCM	43.38	91.26	38.53	91.95	46.68	89.09	50.64	87.83	44.81	90.03
CoOpGL	21.30	95.27	31.66	92.16	40.44	89.31	52.93	84.25	36.58	90.25
LoCoOpMCM (ours)	38.49	92.49	33.27	93.67	39.23	91.07	49.25	89.13	40.17	91.53
LoCoOpGL (ours)	24.61	94.89	25.62	94.59	34.00	92.12	49.86	87.49	33.52	92.14
16-shot (16 labels per class)
CoOpMCM	28.00	94.43	36.95	92.29	43.03	89.74	39.33	91.24	36.83	91.93
CoOpGL	14.60	96.62	28.48	92.65	36.49	89.98	43.13	88.03	30.67	91.82
LoCoOpMCM (ours)	23.06	95.45	32.70	93.35	39.92	90.64	40.23	91.32	33.98	92.69
LoCoOpGL (ours)	16.05	96.86	23.44	95.07	32.87	91.98	42.28	90.19	28.66	93.52
(b) AUROC
Figure 3: Ablation studies on  $K$ . We report average FPR95 and AUROC scores on four OOD datasets in Table 1. The performance is not very sensitive to  $K$  values except for extreme values such as 0 and 1,000.
Evaluation Metrics. For evaluation, we use the following metrics: (1) the false positive rate of OOD images when the true positive rate of in-distribution images is at  $95\%$  (FPR95), (2) the area under the receiver operating characteristic curve (AUROC).
The OOD detection performances are summarized in Table 1. We show that LoCoOp with GL-MCM (LoCoOpGL) can achieve superior OOD detection performance, outperforming the competitive methods by a large margin. In particular, with a 16-shot setting, LoCoOp with GL-MCM reached an AUROC score of  $93.52\%$ , while the other methods failed to reach an AUROC score of  $92\%$ .
In addition, we can see, even in a one-shot setting, LoCoOp with GL-MCM achieved average FPR95 and AUROC scores of 33.52 and 92.14, respectively, which outperforms the zero-shot and fine-tuned methods. Considering that LoCoOp utilizes only one training sample per class while fine-tuned methods use 1,000 training images per class, these results are remarkable.
Table 2: Comparison results with CoCoOp [61]. We report average FPR and AUROC scores on four OOD datasets in Table 1.
Method	Infer time↓	Average
FPR95↓	AUROC↑
CoCoOpMCM [61]	149 ms	35.53	91.99
LoCoOpMCM	2.59 ms	33.98	92.69
LoCoOpGL	5.97 ms	28.66	93.52
Table 3: Comparison in ID accuracy on ImageNet-1K validation data.
Method	Top-1 Accuracy
CoOp	72.1
LoCoOp	71.7
In Fig. 2, we show the results with different numbers of ID labeled samples for CoOp and LoCoOp, where the green line denotes our LoCoOp with MCM, the red one denotes our LoCoOp with GL-MCM, the blue one is CoOp with MCM, and the orange line denotes CoOp with GL-MCM. The detailed accuracy can be found in the supplementary materials. We observed that GL-MCM boosts the performance of LoCoOp much more than CoOp in all settings.
3.3 Ablation studies on hyperparameter K