SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kai Liu $^{1,2*}$ , Zhihang Fu $^{2\dagger}$ , Chao Chen $^{2}$ , Sheng Jin $^{2}$ , Ze Chen $^{2}$ , Mingyuan Tao $^{2}$ , Rongxin Jiang $^{1\dagger}$ , Jieping Ye $^{2}$
<sup>1</sup>Zhejiang University, <sup>2</sup>Alibaba Cloud
The key to OOD detection has two aspects: generalized feature representation and precise category description. Recently, vision-language models such as CLIP provide significant advances in both two issues, but constructing precise category descriptions is still in its infancy due to the absence of unseen categories. This work introduces two hierarchical contexts, namely perceptual context and spurious context, to carefully describe the precise category boundary through automatic prompt tuning. Specifically, perceptual contexts perceive the inter-category difference (e.g., cats vs apples) for current classification tasks, while spurious contexts further identify spurious (similar but exactly not) OOD samples for every single category (e.g., cats vs panthers, apples vs peaches). The two contexts hierarchically construct the precise description for a certain category, which is, first roughly classifying a sample to the predicted category and then delicately identifying whether it is truly an ID sample or actually OOD. Moreover, the precise descriptions for those categories within the vision-language framework present a novel application: CATegory-EXtensible OOD detection (CATEX). One can efficiently extend the set of recognizable categories by simply merging the hierarchical contexts learned under different sub-task settings. And extensive experiments are conducted to demonstrate CATEX's effectiveness, robustness, and category-extensibility. For instance, CATEX consistently surpasses the rivals by a large margin with several protocols on the challenging ImageNet-1K dataset. In addition, we offer new insights on how to efficiently scale up the prompt engineering in vision-language models to recognize thousands of object categories, as well as how to incorporate large language models (like GPT-3) to boost zero-shot applications.
Out-of-distribution (OOD) detection focuses on determining whether an input image is in-distribution (ID) or OOD, while classifying the ID data into respective categories [17]. The key to OOD detection has two aspects: (1) constructing a sufficiently generalized feature representation capability, so that images of arbitrarily different categories can be roughly separated from each other regardless of semantic shifts [67]; and (2) acquiring precise descriptions (namely decision boundary) for each ID category in the image feature space (in CNNs the last fully-connected layer plays this role), so as to determine whether the input image belongs to the corresponding category or just OOD [22].
Specifically, generalized feature representation comes from large-scale and diverse training data for learning-based data-driven models. Therefore, OOD detection works[33, 13, 50] have made great progress on the basis of large-scale pre-trained models such as ResNet [19] and ViT [14]. To achieve
*Work done during Kai Liu's research internship at Alibaba Cloud. Email: kail@zju.edu.cn.
† Corresponding authors. Email: rongxinj@zju.edu.cn, zhihang.fzh@alibaba-inc.com.
(a) Previous approaches
(b) Our method
correctly-identified OOD samples mis-identified OOD samples
Figure 1: Method comparison. Compared to previous approaches, our method utilizes the perceptual context to classify different categories under the current ID task (solid lines), and leverages the spurious context to strictly define the category boundaries independent of the current setting (dashed lines). The hierarchical perceptual and spurious contexts jointly describe the precise and universal boundaries for each category (combination of solid and dashed lines).
the precise description for a certain ID category, on the other hand, we need to let the model acquire as much "prior knowledge" as possible. For instance, if the description comes from a binary classifier that has only seen cat and apple during training, the description of category cat is obviously not precise enough to distinguish a panther image as an OOD: it lacks prior knowledge to distinguish different quadruped mammals.
Recently, vision-language models such as CLIP [41] provided significant advances in the two key issues above for OOD detection. Huge corpora of paired image-text data training brings both powerful feature representation capabilities and more comprehensive prior knowledge than single-vision-modal training [71, 37]. Therefore, CLIP-based OOD detection methods have been proposed successively, such as the zero-shot method MCM [38] and the encoder fine-tuning method NPOS [50]. However, there still exist the issues of generalization of feature representations and precision of category descriptions under the multi-modal framework.
Our observations can be summarized into two aspects: 1. The performance of the zero-shot CLIP-based OOD methods [38] is limited. As CLIP is trained by contrastive learning with informative image-text caption pairs, simply using the category names as all text information constrains the potential image-text discriminability. 2. Even though fine-tuning CLIP's encoder may boost the performance [18, 50], the generalization of multi-modal feature representation is sacrificed. In other words, such methods impair the model's ability to resist data shifts. Although the existing works are promising and inspiring to academia, constructing precise category descriptions via CLIP-like multi-modal features for OOD detection is still in its infancy [18].
Based on the observations above, we take a step towards generalized OOD detection by seeking the precise description prompt for each category and meanwhile maintaining the representation capacity. To this end, we develop a pair of perceptual context and spurious context for each in-distribution category to hierarchically construct the precise category description. CLIP encoders are kept frozen, and the contexts are learned through automatic prompt tuning.
As illustrated in Fig. 1, perceptual contexts first roughly classify a sample to the predicted category (colored regions), and spurious contexts then delicately identify whether it is truly an ID sample or actually spurious OOD (gray regions). Specifically, perceptual contexts perceive the descriptions of inter-category differences across all ID categories, while spurious contexts are relatively independent of the specific ID data task. The latter models a more strictly defined description of the current category itself by training on both real ID data and well-designed adversarial samples. We also introduce a robust sample synthesis strategy that leverages the prior knowledge of the VLM to select the qualified syntheses inspired by [50, 60].
We conduct extensive experiments to show the proposed hierarchical context descriptions are crucial to precisely and universally define each category. As a result, our method consistently outperforms the competitors on the large-scale OOD datasets, while showing comparable or even better generalization than the remarkable zero-shot methods. With the vision-language prompting framework, the precise and universal category descriptions via hierarchical contexts present a novel application: CATegory-EXtensible OOD detection (CATEX). We then merge the context descriptions learned from different
task data, and directly test on the union ID setting. The competitive results demonstrate that the learned descriptions can be used across tasks. Consequently, to illustrate the category-extensibility, we incrementally extend the context descriptions to the whole ImageNet-21K categories[11] at an acceptable GPU memory cost, and achieve superior performance to rivals. In addition, CATEX offers new insights on how to incorporate large language models (like ChatGPT) to boost zero-shot classifications (e.g., implicitly constructing spurious contexts to perform one-class OOD detection).
1. We construct the perceptual and spurious contexts to hierarchically describe each category to perform ID/OOD detection. Our method consistently surpasses the SOTA method on the challenging ImageNet-1K benchmark, leading to an  $8.27\%$  decrease in FPR95.
2. We empirically demonstrate the proposed hierarchical contexts are prone to learning precise category descriptions while keeping the generalized feature representation. When a data shift occurs, our method shows stable robustness on both ID classification and OOD tasks.
3. We present a novel category-extensible (CATEX) OOD detection framework. By simply merging the successively learned hierarchical contexts, we show that precise descriptions enable cross-task ID classification and OOD detection. We hope to offer new insights on how to scale up the prompt engineering in VLMs to recognize thousands of categories, as well as how to incorporate LLMs to boost zero-shot applications.
This paper considers the multi-class classification as the in-distribution (ID) situation, and the OOD detection task is formalized as follows.
Notations. Let  $\mathcal{X} \in \mathbb{R}^{H \times W \times 3}$  denote the input image space and  $\mathcal{Y}^{in} = \{1,2,\dots,C\}$  denote the ID label space of  $C$  categories in total. The labeled training set  $\mathcal{D}_{tr}^{in} = \{(I_i,y_i)\}_{i=1}^n$  is drawn i.i.d. from the joint data distribution  $\mathcal{P}_{\mathcal{X}\mathcal{Y}^{in}}$ . Let  $\mathcal{P}_{\mathcal{X}}$  denote the marginal distribution on  $\mathcal{X}$ , which is referred to as the in-distribution (ID). For ID classification task, let  $f: \mathcal{X} \mapsto \mathbb{R}^C$  denote a function predicting the label of an input sample by  $\hat{y}_i = \operatorname{argmax}_k f[k](I_i)$ .
In-distribution classification. In vision-language models, let  $\mathbf{x} \in \mathbb{R}^d$  denote the  $l_2$ -normalized  $d$ -dimension visual feature of input images via the image encoder:  $\mathbf{x} = \mathcal{I}(I)$ , the mapping function  $f$  can be expressed as  $f(I) = W^{T}\mathcal{I}(I) = W^{T}\mathbf{x}$ , where  $W = [\mathbf{w}_1; \mathbf{w}_2; \dots; \mathbf{w}_C]^T \in \mathbb{R}^{d \times C}$  is the collection of  $l_2$ -normalized text features for  $C$  categories. Specifically, each text feature is extracted from the category description via text encoder:  $\mathbf{w}_k = \mathcal{T}([v_{k,1}; v_{k,2}; \dots; v_{k,m}; \mathrm{CLS}_k]) \triangleq \mathcal{T}([\mathbf{v}_k; \mathrm{CLS}_k])$ , where  $\mathbf{v}_k$  presents the  $m$  word embedding vectors for  $k$ -th ID category and  $\mathrm{CLS}_k$  is the given category name. The ID-label prediction becomes  $\hat{y} = \arg\max_k \mathbf{w}_k^T \mathbf{x} = \arg\max_k \langle \mathbf{w}_k, \mathbf{x} \rangle$ .
Out-of-distribution detection. When an OOD sample  $\mathbf{z}$  with unknown category  $y \notin \mathcal{Y}^{in}$  emerges at test time, it should not be predicted to any known category. Hence, the OOD detection task can be viewed as a binary classification problem. Let  $g: \mathcal{X} \mapsto \mathbb{R}^1$  denote a function distinguishing between the in- v.s. out-of-distribution samples. OOD detection estimates the lower level set  $\mathcal{G} := \{\mathbf{z}: g(\mathbf{z}) \leq \lambda\}$ , where samples with lower scores  $g(\mathbf{z})$  are detected as OOD and vice versa. The threshold  $\lambda$  is typically chosen by ensuring a high fraction of ID data (e.g., 95%) is correctly preserved.
From the data-distribution perspective, the key to OOD detection is to acquire precise descriptions for each ID category to determine the distribution boundary. Inputs beyond the specific boundary should belong to either OOD samples or other ID categories. However, it has always been a challenge [22, 17, 38]. In conventional methods using vision or multi-modal models, one may fine-tune the image encoder to reshape the feature representation space. Then, the classification layer (for vision models) or text features (for VLMs) play the role in category descriptions, where calculated logits measure the relative distances between inputs and corresponding ID categories. However, due to the absence of unseen categories, both the over-fitted feature representations [28] and the ID-label-biased category descriptions [4] are prone to overconfident predictions on unseen OOD data [40].
Figure 2: Illustration of our method. Perceptual context perceives a certain ID category, and spurious context explicitly describes a spurious category around this ID category. Random perturbation is applied to the perceptual context for synthesizing outliers to train the non-trivial spurious context. The hierarchical perceptual and spurious contexts jointly describe the precise category boundary.
To alleviate these problems, we take the vision-language prompting framework [71] to learn precise and universal descriptions for each ID category. First, both the image and text encoders are frozen to preserve the generalized representation capacity. Second, we propose to hierarchically construct the perceptual and spurious context for each category to mitigate label bias.
In the following sections, we present the proposed CATEX by elaborating on the following questions: 1. How do perceptual and spurious contexts model the precise and universal category descriptions (Sec. 3.1)? 2. How to learn such hierarchical contexts (Sec. 3.2)? 3. How to leverage the precise category descriptions for ID classification and OOD detection (Sec. 3.3)? The overview framework of our method is illustrated in Fig. 2.
3.1 Hierarchical Contexts
Recall that empirical risk minimization (ERM) [54] operates under the closed-world assumption. When an OOD sample  $\mathbf{z} \notin \mathcal{P}_{\mathcal{X}}$  emerges, the ID classifier might produce overconfident predictions as  $\hat{y} = \arg\max_{k} \langle \mathbf{w}_k^p, \mathbf{z} \rangle$ , where  $\mathbf{w}_k^p = \mathcal{T}([\mathbf{v}_k^p; \mathrm{CLS}])$  is the encoded text feature for  $k$ -th ID category. Here  $\mathbf{v}_k^p$  is denoted as perceptual context that perceives the original multi-category classifications. However, with  $\mathbf{v}_k^p$  only, whether  $\mathbf{z}$  actually belongs to the  $k$ -th ID category or is just an unseen OOD semantically close to this category is still unclear. Hence, we propose to add a hierarchical context to model such OOD samples as another spurious category. The image-text similarity is measured by  $\langle \mathbf{w}_k^s, \mathbf{z} \rangle$ , where  $\mathbf{w}_k^s = \mathcal{T}([\mathbf{v}_k^s; \mathrm{CLS}])$ . Here  $\mathbf{v}_k^s$  is denoted as spurious context that identifies such semantically similar but spurious OOD samples. We propose to utilize such two types of hierarchical contexts to jointly model the precise and universal boundaries for each category.
First, perceptual context serves to carefully adapt to the current ID classification task. As we propose to model a potentially spurious category for  $C$  ID categories respectively, the total category number is equivalent to grows to  $C + C = 2C$  in the scope. When an ID sample  $(\mathbf{x}_i, y_i)$  emerges, to learn a more strict classification boundary, we put it apart from both other ID categories and their corresponding spurious categories by:
$$
\underset {\mathbf {w} ^ {p}, \mathbf {w} ^ {s}} {\operatorname {a r g m i n}} \quad \frac {1}{n} \sum_ {i} \left[ - \log \frac {e ^ {\langle \mathbf {w} _ {y _ {i}} ^ {p} , \mathbf {x} _ {i} \rangle}}{e ^ {\langle \mathbf {w} _ {y _ {i}} ^ {p} , \mathbf {x} _ {i} \rangle} + \sum_ {k \neq y _ {i}} ^ {C} e ^ {\langle \mathbf {w} _ {k} ^ {p} , \mathbf {x} _ {i} \rangle} + \sum_ {k \neq y _ {i}} ^ {C} e ^ {\langle \mathbf {w} _ {k} ^ {s} , \mathbf {x} _ {i} \rangle}} \right] \tag {1}
$$
Then, spurious context is hierarchically combined to describe the specific category boundary to handle the OOD detection. To formulate the open-world situation, when an ID sample  $\mathbf{x}$  or OOD sample  $\mathbf{z}$  with the predicted category  $\hat{y}$  emerges, the OOD detection risk can be expressed as:
$$
R = \mathbb {E} _ {\mathbf {x} \sim \mathcal {P} _ {\mathcal {X}}} \left[ \mathbb {1} \left\{\left\langle \mathbf {w} _ {\hat {y}} ^ {p} \mathbf {x} \right\rangle <   \left\langle \mathbf {w} _ {\hat {y}} ^ {s}, \mathbf {x} \right\rangle \right\} \right] + \mathbb {E} _ {\mathbf {z} \sim \mathcal {P} _ {\mathcal {X}}} \left[ \mathbb {1} \left\{\left\langle \mathbf {w} _ {\hat {y}} ^ {p}, \mathbf {z} \right\rangle > \left\langle \mathbf {w} _ {\hat {y}} ^ {s}, \mathbf {z} \right\rangle \right\} \right] \tag {2}
$$
For each ID category, if we can proactively draw the spurious OOD samples and their category-predictions as  $\mathcal{D}_{tr}^{s} = \{(\tilde{\mathbf{z}}_{i},\hat{y}_{i})\}_{i = 1}^{n}$ , the risk in Eq. (2) can be empirically minimized by:
$$
\underset {\mathbf {w} ^ {p}, \mathbf {w} ^ {s}} {\operatorname {a r g m i n}} \quad \frac {1}{n} \sum_ {i} \left[ - \log \frac {e ^ {\langle \mathbf {w} _ {y _ {i}} ^ {p} , \mathbf {x} _ {i} \rangle}}{e ^ {\langle \mathbf {w} _ {y _ {i}} ^ {p} , \mathbf {x} _ {i} \rangle} + e ^ {\langle \mathbf {w} _ {y _ {i}} ^ {s} , \mathbf {x} _ {i} \rangle}} \right] + \frac {1}{\tilde {n}} \sum_ {i} \left[ - \log \frac {e ^ {\langle \mathbf {w} _ {\tilde {y} _ {i}} ^ {s} , \tilde {\mathbf {z}} _ {i} \rangle}}{e ^ {\langle \mathbf {w} _ {\tilde {y} _ {i}} ^ {p} , \tilde {\mathbf {z}} _ {i} \rangle} + e ^ {\langle \mathbf {w} _ {\tilde {y} _ {i}} ^ {s} , \tilde {\mathbf {z}} _ {i} \rangle}} \right] \tag {3}
$$
In short, based on Eq. (1), perceptual context first learns to model a more strict inter-class decision boundary. Hierarchically, combining with Eq. (3), perceptual context and spurious context jointly model the precise category boundary, which is the key to OOD detection. More details on the whole training procedure are provided in Appendix A.1.
Intuitively, how to draw the spurious training set  $\mathcal{D}_{tr}^{s}$  is crucial. This paper presents a well-designed sampling strategy to learn the hierarchical contexts.
3.2 Perturbation Guidance
Generating adversarial data samples has been widely studied in recent years [39, 43, 72], among which feature-space sampling is proven more tractable [15, 50]. In general, previous approaches randomly synthesize spurious samples away from ID features' clustering centers (with low likelihood or large distance), and treat them as visual outliers. To improve the quality of spurious syntheses, we provide a perturbation-guided sampling strategy that leverages the prior knowledge of the pre-trained large-scale vision-language models.
In practice, an unseen spurious sample generally shares part of visual characteristics with the ID images, and the rest remains different [18]. Under the vision-language prompting framework, in the learned perceptual context of a certain ID category  $\mathbf{v}_k^p = [v_{k,1}^p; v_{k,2}^p; \dots; v_{k,m}^p])$ , each word embedding  $v_{k,j}^p$  actually describes a visual characteristic for this category [71]. According to the contrastive learning objective during the pre-training process on image-text pairs, if the text is perturbed, it should not be paired with the original image. Correspondingly, if the learned word embeddings are perturbed, the encoded text feature should also not describe images from the original ID category. Thus, as shown in Fig. 3, we can proactively leverage the perturbation to guide the spurious syntheses to boost context learning. Specifically, after randomly applying a pertur
Figure 3:Guiding process.
bation  $u$  (e.g., masking [29]) onto the word  $v_{k,j}^{p}$  of perceptual context, the perturbed text feature becomes  $\mathring{\mathbf{w}}_k^p = \mathcal{T}([v_{k,1}^p;\dots ;u;\dots ;v_{k,m}^p;\mathrm{CLS}_k])$ . Rather than the original ID category,  $\mathring{\mathbf{w}}_k^p$  now describes a kind of unknown spurious category, of which the images should share higher similarities with perturbed text feature  $\mathring{\mathbf{w}}_k^p$  than the original  $\mathbf{w}_k^p$ . Hence, we propose to re-sample the randomly synthesized spurious candidates by:
$$
\mathcal {D} _ {t r} ^ {s} = \left\{\left(\tilde {\mathbf {z}} _ {i}, \hat {y} _ {i}\right): \left\langle \overset {\circ} {\mathbf {w}} _ {\hat {y} _ {i}} ^ {p}, \tilde {\mathbf {z}} _ {i} \right\rangle > \left\langle \mathbf {w} _ {\hat {y} _ {i}} ^ {p}, \tilde {\mathbf {z}} _ {i} \right\rangle \right\} \tag {4}
$$
Combining with Eq. (3) and Eq. (1), the spurious context  $\mathbf{v}_k^s$  is able to learn to describe the synthesized spurious samples with such perturbation. After numerous iterations, random perturbations on perceptual contexts bring diverse spurious syntheses, which lead to learning non-trivial spurious contexts. More details on perturbation guidance (e.g., perturbed ratio) are discussed in Appendix A.2.
After learning the hierarchical perceptual and spurious contexts for each category, we are able to probe the precise and universal ID category boundaries.
3.3 Integrated Inference
After learning the hierarchical perceptual and spurious contexts, to further alleviate the overconfidence predictions on unseen OOD samples, we propose to regularize the vanilla image-text similarities by:
$$
r _ {k} = \left\langle \mathbf {w} _ {k} ^ {p}, \mathbf {x} \right\rangle \times \frac {e ^ {\left\langle \mathbf {w} _ {k} ^ {p} , \mathbf {x} \right\rangle}}{e ^ {\left\langle \mathbf {w} _ {k} ^ {p} , \mathbf {x} \right\rangle} + e ^ {\left\langle \mathbf {w} _ {k} ^ {s} , \mathbf {x} \right\rangle}} \triangleq s _ {k} \times \gamma_ {k} \tag {5}
$$
Eq. (5) is a unified measurement. For ID classification, the category is predicted by  $\hat{y} = \arg\max_{k} r_k$ . For OOD detection, the commonly-used scoring function [38, 50] is adopted:  $g(\mathbf{x}) = -\max_{\substack{\sum_{j}^{C} e^{r_{j}} \\ }}$
The motivation is intuitive. When perceptual context  $\mathbf{w}^p$  produces a relatively higher image-text similarity  $s$  on unseen OOD samples or misclassified ID inputs, the hierarchical spurious context  $\mathbf{w}^s$  could reduce the overconfident similarity by the regularization item  $\gamma$ . More empirical experiments are presented in the following sections.
In this section, we empirically validate the effectiveness of our CATEX on real-word large-scale classification and OOD detection tasks. The setup is described below, based on which extensive experiments and analysis are displayed in Sec. 4.1-Sec. 4.2.
Datasets. Following the common benchmarks in the literature [59, 50, 60, 38], we mainly consider the large-scale ImageNet [11] as the in-distribution data. Subsets of iNaturalist [53], SUN [65], Places [69], and Texture [8] are adopted as the OOD datasets. The categories in each OOD dataset are disjoint with the ID dataset [25, 38, 50]. This paper investigates four practical scenarios in real-world application of ID classification and OOD detection tasks: (1) standard OOD detection, (2) ID-shifted OOD detection, (3) category-extended ID classification and OOD detection, and (4) zero-shot ID classification. Details are presented in Sec. 4.1.
Evaluation metrics. For OOD detection, two metrics are used: (1) FPR95, the false positive rate of OOD samples when the true positive rate of ID samples is  $95\%$ , and (2) AUROC, the area under the receiver operating characteristic curve. For ID classification, we report the mean accuracy (ACC).