SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details. Without loss of generality, our method is implemented based on the vision-language prompting framework [71] with the CLIP model [41], one of the most popular and publicly available pre-trained models. CLIP aligns an image and its corresponding textual description in the feature space through a contrastive objective. We mainly take CLIP-B/16 in our experiments, which comprises of a ViT-B/16 Transformer [14] as the image encoder and a masked self-attention Transformer [55] as the text encoder. During training, all parameters of the image and text encoders are frozen. We only learn a pair of perceptual and spurious contexts for each ID category. Following the default setting [71], each context consists of 16 learnable 512-D prompt embeddings, which are trained for 50 epochs using the SGD optimizer with a momentum of 0.9. The initial learning rate is 0.002, which is decayed by the cosine annealing rule. The contexts are optimized with synthesized samples, and no surrogate OOD datasets are involved during training.
Table 1: OOD detection performance for ImageNet-1k [11] as ID dataset.
Method	OOD Datasets	Average
iNatualist	SUN	Places	Texture
FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑
MCM [38]	32.08	94.41	39.21	92.28	44.88	89.83	58.05	85.96	43.55	90.62
MSP [18]	54.05	87.43	73.37	78.03	72.98	78.03	68.85	79.06	67.31	80.64
Energy [33]	29.75	94.68	53.18	87.33	56.40	85.60	51.35	88.00	47.67	88.90
ViM [57]	32.19	93.16	54.01	87.19	60.67	83.75	53.94	87.18	50.20	87.82
KNN [48]	29.17	94.52	35.62	92.67	39.61	91.02	64.35	85.67	42.19	90.97
SSD+ [45]	59.60	85.54	75.62	73.80	83.60	68.11	39.40	82.40	64.55	77.46
DOE [60]	55.87	85.98	80.94	76.26	67.84	83.05	34.67	88.90	59.83	83.54
VOS [15]	31.65	94.53	43.03	91.92	41.62	90.23	56.67	86.74	43.24	90.86
NPOS [50]	16.58	96.19	43.77	90.44	45.27	89.44	46.12	88.80	37.93	91.22
Ours	10.18	97.88	33.87	92.83	41.43	90.48	33.17	92.73	29.66	93.48
CATEX significantly improves standard OOD detection. We first compare the proposed CATEX with competitive OOD detection methods that also delve into describing the ID category boundary. In particular, MCM [38] is the latest one to adopt the pre-trained CLIP to perform the ID boundary description in a zero-shot way, and others turn to fine-tune the encoders to optimize the ID boundary. The results are presented in Tab. 1, where the best performance is marked **bold**, and the
Table 2: Generalization across ID domains. Models are trained on ImageNet-1K [11] and directly tested on shifted ID datasets.
Method	Target Datasets
ImageNet-A	ImageNet-R	ImageNet-Sketch
ACC↑	FPR95↓	AUROC↑	ACC↑	FPR95↓	AUROC↑	ACC↑	FPR95↓	AUROC↑
MCM [38]	50.61	71.52	80.65	75.95	52.67	89.49	47.42	73.73	80.74
MSP [18]	43.96	86.25	63.25	66.50	82.21	75.07	45.18	91.67	58.45
Energy [33]	43.96	89.78	63.96	66.50	89.55	68.43	45.18	95.56	54.32
VOS [15]	43.79	80.03	73.45	66.83	78.42	80.09	46.06	86.68	68.58
NPOS [50]	50.16	74.57	75.37	73.58	75.64	82.71	50.29	82.87	71.56
Ours	50.87	71.13	81.04	76.67	51.75	89.75	48.59	74.68	80.69
Table 3: Generalization across ID tasks. Models are independently trained on disjoint ImageNet-100 (I) and ImageNet-100 (II), and then tested on the union ImageNet-200 (I  $\cup$  II) without fine-tuning.
Method	ID Datasets
ImageNet-100 (I)	ImageNet-100 (II)	ImageNet-200 (I ∪ II)
ACC↑	FPR95↓	AUROC↑	ACC↑	FPR95↓	AUROC↑	ACC↑	FPR95↓	AUROC↑
MCM [38]	89.00	24.38	95.59	90.48	19.84	96.44	83.35	27.18	94.88
MSP [18]	94.70	41.34	93.39	94.82	38.22	93.75	87.69	60.93	87.26
Energy [33]	94.70	32.11	94.36	94.82	32.95	93.86	87.69	37.78	92.62
VOS [15]	94.68	25.19	95.60	94.72	20.97	96.01	86.14	34.16	93.42
NPOS [50]	94.24 (+0.00)	16.54 (-0.00)	96.62	94.32 (+0.00)	16.84 (-0.00)	96.35	86.23 (+0.00)	25.54 (-0.00)	94.35
Ours	94.12 (-0.12)	10.31 (-6.23)	97.82	94.42 (+0.10)	7.91 (-8.93)	98.31	89.61 (+3.38)	13.34 (-12.20)	97.19
second best is underlined. Accordingly, our CATEX achieves superior OOD detection performance, outperforming competitive rivals by a large margin. Specifically, CATEX consistently surpass the SOTA method NPOS [50] on all of the four OOD datasets, leading to an  $8.27\%$  decrease in FPR95 and  $2.24\%$  increase in AUROC on average. It implies that compared to zero-shot probing and encoder-fine-tuning, learning the hierarchical perceptual and spurious contexts for each category is more effective in acquiring precise ID boundaries. In addition, our method dose not conflict with other post-hoc approaches (such as ReAct [46] and ASH [12]), and appropriate combinations may bring further improvements (e.g.,  $27.56\%$  of FPR95), as discussed in Appendix B.2.
CATEX properly generalizes to ID-shifted OOD detection. If the description of each category is precise enough, it will not only perform well in OOD tasks, but also be discriminative even when the distribution of ID data is shifted. Hence, we evaluate both the ID classification and OOD detection ability of models trained on ImageNet-1K [11] while tested on shifted ID datasets including ImageNet-A [20], ImageNet-R [23], and ImageNet-Sketch [56]. As shown in Tab. 2, CATEX reaches the best ID and OOD performance on both ImageNet-A and ImageNet-R and competitive results on ImageNet-Sketch. It indicates that the generalization ability from the large pre-trained models is well-maintained and even straightened. The fine-tuned models like NPOS are unstable across different datasets, which may be a sign of generalization degradation caused by parameter fine-tuning. Because the data shift is inevitable in real-world applications, we suggest that the proposed hierarchical contexts are instructive for model generalization.
CATEX carefully adapts to Category-extended ID classification and OOD detection. If the hierarchical contexts learn a precise category description, it should prevent overconfident predictions on any unseen samples in open-world but beyond the current ID task scope. We thus conduct a cross-task experiment to further evaluate the open-set discriminability. Specifically, two models are independently trained on ImageNet-100 (I) [38] and another disjoint ImageNet-100 (II) randomly selected from ImageNet-1K. Then, models are directly tested on the union ImageNet-200 (I  $\cup$  II). OOD detection is simultaneously evaluated, and the results are displayed in Tab. 3.
Accordingly, our CATEX achieves the highest performance on the union ImageNet-200 and significantly surpasses the competitors (e.g.,  $3.38\%$  increase on accuracy and  $12.2\%$  decrease on FPR95). In the category-extended scenario, our classification accuracy only drops by  $4\%$ , which is much lower than the fine-tuning methods (e.g.,  $8\%$  decrease from VOS [15]). It implies that fine-tuning the encoder is prone to overfitting on seen training samples, and sacrificing the ability (obtained from
(a) features after finetuning
(b) features w/o finetuning
Figure 4: Feature visualization by t-SNE. (a) Previous approaches that fine-tune the encoders may distort the generalized feature space and make unseen OOD samples inseparable; (b) instead, our method freezes the encoders to maintain the discriminability. (c) Compared with traditional prompting methods using a single perceptual context only, (d) our spurious context provides a better metric for unseen OOD detection.
(c) measure with single context
(d) measure with hierarchical contexts
large-scale pre-training) to distinguish unseen categories. We randomly select five categories from each ImageNet-100 subset respectively, whose feature distribution shown in Fig. 4 is consistent with Tab. 3. Moreover, compared to the SOTA method NPOS, our CATEX outperforms more on the union ImageNet-200 than separated ImageNet-100 subsets (e.g.,  $12.2\%$  v.s.  $6.23\%$  decrease on FPR95). It further demonstrates that in each ImageNet-100 subset, besides our learned perceptual contexts that classify the current 100 categories, the spurious contexts are able to identify the unseen categories (both of the new-coming ID samples and OOD samples), as illustrated in Fig. 4. In this way, the precise boundaries for each category are acquired. On the other hand, to deal with such category-extensible tasks in real-world practice [10, 62, 61], this paper provides a new perspective: separately learning the precise descriptions for the new-coming categories besides the existing ones, and jointly testing on the full ID category scope.
To further evaluate the efficacy of the precise descriptions, we conduct a more challenging larger-scale category-extensible experiments: scaling up ImageNet-1K to ImageNet-21K [11]. In particular, we randomly split the huge ImageNet-21K dataset into several subsets, individually train the hierarchical perceptual and spurious contexts for each category on each subset, and concurrently test on the full ImageNet-21K. Such paradigm is similar to category-incremental learning, and the results are shown in Fig. 5, where we mainly compare with zero-shot CLIP [41] and baseline CoOp [71]. According to Fig. 5, our CATEX consistently outperforms CLIP and CoOp. Specifically, when the total number of categories extends, CoOp has a higher accuracy drop, since its learned contexts on each separated subset share limited discriminability to novel categories across
Figure 5: Scaling up ImageNet-1K [11] to ImageNet-21K [11] with category-incremental learning.
subsets. And training on twenty-thousand categories together is challenging due to the generally prohibitive computational costs, which require over 300 GB GPU memory for the text-encoder on 21K categories. To alleviate this problem, this paper proposes to learn the hierarchical contexts to describe the precise and universal category boundaries, and achieves  $38\%$  accuracy on the full ImageNet-21K with a single V100-32G GPU card. We hope to offer new insights to the community on how to adopt large-scale VLMs to classify numerous categories with limited GPU resources.
CATEX encouragingly boosts zero-shot ID classification. Under the category-extended scenario, the hierarchical contexts help establish the category boundary for each given category, and prevent overconfidence on other unseen categories by adjusting the image-text similarity  $s$  with the spurious-context-regularized item  $\gamma$ , as introduced in Eq. (5). Intuitively, in zero-shot classification scenarios, the regularized score  $r = s \times \gamma$  may also rectify the category predictions and lead to a higher classification accuracy, with proper perceptual and spurious contexts. To verify it, as CLIP's default prompt template "A photo of a [CLS]" contains no visual information, we adopt rich visual descriptions from large language models [37] (such as GPT-3 [3]) as our perceptual context  $\mathbf{w}_p$  to perform classification (denoted as VisDesc). Then, we randomly perturb the visual description to simulate spurious contexts  $\mathbf{w}_s$ , and regularize text-image similarities via Eq. (5). The results shown in Tab. 4 indicate the regularized score  $r$  brings a higher classification accuracy, without any training
X seen ID images X unseen OOD images learned perceptual context learned spurious context distances
Table 4: Zero-shot classification via text-image similarity regularized by simulated spurious contexts.
Method	Prompt Example	ACC-Top1↑	ACC-Top5↑
CLIP	A photo of a goldfish.	63.50	88.99
VisDesc	A yellow fish with a long flowing tail is goldfish.	65.47	90.14
Ours	+ A [MASK] fish with a long [MASK] [MASK] is goldfish.	65.84	90.36
cost. We hope to provide new insights that in the category-extensible setting or a true zero-shot classification scenario, explicitly constructing spurious contexts can perform the one-class OOD detection task, as other categories can also be viewed as OOD for a certain ID category.
4.2 Ablation Studies
In this section, we take ImageNet-100 [38] as the ID dataset, and other settings follow Sec. 4.1. We mainly verify the key contributions of this work in Tab. 5, and the influence on model capacity in Tab. 6. In particular, we evaluate the number of spurious contexts used to help probe the category boundary in Tab. 7. More experiments and discussions are displayed in Appendix A and Appendix B.
Table 5: Ablation study on proposed framework.
Method	ID-ACC ↑	FPR95↓	AUROC↑
baseline [71]	94.12	13.07	97.42
+Hierarchical-Contexts	93.84	11.39	97.66
+Perturbation-Guidance	94.10	10.59	97.81
+Integrated-Inference	94.12	10.31	97.82
Table 6: Ablation study on model capacity.
Model	Method	ID-ACC ↑	FPR95↓	AUROC↑
RN50	MCM	84.26	32.31	94.61
Ours	90.58	19.58	96.25
ViT-L/14	MCM	91.66	20.79	96.35
Ours	96.18	6.97	98.43
The proposed components are effective. In Tab. 5, a baseline model is first constructed following the vision-language prompt-learning framework CoOp [71], which only learns the perceptual contexts with the vanilla softmax cross-entropy loss. Then, we learn the hierarchical perceptual and spurious contexts using initially synthesized OOD samples with Eq. (1) and Eq. (3). According to Tab. 5, the OOD detection performance is improved immediately, whereas the ID classification accuracy slightly drops by  $0.3\%$ . It indicates that the quality of initial OOD syntheses is limited, which puts the learned descriptions at risk of identifying ID inputs as OOD or even other ID categories. On the contrary, the proposed perturbation guidance mechanism in Eq. (4) provides extra constraints on OOD syntheses to improve the quality, which brings consistent gains in both ID classification and OOD detection protocol. Finally, the integrated inference strategy in Eq. (5) brings free improvement on FPR95 without loss of ID accuracy. The efficacy of our proposed method is further demonstrated.
Our method is scalable to model capacity. In Tab. 6, we investigate the scalability of our method with different model capacities, i.e., a lighter model with modified ResNet50 (RN50) and a heavier model with ViT-L/14 as the image encoder. The results imply larger models indeed lead to superior performance, suggesting larger models are endowed with better representation qualities [38, 50]. Interestingly, though the zero-shot method MCM [38] with ViT-L/14 surpasses our method with RN50 by  $1.1\%$  of ID classification accuracy, the OOD performance is inferior to ours (e.g., over  $1.2\%$  increase on FPR95). It further demonstrates that learning precise category descriptions via our hierarchical contexts is still a key to deriving a better OOD detector in the open-world.
Multiple spurious contexts bring more precise descriptions. For each ID category (e.g., cat), the spurious OOD samples can be diverse (e.g., panthers, lions, etc). Thus, we aim to only describe the spurious sample surrounding the ID category, rather than the whole OOD space (which is too complicated). According to common sense, it is intuitive to use more spurious contexts to describe better category boundaries. To verify it, we have tested the number of spurious contexts for each ID category (denoted as W/o Constraints), and the results shown in Tab. 7 implies using more spurious contexts only leads to  $0.1\%$  gain on performance. The reason may be that the learned 2 or more spurious contexts are too redundant without any constraints. To alleviate this problem, we simply add an orthogonal constraint (making the similarities between each two spurious contexts close to zero) (denoted as W/Constraints), and the OOD detection performance is significantly boosted, as displayed in Tab. 7. Therefore, how to effectively and efficiently leverage more spurious contexts to better describe the category boundary deserves further exploration, and we view it as our future work.
Table 7: Ablation study on multiple spurious contexts.
Ns	W/o Constraints	W/Constraints
FPR95↓	AUROC↑	FPR95↓	AUROC↑
1	10.31	97.82	10.31	97.82
2	10.21	97.86	10.17	97.86
4	10.27	97.88	9.89	97.89
8	10.25	97.86	9.76	97.84
OOD detection in computer vision. To keep the safe deployment of multi-category classification models in open-world scenarios, the goal of OOD detection is to derive a binary ID-  $\nu .s.$  -OOD detector for the visual inputs. To reduce the overconfidence on unseen OOD samples [1], a surge of post-hoc scoring functions has been devised based on various information, including output confidence [21, 32, 25], free energy [33, 58, 15], Bayesian inference [30, 34, 5], gradient information [24], model/data sparsity [46, 73, 12], and visual distance [31, 49, 50]. Another promising approach is adding open-set regularization in the training time [36, 22, 26, 52, 63], making models produce lower confidence or higher energy on OOD data. Manually-collected [22, 60] or synthesized outliers [15, 50] are required for auxiliary constraints. Works insofar have mostly focused on regularizing the task-specific classification models trained on downstream scenarios using only visual information. In contrast, this paper explores the OOD detection on category-extensible scenarios, which incorporates rich textual information to probe the task-agnostic category boundaries.
OOD detection with vision-language models (LVMs). Exploring textual information for visual OOD detection with large-scale pre-trained models has recently attracted a surge of interest. One may simply use the vision-language model CLIP [41] to explicitly collect potential OOD labels [18, 16] or conduct zero-shot OOD detection [38] with manually pre-defined text prompts. However, as such manual text prompts lack the description of unseen categories, the prior knowledge learned by VLMs during pre-training is not fully exploited, leading to inferior OOD detection performance. On the contrary, we propose to learn hierarchical contexts to describe the category boundaries against unseen spurious categories via vision-language prompt tuning [71]. RPL [7] and ARPL [6] explored a similar idea of "Reciprocal Points" to constrain a more compact in-distribution feature space, which may distort the generalized representation like NPOS [50]. Our method largely boosts the OOD detection performance, and simultaneously maintains the generalization capacity of pre-trained VLMs.
This paper proposes a novel framework to learn the precise category boundaries via the hierarchical perceptual and spurious contexts. Specifically, the perceptual context first perceives a relatively more strict classification boundary on the current ID task, and then integrates the spurious context to determine a more precise category boundary that is independent of the current task setting. The extensive empirical experiments demonstrate the proposed hierarchical contexts are prone to learning precise category descriptions, which shows robustness to various data shifts. By simply merging the task-independent precise category descriptions, we provide a new perspective to efficiently extend the ID classification scope to twenty-thousand categories at an acceptable resource cost. We also offers new insights to boost zero-shot classification via text-image similarities regularized from an one-class OOD detection perspective.
Limitations and societal impact. To keep the representation capacity of visual features and the generalization ability to identify unseen samples, we freeze the image encoder of the pre-trained CLIP. It limits the upper-bound performance on ID classification and OOD detection of our method. Larger models (e.g., ViT-L/14@336px) might bring higher performance but require more costs. There is a trade-off between efficacy and efficiency. Besides, as we do not fine-tune the encoder, the social biases accumulated during pre-training on uncurated web-scale datasets are inherited. Directly applying our method may cause biased predictions containing stereotypes or racist content. Leveraging debiasing techniques could be the key to alleviating this problem.
This work was supported in part by the Fundamental Research Funds for the Central Universities, and in part by the National Key R&D Program of China under Grant 2020AAA0103902.
A.1 Training Procedure
This section provides a walkthrough example to illustrate the training procedure presented in Sec. 3.1 of the manuscript. As displayed in Fig. A1, we first generate spurious OOD syntheses by perturbation guidance in Sec. 3.2, and then learn in-distribution classification via Eq. (1) and out-of-distribution detection via Eq. (3). Specifically, Fig. A1 captures the training procedure for one category. For the multi-category scenario, please refer to Fig. 2 in the manuscript.
Figure A1: Training illustration for one category. The randomly perturbed context is adopted to select spurious OOD syntheses, where gradients are stopped. Then perceptual context describes the ID images, and spurious context describes those spurious OOD syntheses surrounding the ID category. The two contexts jointly model the category boundary.
A.2 Perturbation Guidance
Detailed perturbation guidance process. For arbitrary  $k$ -th ID category, we use the spurious context to explicitly describe a corresponding spurious category, and a critical consideration is how to synthesize training samples spurious to that  $k$ -th ID category. Recently, generating adversarial data samples have been widely studied, including GAN networks [39, 27], diffusion models [43, 44], image attacks [35, 72], and feature-space sampling [15, 50]. For simplicity, we take the tractable feature-space sampling as NPOS [50] to generate spurious candidates. We calculate the  $k$ -NN distance for each ID sample in the specific category, and sample OOD candidates from a multivariate Gaussian distribution around the ID data points with largest distances (basically away from the clustering center). Then, we leverage the perturbed descriptions of perceptual context to guide the further filtering for high-quality spurious syntheses. Visualization examples are provided in Fig. A2.
Figure A2: Visualization of the Perturbation Guidance process. (a) Given a set of in-distribution image samples, several OOD syntheses are randomly generated in the feature space. (b) We perturb the perceptual context to simulate a kind of OOD category, and select OOD syntheses subject to (more similar to) the perturbed context. (c) After iteratively sample generation and perturbation-guided selection, the synthesized OOD samples are adopted to learn the spurious context.
Perturbation approach. Given the perceptual context  $\mathbf{v}_k^p = [v_{k,1}^p; v_{k,2}^p; \dots; v_{k,m}^p]$  of  $k$ -th ID category, we randomly apply a perturbation  $u$  onto one arbitrary  $v_{k,j}^p$  to produce a perturbed description  $\mathring{\mathbf{w}}_k^p$  through text-encoder. Intuitively, there are two ways to perturb a context  $\mathbf{v}_k^p$ : erasing or replacing the specific visual character  $v_{k,j}^p$ . Consequently, we design three types of perturbation: (1) masking with a placeholder  $u = [\text{MASK}]$ , (2) noise from a Gaussian distribution  $u = \sigma$ , and (3) swapping with another category  $u = v_{k',j}^p$ . And the perturbed text-feature is produced by:  $\mathring{\mathbf{w}}_k^p = \mathcal{T}([v_{k,1}^p; \dots; u; \dots; v_{k,m}^p; \text{CLS}_k])$ . As shown in Fig. A3, all of the three types of perturbed text-features  $\mathring{\mathbf{w}}_k^p$  slightly deviate from the original  $\mathbf{w}_k^p$  while keep the affinity (e.g., shares a  $97\%$  similarity against the original one.) Specifically, the noised  $\mathring{\mathbf{w}}_k^p$  leads to a greater deviation, since the noised visual character  $v_{k,j}^p := u$  is more unpredictable than the masked or swapped ones.
(a) similarity distribution.
(b) t-SNE visualization
Figure A3: Statistics of perturbations, including (a) similarities between original and perturbed text-features, and (b) distribution of original text-feature, perturbed text-features, and image-features.