SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Perturbing ratio. As an important factor, perturbing how many visual characters  $\{v_{j}^{p}\}^{m}$  to simulate a spurious category seems to be very sensitive. A too-small ratio may lead to noneffective perturbation and cause invalid candidates from ID data, while too-large ratios can distort the spurious category description and bring invalid candidates from far OOD. In fact, as previous analysis in Fig. A3 implies perturbing one word/ token  $v_{j}^{p}$  can produce effective deviation on the encoded text feature  $\mathbf{\hat{w}}_k^p$ , Tab. A1 suggests that one-word-perturbation is effective enough to perform perturbation guidance as described in Sec. 3.2. Perturbing 4 or more words even leads to performance degradation, which means severely perturbed contexts may choose the noisy OOD candidates (e.g., random noise), making the learned spurious context unable to capture the true spurious OOD samples and further describe the category boundary.
Table A1: Ablation for perturbing ratio on ImageNet-100 benchmark.
Perturbing ratio	FPR95↓	AUROC↑
1/16	10.31	97.82
2/16	10.27	97.81
4/16	10.47	97.78
8/16	11.02	97.73
16/16	11.70	97.62
Perturbing position. Another hyper-parameter to perform perturbation on perceptual contexts is the perturbing position, i.e., which word/token to perturb? In Sec. 3.2 from the manuscript, we randomly choose the word in perceptual contexts to perturb at each training iteration. Indeed, deciding the optimal word to perturb is challenging, especially when the learned word embeddings do not correspond to actual words in natural language. Though, we have made a simple step to find the "optimal" word/token in the embedding space. After computing the prototype vector by averaging the 16 learned words, we take the most distant (denoted as MaxDist) or closed (denoted as MinDist) word for perturbation guidance for spurious OOD syntheses. However, as shown in Tab. A2, the final OOD detection performance even gets worse. Thus, we argue that randomly choosing the words to perturb at each iteration is still an effective way. After several iterations, we can statistically perturb every word to guide the spurious sample generation, covering the optimal situation.
Table A2: Ablation for perturbing position on ImageNet-100 benchmark.
Method	FPR95↓	AUROC↑
MaxDist	10.42	97.75
MinDist	10.86	97.73
Random	10.31	97.82
A.3 Cross-ID-Domain Generalization
As indicated in the manuscript, the precise category boundary learned by our method shows robust OOD performance when the ID data is shifted. In fact, the shifted ID classification can be further boosted by our proposed integrated inference strategy (Eq. (5) in the manuscript), as shown in Tab. A3. It implies the regularization item  $\gamma$  successfully modulates the relative similarities between input images and learned perceptual descriptions for each category, leading to more precise boundaries.
Table A3: Additionally improved ID accuracy on shifted datasets.
Method	Target Datasets
ImageNet-A	ImageNet-R	ImageNet-Sketch
CATEX	50.87	76.67	48.59
+IntegInfer.	50.98	76.72	48.65
However, our method only takes the secondary place on ImageNet-Sketch [56] on both ID classification (inferior to NPOS [50]) and OOD detection (inferior to MCM [38]). It is mainly because of the huge domain gap between vanilla ImageNet-1K [11] and shifted ImageNet-Sketch. As shown in Fig. A4, compared to the shifted ImageNet-A [20] and ImageNet-R [23], images from ImageNet-Sketch only preserve objects' shape and main texture, while the color information is totally vanished. We leave the generalization to heavily-shifted ID datasets as future work.
Figure A4: Left to right: examples from ImageNet, ImageNet-A, ImageNet-R, and ImageNet-Sketch.
A.4 Cross-ID-Task Generalization
To verify the efficacy of our proposed framework, we conduct a category-extended experiment in Sec.4.1 and Tab.3. Here more implementation details are provided for reproducibility.
Given two models independently trained on the separated ImageNet-100 (I) and ImageNet-100 (II), how to test them on the union ImageNet-200 (I  $\cup$  II) with our CATEX is simple. In the vision-language prompt-tuning framework, the image-encoder  $\mathcal{I}$  and text-encoder  $\mathcal{T}$  are frozen, and we only learn the perceptual and spurious contexts (i.e.,  $\mathbf{v}^p$  and  $\mathbf{v}^s$ ). And the  $l_{2}$ -normalized text-feature can be pre-extracted with the 100 category names in each subset, taking the perceptual descriptions for example, which are denoted as  $\{\mathbf{w}_{\mathbb{I},k}^p = \mathcal{T}(\mathbf{v}_{\mathbb{I},k}^p;\mathrm{CLS}_{\mathbb{I},k})\}_{k = 1}^{100}$  and  $\{\mathbf{w}_{\mathbb{I}\mathbb{I},k}^p = \mathcal{T}(\mathbf{v}_{\mathbb{I}\mathbb{I},k}^p;\mathrm{CLS}_{\mathbb{I}\mathbb{I},k})\}_{k = 1}^{100}$ . During inference, one may concatenate the 200 text-features together as  $\{\mathbf{w}_k^p\}_{k = 1}^{200}$ . Given an input image  $I$ , the  $l_{2}$ -normalized image-feature is extracted by  $\mathbf{x} = \mathcal{I}(I)$ , and the perceptual image-text similarities are computed as  $\mathbf{s}^p = [\langle \mathbf{w}_1^p,\mathbf{x}\rangle ,\langle \mathbf{w}_2^p,\mathbf{x}\rangle ,\dots ,\langle \mathbf{w}_{200}^p,\mathbf{x}\rangle ]\triangleq [s_1^p,s_2^p,\dots ,s_{200}^p ]$ . Similarly, the spurious similarities become  $\mathbf{s}^s = [s_1^s,s_2^s,\dots ,s_{200}^s ]$ . Then we can leverage the measurement defined in Eq. (5) for both ID classification and OOD detection.
As for the competitors, (e.g., VOS [15] and NPOS [50]), two image-encoders are trained separately (denoted as  $\mathcal{I}_{\mathrm{I}}$  and  $\mathcal{I}_{\mathrm{II}}$ ). And for each input image  $I$ , there are two corresponding image-features:
$\mathbf{x}_{\mathrm{I}} = \mathcal{I}_{\mathrm{I}}(I)$  and  $\mathbf{x}_{\mathrm{II}} = \mathcal{I}_{\mathrm{II}}(I)$ . Consequently, there also two sets of image-text similarity vector:  $\mathbf{s}_{\mathrm{I}} = [\langle \mathbf{w}_1,\mathbf{x}_{\mathrm{I}}\rangle ,\langle \mathbf{w}_2,\mathbf{x}_{\mathrm{I}}\rangle ,\dots ,\langle \mathbf{w}_{200},\mathbf{x}_{\mathrm{I}}\rangle ] = \{\langle \mathbf{w}_k,\mathbf{x}_{\mathrm{I}}\rangle \}_{k = 1}^{200}$  and  $\mathbf{s}_{\mathrm{II}} = \{\langle \mathbf{w}_k,\mathbf{x}_{\mathrm{II}}\rangle \}_{k = 1}^{200}$  (the superscript  $^p$  is hidden for simplicity). For compatibility, we choose the one for ID classification and OOD detection according to its highest image-text similarity.  $\mathbf{s} = \left\{ \begin{array}{ll}\mathbf{s}_{\mathrm{I}} & max(\mathbf{s}_{\mathrm{I}}) > max(\mathbf{s}_{\mathrm{II}})\\ \mathbf{s}_{\mathrm{II}} & otherwise \end{array} \right.$ . Now, the performance of our method and other rivals are evaluated under the same measurements.
Note that since we only take one image encoder throughout, the inference time is fixed (because the text-features can be pre-extracted). In contrast, applying other methods brings multiple time cost (e.g., twice slower than ours in this case). When the training subsets extend intensely (e.g., from ImageNet-1K to ImageNet-21K in our manuscript), our method still keeps a fast speed (e.g., 100FPS on V100) during inference, which can even enable real-time applications in practice.
A.5 Software and Hardware
We use Python 3.7.13 and PyTorch 1.8.1, and 2 NVIDIA V100-32G GPUs.
B.1 Comparison to CoCoOp
Compared with the vanilla vision-language prompt tuning method CoOp [71], the newer version CoCoOP [70] was explicitly designed to deal with out-of-distribution issues through image conditional contexts. However, Tab. A4 indicates that CoCoOp is surprisingly much worse than our method. The reason may be that during training, CoCoOp only takes ID images as context conditions, while neither OOD samples nor OOD contexts are involved. Thus, when employed in the open-world and asked to reject OOD samples not belonging to any ID category, CoCoOp failed. The image conditions even aggravate the overconfidence in OOD samples.
Table A4: Comparison to CoCoOp on ImageNet-100 benchmark.
Method	ID-ACC↑	FPR95↓	AUROC↑
CoCoOp	92.90	39.22	92.69
Ours	94.12	10.31	97.82
B.2 Combination with Post-hoc Enhancements
Recently, post-hoc OOD detection methods that enhance the single-vision-modal networks (e.g., ResNet [19] and ViT [14]) have been widely studied [9, 46, 47, 73, 12]. In this section, we make a step towards combining vision-language models with previous post-hoc enhancements for better OOD performance. The results are shown in Tab. A5, where ReAct [46] achieves a remarkable improvement. It indicates that pruning the extreme feature values according to the unified distributional statistics may be more suitable for VLMs to reduce the overconfidence on OOD samples. We hope this can bring new insights to the community.
Table A5: Combination with post-hoc methods.
Cmobine	ImageNet-100	ImageNet-1K
FPR95↓	AUROC↑	FPR95↓	AUROC↑
None	10.31	97.82	29.66	93.48
ReAct [46]	10.06	97.82	27.56	93.77
BATS [73]	10.16	97.84	29.37	93.59
ASH [12]	10.19	97.81	29.14	93.27
B.3 Inference-time perturbation
Now that every perturbation can directly produce the description (i.e., text-feature) of an unknown spurious category, one may try to take the perturbed description as a substitute for the learned spurious contexts to execute OOD detection. That is, use the perturbed  $\mathbf{w}_k^p$  to replace the learned  $\mathbf{w}_k^s$  in Eq. (5) in the manuscript. And the results are shown in Tab. A6, where ImageNet-100 [38] is the ID dataset. Given a baseline model [71] learned with perceptual contexts only, simply using the perturbed descriptions (denoted as  $+Perturb$ -Desc.) brings slight improvements (e.g., 0.2% decrease on FPR95). The insignificant advantage is not due to the limited capacity of only one perturbed description for each ID category. Because ensembling [38] several perturbed descriptions for an ID category at once (denoted as  $+Perturb$ -Ensem.) dose not bring remarkable improvements. In contrast, our proposed CATEX can significantly enhance the OOD detection performance, which demonstrates it is still necessary to explicitly learn the spurious contexts for each ID category.
Table A6: Comparison with directly using perturbed descriptions for OOD detection.
Method	FPR95↓	AUROC↑
baseline [71]	13.07	97.42
+Perturb-Desc.	12.84	97.43
+Perturb-Ensem.	12.87	97.45
CATEX (Ours)	10.31	97.82
B.4 Experiments on CIFAR Benchmarks
To further verify the robustness of our method, we conduct additional experiments on CIFAR-10 and CIFAR-100 datasets, and evaluate the OOD detection performance on SCOOD [66] benchmark. We train our CATEX for 20 epochs, and the other settings are the same as Sec.4 in the manuscript. The results are shown in Tab. A7 and Tab. A8, where "Surr." means the extra TinyImages80M [51] is adopted for surrogate OOD training set. Accordingly, our CATEX consistently outperforms the competitors as well, and even surpasses those who adopts the extra OOD training data. It implies the pre-trained knowledge for large-scale CLIP [41] model leveraged by our method is capable enough to detect the OOD samples in the open-world. The efficacy of our CATEX is further demonstrated.
Table A7: Performance on CIFAR-10.
Method	Surr.	ID-ACC ↑	FPR95↓	AUROC↑
MCM [38]	✗	90.79	23.14	94.68
ODIN [32]	✗	95.36	52.00	82.00
Energy [33]	✗	95.36	50.03	83.83
OE [22]	✓	94.90	50.53	88.93
UDG [66]	✓	94.71	36.22	93.78
CATEX	✗	95.57	21.17	95.33
Table A8: Performance on CIFAR-100.
Method	Surr.	ID-ACC ↑	FPR95↓	AUROC↑
MCM [38]	✗	66.91	71.93	79.39
ODIN [32]	✗	81.84	81.89	77.98
Energy [33]	✗	81.84	83.66	79.31
OE [22]	✓	81.31	80.06	78.46
UDG [66]	✓	80.89	75.45	79.63
CATEX	✗	81.99	67.95	84.04
B.5 Error Bars
To verify the robustness, we repeat the training of our method and the rivals on ImageNet-100 [38] with CLIP-B/16 for 3 times, and the results are shown in Tab. A9. Our CATEX consistently outperforms the rivals on OOD detection by a significant margin.
Table A9: Error Bars on ImageNet-100 after 3 runs
Method	ACC↑	FPR95↓	AUROC↑
MSP [18]	94.77 (±0.05)	41.90 (±0.61)	93.38 (±0.05)
Energy [33]	94.77 (±0.05)	31.89 (±0.50)	94.53 (±0.18)
VOS [15]	94.75 (±0.07)	24.48 (±0.71)	96.04 (±0.36)
NPOS [50]	94.34 (±0.12)	17.32 (±0.87)	96.46 (±0.13)
CATEX	94.11 (±0.03)	10.97 (±0.79)	97.75 (±0.07)
Performance improvement. To further evaluate the improvement brought by our method (e.g.,  $8\%$  decrease of FPR95 against NPOS), we conduct a comparative experiment on ImageNet-1K. To provide a unified analysis across two models, we take a third-party ResNet-50 model [64] (pre-trained on ImageNet-1K classification only) to produce the Maximum SoftMax Probability for each OOD sample that is correctly detected by our CATEX while wrongly viewed as ID samples by NPOS. According to Fig. A5, our method consistently improves the OOD detection on each interval, where the high-probability OOD (generally hard samples) detection is significantly enhanced. It indicates that properly leveraging the prior knowledge from pre-trained VLMs can alleviate the OOD problem when the fine-tuned visual features are indistinguishable, which is consistent with our motivation.
Figure A5: Corrected OOD detections compared with NPOS. The softmax probability predictions on those OOD samples are produced by another pre-trained ResNet-50 [64] classifier.
Failure cases. As our method still gets  $29\%$  FPR95 on ImageNet-1K, we provide some failure cases in Fig. A6, which can be summarized into three kinds:
- Noisy label, where the ID objects (e.g., dam) also exist in some OOD images from the test set. And the dataset composition may need a further examination.
- Similar texture, shared by some OOD samples (e.g., flower) against ID images (e.g., starfish), and the pre-trained encoders of CLIP are unable to distinguish their features. Applying image-level spurious OOD syntheses (e.g., image attacks [35, 72]) may reduce the texture-bias.
- Same background (e.g., sky) that seizes a large proportion of the image may lead to similar feature representations. Adopting image-level automatic masking techniques [2, 68] to synthesize spurious OOD samples may alleviate such problem.
Similar failure cases are also observed in recent SOTA methods, which reveal the unsolved challenges of OOD detection and suggest the potential directions for future works.
D Datasets and Baselines
For reproducibility, we present the details of datasets and baselines as follows.
ImageNet-100 (I). Following MCM [38], we take the randomly-sampled 100 classes from ImageNet-1K [11] as the ImageNet-100 (I) subset, which contains the following categories: n03877845, n03000684, n03110669, n03710721, n02825657, n02113186, n01817953, n04239074, n02002556, n04356056, n03187595, n03355925, n03125729, n02058221, n01580077, n03016953, n02843684, n04371430, n01944390, n03887697, n04037443, n02493793, n01518878, n03840681, n04179913, n01871265, n03866082, n03180011, n01910747, n03388549, n03908714, n01855032, n02134084, n03400231, n04483307, n03721384, n02033041, n01775062, n02808304, n13052670, n01601694, n04136333, n03272562, n03895866, n03995372, n06785654, n02111889, n03447721, n03666591, n04376876, n03929855, n02128757, n02326432, n07614500, n01695060, n02484975, n02105412, n04090263, n03127925, n04550184, n04606251, n02488702, n03404251, n03633091, n02091635, n03457902, n02233338, n02483362, n04461696, n02871525, n01689811, n01498041, n02107312, n01632458, n03394916, n04147183, n04418357, n03218198, n01917289, n02102318, n02088364, n09835506, n02095570, n03982430, n04041544, n04562935, n03933933, n01843065, n02128925, n02480495, n03425413, n03935335, n02971356, n02124075, n07714571, n03133878, n02097130, n02113799, n09399592, n03594945.
ImageNet-100 (II). Disjoint from ImageNet-100 (I), ImageNet-100 (II) contains another 100 classes randomly sampled from ImageNet-1K: n02096177, n03769881, n01629819, n04033995, n04357314, n02101388, n02328150,
Figure A6: Failed OOD detections of our CATEX
n03729826, n02655020, n01985128, n02109525, n07715103, n02099429, n04517823, n02088632, n03207743, n03657121, n02948072, n02106662, n01631663, n09229709, n03793489, n03776460, n07860988, n02129604, n03483316, n02107574, n07716358, n04208210, n02107908, n04372370, n02119022, n12144580, n01693334, n04548280, n03785016, n03535780, n03599486, n02859443, n04335435, n02110341, n03902125, n04146614, n01774750, n03314780, n03045698, n01697457, n02869837, n02276258, n04081281, n03956157, n02487347, n04311174, n02094114, n04409515, n03028079, n03384352, n04532106, n02087394, n04612504, n02100583, n11939491, n02107142, n01669191, n12998815, n04522168, n02894605, n03529860, n10148035, n01677366, n03775071, n03208938, n04238763, n02363005, n02804414, n02106382, n03950228, n02128385, n02028035, n04099969, n02481823, n01729322, n02939185, n02483708, n04162706, n03857828, n02093647, n02927161, n03160309, n02840245, n03920288, n07871810, n04404412, n03947888, n04509417, n02086910, n02256656, n02412080, n02410509, n03584829.
ImageNet-21K. The ImageNet-21K dataset on which we conduct the category-extended experiment is the official winter 2021 released version *. For pre-processing, we follow Ridnik et al [42] to clean invalid classes, allocating 50 images per class for validation, and crop-resizing all the images to 224 resolution. Training settings are the same as Sec.4 in our manuscript.
OD datasets. Following the literature [59, 50, 60, 38], we mainly consider subsets of iNaturalist [53], SUN [65], Places [69], and Texture [8] as the OOD datasets, which contains 35640 images in total.
Baselines. To evaluate the baselines on our experiment settings, we re-implement the most representative and relevant methods, including MSP [21, 18], Energy [33], VOS [15], and NPOS [50]. For a fair comparison, we train all the baselines with NPOS's codebase  $\dagger$ , and only fine-tune the last two transformer blocks of image encoder [50].
- For MSP and Energy, we train a single model with standard cross-entropy loss function for ID classification only, and infer with respective OOD metrics.
- For VOS, we take the likelihood-based sampling strategy to generate spurious OOD syntheses, and train the model with uncertainty regularization as suggested [15].
- For NPOS, we take the non-parametric distance-based sampling strategy to generate spurious OOD syntheses, and train the model with open-set ERM as suggested [50].
*https://image-net.org/
$\dagger$  https://github.com/deeplearning-wisc/npos