SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the left term is cross-entropy loss and the right term is Kullback-Leibler divergence to the uniform distribution, which can be expressed as  $-\frac{1}{C}\sum_{c}\mathsf{softmax}_{c}f(\cdot)$ .  $\mathcal{D}_{out}^{OE}$  refers to the generated textual outliers. Our loss aims to increase the classification performance on train data and reduce the confidence on  $\mathcal{D}_{out}^{OE}$ . We add Gaussian noise to the textual outliers to reduce the modality gap [36].
4.3 Test time OoD detection
When testing, we use Energy score [29] for OoD detection  $S(\mathbf{x},f) = -E(\mathbf{x};f)$ , where
$$
E (\mathbf {x}; f) = - T \cdot \log \sum_ {i: y _ {i} = c} ^ {C} e ^ {f _ {i} (x) / T}. \tag {5}
$$
Energy score is known to be less susceptible to the overconfident issue. It is worth mentioning that the sign of the energy function, denoted as  $E(\mathbf{x})$ , is reversed to align with the convention where samples with higher scores are classified as ID, while samples with lower scores are classified as OoD. Our method can be used with any OoD score that utilizes output probabilities.
In this section, we conduct extensive experiments to showcase the effectiveness of our novel technique for textual outlier exposure. Through comprehensive experiments, we evaluate and investigate our approach from three main perspectives: (1) comparison of our method with existing methods in outlier exposure, (2) comparison of performance among proposed textual outliers, and (3) assessment of the proposed textual outlier exposure in challenging OoD detection scenarios.
Datasets. We use the large-scale ImageNet-1K [6] OoD detection benchmark proposed by Huang et al. [20]. We conduct experiments on four OoD test datasets: subsets of iNaturalist [47], SUN [52], Places [58], Texture [4]. The categories of OoD datasets are disjoint from those of the ID dataset.
Experimental setting. The CLIP model used in this paper is adapted from OpenAI's public repository, with ViT-B/16 serving as the default vision and language backbone. For word-level textual outliers, the BERT [7] large model with 24 layers and a hidden size of 1024 from huggingface [51] is used. The model is trained on the MS-COCO [28] dataset using the Adam optimizer [22] with a fixed learning rate of  $10^{-5}$  for 100 epochs. A simple teacher-forcing algorithm is employed to retrieve text during the training process. To generate description-level textual outliers, the GPT-3-text-davinci-002 model is used. For caption-level textual outliers, BLIP-2 opt 2.7b from huggingface is used. The Adam optimizer is used to train our linear classifier for OoD detection, and batch size, learning rate, and other hyperparameters are tuned on the validation set. We use 0.5 for  $\lambda$  in our training loss, and batch size for both ID and train time outlier is 32. The temperature value  $T$  for Energy score is set to 1, as per the original paper. Model checkpoints with the highest validation accuracy are evaluated on the test set. We employ a value of 30 for  $k$ , which is used in word-level outliers filtering, and a value of 25 for  $\delta$ . Furthermore, a filtering ratio of 15 is utilized as the  $p$  for caption-level outlier analysis. The ablation study examining these parameters can be found in the Appendix B.
Table 5: Comparison of proposed textual outliers and competitive baselines from visual outlier exposure on the ImageNet-1K dataset. The best result in each column is in bold.
OoD Data	iNaturalist	SUN	Places	Texture	Average
Metrics	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	ID Acc	FPR95	AUROC
Visual OE
OE	78.31	75.23	80.10	76.55	70.41	81.78	66.38	82.04	75.51	73.80	78.90
CSI	75.85	82.63	90.62	47.83	94.90	44.62	85.85	87.11	74.27	86.80	65.54
SSD+	59.60	85.54	75.62	73.80	83.60	68.11	39.40	82.40	78.80	64.55	77.46
MixOE	80.51	74.30	74.62	79.81	84.33	69.20	58.00	85.83	74.62	74.36	77.28
VOS	94.83	57.69	98.72	38.50	87.75	65.65	70.20	83.62	74.43	87.87	61.36
DOE	55.87	85.98	80.94	76.26	67.84	83.05	34.67	88.90	75.50	59.83	83.54
Textual OE
None	84.74	74.02	92.52	63.85	99.21	41.72	99.01	27.30	72.43	93.87	51.72
Word	31.72	94.56	58.76	86.73	67.68	83.51	80.12	77.10	76.60	59.57	85.47
Desc.	33.12	94.28	54.72	87.70	72.47	81.82	86.03	70.72	77.48	61.58	83.63
Caption	29.61	94.74	57.12	87.34	66.82	83.71	79.29	77.76	76.00	58.21	85.88
5.1 Comparison with competitive baseline approaches
Textual outliers outperform previous advanced methods. To compare the effectiveness of textual outlier exposure method with other outlier exposure methods, we consider the following baselines: OE [18], CSI [45], SSD+ [41], MixOE [55], VOS [8], and DOE [50]. The performances of compared methods in Table 5 are borrowed from those reported in Wang et al. [50]. All of these baseline methods utilize CNN architectures, specifically ResNet [16] or WideResNet [54].
Our textual outlier methods at every level consistently achieve superior OoD detection performance, surpassing competing approaches in terms of AUROC. Caption- and word-level textual outliers succeed at improving the FPR95 metric as well. We point out that the caption-level textual outliers reduce FPR95 from  $87.87\%$  (VOS) to  $58.21\%$  (ours), resulting in a substantial improvement of  $29.66\%$ . This significant performance gap highlights the effectiveness of our textual outlier generation method for model regularization. The results on the ImageNet-100 benchmark and comparison with post-hoc methods can be found in the Appendix A. We include experimental results for CLIP-RN50 and RN50x4 in the Appendix D.4 and D.5 to demonstrate that the observed performance gains do not stem from different architectures, namely ResNet and ViT. The performance comparison among the proposed textual outliers is included in the lower half of Table 5. We also report the results of using CLIP without outlier exposure (None) to demonstrate that the performance improvement is not simply due to the inductive bias of the CLIP embedding space. Among the three levels of textual outliers, the caption-level textual outliers demonstrate superior performance on average.
Based on these results, we can deduce that textual outliers must meet the characteristics summarized below to function effectively for outlier exposure. While the near-distribution criterion is in line with previous insights from the outlier exposure literature, the other two are unique to textual outliers.
- Near-Distribution: Textual outliers should be situated near the boundary samples of the training distribution, as several approaches in visual outlier exposure have emphasized [23, 33, 3]. This allows the neural network to compactly estimate the decision boundary between ID and OoD data [8].
- Descriptiveness: Textual outliers should possess rich linguistic semantics to include comprehensive and diverse explanations of OoD data. Word-level textual outliers, which are relatively terse, often fall short of the other two forms of textual outliers in this aspect.
- Inclusion of Visual Semantics: Textual outliers should incorporate visual elements, such that generated textual outliers do not deviate far from visual boundary samples of ID data.
The characteristics satisfied by textual outliers at each level are detailed in the Appendix F.3.
Table 6: Effectiveness of textual outliers exposure in hard OoD situation, a more challenging OoD detection scenario. Proposed textual outliers are compared against strong baseline approaches on three different ID/OOD dataset combinations. The best result in each column is in bold.
Dataset	ID
OoD	ImageNet10
ImageNet20	ImageNet20
ImageNet10	CUB100
CUB100
FPR	AUROC	FPR	AUROC	FPR	AUROC
Visual OE	OE	93.20	54.51	89.80	57.63	87.20	60.70
VOS	91.00	60.42	93.50	44.38	94.79	49.25
Textual OE	Word	4.60	98.92	9.00	98.39	80.34	71.62
Desc.	3.60	99.02	7.20	98.69	82.56	72.33
Caption	5.30	98.85	8.60	98.43	85.92	64.79
5.2 Textual outliers in hard OoD situation
Description-level textual outliers outperforms in hard OoD. Furthermore, we explore the effectiveness of textual outliers in handling challenging OoD inputs. In these scenarios, the OoD samples closely resemble the ID samples in terms of their semantic content. To evaluate the performance of OoD detection in such a semantically hard scenario, we first utilize the ImageNet10 vs. ImageNet20 benchmark proposed by Ming et al. [34]. This task involves distinguishing between high-resolution images with semantically similar categories, such as dog vs. wolf. Our experimental results, shown in Table 6 (first and second columns), demonstrate the superiority of our textual outliers over OE, achieving an  $89.6\%$  p improvement in FPR95 for ImageNet10 (ID) vs. ImageNet20 (OoD), and a  $82.6\%$  p improvement when ID and OoD data are switched.
Additionally, we conduct experiments on the CUB dataset [49] benchmark, which Vaze et al. [48] proposed (third column in Table 6). This benchmark is designed to discriminate between ID and OoD samples that possess numerous shared attributes, thereby introducing difficulties in accurately identifying OoD instances. As shown in Table 6, our description-based textual outliers outperform the auxiliary dataset and synthesis methods. While descriptions are known to improve classification performance, they can also be effective outliers in situations where class names are not considered. The class labels for the hard OoD benchmarks are provided in the Appendix F.2.
OoD detection. A considerable body of research has been dedicated to designing scoring functions for detecting samples that lie outside the training categories. These methods include Maximum Softmax Probability [17], ODIN score [27], Mahalanobis-based score [24], Energy score [29], gradient-based score [19], and non-parametric KNN score [44]. While there have been studies proposing OoD scores using CLIP's textual information [32, 12, 10], none of them have incorporated textual information into outlier exposure during training. An alternative approach to address the out-of-distribution (OoD) detection problem is through training-time regularization, as discussed in previous research [23, 18]. These methods encourage models to provide predictions with lower confidence or higher energies, aiming to improve OoD detection. However, most of these approaches rely on the availability of an auxiliary outlier dataset. To overcome this limitation, recent studies have explored the synthesis of virtual outliers in the feature space, as seen in works such as [8, 46]. This approach aims to generate synthetic outliers without the need for an additional dataset. Our proposed textual outliers differ from previous methods in that they are defined in the input space and do not rely on an auxiliary dataset.
Multi-modal models. The adoption of pre-trained VLMs in multi-modal tasks has attracted considerable interest and yielded impressive results. Prominent among these approaches are dual-stream models such as CLIP [37], ALIGN [21], and FILIP [53]. These models utilize distinct encoders for text and image data and optimize them through contrastive objectives to align semantically similar features across heterogeneous modalities. The success of CLIP-like models inspired numerous subsequent investigations [26, 56, 59], aiming to enhance data efficiency and adaptability for various downstream tasks. Recent progress in this field, exemplified by innovations such as DALL-E 2 [38], ClipCap [35], and related studies [5, 13], has pushed the boundaries of exploration in multi-modal contrastive representation spaces, uncovering their untapped potential. To the best of our knowledge, this is the first time that joint embedding has been utilized for outlier exposure.
In this paper, we propose a novel outlier framework, transitioning from the traditional single-modal paradigm to a multi-modal regime. Unlike existing methods that rely on image data, we leverage textual outliers during the training process. We analyze the advantages of textual outliers compared to visual outliers and propose three levels of textual outliers. Our proposed textual outliers significantly enhance the neural network's capability to distinguish between ID and OoD data, leading to superior OoD detection performance while maintaining the classification performance on ID data. Furthermore, our analysis of each level of outliers provides valuable insights into the characteristics of effective textual outliers. We anticipate that our work will serve as a source of inspiration for future research on OoD detection methods that leverage textual outlier exposure. Limitations. Our proposed textual outlier method involves a heuristic selection process to refine outputs of generative models. In future work, it is necessary to reduce the dependence on heuristic methods and explore alternative approaches.
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2022R1A3B1077720 and 2022R1A5A708390811), Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) [2021-0-01343: Artificial Intelligence Graduate School Program (Seoul National University) and 2022-0-00959] and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2023.
[1] Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. The unreasonable effectiveness of clip features for image captioning: an experimental analysis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4662-4670, 2022.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[3] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution detection using outlier mining. In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), 2021.
[4] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[5] Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-language representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX, pages 558-577. Springer, 2022.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[8] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. Proceedings of the International Conference on Learning Representations, 2022.
[49] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[50] Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye HAO, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In The Eleventh International Conference on Learning Representations, 2023.
[51] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[52] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[53] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022.
[54] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016.
[55] Jingyang Zhang, Nathan Inkawich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5531-5540, January 2023.
[56] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021.
[57] Yuhui Zhang, Jeff Z. HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In The Eleventh International Conference on Learning Representations, 2023.
[58] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452-1464, 2018.
[59] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.