SlowGuess/NeurIPS_2023_OCR huggingface dataset:
benchmark are from iNaturalist [23], SUN [33], Places [34] and Textures [35], including fine-grained images, scene-oriented images, and textural images. We also evaluate CIFAR10 and CIFAR100 benchmarks [7], which are routinely used in literature. Correspondingly, OOD datasets are SVHN [36], TinyImageNet [9], LSUN [37], Places [34] and Textures [35].
Baselines. We consider various kinds of mainstream post-hoc OOD detection methods as baselines, including Maximum Softmax Probability (MSP) [7], ODIN [9], Energy-based method [5], Mahalanobis [14], ReAct [11], GradNorm [10], Rankfeat [15], ASH [30] and KNN [29]. We use FPR95 (the false positive rate of OOD examples when the true positive rate of ID examples is  $95\%$ ) and AUROC (the area under the receiver operating characteristic curve) as evaluation metrics.
In our main results, all methods can be directly used for pre-trained models and for a fair comparison, auxiliary OOD data is unavailable for tuning.
Evaluation on CIFAR benchmarks. In Tab. 1, we evaluate GAIA methods on CIFAR10 and CIFAR100 benchmarks. The results show that both GAIA-A and GAIA-Z exhibit superior performance. And we also note that advanced post-hoc methods such as Rankfeat and Gradnorm tend to encounter performance degradations on limited label space with small architectures. For ID dataset CIFAR10, baseline ASH performs the best with an average FPR95 of  $26.36\%$  on ResNet34 and ODIN performs  $37.09\%$  on WideResNet40 (WRN40). Our method GAIA-Z significantly outperforms ASH on ResNet34 by  $23.10\%$  improvement and outperforms ODIN on WideResNet by  $19.13\%$  improvement. Moreover, GAIA-A achieves the second best performance after GAIA-Z. For CIFAR100, GAIA-Z attains an average FPR95 of  $29.10\%$  and average AUROC of  $94.93\%$  on ResNet34, surpassing the best baseline ReAct by a margin of  $45.41\%$  FPR95 and  $13.17\%$  AUROC. GAIA-Z achieves surprising performance on CIFAR benchmarks by utilizing the zero-deflation abnormality.
Methods
Space	Methods	iNaturalist	SUN	Places	Textures	Average
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
Output	MSP [7]	63.69	87.59	79.98	78.34	81.44	76.76	82.73	74.45	76.96	79.29
ODIN [9]	62.69	89.36	71.67	83.92	76.27	80.67	81.31	76.30	72.99	82.56
Energy [5]	64.91	88.48	65.33	85.32	73.02	81.37	80.87	75.79	71.03	82.74
Feature	Mahalanobis [14]	96.34	46.33	88.43	65.20	89.75	64.46	52.23	72.10	81.69	62.02
ReAct [11]	44.52	91.81	52.71	90.16	62.66	87.83	70.73	76.85	57.66	86.67
KNN [29]	59.08	86.20	69.53	80.10	77.09	74.87	11.56	97.18	54.32	84.59
Rankfeat (Block4)[15]	46.54	81.49	27.88	92.18	38.26	88.34	46.06	89.33	39.69	87.84
Rankfeat (Block3+4)[15]	41.31	91.91	29.27	94.07	39.34	90.93	37.29	91.70	36.80	92.15
ASH-B@90 [30]	22.22	96.15	35.43	92.53	47.73	89.61	23.33	95.43	32.18	93.43
Gradient	GradNorm [10]	50.03	90.33	46.48	89.03	60.86	84.82	61.42	81.07	54.70	86.31
GAIA-A (Ours)	29.47	93.52	31.24	92.42	48.55	88.94	40.41	92.71	37.42	91.90
GAIA-Z (Ours)	65.09	84.15	64.23	84.31	71.02	81.16	11.32	97.93	52.92	86.89
Table 2: Main Results on ImageNet-1K [28]. OOD detection performance comparison between GAIA and advanced baselines on pre-trained Google BiT-S [38] model. Our methods only use layers from Block4 and all methods are post hoc that can be directly used for pre-trained models. The best results for each Methods Space are all in Bold.
Evaluation on ImageNet-1K benchmark. In Tab. 2, we compare GAIA with other post hoc baselines on pre-trained Google BiT-S model [38]. For our methods, both GAIA-A and GAIA-Z use layers from the last block (Block4), and no hyperparameters are required. GAIA-A performs well with an average FPR95 of  $37.42\%$  and an average AUROC of  $91.90\%$ . Compared to other gradient-based OOD methods, GAIA-A outperforms GradNorm by  $17.28\%$  in FPR95. Besides, GAIA-Z excels in handling the OOD dataset of textures with  $11.32\%$  FPR95, despite not achieving the best overall performance. While ASH achieves competitive results on the ImageNet dataset through careful parameter tuning, it is highly sensitive to its hyperparameters and lacks empirical parameters. In contrast, GAIA methods don't require parameter adjustments and directly achieve good results.
Our ablation study begins by validating the effectiveness of each step of the methods. We first verify the effect of the Frobenius norm (2-norm). Then we explore the aggregation's effectiveness on the label space and the input space.
Influence of Frobenius norm. In Eq. 12, we use  $\| \boldsymbol{\Lambda}\|_{F}$  to calculate the final OOD score. To verify its effectiveness, we evaluate different norms of  $\boldsymbol{\Lambda}$  on the above three benchmarks. As shown in Fig. 4, the Frobenius norm performs the best. Compared to 1-norm, Frobenius norm particularly demonstrates significant improvements. This is because the Frobenius norm can exclude the influence of numerous smaller values. As the number of layers in the model increases, the accumulation of insignificant small values in the shallow layers can weaken the scoring impact of extreme values OOD data. However, we can observe that as the value of  $p$  increases, the influence of extreme values will also be affected.
Figure 4: Ablation studies on Frobenius norm of matrix  $\Lambda$ .
Influence of label space aggregation. In GAIA-A, we employ division to fuse the inner component  $\mathbb{E}_{\mathrm{inner}}[\epsilon |A^{kl}]$  and the output component  $\mathbb{E}_{\mathrm{output}}[\epsilon |A^{kl}]$  to obtain the final OOD scores. As shown in Fig. 5, we visualize the score distributions of the individual components and the fused scores, and observe that the performance of the inner and output components in OOD and ID data are contrasting. After dividing and merging the two components, the fusion resulted in a greater concentration of ID data, tending towards a narrower distribution. However, the impact on the distribution of OOD data was relatively minor, thereby widening the score differences between them. In Tab. 3, we compare the
Figure 5: The distribution of the OOD scores in three settings (inner only, output only and fusion). All scores are non-negative for comparison.
OOD detection performance with and without (w/o) the fusion strategy. Experiments demonstrated a improvement with the implementation of this strategy.
Methods	iNaturalist	SUN	Places	Textures	Average
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
w/o fusion (top 1 label)	74.44	74.04	77.30	77.60	82.07	71.11	50.14	89.57	70.99	78.08
w/o fusion (output only)	47.50	92.54	69.87	84.47	74.52	82.00	76.17	78.83	67.02	84.46
w/o fusion (inner only)	59.45	81.98	52.24	86.51	62.74	80.20	52.45	89.77	56.72	84.62
fusion (logsoftmax + division)	29.47	93.52	31.24	92.42	48.55	88.94	40.41	92.71	37.42	91.90
Influence of input space aggregation across different layers (blocks). Given that both ResNet34 and Google BiT-S models have four blocks, we analyze the performance of our methods across different blocks to elucidate the influence of feature layers. As shown in Tab. 4, deeper layers possess a higher power in distinguishing between ID and OOD data. It indicates that as the network becomes shallower, the feature maps progressively contain a diminishing amount of relevant information w.r.t. the prediction decision [39]. For CIFAR benchmarks, information from Block3+4 is sufficient for detection, and for ImageNet-1K benchmark, only using Block4 can achieve the best performance.
Table 3: Ablation studies on fusion strategy. top 1 label means utilizing the predictive output only.
Blocks	CIFAR10	CIFAR100	ImageNet
GAIA-A	GAIA-Z	GAIA-A	GAIA-Z	GAIA-A	GAIA-Z
FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑
Block 1	64.52	83.28	57.15	67.06	77.89	79.72	58.38	66.46	86.59	61.26	92.38	49.62
Block 2	62.19	86.26	50.71	86.69	77.30	80.50	52.40	86.96	87.21	58.39	88.56	58.64
Block 3	44.56	91.07	22.17	95.71	71.28	84.67	44.18	89.01	63.34	80.81	73.28	78.96
Block 4	12.90	97.54	6.42	98.75	69.16	86.40	49.97	91.28	37.42	91.90	52.92	86.89
Block 3+4	12.70	97.53	3.55	99.26	68.98	86.42	27.86	95.24	41.91	91.03	58.39	86.91
All blocks	12.73	97.53	3.26	99.28	68.98	86.42	29.05	94.92	42.38	90.86	63.28	86.02
Table 4: Ablation studies of the influence on different blocks with average FPR95 and AUROC.
Among all attempts so far, post-hoc methods [5, 9-11, 14, 15] are preferable in the wild due to their advantages of being easy to use without modifying the training procedure and objective. An initial solution proposed by Hendrycks and Gimpel [7] utilizes maximum softmax probability (MSP). While due to the tendency of networks to display overconfident softmax scores when predicting OOD inputs [40, 41], it renders a non-trivial dilemma to separate ID and OOD data. Then ODIN [9] introduces temperature factors and input perturbations to enhance detection performance. In a different approach, Energy [5] is proposed to utilize the energy score as an informative indicator. ReAct [11] proposes that OOD examples result in abnormal model activation and suggests clamping the activation values above a threshold. Rankfeat [15] leverages the differences in singular value distributions, which still focuses on abnormal activations of the model. Another relevant study to this paper is gradient-based OOD detection. In the early work, ODIN [9] first implicitly utilizes gradients as perturbations to increase the softmax score of any given input. Recently, Lee and AlRegib [16], Huang et al. [10] and Igoe et al. [17] use the gradients of parameters as the measurement, which emphasizes the importance of the loss function. In this paper, we delve into investigating attribution abnormality and utilize attribution gradients for OOD detection.
In this section, we discuss the comparison of our methods with other gradient-based OOD detection methods, as well as the limitation on transformer-based models.
7.1 Comparison with Other Gradient-based Methods
A crucial distinction between other gradient-based OOD detection methods and ours lies in the utilization of attribution methods to interpret the anomalous behavior of OOD examples. Specifically, we investigate and aggregate the abnormal patterns exhibited by attribution gradients at the feature level. Compared to ODIN [9], GAIA directly leverages the uncertainty derived from the gradients of input features, providing a more intuitive and efficient solution. Furthermore, rather than focusing solely on the softmax output, we delve into the intermediate statistics to uncover more fundamental discrepancies. Compared to GradNorm [10], ExGrad [17] and Lee and AlRegib [16], our approaches focus on attribution gradients and demonstrate superior performance. The comparative performance is presented in Tab. 5. Additionally, GAIA supports batch processing, as the attribution gradients are independent for each input feature, while gradients of parameters are unique to the network. This means that our method can handle multiple samples simultaneously, providing a parallel processing advantage over these methods that can only process one sample at a time.
Methods	Batch processing	iNaturalist	SUN	Places	Textures	Average
AUROC ↑	AUROC ↑	AUROC ↑	AUROC ↑	AUROC ↑
Lee and AlRegib [16]	72.30	82.61	74.00	84.16	78.27
GradNorm [10]	90.33	89.03	84.82	81.07	86.31
ExGrad [17]	76.90	66.60	68.90	65.10	69.40
GAIA-A (Ours)	✓	93.52	92.42	88.94	92.71	91.90
GAIA-Z (Ours)	✓	84.15	84.31	81.16	97.93	86.89
Table 5: Comparison with other gradient-based methods. To ensure a fair comparison with Lee and AlRegib [16], the gradients of uniform noise are used as a surrogate, as suggested in [10].
7.2 Limitation on Transformer-based Models
Newer models like Vision Transformers (ViT) [42], which are based on transformers, excel in feature extraction. However, they may not align well with image-specific characteristics. For instance, ViTs employ positional encoding to capture spatial information, posing challenges for attribution. Due to this reason, existing attribution algorithms are rarely applied to ViTs, resulting in poorer performance for GAIA. While the attention mechanism in transformer-based models can also offer directions for visual explanations. In our future work, we will research the uncertainty in the attention matrix to enhance OOD detection performance on transformer-based models.
This paper targets bridging the gap between OOD detection and visual interpretation by utilizing the uncertainty of a model in explaining its own predictions. We further examine how attribution gradients contribute to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection. Then, we propose GAIA, a simple and effective framework for abnormality aggregation. The effectiveness of our framework is validated through experiments.
Societal impact and limitations. Through this work, we aim to provide a new perspective to improve the performance of OOD detection and ensure the safety and reliability of machine learning applications. However, the utilization of attribution gradients in this paper is relatively simplistic. We believe there is still significant research potential in this area. Moreover, the limitation on transformer-based models remains a topic for further investigation.
Research is supported by the Key Research and Development Program of Guangdong Province (grant No. 2021B0101400003). This work was done while Jinggang Chen was interning at Ping An Technology and the corresponding authors are Xiaoyang Qu and Jianzong Wang from Ping An Technology (Shenzhen) Co., Ltd.
[35] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[36] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[37] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[38] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, pages 491-507. Springer, 2020.
[39] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Computing Surveys, 51(5):1-42, 2018.
[40] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for recognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 427-436, 2015.
[41] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41-50, 2019.
[42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.