SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Performance Upper Bound Analysis We propose VRA and VRA+ to approximate the optimal operation for OOD detection. But is it necessary to design other functions to get a better approximation? To answer this question, we need to reveal whether  $g^{*}(\cdot)$  can reach the upper-bound performance. The core of estimating  $g^{*}(\cdot)$  is to estimate the probability density functions of  $p_{\mathrm{in}}$  and  $p_{\mathrm{out}}$ . To this end, we consider two ideal cases: VRA-True and VRA-Fake-True. In the first case, we assume that all ID and OOD data are known in advance; in the second case, we randomly select 1% of ID and OOD data from the entire dataset. Both cases leverage histograms to estimate  $p_{\mathrm{in}}$  and  $p_{\mathrm{out}}$  and use Eq. 13 to calculate  $g^{*}(\cdot)$ . Considering that histograms provide a piecewise form of  $g^{*}(\cdot)$ , we directly use the piecewise function to represent  $g^{*}(\cdot)$ . In Table 4, we observe that both ideal cases can achieve near-perfect results. Therefore,  $g^{*}(\cdot)$  that increases the gap between ID and OOD can generate more discriminative features for OOD detection. In the future, we will explore other functions that can better describe the optimal operation for better performance.
Parameter Sensitivity Analysis VRA uses two hyper-parameters  $(\eta_{\alpha}$  and  $\eta_{\beta})$  to adaptively adjust thresholds for low and high activations. In this section, we conduct parameter sensitivity analysis and reveal their impact on OOD detection. In Figure 2, we observe that our method does not perform well when  $\eta_{\alpha}$  and  $\eta_{\beta}$  are inappropriate. A large  $\eta_{\alpha}$  suppresses too many low activations, while a large  $\eta_{\beta}$  suppresses too few high activations. Therefore, it is necessary to choose proper  $\eta_{\alpha}$  and  $\eta_{\beta}$  for VRA.
Role of Adaptively Adjusted Strategy In this paper, we adopt an adaptive strategy to automatically determine  $\alpha$  and  $\beta$ . To verify its effectiveness, we compare this adaptive strategy with another strategy that uses fixed  $\alpha$  and  $\beta$  for different features. To determine these hyper-parameters, we use Gaussian
Table 5: Role of adaptively adjusted strategy. We use DenseNet-101 [28] for CIFAR.
ID	Strategy	Hyper-parameters	OOD Performance
α	β	ηα	ηβ	FPR95 ↓	AUROC ↑
CIFAR-10	assign α, β	0.50	1.50	-	-	19.44	96.34
assign ηα, ηβ	-	-	0.60	0.95	17.74	96.47
CIFAR-100	assign α, β	0.50	1.50	-	-	56.35	86.09
assign ηα, ηβ	-	-	0.60	0.95	47.12	90.21
Table 6: Compatibility with different backbones. All methods are pretrained on ImageNet.
Backbone	MSP	Energy	ReAct+Energy	VRA+Energy
FPR95 ↓ AUROC ↑	FPR95 ↓ AUROC ↑	FPR95 ↓ AUROC ↑	FPR95 ↓ AUROC ↑
ResNet-18 [29]	69.70	80.61	58.59	80.40	36.36	92.17	34.87	92.58
ResNet-34 [29]	68.84	81.19	57.20	86.84	32.23	93.08	30.63	93.46
ResNet-50 [29]	66.95	81.99	58.40	86.17	31.43	92.95	25.49	94.57
ResNet-101 [29]	64.70	82.47	54.84	87.29	31.68	93.03	25.80	94.36
ResNet-152 [29]	61.35	83.74	50.39	88.61	26.57	94.22	22.21	95.20
VGG-16 [33]	67.94	81.60	54.33	88.17	67.81	83.68	32.99	92.59
VGG-16-BN [33]	65.92	82.00	50.49	89.03	59.02	86.34	35.12	92.05
EfficientNetV2 [34]	57.57	83.96	75.29	71.10	48.28	88.01	43.81	89.76
RegNet [35]	65.37	82.85	59.46	85.51	34.65	92.53	26.18	94.55
MobileNetV3 [36]	67.99	82.14	60.49	87.80	60.72	87.82	56.65	89.30
noise images as the validation set, in line with previous works [8]. Experimental results in Table 5 demonstrate that our adaptive strategy outperforms this fixed strategy. The reason lies in that different features have distinct statistical distributions. Using fixed thresholds for different features will limit the performance of OOD detection.
Compatibility with Backbones In this section, we further verify the compatibility of our method with different backbones. For a fair comparison, all methods are pretrained on ImageNet, and we report the average results on four OOD datasets of ImageNet. Compared with competitive post-hoc strategies, experimental results in Table 6 demonstrate that our method can achieve the best performance under different network architectures. These results validate the effectiveness and compatibility of our method. Meanwhile, we observe some interesting phenomena in Table 6. ReAct [8] points out that mismatched BatchNorm [31] statistics between ID and OOD lead to model overconfidence on OOD data. In Table 6, VGG-16 and VGG-16-BN refer to models without and with BatchNorm, respectively. We observe that no matter with or without BatchNorm, ReAct cannot achieve better performance than Energy, consistent with previous findings [32]. Therefore, BatchNorm may not be the only reason for model overconfidence, and the network architecture also matters. Furthermore, Energy [23] generally outperforms MSP [20] with the exception of EfficientNetV2, which also reveals its limitation in compatibility. In the future, we will conduct an in-depth analysis to reveal the reasons behind these phenomena.
4 Further Analysis
Combining features with logit outputs can achieve better performance in OOD detection [37]. Therefore, we design another variant of VRA called VRA++ whose scoring function is defined as:
$$
\lambda_ {v} \sum_ {i = 1} ^ {m} g \left(z _ {i}\right) + \log \sum_ {i = 1} ^ {c} e ^ {l _ {i}}, \tag {16}
$$
where  $z_{i}, i \in [1, m]$  represents the  $i$ -th feature and  $l_{i}, i \in [1, c]$  represents the  $i$ -th logit output. This scoring function consists of two items: (1) Since we have maximized the gap between ID and OOD  $\mathbb{E}_{\mathrm{in}}[g(z_i)] - \mathbb{E}_{\mathrm{out}}[g(z_i)]$ , we directly use the sum of all rectified features  $\sum_{i=1}^{m} g(z_i)$  as the indicator; (2) We also calculate the energy score on logit outputs for OOD detection. These items are combined using a balancing factor  $\lambda_v$ . Unlike VRA using piecewise functions, we further test the performance of the quadratic function  $g(z) = -z^2 + \alpha_v z$ . By choosing a proper  $\alpha_v$ , this quadratic function can
Table 7: Performance of VRA++. All methods are based on BiT [39] and pretrained on ImageNet.
Method	OpenImage-O	Texture	iNaturalist	ImageNet-O	Average
FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑
MSP [20]	73.72	84.16	76.65	79.80	64.09	87.92	96.85	57.12	77.83	77.25
ODIN [21]	72.83	85.64	74.07	81.60	70.75	86.73	96.85	63.00	78.63	79.24
Mahalanobis [22]	64.32	83.10	14.05	97.33	64.95	85.70	70.05	80.37	53.34	86.63
Energy [23]	73.42	84.77	73.91	81.09	74.98	84.47	96.40	63.59	79.68	78.48
ReAct [8]	54.97	88.94	50.25	90.64	48.60	91.45	91.70	67.07	61.38	84.52
ViM [37]	43.96	91.54	04.69	98.92	55.71	89.30	61.50	83.87	41.47	90.91
VRA++	34.94	93.55	05.02	98.76	22.25	96.37	60.45	84.21	30.67	93.22
Table 8: Comparison of VRA variants. "Net1" and "Net2" refer to ResNet-50 and ResNetv2-101.
Method	CIFAR-10	CIFAR-100	ImageNet (Net1)	ImageNet (Net2)
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
VRA	17.74	96.47	47.12	90.21	25.49	94.57	34.95	93.34
VRA+	15.89	96.90	43.31	90.61	23.32	94.96	30.85	93.78
VRA++	15.52	96.87	35.20	91.80	18.63	95.75	25.92	94.60
also simulate suppression and amplification operations. Finally, our scoring function is defined as:
$$
- \lambda_ {v} \sum_ {i = 1} ^ {m} \left(z _ {i} ^ {2} - \alpha_ {v} z _ {i}\right) + \log \sum_ {i = 1} ^ {c} e ^ {l _ {i}}. \tag {17}
$$
Among all methods, ViM [37] is a powerful strategy that combines features and logit outputs. For a fair comparison with ViM, we use the same ID data (ImageNet), OOD data (OpenImage-O [37], Texture [12], iNaturalist [18], and ImageNet-O [38]), and network architecture (BiT [39]). Experimental results in Table 7 demonstrate that VRA++ achieves better performance than ViM, verifying the scalability and high potential of our method. Meanwhile, VRA++ generally achieves the best performance among all variants (see Table 8). These results further demonstrate the necessity of combining features and logit outputs in OOD detection.
Post-hoc Method Post-hoc strategies are an important branch of OOD detection. Due to their ease of implementation, they have attracted increasing attention from researchers. Among them, MSP [20] was the most basic post-hoc strategy, which directly leveraged the maximum value of the posterior distribution as the indicator. Since then, researchers have proposed various post-hoc approaches. For example, ODIN [21] used temperature scaling and input perturbations to improve the separability of ID and OOD data. Energy [23] replaced the softmax confidence score in MSP [20] with the theoretically guaranteed energy score. Mahalanobis [22] used the minimum distance from the class centers to identify OOD data. KNN [24] was a nonparametric method that explored K-nearest neighbors. More recently, researchers have found that the reason behind model overconfidence in OOD data lies in abnormally high activations of a small number of neurons. To address this, Dice [10] used weight sparsification, while ReAct [8] exploited activation truncation. Different from these works, we further demonstrate that abnormally low activations also affect OOD detection performance. This motivates us to propose VRA to rectify the activation function.
Activation Function Activation functions are an important part of neural networks [40, 41]. Previously, researchers found that neural networks with the ReLU activation function produced abnormally high activations for inputs far from the training data, harming the reliability of deployed systems [42]. To address this problem, ReAct used a truncation operation to rectify activation functions. In this paper, we propose a more powerful rectified activation function for OOD detection. Experimental results on multiple benchmark datasets demonstrate the effectiveness of our method.
Variational Method The variational method is often used to solve for the functional extreme value. Its most famous application in neural networks is the variational autoencoder [43], which solves for the functional extreme value by trading off reconstruction loss and Kullback-Leibler divergence. It
has also been applied to other complex scenarios [44] and multimodal tasks [45]. In this paper, we use the variational method to find the operation that can maximize the gap between ID and OOD.
Distributions of ID and OOD data impact the performance of VRA. In the future, we will conduct a theoretical analysis to explain the reason behind this phenomenon. Meanwhile, analogous to previous works such as ReAct and ASH, this paper mainly focuses on the pre-trained classifiers with ReLU-based activations. Although we have explored some other architectures in the appendix, future experiments in more structures are also needed.
In this paper, we treat  $\max_g\mathbb{E}_{\mathrm{in}}[g(z)] - \mathbb{E}_{\mathrm{out}}[g(z)]$  as the core objective function derived from ReAct and  $\min_g\mathbb{E}_{\mathrm{in}}[(g(z) - z)^2 ]$  as the regularization term. However, there may be better regularization terms that can not only guarantee the existence of the optimal solution but also ensure that the expression of the optimal solution is easy to implement and has good interpretability. Therefore, we will explore other regularization terms for OOD detection. Meanwhile, this paper uses simple piecewise functions to approximate the complex optimal operation. In the future, we will explore other functional forms that can better describe the optimal operation.
This paper proposes a post-hoc OOD detection strategy called VRA. From the perspective of the variational method, we find the theoretically optimal operation for maximizing the gap between ID and OOD. This operation reveals the necessity of suppressing abnormally low and high activations and amplifying intermediate activations in OOD detection. Therefore, we propose VRA to mimic these suppression and amplification operations. Experimental results show that our method outperforms existing post-hoc strategies and is compatible with different scoring functions and network architectures. In the ideal case of knowing a small fraction of OOD samples, we can achieve near-perfect performance, demonstrating the strong potential of our method. Meanwhile, we verify the effectiveness of our adaptively adjusted strategy and reveal the impact of different hyper-parameters.
This work is supported by the National Natural Science Foundation of China (No.62201572, No.61831022, No.62276259, No.U21B2010, No.62271083), Beijing Municipal Science & Technology Commission, Administrative Commission of Zhongguancun Science Park (No.Z211100004821013), Open Research Projects of Zhejiang Lab (NO. 2021KH0AB06), CCF-Baidu Open Fund (No.OF2022025).
[49] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[50] Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue, Xiang Tian, Yaowu Chen, et al. Boosting out-of-distribution detection with typical features. Advances in Neural Information Processing Systems, 35:20758-20769, 2022.
[51] Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection. Advances in Neural Information Processing Systems, 35:17885–17898, 2022.
A Contrastive Learning Trained Backbones
Many post-hoc methods are conducted on classification-based backbones [8, 27]. In this section, we further perform experiments on backbones trained on contrastive learning. We use the pre-trained model from CIDER[46] and apply VRA to the normalized features, followed by KNN[24] for OOD detection. In Table 9, we observe that for the contrastive-learning-based backbone, modifying the feature values and suppressing larger and smaller activation values are also beneficial to OOD detection.
Table 9: Comparison with contrastive learning trained backbones (SSD+[47], CSI[48], and CIDER). We use ResNet-34 for CIFAR-100. Results of SSD+, CSI and CIDER are reported from CIDER [46].
Method	SVHN	Places365	LSUN	iSUN	Textures	Average
FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑
SSD+ [1]	31.2	94.2	77.7	79.9	79.4	85.2	80.9	84.1	66.6	86.2	67.2	85.9
CSI [2]	44.5	92.7	79.1	76.3	75.6	83.8	76.6	85.0	61.6	86.5	67.5	84.8
CIDER [3]	23.1	95.2	79.6	73.4	16.2	96.3	71.7	83.0	43.9	90.4	46.9	87.7
CIDER+VRA	20.5	95.5	78.9	71.0	11.8	97.4	61.3	83.5	36.4	92.1	41.9	87.9
B Compatibility with VIT[49]
Previous works [8, 10, 27] usually only consider the condition where the activation values are nonnegative. To further illustrate the potential of VRA, we conduct experiments on VIT [49] with a large number of negative activation values.
For positive values, ReAct tries to increase  $\mathbb{E}_{\mathrm{in}}[g(z)] - \mathbb{E}_{\mathrm{out}}[g(z)]$ . It uses an operation to make OOD data suppress more than ID data. Therefore, for negative values, we should also make OOD data suppress more than ID data. This process results in an increase of  $\mathbb{E}_{\mathrm{in}}[-g(z)] - \mathbb{E}_{\mathrm{out}}[-g(z)]$ . To unify the positive and negative cases, we should maximize  $\operatorname{sgn}(z)(\mathbb{E}_{\mathrm{in}}[g(z)] - \mathbb{E}_{\mathrm{out}}[g(z)])$ , where  $\operatorname{sgn}(\cdot)$  is the sign function. Similar to Eq. 3~13, we can get the optimal activation:
$$
g ^ {*} (z) = z + \lambda \operatorname {s g n} (z) \left(1 - \frac {p _ {\text {o u t}} (z)}{p _ {\text {i n}} (z)}\right).
$$
(a) empirical PDF on ImageNet (VIT)
(b) activation functions on ImageNet (VIT)
Figure 3: Empirical PDFs for  $p_{\mathrm{in}}(\cdot)$  and  $p_{\mathrm{out}}(\cdot)$  and visualization of different activation functions. We use VIT for ImageNet.
We visualize the optimal function  $g^{*}(\cdot)$  on different ID and OOD data in Figure 3. We observe a different  $g^{*}(\cdot)$  in VIT compared to ReLU-based backbones. Specifically, we should amplify activations with low absolute features and suppress activations with high absolute features. To mimic this operation, we design a new piecewise function called VRA-VIT:
$$
\operatorname {V R A - V I T} (z) = \left\{ \begin{array}{l} - \alpha , z \leq - \alpha \\ \beta z, - \alpha \leq z \leq \alpha \\ \alpha , z \geq \alpha \end{array} \right.,
$$
Table 10: Compatibility with VIT. We conduct experiments on ImageNet.
Backbone	Method	iNaturalist	SUN	Places	Textures	Average
FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑
VIT-B-16	Energy	64.08	79.24	72.77	70.24	74.31	68.44	58.47	79.30	67.41	74.30
Energy + ReAct	61.52	85.66	70.87	77.40	72.16	75.84	55.41	84.05	64.99	80.74
Energy + VRA-VIT	55.80	89.96	66.42	84.02	68.29	82.46	53.44	86.57	60.99	85.76
VIT-B-32	Energy	74.16	81.73	81.72	72.58	81.77	71.47	69.50	79.81	76.69	76.40
Energy + ReAct	73.39	84.23	80.08	76.01	80.63	74.69	66.77	82.73	75.22	79.41
Energy + VRA-VIT	65.49	88.03	76.88	80.63	76.49	79.59	65.98	84.20	71.21	83.11
VIT-L-16	Energy	60.16	80.31	75.23	69.77	78.21	69.02	60.30	79.25	68.48	74.59
Energy + ReAct	55.15	87.56	77.21	77.65	73.98	77.07	58.71	84.37	64.99	81.67
Energy + VRA-VIT	57.24	90.49	71.07	82.84	71.44	82.31	62.45	86.16	65.55	85.45
VIT-L-32	Energy	69.15	78.43	76.32	70.71	77.86	69.39	65.53	77.41	72.22	73.98
Energy + ReAct	68.54	83.12	74.58	76.51	76.32	75.09	63.00	83.12	70.61	79.09
Energy + VRA-VIT	61.00	89.61	71.21	83.14	71.65	82.02	62.70	84.82	66.64	84.75
where  $\alpha > 0$  controls the threshold for determining low and high activations, and  $\beta > 0$  controls the gradient. Results in Table 10 show the effectiveness of VRA, where we set  $\alpha = 1$  and  $\beta = 1.5$ .
C More variance of VRA
The theoretical derivation of VRA and analysis of ID and OOD activations allow us to design other variants of VRA. In this section, we denote these new variants as VRA-B (binary) and VRA-G (change the gradient of the medium):
$$
\operatorname {V R A - B} (z) = \left\{ \begin{array}{l} 0, z \leq \alpha \\ \alpha , z > \alpha \end{array} \right.,
$$
$$
\operatorname {V R A - G} (z) = \left\{ \begin{array}{l} 0, z <   \alpha \\ k z, \alpha \leq z \leq \beta \\ \beta , z > \beta \end{array} \right.,
$$
where  $\alpha$  and  $\beta$  are thresholds and  $k$  controls the scaling factor. For VRA-G, we treat the  $\eta_{\alpha}$ -quantile (or  $\eta_{\beta}$ -quantile) of activations estimated on ID data as  $\alpha$  (or  $\beta$ ), in line with VRA. In Table 11, we investigate the performance of different VRA-based variants. For a fair comparison, we use the same ID data, OOD data, and backbone. Experimental results demonstrate that VRA++ outperforms other variants in OOD detection.
Table 11: Performance of different VRA-based variants. We use DenseNet-101 for CIFRA-10.
Method	CIFAR-10
FPR95 ↓	AUROC ↑
VRA-B (α = 0.5)	30.01	94.02
VRA-B (α = 0.6)	20.37	95.84
VRA-B (α = 0.7)	21.18	95.64
VRA-B (α = 0.8)	35.84	93.31
VRA-G (k = 0.5)	28.14	94.12
VRA-G (k = 0.8)	19.62	96.11
VRA-G (k = 1.0)	17.74	96.47
VRA-G (k = 1.2)	17.32	96.64
VRA-G (k = 1.5)	16.81	96.66
VRA-G (k = 2.0)	17.05	96.56
D Discussion with BATs[50] and RankFeat[51]
BATS[50] is an OOD detection method that relies on BatchNorm. The main difference between BATs and our VRA lies in the following aspects. (1) BATs exploits the phenomenon that the features of the
training data more frequently fall in the interval  $[\mu -\lambda \sigma ,\mu +\lambda \sigma ]$  . Differently, VRA does not depend on BatchNorm. It tries to find the theoretically optimal operation to maximize the gap between ID and OOD. (2) BATs relies on BatchNorm and uses the mean and standard deviation in BatchNorm to determine thresholds for low and high activations. Differently, VRA does not rely on BatchNorm and determines thresholds for low and high activations using quantiles  $\eta_{\alpha}$  and  $\eta_{\beta}$  . Experimental results in Table 6 show that VRA can achieve good performance no matter with or without BatchNorm (see VGG-16 and VGG-16-BN). Therefore, our VRA has better compatibility with different backbones compared with BATs. (3) BATs only suppresses high and low activations, but  $\mathrm{VRA + }$  and  $\mathrm{VRA + + }$  further amplify intermediate activations. According to Table 12, this amplification process further improves the OOD detection performance.
Table 12: Comparison with BATs. For ImageNet, we use the pretrained ResNet50. For CIFAR, we use the pre-trained DenseNet-121.
Method	CIFAR-10	CIFAR-100	ImageNet
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
BATS	24.30	95.32	59.32	86.79	27.11	94.28
VRA	17.74	96.47	47.12	90.21	25.49	94.57
VRA+	15.89	96.90	43.31	90.61	23.32	94.96
VRA++	15.52	96.87	35.20	91.80	18.63	95.75
RankFeat [51] finds that the feature maps of OOD samples often have large singular values, and removing these values are beneficial to OOD detection. We compare VRA with RankFeat, and the experimental results are shown in Table 13. We observe that our method has a significant improvement compared to RankFeat, especially on the OOD datasets iNaturalist and Texture.
Table 13: Comparison with RankFeat. We use ResNetv2-101 for ImageNet. The result of RankFeat is reported from RankFeat[51].
Method	iNaturalist	SUN	Places	Textures	Average
FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑
RankFeat (Block 4)	46.54	81.49	27.88	92.18	38.26	88.34	46.06	89.33	39.69	87.84
RankFeat (Block 3)	49.61	91.42	39.91	92.01	51.82	88.32	41.84	91.44	45.80	90.80
RankFeat (Block 3+4)	41.31	91.91	29.27	94.07	39.34	90.93	37.29	91.70	36.80	92.15
VRA	27.26	95.68	34.53	93.27	47.31	90.19	30.69	94.22	34.95	93.34
VRA+	20.81	97.70	32.89	92.68	45.83	90.01	23.88	95.43	30.85	93.71
E Upper Bound Analysis
Similar to RankFeat, we provide theoretical analysis from the perspective of boundary analysis. Consider output logits  $f(\mathbf{z}) = \mathbf{W}\mathbf{z} + \mathbf{b}$ , where  $\mathbf{z}$  is the feature vector of the penultimate layer. We denote the  $i^{th}$  element in  $f(\mathbf{z})$  as  $f_{i}$ , and  $c$  is the number of categories. We compute the energy score of the output logits  $f(\mathbf{z})$  as the OOD score, consistent with RankFeat:
$$
S (\mathbf {z}) = \log \sum_ {i = 1} ^ {c} e ^ {f _ {i}} = \log \sum_ {i = 1} ^ {c} e ^ {f _ {i} - \max _ {i = 1} ^ {c} f _ {i}} + \max _ {i = 1} ^ {c} f _ {i} \leq \log c + \| f (\mathbf {z}) \| _ {\infty},
$$
For any norm  $\| \cdot \| _p(p > 0)$ , there exist  $k_{1} > 0$  and  $k_{2} > 0$  satisfying:
$$
k _ {1} \| f (\mathbf {z}) \| _ {p} \leq \| f (\mathbf {z}) \| _ {\infty} \leq k _ {2} \| f (\mathbf {z}) \| _ {p}.
$$
Then, we get:
$$
S (\mathbf {z}) \leq k _ {2} \| \mathbf {W} \mathbf {z} + \mathbf {b} \| _ {p} + \log c.
$$
According to the triangle inequality  $\| \mathbf{W}\mathbf{z} + \mathbf{b}\| _p\leq \| \mathbf{W}\mathbf{z}\| _p + \| \mathbf{b}\| _p$  and consistence of matrix norms  $\| \mathbf{W}\mathbf{z}\| _p\leq \| \mathbf{W}\| _p\| \mathbf{z}\| _p$ , we can get:
$$
S (\mathbf {z}) \leq k _ {2} \| \mathbf {W} \| _ {p} \| \mathbf {z} \| _ {p} + k _ {2} \| \mathbf {b} \| _ {p} + \log c.
$$
According to the above inequality, maximizing  $\mathbb{E}_{\mathrm{in}}[z] - \mathbb{E}_{\mathrm{out}}[z]$  is equivalent to maximizing the upper bound gap between ID and OOD. Therefore, the basic idea of VRA is similar to RankFeat. Specifically, RankFeat removes the rank-1 matrix from the high-level feature and increases the upper bound gap between ID and OOD; VRA maximizes  $\mathbb{E}_{\mathrm{in}}[z] - \mathbb{E}_{\mathrm{out}}[z]$  to maximize the upper bound gap between ID and OOD. Therefore, we can prove VRA from the similar perspective of RankFeat.
F Visualization on CIFAR
Similar to Figure 1, we plot the activation distribution of ID and OOD data for the DenseNet-101 trained on CIFAR. As shown in Figure 4, we observe a similar distribution pattern to the ResNet-50 trained on ImageNet. Our designed modification function also exhibits the property of suppressing both ends and amplifying in the middle.
(a) empirical PDF on CIFAR-10
(b) empirical PDF on CIFAR-100
(c) activation functions on CIFAR-10
(d) activation functions on CIFAR-100
Figure 4: Empirical PDFs for  $p_{\mathrm{in}}(\cdot)$  and  $p_{\mathrm{out}}(\cdot)$  and visualization of different activation functions. We treat CIFAR as ID data and use DenseNet-101 as the backbone.
G Distribution of Energy Score