SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mingyu Xu $^{1,2}$ , Zheng Lian $^{1*}$ , Bin Liu $^{1,2}$ , Jianhua Tao $^{3,4}$
<sup>4</sup>Beijing National Research Center for Information Science and Technology, Tsinghua University {xumingyu2021, lianzheng2016}@ia.ac.cn
Out-of-distribution (OOD) detection is critical to building reliable machine learning systems in the open world. Researchers have proposed various strategies to reduce model overconfidence on OOD data. Among them, ReAct is a typical and effective technique to deal with model overconfidence, which truncates high activations to increase the gap between in-distribution and OOD. Despite its promising results, is this technique the best choice? To answer this question, we leverage the variational method to find the optimal operation and verify the necessity of suppressing abnormally low and high activations and amplifying intermediate activations in OOD detection, rather than focusing only on high activations like ReAct. This motivates us to propose a novel technique called "Variational Rectified Activation (VRA)", which simulates these suppression and amplification operations using piecewise functions. Experimental results on multiple benchmark datasets demonstrate that our method outperforms existing post-hoc strategies. Meanwhile, VRA is compatible with different scoring functions and network architectures. Our code is available at https://github.com/zeroQiaoba/VRA.
Systems deployed in the real world often encounter out-of-distribution (OOD) data, i.e., samples from an irrelevant distribution whose label set has no interaction with the training data. Most of the existing systems tend to generate overconfident estimations for OOD data, seriously affecting their reliability [1]. Therefore, researchers propose the OOD detection task, which aims to determine whether a sample comes from in-distribution (ID) or OOD. This task allows the model to reject recognition when faced with unfamiliar samples. Considering its importance, OOD detection has attracted increasing attention from researchers and has been applied to many fields with high-safety requirements such as autonomous driving [2] and medical diagnosis [3].
In OOD detection, existing methods can be roughly divided into two categories: methods requiring training and post-hoc strategies. The first category identifies OOD data by training-time regularization [4, 5] or external OOD samples [6, 7]. But they require more computational resources and are inconvenient in practical applications. To this end, researchers propose post-hoc strategies that directly use pretrained models for OOD detection. Due to their ease of implementation, these methods have attracted increasing attention in recent years. Among them, React [8] is a typical post-hoc strategy that truncates abnormally high activations to increase the gap between ID and OOD, thereby improving detection performance. But is this operation the best choice for widening the gap?
To answer this question, we use the variational method to solve for the optimal operation. Based on this operation, we reveal the necessity of suppressing abnormally low and high activations and amplifying intermediate activations in OOD detection. Then, we propose a simple yet effective strategy called "Variational Rectified Activation (VRA)", which mimics suppression and amplification operations using piecewise functions. To verify its effectiveness, we conduct experiments on multiple benchmark datasets, including CIFAR-10, CIFAR-100, and the more challenging ImageNet. Experimental results demonstrate that our method outperforms existing post-hoc strategies, setting new state-of-the-art records. The main contributions of this paper can be summarized as follows:
- (Theory) From the perspective of the variational method, we find the best operation for maximizing the gap between ID and OOD. This operation verifies the necessity of suppressing abnormally low and high activations and amplifying intermediate activations.
- (Method) We propose a simple yet effective post-hoc strategy called VRA. Our method is compatible with various scoring functions and network architectures.
- (Performance) Experimental results on benchmark datasets demonstrate the effectiveness of our method. VRA is superior to existing post-hoc strategies in OOD detection.
Let  $\mathcal{X}$  be the input space and  $\mathcal{Y}$  be the label space with  $c$  distinct categories. Consider a supervised classification task on a dataset containing  $N$  labeled samples  $\{\mathbf{x},y\}$ , where  $y\in \mathcal{Y}$  is the ground-truth label for the sample  $\mathbf{x}\in \mathcal{X}$ . Ideally, all test samples come from the same distribution as the training data. But in practice, the test sample may come from an unknown distribution, such as an irrelevant distribution whose label set has no intersection with  $\mathcal{Y}$ . In this paper, we use  $p_{\mathrm{in}}$  to represent the marginal distribution of  $\mathcal{X}$  and  $p_{\mathrm{out}}$  to represent the distribution of OOD data. In this paper, we aim to determine whether a sample comes from ID or OOD.
Among all methods, ReAct is a typical and effective post-hoc strategy [8]. Suppose  $h(\mathbf{x}) = \{z_i\}_{i=1}^m$  is the feature vector of the penultimate layer and  $m$  denotes the feature dimension. For convenience, we use  $z$  as shorthand for  $z_i$ . ReAct truncates activations above a threshold  $c$  for each  $z$ :
$$
g (z) = \min  (z, c), \tag {1}
$$
where  $c = \infty$  is equivalent to the model without truncation. ReAct has demonstrated that this truncation operation can increase the gap between ID and OOD [8]:
$$
\mathbb {E} _ {\text {i n}} [ g (z) ] - \mathbb {E} _ {\text {o u t}} [ g (z) ] \geq \mathbb {E} _ {\text {i n}} [ z ] - \mathbb {E} _ {\text {o u t}} [ z ]. \tag {2}
$$
Despite its promising results, is this strategy the best option for widening the gap between ID and OOD? In this paper, we attempt to answer this question with the help of the variational method.
2.3 VRA Framework
To find the best operation, we should optimize the following objectives:
- Maximize the gap between ID and OOD.
- Minimize the modification brought by the operation to maximally preserve the input.
The final objective function is calculated as follows:
$$
\min  _ {g} \mathcal {L} (g) = \mathbb {E} _ {\text {i n}} [ (g (z) - z) ^ {2} ] - 2 \lambda (\mathbb {E} _ {\text {i n}} [ g (z) ] - \mathbb {E} _ {\text {o u t}} [ g (z) ]), \tag {3}
$$
where  $\lambda$  controls the trade-off between two losses. To solve for Eq. 3, we first make a mild assumption to ensure the function space  $\mathcal{G}$  is sufficiently complex.
Assumption 1 We assume  $\mathbb{E}_{in}[|z|]$ ,  $\mathbb{E}_{out}[|z|]$ ,  $\mathbb{E}_{in}[z^2]$ , and  $\mathbb{E}_{out}[z^2]$  exist. Let  $\mathcal{G}$  be a Hilbert space:
$$
\mathcal {G} = \left\{g (z) \mid \mathbb {E} _ {i n} [ | g (z) | ], \mathbb {E} _ {o u t} [ | g (z) | ], \mathbb {E} _ {i n} [ g (z) ^ {2} ], \mathbb {E} _ {o u t} [ g (z) ^ {2} ] <   \infty \right\}. \tag {4}
$$
This space is sufficiently complex containing most functions, such as identity functions, constant functions, and all bounded continuous functions. Then, we define the inner product of  $\mathcal{G}$  as follows:
$$
<   g _ {a} (z), g _ {b} (z) > = \int g _ {a} (z) g _ {b} (z) p _ {i n} (z) d z. \tag {5}
$$
Combining this assumption, the equivalent equation of Eq. 3 is:
$$
\min  _ {g \in \mathcal {G}} \mathcal {L} (g) = \int \left(g (z) - z\right) ^ {2} p _ {\text {i n}} (z) - 2 \lambda g (z) \left(p _ {\text {i n}} (z) - p _ {\text {o u t}} (z)\right) d z. \tag {6}
$$
Then, we leverage the variational method to solve for the functional extreme value. We mark  $g^{*}(\cdot)$  as the optimal solution.  $\forall f(\cdot)\in \mathcal{G}$  and  $\forall \epsilon \in \mathbb{R}$ , we then have:
$$
\mathcal {L} \left(g ^ {*}\right) \leq \mathcal {L} \left(g ^ {*} + \epsilon f\right). \tag {7}
$$
$$
\begin{array}{l} \int \left(g ^ {*} (z) - z\right) ^ {2} p _ {\text {i n}} (z) - 2 \lambda g ^ {*} (z) \left(p _ {\text {i n}} (z) - p _ {\text {o u t}} (z)\right) d z (8) \\ \leq \int \left(g ^ {*} (z) + \epsilon f (z) - z\right) ^ {2} p _ {\text {i n}} (z) - 2 \lambda \left(g ^ {*} (z) + \epsilon f (z)\right) \left(p _ {\text {i n}} (z) - p _ {\text {o u t}} (z)\right) d z. (9) \\ \end{array}
$$
$$
\epsilon^ {2} \int f ^ {2} (z) p _ {\text {i n}} (z) d z + 2 \epsilon \int f (z) \left(g ^ {*} (z) - z - \lambda \left(1 - \frac {p _ {\text {o u t}} (z)}{p _ {\text {i n}} (z)}\right)\right) p _ {\text {i n}} (z) d z \geq 0. \tag {10}
$$
Combining Assumption 1 and the arbitrariness of  $\epsilon$ , we can get:
$$
\int f (z) \left(g ^ {*} (z) - z - \lambda \left(1 - \frac {p _ {\text {o u t}} (z)}{p _ {\text {i n}} (z)}\right)\right) p _ {\text {i n}} (z) d z = 0. \tag {11}
$$
Considering Assumption 1 and the arbitrariness of  $f(z)$ , we have:
$$
g ^ {*} (z) - z - \lambda \left(1 - \frac {p _ {\text {o u t}} (z)}{p _ {\text {i n}} (z)}\right) = 0. \tag {12}
$$
Therefore, the optimal activation function is:
$$
g ^ {*} (z) = z + \lambda \left(1 - \frac {p _ {\text {o u t}} (z)}{p _ {\text {i n}} (z)}\right). \tag {13}
$$
To verify its effectiveness, we compare the optimal function  $g^{*}(\cdot)$  with the unrectified function  $g(z) = z$ . Since  $g^{*}(\cdot)$  is the optimal solution, it should get a smaller value in Eq. 3:
$$
\mathbb {E} _ {\text {i n}} \left[ (g ^ {*} (z) - z) ^ {2} \right] - 2 \lambda \left(\mathbb {E} _ {\text {i n}} [ g ^ {*} (z) ] - \mathbb {E} _ {\text {o u t}} [ g ^ {*} (z) ]\right) \leq \mathbb {E} _ {\text {i n}} \left[ (z - z) ^ {2} \right] - 2 \lambda \left(\mathbb {E} _ {\text {i n}} [ z ] - \mathbb {E} _ {\text {o u t}} [ z ]\right). \tag {14}
$$
The equivalent equation of Eq. 14 is:
$$
\left(\mathbb {E} _ {\text {i n}} \left[ g ^ {*} (z) \right] - \mathbb {E} _ {\text {o u t}} \left[ g ^ {*} (z) \right]\right) - \left(\mathbb {E} _ {\text {i n}} [ z ] - \mathbb {E} _ {\text {o u t}} [ z ]\right) \geq \frac {1}{2 \lambda} \mathbb {E} _ {\text {i n}} \left[ \left(g ^ {*} (z) - z\right) ^ {2} \right]. \tag {15}
$$
It shows that  $g^{*}(\cdot)$  enlarges the gap between ID and OOD by at least  $\frac{1}{2\lambda}\mathbb{E}_{\mathrm{in}}[(g^{*}(z) - z)^{2}]\geq 0$
2.4 Practical Implementations
Through theoretical analysis, we have found the optimal operation  $g^{*}(\cdot)$  that can maximize the gap between ID and OOD. But in practice, this operation depends on the specific expressions of  $p_{\mathrm{in}}$  and  $p_{\mathrm{out}}$ . Estimating these expressions is a challenging task given that OOD data comes from unknown distributions [9]. This drives us to look for more practical implementations.
(a) empirical PDF on iNaturalist
(b) empirical PDF on SUN
(c) empirical PDF on Places
(d) activation functions on iNaturalist
(e) activation functions on SUN
(f) activation functions on Places
Figure 1: Empirical PDFs for  $p_{\mathrm{in}}(\cdot)$  and  $p_{\mathrm{out}}(\cdot)$ , and visualization of different activation functions. We treat ImageNet as ID data and select multiple OOD datasets for visualization.
For this purpose, we treat ImageNet as ID data and select multiple OOD datasets. We first use histograms to approximate the probability density functions of  $p_{\mathrm{in}}$  and  $p_{\mathrm{out}}$ . Then, we compute  $g^{*}(\cdot)$  and compare it with ReAct, whose threshold is set to the  $90^{th}$  percentile of activations estimated on ID data, consistent with the original paper [8]. Experimental results are shown in Figure 1. Compared with the model without truncation, we observe that ReAct suppresses high activations (see Figure 1(d)~1(f)). Unlike ReAct, the optimal operation  $g^{*}(\cdot)$  further demonstrates the necessity of suppressing abnormally low activations in OOD detection. To mimic such operations, we design a piecewise function called VRA:
$$
\operatorname {V R A} (z) = \left\{ \begin{array}{l} 0, z <   \alpha \\ z, \alpha \leq z \leq \beta \\ \beta , z > \beta \end{array} \right.,
$$
where  $\alpha$  and  $\beta$  are two thresholds for determining low and high activations. Obviously,  $\alpha = 0$  and  $\beta = \infty$  represent models without activation truncation;  $\alpha = 0$  and  $\beta > 0$  represent models equivalent to ReAct. Therefore, our method is a more general operation. Since different features have distinct distributions, we propose an adaptively adjusted strategy to determine  $\alpha$  and  $\beta$ . Specifically, we predefine  $\eta_{\alpha}$  and  $\eta_{\beta}$  satisfying  $\eta_{\alpha} < \eta_{\beta}$ . Then, we treat the  $\eta_{\alpha}$ -quantile (or  $\eta_{\beta}$ -quantile) of activations estimated on ID data as  $\alpha$  (or  $\beta$ ). Meanwhile, we observe that  $g^{*}(\cdot)$  amplifies intermediate activations in Figure 1(d)~1(f). Therefore, we propose another variant of VRA called VRA+, which further introduces a hyper-parameter  $\gamma$  to control the degree of amplification:
$$
\mathrm {V R A} + (z) = \left\{ \begin{array}{l} 0, z <   \alpha \\ z + \gamma , \alpha \leq z \leq \beta \\ \beta , z > \beta \end{array} \right..
$$
Corpus Description In line with previous works, we consider different OOD datasets for distinct ID datasets [8, 10]. For CIFAR benchmarks [11] as ID data, we use the official train/test splits for ID data and select six datasets as OOD data: Textures [12], SVHN [13], Places365 [14], LSUN-Crop [15], LSUN-Resize [15], and iSUN [16]; for ImageNet [17] as ID data, it is more challenging than
Table 1: Main results on CIFAR benchmarks. In this table, we compare detection performance with competitive post-hoc strategies. All methods are pretrained on ID data. We report the results for each dataset, as well as the average results across all datasets. "FR." and "AU." are abbreviations of FPR95 and AUROC. Top3 results are marked in red, and darker colors indicate better performance.
Method	SVHN	LSUN-C	LSUN-R	iSUN	Textures	Places365	Average
FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑
ID Dataset: CIFAR-10; Backbone: DenseNet-101 [28]
MSP [20]	47.27	93.48	33.57	95.54	42.10	94.51	42.31	94.52	64.15	88.15	63.02	88.57	48.74	92.46
ODIN [21]	25.29	94.57	04.70	98.86	03.09	99.02	03.98	98.90	57.50	82.38	52.85	88.55	24.57	93.71
Mahalanobis [22]	06.42	98.31	56.55	86.96	09.14	97.09	09.78	97.25	21.51	92.15	85.14	63.15	31.42	89.15
Energy [23]	40.61	93.99	03.81	99.15	09.28	98.12	10.07	98.07	56.12	86.43	39.40	91.64	26.55	94.57
ReAct [8]	41.64	93.87	05.96	98.84	11.46	97.87	12.72	97.72	43.58	92.47	43.31	91.03	26.45	95.30
KNN [24]	13.51	96.68	30.95	93.82	11.37	97.72	10.79	97.91	24.50	95.19	63.88	85.00	25.83	94.39
DICE [10]	25.99	95.90	00.26	99.92	03.91	99.20	04.36	99.14	41.90	88.18	48.59	89.13	20.84	95.25
SHE [25]	28.12	94.72	00.76	99.84	09.73	98.15	10.99	97.95	51.98	83.07	59.35	84.16	26.82	92.98
ASH[27]	30.14	95.29	2.82	99.34	7.97	98.33	8.46	98.29	50.85	88.29	40.46	91.76	23.45	95.22
VRA	18.75	96.68	01.32	99.63	05.80	98.69	05.70	98.69	34.89	93.42	39.98	91.69	17.74	96.47
VRA+	13.54	97.45	02.03	99.56	06.37	98.72	06.15	98.71	27.07	95.03	39.97	91.96	15.85	96.91
ID Dataset: CIFAR-100; Backbone: DenseNet-101 [28]
MSP [20]	81.70	75.40	60.49	85.60	85.24	69.18	85.99	70.17	84.79	71.48	82.55	74.31	80.13	74.36
ODIN [21]	41.35	92.65	10.54	97.93	65.22	84.22	67.05	83.84	82.34	71.48	82.32	76.84	58.14	84.49
Mahalanobis [22]	22.44	95.67	68.90	86.30	23.07	94.20	31.38	89.28	62.39	79.39	92.66	61.39	50.14	84.37
Energy [23]	87.46	81.85	14.72	97.43	70.65	80.14	74.54	78.95	84.15	71.03	79.20	77.72	68.45	81.19
ReAct [8]	83.81	81.41	25.55	94.92	60.08	87.88	65.27	86.55	77.78	78.95	82.65	74.04	65.86	83.96
KNN [24]	23.96	93.99	70.98	73.37	76.34	76.69	70.88	78.58	37.75	87.48	95.20	59.70	62.52	78.30
DICE [10]	54.65	88.84	00.93	99.74	49.40	91.04	48.72	90.08	65.04	76.42	79.58	77.26	49.72	87.23
SHE [25]	41.89	90.61	01.06	99.68	78.18	73.97	72.73	76.14	61.49	76.57	85.33	70.53	56.78	81.25
ASH[27]	81.86	83.86	11.60	97.89	67.56	81.67	70.90	80.81	78.24	74.09	77.03	77.94	64.53	82.71
VRA	70.91	87.46	10.73	98.04	38.52	93.49	38.53	93.42	47.64	90.17	76.39	78.66	47.12	90.21
VRA+	62.64	88.70	19.82	96.33	28.44	95.47	28.72	95.18	40.62	91.57	79.78	76.42	43.34	90.61
CIFAR benchmarks due to larger label space and higher resolution images. To ensure non-overlapped categories between ID and OOD, we select a subset from four datasets as OOD data, in line with previous works [8, 10]: iNaturalist [18], SUN [19], Places [14], and Textures [12].
Baselines To verify the effectiveness of our method, we implement the following state-of-the-art post-hoc strategies as baselines: 1) MSP [20] is the most basic method that directly leverages the maximum softmax probability to identify OOD data; 2) ODIN [21] uses temperature scaling and input perturbation to increase the gap between ID and OOD; 3) Mahalanobis [22] calculates the distance from the nearest class center as the indicator; 4) Energy [23] replaces the maximum softmax probability with the theoretically guaranteed energy score; 5) ReAct [8] applies activation truncation to remove abnormally high activations; 6) KNN [24] exploits non-parametric nearest-neighbor distance for OOD detection; 7) DICE [10] leverages sparsification to select the most salient weights; 8) SHE [25] uses the energy function defined in the modern Hopfield network [26]. 9) ASH [27] removed a large portion of a sample's activation at a late layer.
Implementation Details Our method contains three user-specific parameters: the thresholds  $\eta_{\alpha}$  and  $\eta_{\beta}$ , and the degree of amplification  $\gamma$ . We select  $\eta_{\alpha}$  from  $\{0.5, 0.6, 0.65, 0.7\}$ ,  $\eta_{\beta}$  from  $\{0.8, 0.85, 0.9, 0.95, 0.99\}$ , and  $\gamma$  from  $\{0.2, 0.3, 0.4, 0.5, 0.6, 0.7\}$ . Consistent with previous works [8], we use Gaussian noise images as the validation set for hyperparameter tuning. By default, we use DenseNet-101 [28] for CIFAR and ResNet-50 [29] for ImageNet. All experiments are implemented with PyTorch [30] and carried out with NVIDIA Tesla V100 GPU. To compare the performance of different methods, we exploit two widely used OOD detection metrics: FPR95 and AUROC. Among them, FPR95 measures the false positive rate of OOD data when the true positive rate of ID data is  $95\%$ ; AUROC measures the area under the receiver operating characteristic curve.
Main Results To verify the effectiveness of our method, we compare VRA-based methods with competitive post-hoc strategies. Experimental results are shown in Table 1 and Table 2. We observe that our method generally achieves Top3 performance on different datasets and performs the best overall. Different from these baselines, we attempt to maximize the gap between ID and OOD by suppressing abnormally low and high activations and amplifying intermediate activations. These
Table 2: Main results on ImageNet. All methods are pretrained on ImageNet.
Method	iNaturalist	SUN	Places	Textures	Average
FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑	FR.↓	AU.↑
Backbone: ResNet-50 [29]
MSP [20]	54.99	87.74	70.83	80.86	73.99	79.76	68.00	79.61	66.95	81.99
ODIN [21]	47.66	89.66	60.15	84.59	67.89	81.78	50.23	85.62	56.48	85.41
Mahalanobis [22]	97.00	52.65	98.50	42.41	98.40	41.79	55.80	85.01	87.43	55.47
Energy [23]	55.72	89.95	59.26	85.89	64.92	82.86	53.72	85.99	58.40	86.17
ReAct [8]	20.38	96.22	24.20	94.20	33.85	91.58	47.30	89.80	31.43	92.95
KNN [24]	59.08	86.20	69.53	80.10	77.09	74.87	11.56	97.18	54.32	84.59
DICE [10]	25.63	94.49	35.15	90.83	46.49	87.48	31.72	90.30	34.75	90.78
SHE [25]	34.22	90.18	54.19	84.69	45.35	90.15	45.09	87.93	44.71	88.24
ASH[27]	44.57	92.51	52.88	88.35	61.79	85.58	42.06	89.70	50.32	89.04
VRA	15.70	97.12	26.94	94.25	37.85	91.27	21.47	95.62	25.49	94.57
VRA+	15.48	97.08	23.50	94.91	34.62	91.79	19.66	96.08	23.31	94.97
Backbone: ResNetv2-101 [29]
MSP [20]	63.69	87.59	79.98	78.34	81.44	76.76	82.73	75.45	76.96	79.54
ODIN [21]	62.69	89.36	71.67	83.92	76.27	80.67	81.31	76.30	72.99	82.56
Mahalanobis [22]	96.34	46.33	88.43	65.20	89.75	64.46	52.23	72.10	81.69	62.02
Energy [23]	64.91	88.48	65.33	85.32	73.02	81.37	80.87	75.79	71.03	82.74
ReAct [8]	49.97	89.80	65.30	87.40	73.12	85.34	80.82	70.53	67.30	83.27
MOS [4]	09.28	98.15	40.63	92.01	49.54	89.06	60.43	81.23	39.97	90.11
VRA	27.26	95.68	34.53	93.27	47.31	90.19	30.69	94.22	34.95	93.34
VRA+	20.81	97.70	32.89	92.68	45.83	90.01	23.88	95.43	30.85	93.71
Table 3: Compatibility with different scoring functions. For each ID dataset, we report the average results of its OOD datasets. We use DenseNet-101 [28] for CIFAR and ResNet-50 [29] for ImageNet.
Method	CIFAR-10	CIFAR-100	ImageNet	Average
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
MSP [20]	48.73	92.46	80.13	74.36	66.95	81.99	65.27	82.94
MSP + ReAct	48.00	92.77	77.69	76.22	55.63	87.85	60.44	85.61
MSP + DICE	43.72	92.92	76.86	76.39	67.41	82.24	62.66	83.85
MSP + VRA	42.31	93.50	79.69	75.94	47.09	89.62	56.36	86.35
Energy [23]	26.55	94.57	68.45	81.19	58.41	86.17	51.14	87.31
Energy + ReAct	26.45	94.67	62.27	84.47	31.43	92.95	40.05	90.70
Energy + DICE	20.83	95.24	49.72	87.23	34.75	90.77	35.10	91.08
Energy + VRA	17.74	96.47	53.24	88.74	25.49	94.57	32.16	93.26
ODIN [21]	24.57	93.71	58.14	84.49	56.48	85.41	46.40	87.87
ODIN + ReAct	21.00	95.98	54.17	88.62	42.21	91.28	39.13	91.96
ODIN + DICE	26.05	94.62	61.39	83.83	62.89	84.48	50.11	87.64
ODIN + VRA	17.38	96.52	47.12	90.21	32.75	93.39	32.42	93.37
results demonstrate the effectiveness of such suppression and amplification operations in OOD detection. Meanwhile, we observe that  $\mathrm{VRA + }$  generally outperforms VRA, suggesting that the operation closer to the theoretical optimum solution generally can achieve better performance.
We also compare with methods that require training. MOS [4] addresses OOD detection by training-time regularization. Experimental results in Table 2 show that our method outperforms MOS with the same backbone. Meanwhile, VOS [5] is a recently advanced strategy that synthesizes virtual outliers to regularize decision boundaries during training. According to their original paper, it achieves  $95.33\%$  in AUROC and  $22.47\%$  in FPR95 on CIFAR-10. Our method outperforms VOS under the same ID data, OOD data, and network architecture (see Table 1). Therefore, VRA-based methods do not require an expensive training process but can achieve better performance in OOD detection.
Compatibility with Scoring Functions In Table 3, we investigate the compatibility of VRA-based methods with different scoring functions: MSP, Energy, and ODIN. Experimental results demonstrate that our method brings performance improvements for all scoring functions and generally achieves better performance than competitive post-hoc strategies. These results verify the compatibility and effectiveness of our method in OOD detection.
Table 4: Performance upper bound analysis. For each ID dataset, we report the average results over multiple OOD datasets. We use DenseNet-101 [28] for CIFAR and ResNet-50 [29] for ImageNet.
ID	Energy [23]	VRA	VRA-Fake-True	VRA-True
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
CIFAR-10	26.55	94.57	17.74	96.47	13.27	97.75	00.96	99.81
CIFAR-100	68.45	81.19	47.12	90.21	23.62	94.20	01.58	99.69
ImageNet	58.41	86.17	25.49	94.57	13.09	96.89	03.50	99.31
(a) AUROC on CIFAR-10
(b) AUROC on CIFAR-100
(c) AUROC on ImageNet
(d) FPR95 on CIFAR-10
(e) FPR95 on CIFAR-100
(f) FPR95 on ImageNet
Figure 2: Parameter sensitivity analysis. For each ID dataset, we report the average results over multiple OOD datasets. We use DenseNet-101 [28] for CIFAR and ResNet-50 [29] for ImageNet.