VRA: Variational Rectified Activation for Out-of-distribution Detection
Authors: Mingyu Xu
Zheng Lian
Bin Liu
Jianhua Tao
Venue: NeurIPS 2023 poster (2023)
Keywords: Out-of-distribution Detection

Abstract: Out-of-distribution (OOD) detection is critical to building reliable machine learning systems in the open world. Researchers have proposed various strategies to reduce model overconfidence on OOD data. Among them, ReAct is a typical and effective technique to deal with model overconfidence, which truncates high activations to increase the gap between in-distribution and OOD. Despite its promising results, is this technique the best choice? To answer this question, we leverage the variational method to find the optimal operation and verify the necessity of suppressing abnormally low and high activations and amplifying intermediate activations in OOD detection, rather than focusing only on high activations like ReAct. This motivates us to propose a novel technique called ``Variational Rectified Activation (VRA)'', which simulates these suppression and amplification operations using piecewise functions. Experimental results on multiple benchmark datasets demonstrate that our method outperforms existing post-hoc strategies. Meanwhile, VRA is compatible with different scoring functions and network architectures. Our code is available at https://github.com/zeroQiaoba/VRA.

VRA: Variational Rectified Activation for Out-of-distribution Detection
Mingyu Xu Zheng Lian'r ; Bin Liu Jianhua T
The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences DDpparmentt of Automation, Tsinghua University "Bejjing National Research Center for Information Science and Technology, Tsinghua University (xumingyu2021, 1i 1ianzheng2066)01a. ac.
Abstract
Out-of-distributin (OOD) detection is critical to building reliable machine learning systems in the open world. Researchers have proposed various strategies reduce model overconfidence on OOD data. Among them, ReAct is typical and effective technique deal with model overconfidence, which truncates high activations increase the gap between in-distribution and OOD. Despite its promising results, is this technique the best choice? To answer this question, we leverage the variational method to find the optimal operation and verify the necessity of suppressing abnor- mally low and high activations and amplifying intermediate activations OOD detection, rather than focusing only high activations like ReAct. This motivates us to propose a novel technique called "Variational Rectified Activation (VRA)", which simulates these suppression and amplification operations using piecewise functions. Experimental results on multiple benchmark datasets demonstrate that method outperforms existing post-hoc strategies. Meanwhile, VRA is com- patible with different scoring functions and network architectures. Our code is available http://tthuu..oom///ooooaaaaab/a
Introduction
Systems deployed in the real world often encounter out-of- distribution (OOD) data, i.e., samples from an irrelevant distribution whose label set has no interaction with the training data. Most of the existing systems tend generate overconfident estimations for OOD data, seriously affecting their reliability [1] Therefore, researchers propose the OOD detection task, which aims to determine whether sample comes from in-distribution (ID) OOD. This task allows the model to reject recognition when faced with unfamiliar samples. Considering its importance, OOD detection has attracted increasing attention from researchers and has been applied many fields with high-safety requirements such as autonomous driving 22 and medical diagnosis [3.
In OOD detection, existing methods can be roughly divided into two categories: methods requiring training and post-hoc strategies. The first category identifies OOD data by training-time regularization
4 55 external OOD samples [6] [7]. But they require more computational resources and inconvenient in practical applications. To this end, researchers propose post-hoc strategies that directly use pretrained models for OOD detection. Due their ease of implementation, these methods have attracted increasing attention in recent years. Among them, React [ is a typical post-hoc strategy that truncates abnormally high activations increase the gap between ID and OOD, thereby improving detection performance. But is this operation the best choice for widening the gap?
*Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To answer this question, we use the variational method to solve for the optimal operation. Based on this operation, we reveal the necessity of suppressing abnormally low and high activations and amplifying intermediate activations in OOD detection. Then, we propose a simple yet effective strategy called "Variational Rectified Activation (VRA)", which mimics suppression and amplification operations using piecewise functions. To verify its effectiveness, we conduct experiments on multiple benchmark datasets, including CIFAR- 10, CIFAR-I00 and the more challenging ImageNet. Experimental results demonstrate that our method outperforms existing post-hoc strategies, setting new state-of-the-art records. The main contributions of this paper can be summarized as follows:
(Theory) From the perspective of the variational method, we find the best operation for max- imizing the gap between ID and OOD. This operation verifies the necessity of suppressing abnormally low and high activations and amplifying intermediate activations.
(Method) We propose a simple yet effective post-hoc strategy called VRA. Our method is compatible with various scoring functions and network architectures.
(Performance) Experimental results on benchmark datasets demonstrate the effectiveness of our method. VRA is superior to existing post-hoc strategies in OOD detection.
2 Methodology
2.1 Problem Definition
Let X' be the input space and y be the label space with e distinct categories. Consider a supervised classification task on dataset containing N labeled samples (x, y/. where E E is the ground truth label for the sample E X. Ideally, all test samples come from the same distribution as the training data. But in practice, the test sample may come from an unknown distribution, such as an irrelevant distribution whose label set has no intersection with y. In this paper, we use Pin to represent the marginal distribution of and Pout to represent the distribution of OOD data. In this paper, we aim to determine whether a sample comes from ID or OOD.
2.2 Motivation
Among all methods, ReAct is typical and effective post-hoc strategy { Suppose h(x) {2; ) the feature vector of the penultimate layer and m denotes the feature dimension. For convenience, we use as shorthand for ReAct truncates activations above threshold for each
g(=) min(2, c);
where e oc is equivalent to the model without truncation. ReAct has demonstrated that this truncation operation can increase the gap between ID and OOD [8]
[x][g(=)] Epan [a())] > Ei, [x]
E [z].
Despite its promising results, is this strategy the best option for widening the gap between ID and OOD? In this paper, we attempt to answer this question with the help of the variational method.
2.3
VRA Framework
To find the best operation, we should optimize the following objectives:
Maximize the gap between ID and OOD.
Minimize the modification brought by the operation to maximally preserve the input.
The final objective function is calculated as follows:
min L(g) = Ein (g((2) - 2) - 2A [Bm]g(2)] - Eost (G([))
(3)
where \ controls the rafe-oof between two losses. To solve for Eq. we first make a mild assumption to ensure the function space G is sufficiently complex.

---

Assumption
We assume Eg [11, Bowrl2 Bin(=2), and Eowl [2]] exist. Let 9 be a Hilbert space:
g - (g(2 (l9l)))) Eona (l
oc).
(4)
This space is sufficiently complex containing most functions, such as identity functions, constant functions, and all bounded continuous functions. Then, we define the inner product of as follows:
gg((2),g((2) >= (u(())g02))uu(()))
(5)
Combining this assumption, the equivalent equation Eq.
is:
min L(g) / (g(2) =)²² pin(z) (2g(2) Pour Poun()) (z))
(6)
gEg
Then, leverage the variational method solve for the functional extreme value. We mark g-( as the optimal solution. v-(-) E g and Ve E R. we then have:
E(g*) s fig + cf).
can be converted to:
/ (g) (z) - 2)² Pml()) 22g (2) Pout (2))
(8)
(000 +ff))) - 2)² Pmn()) - 22(g + +e(=))) - Pour (2))
(9)
Then, we have:
& $ (2))pm())dz 2x f(=) (())(()))) (0(=)) 2
Pia (1- ))
Pout (2)
Pmn (m(()d= 0.
(10)
Combining Assumption I and the arbitrariness of e, get:
f [(=) -==x))=-x( Poul Powi(2)
Pin (=) )) Pin (z)dz= 0.
(11)
Considering Assumption I and the arbitrariness of f(2), have:
Pour(z) 0. Pin (z)
(12)
Therefore, the optimal activation function is:
Pour (2)
g (2) = =
(13)
Pin (z)
To verify its effectiveness, we compare the optimal function 9-(-) with the unrectified function g(z) z. Since g²() (-) is the optimal solution, should get a smaller value in Eq, 3
(inl((((=)) =) 2A (Eing"* (=)] - Eowrg"" (=)]) S Eii](( z)² 2A (Em,[=] Eout [z]) (14)
The equivalent equation of Eq. 14is: is:
1
(Enng [g" (=)] Eowlg" (x)]l"(<))) (Bin[2]
Evui (2]) 2
En (9 (2) 2)²].
(15)
2X
It shows that g* enlarges the gap between ID and OOD by at least
1 Ein [1 (z) 2) 2 0.
2x
2.4 Practical Implementations
Through theoretical analysis, we have found the optimal operation g g*() that can maximize the gap between ID and OOD. But in practice, this operation depends on the specific expressions of Pmn and Pour Estimating these expressions a challenging task given that OOD data comes from unknown This drives to look for more practical implementations.
distributions [9.

---

Emprica POR
Emprical
Emarnca
(a) empirical PDF on iNaturalist
(b) ompirical PDF on SUN
(c) empirical PDF on Places
ReAct
Wib fruncation
(d) activation functions on INaturalist
(e) activation functions on SUN
(f) activation functions on Places
Figure I: Empirical PDFs for Pin(.) and Pom (.m(.), and visualization of different activation functions. We treat ImageNet ID data and select multiple OOD datasets for visualization.
For this purpose, we treat ImageNet as ID data and select multiple OOD datasets. We first use histograms approximate the probability density functions Pmn and Paut- Then, we compute g (.)
and compare it with ReAct, whose threshold set the 90th percentile of activations estimated
on ID data, consistent with the original paper [8]. Experimental results are shown in Figure [ Compared with the model without truncation, we observe that ReAct suppresses high activations (see
Figure ((f). Unlike ReAct, the optimal operation g²(-) further demonstrates the necessity of suppressing abnormally low activations in OOD detection. mimie such operations, we design piecewise function called VRA:
0, < a 2, =
VRA(:)
B, > 3
where a and 3 are two thresholds for determining low and high activations. Obviously, a = 0 and oo represent models without activation truncation; 0 and 0 represent models equivalent to ReAct. Therefore, our method is more general operation. Since different features have distinct distributions, we propose an adaptively adjusted strategy to determine a and Specifically, we predefine and 1/8 satisfying aa < 173. Then, we treat the 7aa -quuntie (or -1s-quat of activations estimated on ID data (or f3). Meanwhile, we observe that g" amplifies intermediate activations in Figure [(f) Therefore, we propose another variant of VRA called VRA+, which further introduces hyper-parameter 7 to control the degree of amplification:
0, <
VRA+(2)
2+7,0 S S
13,2B
Experiments
3.1 Experimental Setup
Corpus Description In line with previous works, we consider different OOD datasets for distinct ID datasets 8 [10. For CIFAR benchmarks [11] as ID data, we use the official train/test splits for ID
data and select datasets as OOD data: Textures [1], SVHN [3], Places365 [14, LSUN-Crop 11), LSUN-Resiae [15), and iSUN [16]; for ImageNet 17
ID data, is more challenging than

---

Table 1: Main results on CIFAR benchmarks. In this table, we compare detection performance
with competitive post-hoc strategies. All methods are pretrained on ID data. We report the results for each dataset, as well as the average results across all datasets. "FR." and "AU." are abbreviations of
FPR95 and AUROC. Top3 results are marked in red, and darker colors indicate better performance.
Method
SVHN
LSUN-C
LSUN-R
ISUN
Textures
Places365
Average
FR. AU.
FR.
AU.
FR. 4 AU.
FR. 4 AU. T
FR.
AU.
FR.
AU.
FR. J AU. f
ID Dataset:
CIFAR-IO: 10, Backbone: DenseNet-101
28
MSP 220
47.27
93.48
31.57
95.54
40.10
94.51
42.31
94.52
64.15
88.15
63.02
88.57
48.74
92.46
ODIN 21
25.29
94.57
04.70
98.86
03.09
99.02
03.98
98.90
57.50
82.38
52.85
88.55
24.57
93.71
Mahalanobis
06.42
98.31
56.55
86.96
09.14
97.09
09.78
97.25
21.51
92.15
85.14
63.15
31.42
89.15
Energy
03.81
99.15
09.28
98.12
10:07
98.07
56.12
91.64
26.55
ReAct
8
41.64
93.87
05.96
98.84
11.46
97.87
12.72
97.72
43.58
92.47
43.31
91.03
26.45
95.30
KNN 24
13.51
96.68
30.95
93.82
11.37
97.72
10.79
97.91
24.50
95.19
63.88
85.00
25.83
94,39
DICE 10
25.99
95.90
00.26
99.92
03.91
99.20
04.36
99.14
41.90
58.18
48.59
89.13
20.84
95.25
25
94.72
99.84
09.73
98.15
10.99
97.95
51,98
83.07
26.82
92.98
AST 27
30.14
95.29
2.82
99.34
7.97
98.33
8.46
98.29
50.85
88.29
40.46
91.76
23.45
95.22
VRA
18.75
96.68
01.32
99.63
0.5.50
09.69
055770
98.69
34.89
93:42
39.98
91.69
17.74
96.47
VRA+
13.54
97.45
02.03
99.56
06.37
98.72
05.15
98.71
27.07
95.03
39.97
91,96
15.85
96,91
ID Dataset:
CIFAR:
100; Backbone:
DenseNet
101 2X
MSP
20
81.70
75.40
60.49
85.60
85,24
69.18
85,99
70.17
84.79
71.48
82.55
74.31
80.13
74.36
41.35
97.93
65.22
84.22
67.05
82.34
82.32
76.84
58.14
84.49
Mahalanabis
29
22.44
95.67
68.90
86.30
23.07
94.20
31.38
89.28
62.39
79.39
92.66
61.39
50.14
84.37
Energy
22
87.46
81.85
14.72
97.43
70.65
80.14
74.54
78.95
84.15
71.03
79.20
77.72
68.45
81.19
ReAct
8
83.81
81.41
25.55
94.92
60.00 08
87.58 88
65.27
86.55
77.78
78.95
82.65
74.04
86
83.96
KNN 24
23,96
93.99
70.98
73.37
76.34
76.69
70.88
78.58
37.75
87.48
95.20
59.70
62.52
78.30
DICE
10
54.65
88.84
00.93
99.74
49.40
91.04
48.72
90.08
65.04
76.42
79.58
77.26
49.72
87.23
SHE
25
41.89
90.61
01.06
99.68
78. 18
73.97
72.73
16.14
61,49
76.57
85.33
70.53
56,78
81,25
ASH 27
81.86
83.86
11.60
97.89
67.56
81.67
70.90
80.81
78.24
74.09
77.03
77.94
64.53
82,71
VRA
70.91
87.46
10.73
98.04
38.52
93.49
38..53
93.42
477.64
90.17
76.39
78.66
47.12
90.21
VRA+
96.33
95.47
28.72
95.18
43.34
CIFAR benchmarks due to larger label space and higher resolution images. To ensure non-overlapped categories between IDD and OOD, we select a subset from four datasets as OOD data, in line with
previous works g 10 iNaturalist [18). SUN [9]. Places [4]. and Textures [12].
Baselines To verify the effectiveness of our method, implement the following state-of-the-ard post-hoc strategies as baselines: 1) MSP 20) is the most basic method that directly leverages the maximum softmax probability to identify OOD data; 2) ODIN (21) uses temperature scaling and input perturbation to increase the gap between ID and OOD: 3) Mahalanobis (22) calculates the
distance from the nearest class center as the indicator; 4) Energy (23) replaces the maximum softmax
probability with the theoretically guaranteed energy score; 5) ReAct [8] applies activation truncation
to remove abnormally high activations; 6) KNN 24 exploits non-parametric nearest-neighhor distance for OOD detection; 7) DICE [10] leverages sparsification to select the most salient weights; 8) SHE [25] uses the energy function defined in the modern Hopfield network [26]. 9) ASH [27
removed a large portion of a sample's 's activation at a late layer.
Implementation Details Our method contains three user-specific parameters: the thresholds Ta and 178 and the degree of amplification 7. We select 7)2 from {0.5, 0.6, 0.65, 0.7), ns from {0.8, 0.85, 0.9, 0.95, 0.99), and y from (0.2, 0.4, 0.5, 0.6, 0.7}. Consistent with previous works [.. we use Gaussian noise images as the validation set for hyperparameter tuning. By default, we use DenseNet-101 (88 for CIFAR and ResNet-50 29 for ImageNet. All experiments are implemented with PyTorch 300 and carried out with NVIDIA Tesla V100 GPU. To compare the performance of different methods, we exploit two widely used OOD detection metrics: FPR95 and AUROC. Among them, FPR95 measures the false positive rate of OOD data when the true positive rate of ID data is 95%; AUROC measures the area under the receiver operating charactcristic curve.
3.2 Experimental Results and Discussion
Main Results To verify the effectiveness of our method, we compare VRA-based methods with competitive post-hoe strategies. Experimental results are shown Table and Table We observe that our method generally achieves Top3 performance on different datasets and performs the best overall. Different from these baselines, we attempt to maximize the gap between ID and OOD by suppressing abnormally low and high activations and amplifying intermediate activations. These
5

---

Table 2: Main results on ImageNet. All methoos are pretrained on ImageNet.
iNaturalist FR.
SUN
Places
Textures
Average
Method
AU.
FR.
AU.
FR.
AU.
FR.
AU.
FR.
AU.
Backbone: ResNet-50
[29]
MSP 20
54.99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81 1.99
ODIN 21
47.66
89.66
60.15
84.59
67.89
81.78
50.23
85.62
56.48
85,41
Mahalanobis
22
97.00
52.65
98.50
42.41
98.40
41.79
55.80
85.01
87.43
55.47
Energy
55.72
89.95
59.26
85.89
64.92
82.86
53.72
85.99
58.40
86.17
ReAct
8
20.38
96.22
24.20
94.20
33.85
91.58
47.30
89.80
31.43
92.95
KNN 24
59.08
86.20
69.53
80.10
77.09
74.87
11.56
97.18
54.32
84.59
DICE 10
25.63
94.49
35.15
90.83
46.49
87.48
31.72
90.30
34.75
90.78
SHE 225
34.22
90.18
54.19
84.69
45.35
90.15
45.09
87.93
44.71
88.24
ASH
44.57
92.51
52.88
88.35
61.79
85.58
42.06
89.70
50.32
89.04
VRA
15.70
97.12
26.94
94.25
37.85
91.27
21.47
95.62
25.49
94.57
VRA+
15.48
97.08
23.50
94.91
34.62
91.79
19.66
96.08
23.31
94.97
Backbone:
ResNetv2-101 T01
(2
MSP [20]
63.69
87.59
79.98
78.34
81.44
76.76
82.73
75.45
76.96
79.54
ODIN 21
62.69
89.36
71.67
83.92
76.27
80.67
81.31
76.30
72.99
82.56
Mahalanobis
22
96.34
46.33
88.43
65.20
89.75
64.46
52.23
72.10
81.69
62.02
Energy
64.91
88.48
65.33
85.32
73.02
81.37
80.87
75.79
71.03
82.74
ReAct
8
49.97
89.80
65,30
87.40
73.12
85.34
80.82
70.53
67.30
83.27
MOS
09.28
98.15
40.63
92.01
49.54
89.06
60.43
81.23
39.97
90.11
VRA
27.26
95.68
34.53
93.27
47.31
90.19
30.69
94.22
34.95
93.34
VRA+
20.81
97.70
32.89
92.68
45.83
90.01
23.88
95.43
30.85
93.71
Table 3: Compatibility with different scoring functions. For each dataset, we report the average
results of its OOD datasets. We use DenseNet-100 101 (28) for CIFAR and ResNet-50 29] for ImageNet.
CIFAR-10
CIFAR-100
ImageNet
Average
Method
FPR95
AUROC
FPR95 4. AUROC
FPR95 4. AUROC f
FPR95 4 AUROC 1
MSP [20]
48.73
92.46
80.13
74.36
66,95
81.99
65.27
82.94
MSP ReAct
48.00
92.77
77.69
76.22
55.63
87.85
60.44
85.61
MSP DICE
43.72
92.92
76.86
76.39
67.41
82.24
62.66
83.85
MSP VRA
42.31
93.50
79.69
75.94
47.09
89.62
56.36
86.35
Energy 23
26.55
94.57
68.45
81.19
58.41
86.17
51.14
87.31
Energy ReAct
26.45
94.67
62.27
84.47
31.43
92.95
40.05
90.70
Energy + DICE
20.83
95.24
49.72
87.23
34.75
90.77
35.10
91.08
Energy + VRA
17.74
96.47
53.24
88.74
25.49
94.57
32.16
93.26
ODIN 21
24.57
93.71
58.14
84.49
56.48
85,41
46.40
87.87
ODIN + ReAct
21.00
95.98
54.17
88.62
42.21
91.28
39.13
91.96
ODIN + DICE
26.05
94.62
61.39
83.83
62.89
84,48
50.11
87.64
ODIN + VRA
17.38
96.52
47.12
90.21
32.75
93.39
32.42
93.37
results demonstrate the effectiveness of such suppression and amplification operations in OOD detection. Meanwhile, we observe that VRA+ generally outperforms VRA, suggesting that the operation closer to the theoretical optimum solution generally can achieve better performance.
We also compare with methods that require training. MOS 4 addresses OOD detection by training- time regularization. Experimental results in Table[[show that our method outperforms MOS with the same backbone. Meanwhile, VOS a recently advanced strategy that synthesizes virtual outliers to regularize decision boundaries during training. According their original paper. it achieves 95.33% AUROC and 22.47% in FPR95 on CIFAR-10. Our method outperforms VOS under same ID data, OOD data, and network architecture (see Table Therefore, VRA-based methods do not require an expensive training process but can achieve better performance in OOD detection.
Compatibility with Scoring Functions In Table 3, we investigate the compatibility of VRA-bbsed methods with different scoring functions: MSP. Energy, and ODIN. Experimental results demonstrate that our method brings performance improvements for all scoring functions and generally achieves better performance than competitive post-hoc strategies. These results verify the compatibility and effectiveness of our method in OOD detection.

---

Table 4: Performance upper bound analysis. For each dataset, we report the average results over
multiple OOD datasets. We use DenseNet- 101 (28) for CIFAR and ResNet-50 (29] for ImageNet.
Energy 23
VRA
VRA-Fake-
VRA-Ini -rue
ID
FPR95
AUROC f
FPR95 +
AUROC
FPR95 4 AUROC t
FPR95
AUROC f
CIFAR-10
26.55
94.57
17.74
96.47
13.27
97.75
00.96
99.81
100
68,45
81.19
47.12
90.21
23.62
94.20
01.58
99.69
ImageNet
58.41
86.17
25.49
94.57
13.09
96.89
03.50
99.31
111
a AUROC on CIEAR-10
(b) AUROC on CIFAR-101
(c) AUROC on ImageNet
(d) FPR95
FPR95
FPR95 ImageNet
Figure 2: Parameter sensitivity analysis. For each ID dataset, report the average results over multiple OOD datasets. We use DenseNet-101 28 for CIFAR and ResNet-50 for ImageNet.
Performance Upper Bound Analysis We propose VRA and VRA+ to approximate the optimal op- eration for OOD detection. But is it necessary to design other functions to get better approximation? To answer this question, we need to reveal whether (-) can reach the upper-bound performance. The core of estimating (-) is estimate the probability density functions of pm and Paut- To this end, we consider two ideal cases: VRA-True and VRA-Fake-True ID and OOD data known in advance; in the second case, randomly select 1% of ID and OOD data from the entire dataset. Both cases leverage histograms to estimate and Pout and use Eq. [3 calculate 9 (-). Considering that histograms provide piecewise form of (-),
the first case, we assume that all
directly the piecewise function to represent g (-). In Table 4 we observe that both ideal cases can achieve near-pertie peer-perfect results. Therefore, g" (-) that increases the gap between ID and OOD can generate more discriminative features for OOD detection. In the future, we will explore other functions that can better describe the optimal operation for better performance.
Parameter Sensitivity Analysis VRA uses two dyper-parameee (1a and 173)) to adaptively adjust thresholds for low and high activations. this section, we conduct parameter sensitivity analysis and reveal their impact when 7a and ne are inappropriate. large Taa suppresses too many low activations, while large f]8 suppresses too few high activations. Therefore, it necessary to choose proper 7/a and ny for VRA.
OOD detection. In Figure
observe that method does perform well
Role of Adaptively Adjusted Strategy In this paper, we adopt an adaptive strategy to automatically determine a and B. To verify its effectiveness, we compare this adaptive strategy with another strategy that uses fixed and 8 for different features. To determine these paper-parametess we use Gaussian
7

---

Table 5: Role of adaptively adjusted strategy. We use DenseNet- 101
28 for CIFAR.
Hlyer-parameete
OOD Performance
ID
Strategy
Ma
7
FPR95
AUROC 1
assign at
0.50
1.50
19.44
96.34
CIFAR-10
assign To 1]3
0.60
0.95
17.74
96.47
assign B
0.50
1,50
56.35
86.09
CIFAR-100
assign T/o 1/8
0.60
0.95
47.12
90.21
Table 6: Compatibility with different backbones. All methods are pretrained on ImageNet.
MSP
Energy
ReAcy+Energy
VRA+Energy
Backbone
FPR95 L AUROC
FPR95 + AUROC
FPR95 4 AUROC f
FPR95
AUROC f
ResNet-18 18
29
69.70
80.61
58.59
80.40
36,36
92.17
34.87
92.58
ResNet-34
29
68.84
81.19
57.20
86.84
32.23
93.08
30.63
93.46
ResNet-55
29
66.95
81.99
58.40
86.17
31.43
92.95
25.49
94.57
ResNet-101 29
64.70
82.47
54.84
87.29
31.68
93.03
25.80
94.36
ResNet-152 29
61.35
83.74
50.39
88.61
26.57
94.22
22.21
95,20
VGG-16 333
67.94
81.60
54.33
88.17
67.81
83.68
32.99
92.59
VGG-16-BN 33
65.92
82.00
50.49
89.03
59.02
86.34
35.12
92.05
EfficientNetV2
57.57
83,96
75.29
71.10
48.28
88.01
43.81
89.76
RegNet 35
65.37
82.85
59.46
85.51
34.65
92.53
26.18
94.55
MohileNetV3 36
67.99
82.14
60.49
87.80
60.72
87.82
56.65
89.30
noise images the validation set, in line with previous works [8] Experimental results in Table demonstrate that our adaptive strategy outperforms this fixed strategy. The reason lies in that different features have distinct statistical distributions. Using fixed thresholds for different features will limit the performance of OOD detection.
Compatibility with Backbones this section, we further verify the compatibility of our method with different backbones. For a fair comparison, all methods are pretrained on ImageNet, and we report the average results on four OOD datasets of ImageNet. Compared with competitive post-hoc strategies, experimental results in Table 6 demonstrate that our method can achieve the best performance under different network architectures. These results validate the effectiveness and compatibility of our method. Meanwhile, we observe some interesting phenomena in Table 6 ReAct [8] points out that mismatched BatchNorm [31] statistics between ID and OOD lead to OOD data. In Table VGG-16 and VGG-16-BB refer to models without and with BatchNorm, respectively. We observe that no matter with or without BatchNorm, ReAct cannot achieve better performance than Energy, consistent with previous findings 332. Therefore, BatchNorm may not be the only reason for model overconfidence, and the network architecture also matters. Furthermore, Energy (23] generally outperforms MSP [20] with the exception of EfficientNetV2. which also reveals its limitation in compatibility. the future, we will conduct an in-depth analysis reveal the reasons behind these phenomena.
model overconfidence
4 Further Analysis
Combining features with logit outputs can achieve better performance in OOD detection [37. Therefore, we design another variant of VRA called VRA++, whose scoring function is defined as:
m
A
Σ g(2,) + log Σe
ch
(16)
where 2,, E [1, m] represents the i-th feature and 4 [1,c] represents the i-th logit output. This scoring function consists of two items: (1) Since we have maximized the gap between ID and OOD [in(g(2;)] Enst [=+)], we directly use the sum all rectified features 5m ((x) as the indicator:
(2) We also calculate the energy score logit outputs for OOD detection. These items are combined
using a balaneing factor A, Unlike VRA using piecewise functions, we further test the performance of the quadratic function g(z) -2² + a,z. By choosing proper Ce, this quadratic function can

---

Table Performance of + All methods are based on BiT
39 and pretrained on ImageNet.
Openlmage-O
Texture
iNaturalist
ImageNet-O
Average
Method
FR. +
AU.1
FR.
AU.
FR. 4
AU. 1
FR. +
AU.1
FR. 4
AU. 1
MSP 20
73.72
84.16
76.65
79.80
64.09
87.92
96.85
57.12
77.83
77.25
ODIN 21
72.83
85.64
74.07
81.60
70.75
86.73
96.85
63.00
78.63
79.24
Mahalanobis
64.32
83.10
14.05
97.33
64.95
85.70
70.05
80.37
53.34
86.63
Energy 25
73.42
84.77
73.91
81.09
74.98
84.47
96.40
63.59
79.68
78.48
ReAct g
54.97
88.94
50.25
90.64
48.60
91.45
91.70
67.07
61.38
84.52
ViM 37
43.96
91.54
04.69
98.92
55.71
89.30
61.50
83.87
41.47
90.91
VRA++
34.94
93.55
05.02
98.76
22.25
96.37
60.45
84.21
30.67
93.22
Table 8: Comparison of VRA variants. "Netl" and "Net2" refer to ResNet-50 and ResNett221101.
CIFAR-10
CIFAR- 100
ImageNet (Net1)
ImageNet (Ne12)
Method
FPR95 4
AUROC
FPR95 +
AUROC
FPR95 +
AUROC t
FPR95 4
AUROC t
VRA
17.74
96.47
47.12
90.21
25.49
94.57
34.95
93.34
VRA+
15.89
96.90
43.31
90.61
23.32
94.96
30.85
93.78
VRA++
15.52
96.87
35.20
91.80
18.63
95.75
25.92
94.60
also simulate suppression and amplification operations. Finally, our scoring function is defined as:
2
ai 23) + log
(17)
Among all methods, ViM 337 is a powerful strategy that combines features and logit outputs. For fair comparison with ViM,
the same IDD data (ImageNet), OOD data (OpenImage-O
37]. Texture [1]. iNaturalist [8], and ImageNet-O (38). and network architecture (BiT (3)..
Experimental results in Table 7 demonstrate that VRA++ achieves better performance than ViM, verifying the scalability and high potential of our method. Meanwhile, VRA+ generally achieves the best performance among all variants (see Table 8). These results further demonstrate the necessity of combining features and logit outputs in OOD detection.
5 Related Work
Post-hoc strategies are an important branch of OOD detection. Due to their ease of implementation, they have attracted increasing attention from researchers. Among them. MSP [20] was the most basic post-hoc strategy, which directly leveraged the maximum value of the posterior distribution as the indicator. Since then, researchers have proposed various post-hoc approaches. For example, ODIN [2]] used temperature scaling and input perturbations to improve the separability of ID and OOD data. Energy 23 replaced the softmax confidence score in MSP 20 with the theoretically guaranteed energy score. Mahalanobis [22] used the minimum distance from the class centers identify OOD data. KNN 24 was a nonparametric method that explored K-nearest neighbors. More recently, researchers have found that the reason behind model overconfidence in OOD data lies in abnormally high activations of a small number of neurons. To address this, Dice 10 used weight sparsification, while ReAct [8] exploited activation truncation. Different from these works, we further demonstrate that abnormally low activations also affect OOD detection performance. This motivates to propose VRA to rectify the activation function.
Post-hoc Method
Activation Function Activation functions are an important part of neural networks 40 Previously, researchers found that neural networks with the ReLU activation function produced abnormally high activations for inputs far from the training data, harming the reliability of deployed systems (42] To address this problem, ReAct used truncation operation rectify activation functions. In this paper, we propose more powerful rectified activation function for OOD detection. Experimental results on multiple benchmark datasets demonstrate the effectiveness of our method.
Variational Method The variational method is often used to solve for the functional extreme value. Its most famous application in neural networks the variational autoencoder [43, which solves for the functional extreme value by trading off reconstruction loss and Kullback-Loiller divergence. It

---

has also been applied to other complex scenarios 44 and multimodal tasks 45 In this paper, we use the variational method find the operation that can maximize the gap between ID and OOD.
Limitation
Distributions of ID and OOD data impact the performance of VRA. In the future, will conduct a theoretical analysis to explain the reason behind this phenomenon. Mcanwhile, analogous to previous works such ReAct and ASH, this paper mainly focuses on the pre-trained classifiers with ReLU-based activations. Although we have explored some other architectures in the appendix, future experiments in more structures are also needed.
In this paper, we treat max, Enn [g(=]] [=)] as the core objective function derived from ReAct and ming [g) =) as the regularization term. However, there may better regularization terms that can not only guarantee the existence of the optimal solution but also ensure that the expression of the optimal solution easy to implement and has good interpretability. Therefore. we will explore other regularization terms for OOD detection. Mcanwhile, this paper uses simple piecewise functions to approximate the complex optimal operation. In the future, we will explore other functional forms that can better describe the optimal operation.
7 Conclusion
This paper proposes a post-hoc OOD detection strategy called VRA. From the perspective of the variational method, we find the theoretically optimal operation for maximizing the gap between ID and OOD. This operation reveals the necessity of suppressing abnormally low and high activations and amplifying intermediate activations in OOD detection. Therefore, we propose VRA to mimic these suppression and amplification operations. Experimental results show that our method outper- forms existing post-hoc strategies and is compatible with different scoring functions and network architectures. In the ideal case of knowing small fraction of OOD samples, we can achieve near- perfect performance, demonstrating the strong potential of our method. Meanwhile, we verify the effectiveness of our adaptively adjusted strategy and reveal the impact of different hyper-parameters.
8 Acknowledge
This work is supported by the National Natural Science Foundation of China NN..62201572, No.61831022, No.62276259, No.021B2010, No.62271083), Beijing Municipal Science & Technology Commission, Administrative Commission of Zhongguancun Science Park (No.2211 10000482013), Open Research Projects of Zhejiang Lab (NO. 2021 3HOABOO)))) CCF. Baidu Open Fund (No.OF2022025).
10

---

References
[1] Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng. Haoqi Wang. Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. In Proceedings of the Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, pages 1-14, 2022.
[2] Alexander Amini, Ava Soleimany, Sertac Karaman, and Daniela Rus. Spatial uncertainty sampling for end-to-end control. arXiv preprint 1805. 2018.
[3] Tanya Nair, Doina Precup, Douglas L. Arood, and Tal Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Proceedings of the Medical Image Computing and Computer Assisted Intervention, MICCAI, pages 655-665 2018.
[4] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710-8719, 8719, 2021.
[5] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't t know by virtual outlier synthesis. In Proceedings of the International Conference on Learning Representations, pages 2022.
[6] Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximum classifier discrepancy. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 951 9518-9526, 2019.
[7] Jingkang Yang, Haoqi Wang. Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent -fiis--iiittitttionn detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8301-8309, 2021.
[8] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribation detection with rectified activations. In Proceedings of the Advances in Neural Information Processing Systems, pages 144-157, 2021.
[9] Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.
[10] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In Proceedings of the European Conference on Computer Vision pages 691-708, 2022.
[11] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[12] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[13] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, pages 1-9, 2011.
[14] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452-1464 2017.
[15] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arviv: 533663335, 2015.
[16] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint 1rXivv1504.06755, 2015.
11

---

[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 248-255 2588-255222 2009.
[18] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8769-8778, 2018.
[19] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3485-3492 2010.
[20] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribuiie examples in neural networks. In Proceedings of the International Conference on Learning Representations, pages 1-12, 2017.
[21] Shiyu Liang. Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribttion image detection in neural networks. In Proceedings of the 6th International Conference on Learning Representations, pages 1-27, 2018.
[22] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Proceedings of the Advances in Neural Information Processing Systems, pages 7167-7177, 2018
[23] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based dutto--distribution detection. In Proceedings of the Advances in Neural Information Processing Systems, pages 21464-21475, 2020.
[24] Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Out-of Dut-of-distributtin detection with deep nearest neighbors. In Proceedings of the International Conference on Machine Learning, pages 20827-20840, 2022.
[25] Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du Zelin Li, Gang Wang. Shi Han, and Dongmei Zhang. ditt-f-diitiibutti detection based on in-distribution data patterns memorization with modern hopfield energy. In Proceedings of the Eleventh International Conference Learning Representations, pages 1-19, 2023.
[26] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In Proceedings of the International Conference on Learning Representations, pages 1-95, 2021.
[27] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference Learning Representations, 2022.
[28] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 8026-8037, 2019.
[31] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the International Conference on Machine Learning, pages 448-456, 2015.
12

---

[32] Yeonguk Yu, Sungho Shin, Seongju Lee, Changhyun Jun, and Kyoobin Lee. Block selection method for using feature norm in out-of-distrbuutinn detection. arXiv preprint arXiv:2212.02299, 2022.
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: |r09v:1409.1 /556, 2014.
[34] Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In Proceedings of the International Conference on Machine Learning, pages 10096-10106, 2021.
[35] Iliaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428- 10436, 2020.
[36] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1314-1324, 2019.
[37] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of distiiiiittiittion with virtual-logit matching. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4921-4930, 2022.
(38) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardr, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[39] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Proceedings of the European Conference on Computer Vision, pages 491 1-5-07 2020.
[40] Vinod Nair and Geoffrey E Hinton, Rectified linear units improve restricted boltzmann machines.
In Proceedings of the International Conference Machine Learning, pages 807-814, 2010.
[41] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rXiv:1606.08415 2016.
[42] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41-50, 2019.
[43] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations, pages 1-14, 2014.
[44] Bing Yu al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
[45] Gaurav Pandey and Ambedkar Dukkipati. Variational methods for conditional multimodal deep learning. In Proceedings of the International Joint Conference on Neural Networks (IJCN),, pages 308-315, 2017.
[46] Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. Cider: Exploiting hyperspherical embeddings for out-of-distribution detection. arXiv preprini arXiv:2203.04450, 2022.
[47] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2020.
[48] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via
contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 11111181111151 2020.
13

---

[49] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[50] Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue, Xiang Tian, Yaowu Chen, et al. Boosting out-of-distribution detection with typical features. Advances in Neural Information Processing Systems, 07220788-20666 2022.
[51] Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection, Advances in Neural Information Processing Systems, 15:17885-17898, 2022.
14

---

A Contrastive Learning Trained Backbones
Many post-hoc methods are conducted dlassification-bseee backbones 8 27 In this section, further perform experiments on backbones trained on contrastive learning. We use the pre-trained model from [46 and apply VRA the normalized features, followed by KNN 24 for OOD detection. In Table | we observe that for the contrastivelearning baaninng-based backbone, modifying the feature values and suppressing larger and smaller activation values also beneficial to OOD detection.
Table 9: Comparison with contrastive learning trained backbones (SSD+147), CSI and CIDER). We use ResNet-34 for CIFAR-100, Results of SSD+, CSI and CIDER reported from
CIDER 46..
SVIIN
Places365
LSUN
ISUN
Textures
Average
Method
FR.
AU. 1
FR.
AtL
FR.
AU. 1
FR. 4.
AU.
FR.
AU. 1
FR.
AU. t
SSD+ [1]
31.2
94.2
77.7
79.9
79.4
85.2
80.9
84.1
66.6
86.2
67.2
85.9
CSI [2)
44.5
92.7
79.1
76.3
75.6
83.8
76.6
85.0
61.6
86.5
67.5
84.8
[3]
79.6
87.7
CIDER+VRA
20.5
95.5
78.9
71.0
11.8
97.4
61.3
83.5
36.4
92.1
41.9
87.9
B
Compatibility with VIT 49
Previous works 8 10 27 usually only consider the condition where the activation values are non- negative. To further illustrate the potential of VRA, we conduct experiments on VIT [49] with large number negative activation values.
For positive values, ReAct tries to increase Em [=(])) data suppress more than ID data. Therefore, for negative values, we should also make OOD data
[g]]g(=)). It uses operation to make OOD
suppress more than ID data. This process results in an increase of Em [[(]]]] Epar g()]. To
unify the positive and negative cases, we should maximize sgn(=) (EE [g(=] Enwn [g( (2)]) where sgn(-) is the sign function. Similar to Eq. 3~13, we can get the optimal activation:
9 (2) = 2 + Asgn(=) (1 (1
Paul (ou((2)
(in(2)
PG
g
Peir
w/o lsuncation
08
Empirical PDF
0.5
Value
-0.5
Mh
-3.0
0.2
0.0
-20
-1.
3.0
45
-8.5
(a) empirical PDF ImageNet
(b) activation functions ImageNet (VIT)
Figure 3: Empirical PDFs for (-n(:) and Powl (-) and visualization of different activation functions. We use VIT for ImageNet.
We visualize the optimal function g (-) on different ID OOD data in Figure We observe
different (-) in VIT compared to ReLU-based backbones. Specifically, should amplify activations with low absolute features and suppress activations with high absolute features. To mimic this operation, we design a new piecewise function called VRA-VIT.
0 < -a
VRA-VIT(2)
82, - < S
a,
15

---

Table 10: Compatibility with VIT. We conduct experiments on ImageNet.
Backdone
Mothod
INaturalist
SUN
Textures
Average
FR.
AU.
FR.
AU.
FR.
AUL
FR.
AU.
FR.
AU.
Energy
64.08
79.24
72.77
70.24
74.31
68.44
58.47
79.30
67.41
74.30
Finergy + ReAct
85.66
75.84
64.99
Fnergy VRA-VIT
55,80
89.96
66.42
84.02
68.29
82.46
53.44
86.57
60.99
85.76
Energy
V.T.B.32
Energy + ReAct
73.39
84,23
80.08
76.01
80.63
74,69
66.77
82.73
75.22
79.41
Fnergyy +
88.03
79.59
Energy
60.16
80.31
75.23
69.77
78.21
69.02
60.30
79.25
68.48
74.59
VIT--16
Finergy + ReAct
Fnergy + VRA-VIT
57.24
90.49
71.07
$2.84
71.44
82.31
62.45
86. 16
65.55
85.45
Energy
VIT-L-32
Energy + ReAct
68.54
83.12
74.58
76.51
76.32
75.009
633.00
83.12
70.61
79.09
Finergy +
89.61
$2.02
where > 0 controls the threshold for determining low and high activations, and B >> 0 controls the gradient. Results in Table IO show the effectiveness of VRA, where we set = and 3 = 1.5.
C More variance of VRA
The theoretical derivation of VRA and analysis of ID and OOD activations allow us design other variants of VRA. In this section, we denote these new variants as VRA-B (binary ) and VRA-G (change the gradient of the medium):
VRA-B(2) = 10,25a a
10,2>0
0,,2<a
VRA-G(2)
kr,a
B,2> 3
where a and |3 are thresholds and & controls the scaling factor. For VRA-G, we treat the 1/. -uanntile (or "ssquuattlee) of activations estimated on ID data (or 3), in line with VRA. In Table I1. investigate the performance different VRA-based variants. For fair comparison, we use the same ID data, OOD data, and backbone. Experimental results demonstrate that VRA++ outperforms other variants OOD detection.
Table 11: Performance of different VRA-based variants. We use MenseNet-111 for CIFRA-10 10.
CIFAR-10
Method
FPR95
AUROC
VRA-B (a 0.5)
30.01
94.02
VRA-B (a 0.6)
20.37
95.84
VRA-B (a = 0.7)
21.18
95.64
VRA-B (a 0.8)
35.84
93.31
VRA-G (k 0.5)
28.14
94.12
VRA-G (k 0.8)
19.62
96.11
VRA-G (k = 1.0)
17.74
96.47
VRA-G (k 1.2)
17.32
96.64
VRA-G (k = 1.5)
16.81
96.66
VRA-G (k 2.0)
17.05
96.56
D Discussion with BATS 50 and RankFeat 51
BATS 50 is an OOD detection method that relies on BatchNorm. The main difference between BATS and our VRA lies in the following aspects. (1) BATS exploits the phenomenon that the features of the
16

---

training data more frequently fall in the interval [a Ao + λg]. Differently, VRA does not depend on BatchNorm. tries to find the theoretically optimal operation to maximize the gap between ID and OOD. (2) BATS relies on BatchNorm and uses the mean and standard deviation in BatchNorm to determine thresholds low and high activations. Differently, VRA does not rely on BatchNorm and determines thresholds for low and high activations using quantiles la and f]3- Experimental results in Table show that VRA can achieve good performance no matter with or without BatchNorm (see VGG-16 and VGG- 16--1,. Therefore, our VRA has better compatibility with different backbones compared with BATS. (3) BATS only suppresses high and low activations, but VRA+ and VRA++ further amplify intermediate activations. According to Table 12 this amplification process further improves the OOD detection performance.
Table 12: Comparison with BATS. For ImageNet, we use the pretrained ResNet50, For CIFAR, we use the pre-trained DenseNet-12 121.
CIFAR-10 1-
CIFAR-I00
ImageNet
Method
FPR95 + AUROC f
FPR95
AUROC
FPR95 4
AUROC
BATS
24.30
95.32
59.32
86.79
27.11
94.28
VRA
17.74
96.47
47.12
90.21
25.49
94.57
VRA+
15.89
96.90
43.31
90.61
23.32
94.96
VRA++
15,52
96.87
35.20
91.80
18.63
95.75
RankFeat [51] finds that the feature maps of OOD samples often have large singular values, and removing these values are beneficial to OOD detection. We compare VRA with RankFeat, and the experimental results are shown in Table [3] We observe that our method has significant improvement compared to RankFeat, especially the OOD datasets iNaturalist and Texture.
Table 13: Comparison with RankFeat. We use ResNetv2-101 for ImageNet. The result of RankFeat is reported from RankFeat [5]].
iNaturalist
SUN
Places
Textures
Average
Method
FR. 4
AU.
FR. 4
AU. +
FR. +
AU. f
FR. 4
AU. f
FR. +
AU.
RankFeat (Block 4)
46.54
81.49
27.88
92.18
38.26
88.34
46.06
89.33
39.69
87.84
RankFeat (Block 3)
49.61
91.42
39.91
92.01
51.82
88.32
41.84
91.44
45.80
90.80
RankFeat (Block 3+4)
41.31
91.91
29.27
94.07
39.34
90.93
37.29
91.70
36.80
92.15
VRA VRA+
27.26
95.68
34.53
93.27
47.31
90.19
30.69
94.22
34.95
93.34
20.81
97.70
32.89
92.68
45.83
90.01
23.88
95.43
30.85
93.71
E Upper Bound Analysis
Similar RankFeat, we provide theoretical analysis from the perspective of boundary analysis. Consider output logits [z) Wz + b, where is the feature vector of the penultimate layer. We denote the ith element in fzz) as fi, and c the number of categories. We compute the energy score of the output logits fzz as the OOD score, consistent with RankFeat:
S(z) = log
log
g max fo + max fi S log + [l
==
For any norm [1-11p(p > 0), there exist k > and lo satisfying:
k flllf(z)ll < [lf(z))l S (z))l
Then, get:
S(z) < 1z Wz + blpp + logc.
According to the triangle inequality W2 + b p
< Wxllp + ||bl|p and consistence of matrix norms
|>|lp S ||||||| we can get:
S(z) < 1o
W | ||1 + 12 + log
17

---

According to the above inequality, maximizing E. [m
Eou[2] is equivalent to maximizing the upper bound gap between ID and OOD. Therefore, the basic idea of VRA is similar RankFeat. Specifically, RankFeat removes the rank-1 matrix from the high-level feature and increases the upper bound gap between and OOD; VRA maximizes Em [z] [a [a] to maximize the upper bound gap between ID and OOD. Therefore, we can prove VRA from the similar perspective of RankFeat.
F Visualization on CIFAR
Similar to Figure D we plot the activation distribution of ID and OOD data for the DenseNet- 101 trained on CIFAR. As shown in Figure 4 we observe similar distribution pattern to the ResNet-50 trained on ImageNet. Our designed modification function also exhibits the property of suppressing both ends and amplifying in the middle.
Di
16
De
Past
Paul
Empirical PDF
Empirica
0.4
0.1
02
(a) empirical PDF CIFAR-10
(b) empirical PDF CIFAR-100
0.8
(-)
truncation
0.5
value
0.2
0.2
[-]
0.1
w/c truncation
0.
0.2
0.
0.4
0.5
0.6
(c) activation functions CIFAR-10
(d) activation functions CIFAR-100
Figure 4: Empirical PDFs for (.)(.) and Pour (-) and visualization of different activation functions. We
treat CIFAR ID data and DenseNet-101 101 the backbone.
G Distribution of Energy Score
We plot the distribution energy scores before and after VRA modification in Figure After using VRA modification, the overlap between the energy scores of ID and OOD data becomes less. This phenomenon indicates that VRA makes it easier distinguish ID and OOD data.
18

---

ID
0OD
0.20
05-
Empirical PDF
Empirical
0.00
0.
Scores
Scores
(a) before rectificatiin (overlap: 20.25%)
(b) after rectification (overlap: 12.52%)
Figure 5: Distribution of scores before and after variational rectification. We treat ImageNet ID) data and use ResNet-50 as the backbone.
19


Reviews:
[{"review_id": "6IiEJzGZoT", "reviewer": "Reviewer_GN4S", "summary": "In this paper, the authors explore the out-of-distribution detection ability. They propose Variational Rectified Activation based on ReAct. Specifically, they suppress both the high and low values of the penultimate layer rather than only focusing on high values. Their performance on several benchmark dataset surpasses existing post-hoc strategies. ", "questions": "1. In equation 3, why it is needed to maximally preserve the input? And with this term, why the optimal operation maximize the gap between ID and OOD?\n\n2. In figure 1, your optimal g function does not seem to agree with VRA function. Have you ever try to adjust the gradient when z is between alpha and beta?", "limitations": "It seems that the paper did not include a limitation section or paragraph explicitly. ", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. They give a theoretical analysis to the proposed method.\n2. They compare VRA-based methods with competitive post-hoc strategies, and performs the best.", "weaknesses": "1. The theory part should be more clear about the relationship between the objective you minimize and the final score.\n2. Please add more baselines to demonstrate the effectiveness of VRA, comparing your method with contrastive learning trained backbones (SSD+[1], CSI[2], CIDER[3]), and other state-of-the-art post-hoc methods (maxlogit[4] and HEAT[5]). \n\n\n\n[1] https://openreview.net/forum?id=v5gjXpmR8J\n\n[2] https://openreview.net/forum?id=o5RKoLQlK4olF\n\n[3] https://openreview.net/forum?id=aEFaE0W5pAd\n\n[4] https://arxiv.org/abs/1911.11132\n\n[5] https://openreview.net/forum?id=tpCynHFviX", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer GN4S\nWe thank the reviewer's appreciation of our theoretical analysis and rich experimental results. We try to address your concerns as follows:\n\n**Q1**: The theory part should be more clear about the relationship between the objective you minimize and the final score.\n\n**A1**: ReAct has proved that increasing the gap between ID and OOD (see Eq. 2) can improve the final score in OOD detection. We adopt the same objective function as ReAct and try to find the optimal operation to widen the gap. Therefore, the theoretical proof of the relationship between the objective function and final scores is equivalent to ReAct. Due to page limitations, we have not included these parts.\n\n**Q2**: Please add more baselines to demonstrate the effectiveness of VRA, comparing your method with contrastive learning trained backbones (SSD+, CSI, CIDER), and other state-of-the-art post-hoc methods (MaxLogit and HEAT).\n\n**A2**: Due to the good compatibility of VRA, for a fair comparison, we test the performance of VRA under the same experimental settings as the baselines (including ID data, OOD data, and network architecture). For contrastive-learning-based methods, we conduct experiments under the same settings as CIDER. For post-hoc methods, we conduct experiments under the same settings as HEAT. Experimental results in Tables 1$\\sim$2 demonstrate that VRA outperforms existing OOD detection strategies.\n\n\n**Table1: Comparison with contrastive learning trained backbones. We use ResNet-34 for CIFAR-100.**\n\n| Method | FR $\\downarrow$ / AU$\\uparrow$ |\n| ------ | --------------------------------------- |\n| SSD+   | 67.2 / 85.9                             |\n| CSI    | 67.5 / 84.8                            |\n| CIDER  |  46.9 / 87.7                           |\n| VRA    | **41.9** / **87.9**                    |\n\n\n\n**Table 2: Comparison with post-hoc methods. We use ResNet-50 on ImageNet.**\n\n| Method   | FR $\\downarrow$ / AU$\\uparrow$ |\n| -------- | --------------------------------------- |\n| MaxLogit |  58.0 / 87.0                           |\n| HEAT     |  34.4 / 92.6                           |\n| VRA      | **25.5** / **94.6**                      |\n\n\n**Q3**: In equation 3, why it is needed to maximally preserve the input? And with this term, why the optimal operation maximize the gap between ID and OOD?\n\n**A3**: We apologize for this unclear description. In this paper, we treat $\\max_g  \\mathbb{E}{\\text{in}}[g(z)] - \\mathbb{E}{\\text{out}}[g(z)]$ as the core objective function derived from ReAct and $\\min_g  \\mathbb{E}{\\text{in}}[(g(z)-z)^2]$ as the regularization term. The motivation behind this regularization term is twofold: (1) Many post-hoc methods do not change the feature space but also achieve promising results on OOD detection (such as MSP, Energy, and ODIN). Therefore, $g(z)=z$ is an acceptable operation. (2) If we only maximize $\\mathbb{E}{\\text{in}}[g(z)] - \\mathbb{E}{\\text{out}}[g(z)]$ without the regularization term, the optimal solution will not exist. The proof is by contradiction. Suppose there is an optimal operation $g^*(\\cdot)$. Then, we can easily get a better operation $2g^*(\\cdot)$ with larger $\\mathbb{E}{\\text{in}}[g(z)] - \\mathbb{E}{\\text{out}}[g(z)]$. Therefore, this regularization term can guarantee the existence of the optimal solution. There may be better regularization terms. In the future, we will explore other regularization terms for OOD detection.\n\nIn Eq. 14$\\sim$15, we prove that the optimal operation $g^*(\\cdot)$ driven by our objective function can widen the gap between ID and OOD by at least $\\frac{1}{2\\lambda} \\mathbb{E}{\\text{in}}[(g^*(z)-z)^2] \\geq 0$. Therefore, adding this term does not change the core objective of VRA. Meanwhile, Table 4 (see the submitted paper) shows that $g^*(\\cdot)$ can achieve near-perfect results on OOD detection. Therefore, although this regularization term slightly changes the objective function, it does not significantly affect the performance of OOD detection but ensures the existence of the optimal solution. Just like the regularization term in neural networks. Although the regularization term changes the loss function, it prevents overfitting and improves the generalization ability of neural networks.\n\n**Q4**: In figure 1, your optimal g function does not seem to agree with VRA function. Have you ever try to adjust the gradient when z is between alpha and beta?\n\n**A4**: Good idea! In this paper, VRA+ introduces a hyper-parameter $\\gamma$ to amplify intermediate activations. However, we have not considered adjusting the gradient at $\\alpha \\leq z \\leq \\beta$. We denote this new variant as VRA-G:\n\\begin{equation}\n    \\text{VRA-G}(z)=\\begin{cases}\n        0, z< \\alpha\\nonumber \\\\\\\\\n        kz, \\alpha \\leq z \\leq \\beta \\nonumber \\\\\\\\\n        \\beta, z > \\beta \\\\\\\\\n    \\end{cases},\n\\end{equation}\nwhere $k$ controls the gradient. Same with VRA, we treat the $\\eta_\\alpha$-quantile (or $\\eta_\\beta$-quantile) of activations estimated on ID data as $\\alpha$ (or $\\beta$). It is worth noting that VRA-G with $k=1.0$ is equivalent to VRA. Experimental results in Table 3 show that setting the gradient a little steeper (i.e., $k>1$) can amplify intermediate activations and achieve better performance. But over-amplification leads to performance degradation.\n\n\n\n**Table 3: Performance of VRA-G. We use DenseNet-101 for CIFAR-10.**\n\n| Method          | CIFAR-10 (FR $\\downarrow$ / AU$\\uparrow$) |\n| --------------- | ----------------------------------------- |\n| VRA             | 17.74 / 96.47                        |\n| VRA+            | **15.89** / **96.90**                   |\n| VRA-G $(k=0.5)$ | 28.14 / 94.12                         |\n| VRA-G $(k=0.8)$ | 19.62 / 96.11                        |\n| VRA-G $(k=1.0)$ | 17.74 / 96.47                        |\n| VRA-G $(k=1.2)$ | 17.32 / 96.64                       |\n| VRA-G $(k=1.5)$ | **16.81** / **96.66**                 |\n| VRA-G $(k=2.0)$ | 17.05 / 96.56                        |\n\n\n\n"}, {"review_id": "cvjAQ0oDpt", "reviewer": "Reviewer_Tdya", "summary": "This work leverages the variational method to find optimal operation and proposes a new technique variational rectified activation (VRA) for out-of-distribution (OOD) detection. This paper finds the best operation for OOD detection and verifies the necessity of suppressing abnormally low and high activations and amplifying intermediate activations. The proposed VRA method is compatible with different network architectures and scoring functions. Extensive experiments on a number of benchmark datasets show the effectiveness of the proposed method. ", "questions": "Refer to the detailed comments on weaknesses.", "limitations": "This work does not present any potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work tackles out-of-distribution (OOD) detection tasks, which is important for building reliable machine learning models in the real world.\n2. The key idea of variational rectified activation that mimics suppression and amplification operations using piecewise functions is simple and flexible to compatible with different scoring functions and network architectures. This work also provides a theoretical understanding of the VRA method for OOD detection from the perspective of the variational method.\n3. This paper conducts extensive experiments on several benchmark datasets to validate the effectiveness of the proposed VRA method.", "weaknesses": "1. Can the authors report the mean and std for the main results in the experiment section?\n2. It would be interesting if the authors can compare the visualization of the distribution of ID and OOD uncertainty scores before and after variational rectification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Tdya\n\nWe thank the reviewer's appreciation of our clear writing and sufficient experimental results. We try to address your concerns as follows:\n\n**Q1**: Can the authors report the mean and std for the main results in the experiment section?\n\n**A1**: Thanks for your valuable comments. Since the standard deviations are relatively small, we did not include them in the main results. In Table 1, we take the main results on CIFAR-10 as an example.\n\n**Table 1: Mean and std for the main results. We use DenseNet-101 for CIFAR-10.**\n| Method        | CIFAR-10 (FR $\\downarrow$ / AU$\\uparrow$) |\n| ------------- | ----------------------------------------- |\n| MSP           |  48.74$\\pm$0.00 / 92.46$\\pm$0.00     |\n| ODIN          |  24.57$\\pm$0.00 / 93.71$\\pm$0.00     |\n| Mahalanobis   |  31.42$\\pm$0.00 / 89.15$\\pm$0.00     |\n| Energy        |  26.55$\\pm$0.00 / 94.57$\\pm$0.00     |\n| ReAct         |  26.45$\\pm$0.00 / 95.30$\\pm$0.00     |\n| KNN           |  25.83$\\pm$0.00 / 94.39$\\pm$0.00     |\n| DICE          |  20.84$\\pm$1.58 / 95.25$\\pm$0.24     |\n| SHE           |  26.82$\\pm$0.00 / 92.98$\\pm$0.00     |\n| VRA  |  17.74$\\pm$0.00 / 96.47$\\pm$0.00     |\n| VRA+ |  15.85$\\pm$0.00/ 96.91$\\pm$0.00  |\n\n\n\n\n\n**Q2**: It would be interesting if the authors can compare the visualization of the distribution of ID and OOD uncertainty scores before and after variational rectification.\n\n**A2**: Thanks for your valuable suggestion, In Figure 1 (see attached PDF in Global Response), we visualize the distribution of ID and OOD uncertainty scores before and after variational rectification. We observe that our rectification operation can reduce the overlap between ID and OOD data, verifying the effectiveness of our strategy."}, {"review_id": "2aWsQPimkz", "reviewer": "Reviewer_J9PQ", "summary": "The authors propose a technique called \"Variational Rectified Activation (VRA)\", which simulates these suppression and amplification operations using piecewise functions. Theoretical analysis is provided to illustrated VRA. Extensive experiments demonstrate the effectiveness and generalization for the proposed method.", "questions": "1. Please compare the VRA framework with [1] and discuss the similarities and differences between the two.\n2. In Fig.1, I'm curious about how the optimal function is look like if you use ViT as the classification model, and CIFAR as ID dataset. Could you provide more visualizations in the appendix.\n\n&nbsp;\n\n[1] Boosting Out-of-distribution Detection with Typical Features. NeurIPS 2022.", "limitations": "Please see the comments above.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It is technically sound to use variational methods to find the optimal function that maximizes the gap between ID and OOD.\n2. The whole method is simple and effective, and performs well in different datasets and models. The proposal is also compatible with different scoring functions.\n3. The paper is well written, well-structured and easy to understand.\n", "weaknesses": "1. The proposed method of simultaneously truncating high and low activations is very similar to the existing method in [1], which also corrects features by suppressing high and low activations, thus limiting the novelty of this proposal.\n2. The formula in Eq.3 confuses me a bit. Why maximizing Ein(z) - Eout(z) maximizes the gap between ID and OOD. Why not maximize the squared difference? What happens if you maximize Eout(z) - Ein(z)?\n3. The VRA piecewise function is designed based on the feature histograms in one model and one ID dataset, which may not hold for other datasets and model architectures. For example，for a Transformer-based model, the features in the penultimate layer contains many negative values，how the piecewise function should be in Transformer-based function. \n\n&nbsp;\n\n[1] Boosting Out-of-distribution Detection with Typical Features. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer J9PQ \nWe sincerely thank the reviewer for your detailed comments and suggestions. We try to address each comment as satisfactorily as possible:\n\n**Q1**: The proposed method of simultaneously truncating high and low activations is very similar to the existing method in BATS, which also corrects features by suppressing high and low activations, thus limiting the novelty of this proposal. Please compare the VRA framework with BATS and discuss the similarities and differences between the two.\n\n**A1**: Thank you for your valuable comments. Although VRA and BATS  share some similarities, they are quite different in motivation, method, and results:\n\n*Motivation*: BATS relies on BatchNorm. For OOD detection, BATS exploits the phenomenon that the features of the training data more frequently fall in the interval $[ \\mu -\\lambda  \\sigma, \\mu + \\lambda \\sigma ]$. Differently, VRA does not depend on BatchNorm. It tries to find the theoretically optimal operation to maximize the gap between ID and OOD. \n\n*Method*: BATS relies on BatchNorm and uses mean and std to determine thresholds for low and high activations. Differently, VRA does not rely on BatchNorm and determines thresholds for low and high activations by quantiles. And the supressing is different, we suppress towards zero, they suppress towards mean. Experimental results in Table 6 of our main text show that VRA can achieve good performance no matter with or without BatchNorm (see VGG-16 and VGG-16-BN). Therefore, our VRA has better compatibility with different backbones compared with BATS. In addition, VRA+ and VRA++ further amplify intermediate activations.  Table 1 shows amplification process further improves the OOD detection performance.\n\n*Results*: We further compare the performance of VRA-based methods and BATS, we use DenseNet-121 for CIFAR and ResNet50 for ImageNet. Experimental results in Table 1 demonstrate that VRA-based methods outperform BATS in OOD detection, verifying the effectiveness of our method. We will add these comparison results in the revised paper.\n\n**Table 1: Comparison with BATS. (FR / AU)**\n| Method |CIFAR-10 | CIFAR-100 | ImageNet |\n| - | - | - | - |\n|BATS | 24.30 /95.32|59.32/86.79|27.11/94.28|\n|VRA|17.74/96.47|47.12/90.21|25.49/94.57|\n|VRA+ |15.89 /96.90|43.31/90.61|23.32/94.96|\n|VRA++|15.52/96.87|35.20/91.80|18.63/95.75|\n\n**Q2**: The formula in Eq. 3 confuses me a bit. Why maximizing $E_{in}(z) - E_{out}(z)$ maximizes the gap between ID and OOD. Why not maximize the squared difference? What happens if you maximize $E_{out}(z) - E_{in}(z)$?\n\n**A2**: The objective function $max  E{in}g(z) - E_{out}g(z)$ is derived from ReAct, a well-known and effective OOD detection technique. ReAct proves that increasing $E_{in}g(z) - E_{out}g(z)$ leads to better OOD detection performance. In this paper, we extend the idea of ReAct and propose a new operation VRA to maximize $E_{in}g(z) - E_{out}g(z)$. Experimental results prove that we can achieve better performance than ReAct in OOD detection.\n\n**Q3**: The VRA piecewise function is designed based on the feature histograms in one model and one ID dataset, which may not hold for other datasets and model architectures. \n\n**A3**: Thanks for your valuable comments. In fact, we design VRA based on the results of multiple models and multiple ID datasets. Due to page limitations, we did not include all visualization results in this paper. In Figure 2 (see attached PDF in Global Response), we provide more visualization results and obverse the same phenomenon. We will add these visualization results in the appendix.\n\n**Q4**: For a Transformer-based model, how the piecewise function should be in the Transformer-based function. I'm curious about how the optimal function looks like if you use ViT as the classification model. Could you provide more visualizations in the appendix?\n\n**A4**: Good question! Our core function comes from ReAct, which relies on ReLU-based backbones where the penultimate layer does not contain negative values. For the case of many negative values, we conduct the following analogy analysis. \n\nFor positive values, ReAct tries to increase $E_{in}g(z) - E_{out}g(z)$. It uses an operation to make OOD data suppress more than ID data. Therefore, for negative values, we should also make OOD data suppress more than ID data. This process results in an increase of $E_{in}[-g(z)] - E_{out}[-g(z)]$. To unify the positive and negative cases, we should maximize $\\mbox{sgn}(z)(E_{in}g(z) - E_{out}g(z))$, where $\\mbox{sgn}(\\cdot)$ is the sign function. Similar to Eq. 3$\\sim$13, we can get the optimal activation:\n\\begin{align}\n    g^*(z) = z + \\lambda \\mbox{sgn}(z) \\left(1 - \\frac{p_{out}(z)}{p_{in}(z)}\\right).  \n\\end{align}\n\nWe visualize the optimal function $g^*$ on different ID and OOD data. Due to page limitations, we will put these visualization results in the appendix. We observe a different $g^*$ in ViT compared to ReLU-based backbones. Specifically, we should amplify activations with low absolute features and suppress activations with high absolute features. To mimic this operation, we design a new piecewise function called VRA-ViT:\n\\begin{equation}\n    \\text{VRA-ViT}(z)=\\begin{cases}\n        -\\alpha, z \\leq -\\alpha \\nonumber  \\\\\\\\\n        \\beta z, -\\alpha <  z < \\alpha\\nonumber \\\\\\\\\n        \\alpha,z \\ge \\alpha\n        \\nonumber \n    \\end{cases},\n\\end{equation}\nwhere $\\alpha>0$ controls the threshold for determining low and high activations, and $\\beta>0$ controls the gradient. Experimental results in Table 2 show the effectiveness of VRA.\n\n**Table 2: Compatibility with ViT on ImageNet.**\n|Method|Backbone|FR / AU|\n| - | - | - |\n|Energy|B/16|67.41/74.30|\n|+ReAct||64.99?80.74|\n|+VRA-ViT||60.99/85.76|\n|Energy|B/32|76.69/76.40|\n|+ReAct||75.22/79.41|\n|+VRA-ViT||65.98/84.20|71.21/ 83.11|\n|Energy|L/16|68.48/74.59|\n|+ReAct||64.99 / 81.67|\n|+VRA-ViT||62.45 / 86.16| 65.55 / 85.45|\n|Energy|L/32|72.22 / 73.98 |\n|+ReAct||70.61/79.09 |\n|+VRA-ViT|| 66.64/ 84.75 |\n"}, {"review_id": "i4F1ubSmfk", "reviewer": "Reviewer_JjUp", "summary": "This paper conducts a fine-grained analysis of ReAct, a simple but effective OOD method by rectifying high activation. The authors perform a variational method to analyze the optimal activation function by clipping and amplifying features. Strong empirical results are presented to support the effectiveness of the proposed methods. ", "questions": "Please see weakness and limitations.", "limitations": "My primary concern is the availability of OOD statistics required by this method. I suggest the authors at least explicitly mention it in the limitation section.  ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is well-motivated both intuitively and theoretically. ReAct presented a promising solution to distinguish OOD samples by clipping activation. So it is natural to perform a fine-grained analysis of how to better clip activations. Actually, ASH [1] has a similar idea but this paper takes a step further to find the optimal solution of clipping from a theoretical point of view. The variational method seems valid and the assumption of Hilbert space is mostly valid for activations of deep neural networks. The theoretical analysis well motivates the methodology and shaping.\n\n2. The authors propose two different variants of the method: one by simply clipping as React, and the other with middle activation amplification. This allows more possibilities to perform OOD detection.\n\n3. The empirical results are very strong. In the ImageNet benchmark, it even outperforms MOS which is a training-needed approach.\n\n\n>[1] Extremely Simple Activation Shaping for Out-of-Distribution Detection. ICLR 23. ", "weaknesses": "1. My main concern is that the methodology seems to need to calculate $\\rho_{out}$ in advance. However, the statistics of OOD data are usually unknown to the model. So having access to the density might reveal some information about OOD data, which might be unfair to the baselines. This might limit the real-world usage of the method.\n\n2. There are some very relevant OOD papers missing in the reference [1,2,3,4]. As stated before, ASH[1] actually has done the similar fine-grained analysis of ReAct. Furthermore, the concept of shaping in ASH covers more operations such as pruning, binary, and scale. RankFeat [2] is also motivated by ReAct and removes the dominant single value of the last feature map. Moreover, it analyzes the lower bound of their method and ReAct. The bound analysis seems to apply to the method of this paper. The comparison and discussion with [1,2] are thus needed in the paper.\n\n3. Some more operations such as binary and scaling can be explored. Currently, only amplification and pruning are supported. \n\n>[1] Extremely Simple Activation Shaping for Out-of-Distribution Detection. ICLR 23. \n>\n>[2] RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer JjUp \nWe thank the reviewer's appreciation of the writing and sufficient experimental results. We try to address your concerns as follows:\n\n**Q1**: My main concern is that the methodology seems to need to calculate $p_{out}$ in advance. However, the statistics of OOD data are usually unknown to the model. So having access to the density might reveal some information about OOD data, which might be unfair to the baselines. My primary concern is the availability of OOD statistics required by this method. I suggest the authors at least explicitly mention it in the limitation section.\n\n**A1**: Thanks for your valuable comments. As you pointed out, VRA does need to know $p_{out}$. If we know $p_{out}$ in advance, we can achieve near-perfect OOD detection results (see Table 4). But in real-world scenarios, we cannot obtain real OOD data in advance. For a fair comparison with baselines, we use Gaussian noise images as virtual OOD data in our implementation (see Section 3.1). According to Tables 1$\\sim$2, we can still achieve state-of-the-art performance in OOD detection. Therefore, our comparison with baselines is fair. In the revised paper, we will add more discussions in the limitation section.\n\n**Q2**: There are some very relevant OOD papers missing in the reference. As stated before, ASH actually has done a similar fine-grained analysis of ReAct. Furthermore, the concept of shaping in ASH covers more operations such as pruning, binary, and scale. RankFeat is also motivated by ReAct and removes the dominant single value of the last feature map. The comparison and discussion with are thus needed in the paper.\n\n**A2**: Thank you for your valuable comments. Although VRA, ASH and RankFeat share some similarities, they are quite different in motivation, method, and results:\n\n*Motivation*: ASH argues that representations produced by over-parameterized neural networks are excessive for the task at hand, and therefore could be greatly simplified without much deterioration on the original performance while resulting\nin a surprising gain in OOD detection. RankFeat observes that the OOD feature matrix tends to have a larger dominant singular value than the ID feature, and the class predictions of OOD samples are largely determined by it. Different from ASH and RankFeat, VRA tries to find the theoretically optimal operation to maximize the gap between ID and OOD. Therefore, the motivations behind these methods are quite different.\n\n*Method*: Although all methods are extended from ReAct, their implementation process is different. ASH uses sample-wise pruning to remove a majority of the activation of the entire representation. Same to ASH, RankFeat is a also sample-wise pruning strategy. It removes the rank-1 matrix composed of the largest singular value and the associated singular vectors from the high-level feature. Differently, VRA follows ReAct and uses feature-wise pruning. According to Tables 1$\\sim$2, our method can achieve the best results among all baselines.\n \n*Results*: We compare the performance of VRA-based methods with ASH and RankFeat. For a fair comparison, we use the same ID data, OOD data, and network architecture. Experimental results in Tables 1$\\sim$2 demonstrate that VRA-based methods outperform previous works in OOD detection, verifying the effectiveness of our method. We will add these comparison results in the revised paper.\n\n**Table 1: Comparison with ASH. We use DenseNet-101 for CIFAR and ResNet-50 for ImageNet.**\n\n|Method|CIFAR-10(FR/AU)|CIFAR-100(FR/AU)|ImageNet(FR/AU)|Average(FR/AU)|\n|-|-|-|-|-|\n|ASH-P|23.45/95.22|64.53/82.71|50.32/89.04|46.10/88.99|\n|ASH-B|20.23/96.02|48.73/88.04|22.73/95.06|30.56/93.04|\n|ASH-S|15.05/96.61|41.40/90.02|22.80/95.12|26.42/93.92|\n|VRA|17.74/96.47|47.12/90.21|25.49/94.57|30.12/93.08|\n|VRA+|15.89/96.90|43.31/90.61|23.32/94.96|27.51/94.16|\n|VRA++|15.52/96.87|35.20/91.80|18.63/95.75|23.12/94.81|\n\n**Table 2: Comparison with RankFeat. We use ResNetv2-101 for ImageNet.**\n\n| Method   | FR/ AU |\n| - | - |\n| RankFeat (Block 4)|39.69/87.84|\n| RankFeat (Block 3)|45.80/90.80|\n| RankFeat (Block 3+4)|36.80/92.15|\n| VRA|34.95/93.34|\n| VRA+|**30.85**/**93.71**|\n\n**Q3**: RankFeat analyzes the lower bound of their method and ReAct. The bound analysis seems to apply to the method of this paper.\n\n**A3**: Thanks for your valuable comments. We have followed a similar method for analysis, but the space here is too small to write, and we will show it in our subsequent discussion and appendix.\n\n**Q4**: Some more operations such as binary and scaling can be explored. Currently, only amplification and pruning are supported.\n\n**A4**: Good idea! In this paper, we have not considered more operations such as binary and scaling. We denote these new variants as VRA-B and VRA-G:\n\\begin{equation}\n    \\text{VRA-B}(z)=\\begin{cases}\n        0, z \\leq \\alpha \\nonumber  \\\\\\\\\n        \\alpha, z> \\alpha\\nonumber \n    \\end{cases},\n\\end{equation}\n\n\\begin{equation}\n    \\text{VRA-G}(z)=\\begin{cases}\n        0, z< \\alpha\\nonumber \\\\\\\\\n        kz, \\alpha \\leq z \\leq \\beta \\nonumber \\\\\\\\\n        \\beta, z > \\beta \\\\\\\\\n    \\end{cases},\n\\end{equation}\nwhere $\\alpha$ and $\\beta$ are thresholds and $k$ controls the scaling factor. For VRA-G, we treat the $\\eta_\\alpha$-quantile (or $\\eta_\\beta$-quantile) of activations estimated on ID data as $\\alpha$ (or $\\beta$), in line with VRA. In Table 3, we investigate the performance of different VRA-based variants. Experimental results demonstrate that VRA++ outperforms other variants in OOD detection. \n\n**Table 3: Performance of different VRA-based variants with DenseNet-101 on CIFAR-10.**\n|Method|FR/AU|\n|-|-|\n|VRA|17.74/96.47|\n|VRA+|15.89/**96.90**|\n|VRA++|**15.52**/96.87|\n|VRA-B$(\\alpha=0.5)$|30.01/94.02|\n|$\\alpha=0.6$|**20.37**/**95.84**|\n|$\\alpha=0.7$|21.18/95.64|\n|$\\alpha=0.8$|35.84/93.31|\n|VRA-G(k=0.5)|28.14/94.12|\n|k=0.8|19.62/96.11|\n|k=1.0|17.74/96.47|\n|k=1.2|17.32/96.64|\n|k=1.5|**16.81**/**96.66**|\n|k=2.0|17.05/96.56|"}]

Author rebuttal:
# Global Response
Dear Reviewers, Area Chairs, and Program Chairs:

We would like to express our gratitude to all reviewers for taking their valuable time to review our paper. We sincerely appreciate all reviewers for their positive comments on our theoretical analysis and technique soundness. For example, Reviewer JjUp points out that "the theoretical analysis well motivates the methodology and shaping". Reviewer JjUp notes that the proposed method is "well-motivated both intuitively and theoretically". We also thank the reviewers for noting that this paper is "well-written, well-structured, and easy to understand" (J9PQ), the proposed method is "promising" (JjUp) and "allows more possibilities to perform OOD detection" (JjUp), the performance is "competitive" (GN4S), "effective" (Tdya, J9PQ) and "strong" (JjUp). Meanwhile, we appreciate the reviewers for pointing out the shortcomings. Your valuable comments help us improve this paper. We try to address each comment as satisfactorily as possible. In the response, we compare with more baselines (including SSD+, CSI, CIDER, MaxLogit, HEAT, BATS, ASH, and RankFeat), verify the compatibility of our method with more backbones (such as ViT), present more visualization results (including the distribution of ID and OOD uncertainty scores before and after variational rectification, and the optimal operation in CIFAR-10 and CIFAR-100), and study more VRA-based variants (such as VRA-B and VRA-G). Please find the responses to each reviewer’s comments below.

We kindly ask the reviewers to take the above clarifications into account when considering score adjustments. We welcome any further discussion with the reviewers.

Best regards,

Paper4986 Authors

Meta-review: This paper has received relatively diverging reviews. Reviewers Tdya and JjUp consider that the paper has enough contributions to be accepted, while reviewers GN4S and J9PQ expressed certain concerns. In particular, reviewer GN4S initially asked for clarifications on several questions and additional baselines. After these questions were answered, reviewer GN4S further raised concerns in novelty, contribution, related work, and technology, but did not further detail these concerns nor follow up with more discussions. Reviewer J9PQ  mainly had concerns in novelty (difference between this work and ReAct/BAST/ASH) and want to get clarifications on Eq. 3 and the choice of hyper-parameters. The last two are sufficiently addressed by the author response. In addition, AC considers the novelty of this work enough upon reading through the paper and all the discussions. Even though the work is an extension of ReAct, the contributions are motivated and empirically backed by the experiments. The AC thus recommends acceptance of this work to NeurIPS. 

The decision of this paper was communicated with the SAC, and was made based on the discussion.

The authors are highly encouraged to incorporate the feedback and rebuttal discussions/experiments into the final version and revise accordingly.
Decision: Accept (poster)