ReAct: Out-of-distribution Detection With Rectified Activations
Authors: Yiyou Sun
Chuan Guo
Yixuan Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: OOD detection

Abstract: Out-of-distribution (OOD) detection has received much attention lately due to its practical importance in enhancing the safe deployment of neural networks. One of the primary challenges is that models often produce highly confident predictions on OOD data, which undermines the driving principle in OOD detection that the model should only be confident about in-distribution samples. In this work, we propose ReAct—a simple and effective technique for reducing model overconfidence on OOD data. Our method is motivated by novel analysis on internal activations of neural networks, which displays highly distinctive signature patterns for OOD distributions. Our method can generalize effectively to different network architectures and different OOD detection scores. We empirically demonstrate that ReAct achieves competitive detection performance on a comprehensive suite of benchmark datasets, and give theoretical explication for our method’s efficacy. On the ImageNet benchmark, ReAct reduces the false positive rate (FPR95) by 25.05% compared to the previous best method.

ReAct: Out-of-distribution Detection With Rectified Activations
Yiyou Sun
Chuan Guo Facebook AI Research chuanguo@fb. com
Department of Computer Sciences University of Wisconsin-Madiion sunyiyou6cs. wisc. edu
Yixuan Li Department of Computer Sciences University of Wisconsin-Madison sharonl #cs. wisc edu
Abstract
Duttof-distributiin (OOD) detection has received much attention lately due its practical importance in enhancing the safe deployment of neural networks, One of the primary challenges that models often produce highly confident predictions on OOD data, which undermines the driving principle in OOD detection that the model should only be confident about in-distribution samples. In this work, we propose ReAct-a simple and effective technique for reducing model overcon- fidence OOD data. Our method is motivated by novel analysis on internal activations of neural networks, which displays highly distinctive signature patterns for OOD distributions. Our method can generalize effectively to different network architectures and different OOD detection scores. We empirically demonstrate that comprehensive suite of
ReAct achieves competitive detection performance benchmark datasets, and give theoretical explication for our method's s efficacy. On the ImageNet benchmark, ReAct reduces the false positive rate (FPR95) by 25.05% compared the previous best method"
Introduction
Neural networks deployed in real-world systems often encounter out-of-distribution (OOD) inputs unknown samples that the network has not been exposed during training. Identifying and handling these OOD inputs can be paramount in safety-critical applications such as autonomous driving O and health care. For example, autonomous vehicle may fail to recognize objects the road that do not appear in its object detection model's training set, potentially leading to a crash. This can be prevented if the system identifies the unrecognized object OOD and warns the driver in advance.
driving idea behind OOD detection that the model should be much more uncertain about samples outside of its training distribution. However, Nguyen et al. 3% revealed that modern neural networks produce overconfident predictions of in- in-distribution (ID) and OOD data a non non-trivial task. Indeed, much of the prior work on OOD detection focused on defining more suitable measures of OOD uncertainty [12 19
OOD inputs. This phenomenon renders the separation
28
29
BI [3
Despite the improvement, is arguable that continued research progress OOD detection requires insights into the fundamental cause and mitigation of model overconfidence OOD data.
In this paper, start by revealing important observation that OOD data trigger unit activation patterns that are significantly different from ID data. Figure b) shows the distribution of activations
Code available at: http:p////ggggt....oo///e/eeaaea
35th Conference on Neural Information Processing Systems (NeurIPS 202.1).

---

Before Rectification
After Rectification FPR95: 20.38%
OOD ID
FPR95: 55.72%
Frequency
Unit Activation
Frequency
OOD Scores (a)
Unit Indices (b)
OOD Scores
uncertainty scores before truncation, (b) the distribution per-unit activations in the penultimate layer for ID and OOD data, and (c) the distribution of OOD uncertainty scores (33) after rectification. Applying ReAct drastically improves the separation of ID and OOD data. See text for details.
Figure 1: Plots showing (a) the distribution of ID (ImageNet ? and OOD (iNaturalist 16]
in the penultimate layer of ResNet-50 trained on ImageNet corresponds to single unit. The mean and standard deviation are shown by the solid line and shaded area, respectively. The mean activation for ID data (blue) is well-behaved with a near-constant mean and standard deviation. In contrast, for OOD data (gray), the mean activation has significantly larger variations across units and biased towards having sharp positive values (i.e., positively skewed). As a result, such high unit activation can undesirably manifest in model output, producing overconfident predictions on OOD data. A similar distributional property holds for other OOD datasets as well.
? Each point on the horizontal axis
The above observation naturally inspires a simple yet surprisingly effective Beathod-Rectine Activations (dubbed ReAct) for OOD) detection. In particular, the outsized activation of few selected hidden units can be attenuated by rectifying the activations at an upper limit e >> O. Conveniently, this can be done on pre-trained model without any modification to training. The dashed horizontal line in Figure [b) shows the cutoff point c, and its effect on the OOD uncertainty score is shown in Figure lc.. After rectification, the output distributions for ID and OOD data become much more well-separated and the false positive rate (FPR) is significantly reduced from 55.72% to 20.38%. Importantly, this truncation largely preserves the activation for in-distribution data, and therefore ensures the classification accuracy on the original task is largely comparable.
We provide both empirical and theoretical insights, characterizing and explaining the mechanism by which ReAct improves OOD detection. We perform extensive evaluations and establish state-of- the-art performance on suite of common OOD detection benchmarks, including CIFAR-10 and CIFAR-100, as well as a large-scale ImageNet dataset 7 ReAct outperforms the best baseline method
by large margin, reducing the average FPR95 by up to 25.05%. We further analyze theoretically and show that ReAct is more beneficial when OOD activations are more chaotic (i.e., having larger variance) and positively skewed compared to ID activations, a behavior that is typical of many OOD datasets (cf. Figure Tp. In summary, our key results and contributions are:
1. We introduce ReAct -a simple and effective post hoc OOD detection approach that utilizes activation truncation. We show that ReAct can generalize effectively to different network architectures and works with different OOD detection methods including MSP (15). ODIN 31. and energy score 33
2. We extensively evaluate ReAct on suite of OOD detection tasks and establish a state- of-the-art performance among post hoc methods. Compared to the previous best method, ReAct achieves an FPR95 reduction of 25.05% on large-scale ImageNet benchmark.
3. We provide both empirical ablation and theoretical analysis, revealing important insights that abnormally high activations on OOD data can harm their detection and how ReAct effectively mitigates this issue. We hope that insights inspire future research to further examine the internal mechanisms of neural networks for OOD detection. Code and dataset will be released for reproducible research.

---

2 Background
The problem of out-ef-disrribution detection for classification is defined using the following setup. Let X' (resp. }) be the input (resp. output) space and let P be a distribution over X' x y. Let f x + DD be a neural network trained on samples drawn from P to output a logit vector, which is used predict the label of imput sample. Denote by Di the marginal distribution of P for X, which represents the distribution
in-distribution data. At test time, the environment can present distribution Dout over X' of out-of -ittr--distritttion data. The goal of OOD detection is to define decision function G such that for given test input X::
f) - S x Dout: if ~ Din-
G(x: G(x:f)=
The difficulty of OOD detection greatly depends on the separation between Din and Dout In the extreme case, Din = Doutt then the optimal decision function is random coin flip. In practice, Dout is often defined by distribution that simulates unknowns encountered during deployment time, such as samples from an irrelevant distribution whose label set has no intersection with y and
therefore should not be predicted by the model.
3 Method
We introduce a simple and surprisingly effective technique, Rectified Activations (ReAct), for improving OOD detection performance. Our key idea is to perform post hoc modification to the unit activation, so to bring the overall activation pattern closer to the well-behaved case. Specifically, we consider pre-trained neural network parameterized by 9, which encodes an input x E Rd to feature space with dimension m. We denote by h(x) R" the feature vector from the penultimate layer of the network. A weight matrix W R connects the feature h(x) to the output f(x), where is the total number of classes in y = {1.2...,,K] K}.
ReAct: Rectified Activation. We propose the ReAct operation, which is applied on the penultimate layer of a network:
h(x) = ReAct(!(x);c),,
where ReAct(#; e) min(xr, c) and is applied element-wise to the feature vector h(x). In effect, this operation truncates activations above e to limit the effect of noise. The model output after rectified activation is given by:
|ReAct (): W h(x) + bb,
(2)
where I K the bias vector. A higher e indicates a larger threshold of activation truncation.
When = the output becomes equivalent to the original output fxx 0) without rectification, where fxx 0) W h(x) + b. Ideally, the rectification parameter should be chosen to sufficiently preserve the activations for ID data while rectifying that of OOD data. In practice, we set based on the p-th percentile of activations estimated on the ID data. For example, when p = 90, it indicates that 90% percent of the ID activations are less than the threshold We discuss the effect of percentile in detail in Section
OOD detection with rectified activation. During test time, ReAct can be leveraged by variety of downstream OOD scoring functions relying on fReAd 0):
S(x; f (ReAct) 2 GA [ ReAd) = (out S(x; ReAct) < X
in
(3)
where a thresholding mechanism is exercised to distinguish between ID and OOD during test time. To align with the convention, samples with higher scores S S(x; f) are classified ID and vice versa. The threshold À is typically chosen so that a high fraction of ID data (e.g.. 95%) is correctly classified. ReAct can be compatible with several commonly used OOD scoring functions derived from the model output 0), including the softmax confidence [5). ODIN score (3I), and the energy score 33 For readers' convenience,
provide an expansive description of these methods in Appendix B In Section we default to using the energy score (since it is hyperparameter-fiee and does not require fine-tuning), but demonstrate the benefit of using ReAct with other OOD scoring functions too.
3

---

OOD Datasets
Model
Methods
INaturalist
SUN
Places
Textures
Average
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
MSP 151
54,99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81.99
ODIN R31
47.66
89.66
60.15
84.59
67.89
81.78
50.23
85.62
56,48
85.41
ResNet
MahalanoMi AD
97.00
98.50
42.41
98.40
41.79
55.80
85.01
55.47
baergy
55.72
89.95
59.26
85.89
64.92
82.86
53.72
85.99
58.41
86.17
ReAct (Ours)
20.38
24.20
94.20
33.85
47.30
31.43
92.95
MSP 1k]
64.29
65.32
77.02
77.10
79.23
76.27
73.51
77.30
73.51
79.00
55.39
57.36
MohileNet
Mahalanobis
A0
62.11
81.00
47.82
86.33
52.09
83.63
92.38
33.06
63.60
71.01
Energy [3]
59.50
88.91
62.65
84,50
69.37
81.19
58.05
85.03
62.39
84.91
ReAct (Oars)
42.40
91.53
47.69
88.16
51.56
86.64
38.42
91.53
45.02
89.47
Table I: Main results. Comparison with competitive post hoc out-of-distribtiion detection methods. All methods are based on model trained on ID data only (ImageNet-1kg Ik), without using any auxiliary outlier data. f indicates larger values are better and indicates smaller values are better. All values are percentages.
4 Experiments
In this section, we evaluate ReAct on suite of OOD detection tasks. We first evaluate on large-scale OOD detection benchmark based on ImageNet [[] Section 4.1p. and then proceed in Section 4.2 with CIFAR benchmarks 27
4.1 Evaluation on Large-scale ImageNet Classification Networks
We first evaluate ReAct a large-scale OOD detection benchmark developed in IS
Compared to the CIFAR benchmarks that are routinely used in literature, the ImageNet benchmark more challenging due to larger label space (K = 1, 000). Moreover, such large-scale evaluation is more relevant to real-world applications. where the deployed models often operate on images that have high resolution and contain more classes than the CIFAR benchmarks.
Setup. We use pre-trained ResNet-50 model 12 for ImageNet-1k: At test time, all images resized to 224 x 224. We evaluate on four test OOD datasets from (subsets of) Places365 (62), Textures (5, iNaturalist [16], and SUN (55] with non-overlapping categories w.r.t ImageNet. Our evaluations span a diverse range of domains including fine-grained images, scene images, and textural images (see Appendix C for details). We use validation set of Gaussian noise images, which are generated by sampling from Trom.((O, 1) for each pixel location. To ensure validity, we further verify the activation pattern under Gaussian noise, which exhibits a similar distributional trend with positive skewness and chaoticness; see Figure in Appendix G for details. We select p from {10, 65, 80, 85, 90, 95, 99} based on the FPR95 performance. The optimal p is 90. All experiments are based on the hardware described in Appendix D
Comparison with competitive OOD detection methods. Table we compare ReAct with OOD detection methods that are competitive in the literature, where ReAct establishes the state-of-the-art performance. For a fair comparison, all methods use the pre-trained networks post hoc. We report performance for each OOD test dataset, well as the average of the four. ReAct outperforms all baselines considered, including Maximum Softmax Probability [55. ODIN BT, Mahalanobis distance [29] and energy score 33 Noticeably, ReAct reduces the FPR95 by 25.05% compared to
the best baseline [31 on ResNet. Note that Mahalanobis requires training separate binary classifier, and displays limiting performance since the increased size label space makes the class-conditional Gaussian density estimation less viable. In contrast, ReAct is much easier use in practice, and can be implemented through simple post hoc activation rectification.
Effect of rectification threshold c. We now characterize the effect of the rectification parameter which can be modulated by the percentile p described in Section 3 In Table 2 summarize the OOD detection performance, where we vary that over-activation does compromise the ability to detect OOD data, and ReAct can effectively alleviate this problem. Moreover, when p is sufficiently large. ReAct can improve OOD detection while maintaining comparable ID classification accuracy. Alternatively, once sample is detected to be ID one can always use the original activation hxx), which is guaranteed to give identical classification accuracy. When p is too small, OOD performance starts to degrade as expected.
[10, 65, 80, 85, 90, 95, 99). This ablation confirms

---

Rectification percentile
FPR95
AUROC
AUPR
ID ACC.
Rectification threshold
No ReAct [8
58.41
86.17
96.88
75.08
99
44.57
90.45
97.96
75.12
2.25
95
35.39
92.39
98.37
74.76
1.50
p 90
31,43
92.95
98.50
73.75
1.00
85
34,08
92.05
98.35
72.91
0.84
p 80
41.51
89.54
97.91
71.93
0.72
p= 65
74.62
74.14
94.39
67.14
0.50
p= 10
74.70
57.55
86.06
1.22
0.06
Table 2: Effect rectification threshold inference. Model is trained on ImageNet using ResNet-50 [1]] numbers are percentages and are averaged over OOD test datasets.
Efffec on other network architectures. We show that ReAct is effective on a different architecture Table In particular, we consider a lightweight model MobileNet-v2 [e which can be suitable for OOD detection on-device mobile applications. Same as before, we apply ReAct on the output of the penultimate layer, with the rectification threshold chosen based on the 90-th percentile. Our method reduces the FPR95 by 9.18% compared to the best baseline [1]
What about applying ReAct on other layers? Our results suggest that applying ReAct on the penultimate layer is the effective, since the activation patterns most distinctive. To this, we provide the activation and performance study for intermediate layers in Appendix and Table 6 Interestingly, early layers display less distinctive signatures between ID and OOD data. This is expected because neural networks generally capture lower-level features in early layers (such Gabor filters [ill in layer 1), whose activations can be very similar between ID and OOD. The semantic-lewe features only emerge as with deeper layers, where ReAct is the most effective.
D (see Figure 6
4.2 Evaluation on CIFAR Benchmarks
Datasets. We evaluate CIFAR- 10 and CIFAR-I0K 100 e datasets in-distribution data, using the standard split with 50,000 training images and 10,000 test images. For OOD data, we consider six common benchmark datasets: Textures 5
SVHN 57 Places365
62 LSUN-Crop [60].
LSUN-Resize 160] and iSUN (57).
Experimental details. We train a standard ResNet-18 11 model on in-distribution data. The feature dimension of the penultimate layer is 512. For both CIFAR-10 and CIFAR-100, the models trained for 100 epochs. The start learning rate 0. and decays by a factor of at epochs 50, 75, and 90. For threshold c, we use the 90-th percentile of activations estimated on the ID data.
ReAct is compatible with various OOD scoring functions. We show in Table 3 that ReAct is flexible method that is compatible with alternative scoring functions S(x; fRcAct] J. To see this, we consider commonly used scoring functions, and compare the performance both with and without using ReAct respectively.
particular, we consider softmax confidence [[5]. ODIN score BI
well as energy score [33 all of which derive OOD scores directly from the output f(x). In particular, using ReAct on energy score yields the best performance, which is desirable as energy hyperparameter-free free OOD score and is easy to compute in practice. Note that Mahalanobis [29] estimates OOD score using feature representations instead of the model output fxx,, hence less compatible with ReAct. On all three in-distribution datasets, using ReAct consistently outperforms the counterpart without rectification. Results in Table 3 are based on the average across multiple OOD test datasets. Detailed performance for each OOD test datasel is provided in Appendix, Table 7
CIFAR-10 AUROC
CIFAR-100 AUROC
ImageNet AUROC
Method
FPR95
AUPR
FPR95
AUPR
FPR95
AUPR
Softmax
[S
56.71
91.17
79.11
80.72
76.83
78.41
66.95
81.99
95.76
Softmax score ReAct
53.81
91.70
92.11
75.45
80.40
84.28
58.28
87.06
97.22
Energy 68
35.60
93.57
95.01
71.93
82.82
86.28
58.41
86.17
96.88
E'eeryy+Rede ReAct
32.91
95.53
31.43
92.95
ODIN 3TI
31.10
93.79
94.95
82.88
86.25
56.48
96.61
RRAAt
28.81
94.04
94.82
59.91
85.23
87.53
44.10
90.70
98.04
Table 3: Ablation results. ReAct is compatible with different OOD scoring functions. For each ID dataset, the same model and compare the performance with and without ReAct respectively. indicates larger values are better and L. indicates smaller values are better. All values are percentages and are averaged over multiple OOD datasets. Detailed performance each OOD test dataset available Table Malee/ii in Appendix.
5

---

Frequency
Frequency
Frequency
Frequency
(a) INaturalist
(b) SUN
(c) Places
(d) Textures
Figure Positively skewed distribution of μ4 (mean each unit)
penultimate layer four OOD
datasets (iNaturalist [6). SUN [55]. Places [2] Textures [5)). Model is trained on ImageNet. Note that the right tail has higher density than the left tail. By left and right tail we refer to the samples that are to the left
and right of the median, which is close to the mode in the case of skewed distribution.
5 Theoretical Analysis
To better understand the effect of ReAct, we mathematically model the ID and OOD activations as rectified Gaussian distributions and derive their respective distributions after applying ReAct. These modeling assumptions are based on the activation statistics observed on ImageNet in Figure [ In the following analysis, we show that ReAct reduces mean OOD activations more than ID activations since OOD activations are more positively skewed (see Appendix A for derivations).
be the activations for the penultimate layer. We assume each 2; ~ ~ (1, ai for some om > O. Here (" = max(0, ofa)) denotes the rectified Gaussian distribution, which reflects the fact that activations after ReLU have no negative components. Before truncation with ReAct, the expectation of z; is given by:
ID activations. Let hxx) (=1, (21,....Zmn) 2m)
[m] = 1
1
Jin
Omn
where o and & denote the edf and pdf of the standard normal distribution, respectively. After rectification with ReAct, the expectation of & min(z;, is:
④ ( c
Em 2.]
+
T
1
c
i
Fin
Fm
Oin
Oin
The reduction in activation after ReAct is:
c 11
c
E [2: _z]
Fm
+
(c 11)
(4)
Oin
OOD activations. We model OOD activations as being generated by two-stage process: Each OOD distribution defines a set of μ 's that represent the mode of the activation distribution for unit i, and the activations 2, given (4; is represented by 2/|4,; ~ *N (14, with r > 0. For instance, the dark gray line in Figure shows the set f4; on the iNaturalist dataset, and the light gray area depicts the distribution of 2:/p.. One commonality across different OOD datasets is that the distribution of M4 positively skewed. The assumption of positive skewness motivated by our observation on real OOD data. Indeed, model for four OOD datasets, all of which display strong positive-skewness, i.e., the right tail has much higher density than the left tail. This observation is surprisingly consistent across datasets and model architectures. Although more in-depth understanding of the fundamental cause of positive skewness is important, for this work, we chose to rely on this empirically verifiable assumption and instead focus analyzing method ReAct.
2hows the empirical distribution of on an ImageNet pre-trained
Utilizing the mositive-skewneee property of μs, we analyze the distribution of 2; after marginalizing out (4s, which corresponds to averaging across different 24's induced by various OOD distributions. Let ar N N(Ja, 7²) so that 2; zi|m max(x; (fr/ /1, 0). Since æ; (1; is symmetric and μ4 is positively- skewed, the marginal distribution of is also positively-skewed" which we model with the epsilon- norm-normal (ESN) distribution 35 Specifically, we assume that T, ~ ESN(u, d e), which has
"This can be argued rigorously using Pearson's mode skewness coefficient if the distribution of μr is unimodal.

---

OOD
(epsilon=0))
Figure 3: Plot showing the relation- ship between the skewness param- eter and the chaotic-ness param- eter a on activation reduction af- ter applying ReAct. The function E[ Σg] is increasing in both -6 and a, which suggests that ReAct has greater reduction effect for activation distributions with posi- tive skewness (e 0) and chaotic- ness-two signature characteristics of OOD activation.
14
1.0
0.8
0.5
1D
opsilon=0
0.2
-0.5
255--5
Log
-0.2
-0.1
0.25
9.50
0
0.0
0.00
the following density function:
[0(( 11// o e))/ if μ4 60(( -1) ou) e))/Fout
((x)
(5)
if μ.
with [-1,1] controlling the skewness. In particular, the ESN distribution is positively-skewed
when < 0. It follows that 2, = max(x;;0) 0), with expectation:
Eou[ [2:] - (1 (1++0) c) (I e)o
1
+ (1 (1+
-out
4e
Oout.
(6)
(1 + E)oout
22Ti
Expectation after applying ReAct becomes:
Evar [ 1 (1 +)Φ
1
1 (1 ce)
1 +
c c-M 1
(c
1))
(1 + e)
c)o
+ +111 (1+c) c)
-4 (1+c)) c)oout
-11 -1-1)36
H
4e
out
(7)
(1 c)Jomr
2TT
Hence:
)]
Exu[2 2,]
c)3g
c μ
Ooul (1 1 +b
(1 - c)oour (1-c)oou
c-
.cc- p))
(8)
which recovers Lqquation 4 when e 0 and Cout = Ojn-
Remark 1: Activation reduction on OOD is more than ID. Figure 3 shows plot of B[2; =i] for μ 0.5 and 1. Observe that decreasing (more positive-skewness or increasing (more chaotic-ness) leads to a larger reduction in the mean activation after applying ReAct. For example, under the same o, larger Eour 2g] Em. [2 2,] can be observed by the gap of 2-axis value between e 0 and < 0 (e.g.. é -0.4). This suggests that rectification on average affects OOD activations more severely compared ID activations.
Output reduction on OOD more than ID. derive the effect on the distribution of model output, consider output logits f(z) = Wz + b and assume without loss of generality that W1 > 0 element-wise. This can be achieved by adding a positive constant W without changing the output probabilities or classification decision. Let d = Evnn (z z]
Remark 2:
Ein (z z] > 0. Then:
[f f()) Eout [W (( 2 WE [z z] W (Enn z] + 81)
= E., (z 2)] + SW1
> [f -f())-
Hence the increased separation between OOD and ID activations transfers to the output space as well. Note that the condition of W1 is sufficient but not necessary for this result to hold. In fact, experiments in Section do not require this condition. However, verified empirically that ensuring W1 > 0 by adding positive constant to W and applying ReAct does confer benefits to OOD detection, which validates theoretical analysis.

---

Why ReAct improves the OOD scoring functions? Our theoretical analysis above shows that ReAct suppresses logit output for OOD data more so than for ID data. This means that for detection scores depending on the logit output (e.g., energy score [33 ). the gap between OOD and ID score will be enlarged after applying ReAct, which makes thresholding more capable of separating OOD) and ID inputs; see Figure I a) and (c) for a concrete example showing this effect.
Discussion and Further Analysis
Why do OOD samples trigger abnormal unit activation patterns? So far we have shown that OOD data can trigger unit activation patterns that are significantly different from ID data, and that
ReAct can effectively alleviate this issue (empirically in Section A and theoretically in Section Yet a question left in mystery is why such a pattern occurs in modern neural networks? Answering this
question requires carefully examining the internal mechanism by which the network is trained and
evaluated. Here we provide one plausible explanation for the activation patterns observed in Figure [
with the hope of shedding light for future research.
Intriguingly, our analysis reveals an important insight that batch normalization (BatchNorm) ? common technique employed during model training- is in fact both blessing (for ID classification) and curse (for OOD detection). Specifically, for unit activation denoted by z, the network estimates the running mean Em (2) and variance (a), over the entire ID training set during training. During inference time, the network applies BatchNorm statistics Enn(2) and (ar,a which helps normalize
the activations for the test data with the same distribution Din:
2 - Em
. +
VVam 2 +E
However, our key observation is that using mismatched BatchNorm statistics-that are estimated on Di, yet blindly applied to the OOD Dont can trigger abmormally high unit activations. As a thought experiment, we instead apply the true BatchNorm statistics estimated on a batch of OOD images and we observe well-behaved activation patterns with near-constant mean and standard deviations like the ones observed on the ID data (see Figure 4 top). Our study therefore reveals one of the fundamental causes for neural networks produce overconfident predictions for OOD data. After applying the true statistics (estimated on OOD), the output distributions between ID and OOD data become much more separable. While this thought experiment has shed some guiding light, the solution of estimating BatchNorm statistics on a batch of OOD data is not at all satisfactory and realistic. Arguably, it poses a strong and impractical assumption of having access to a batch of OOD data during test time, Despite its limitation, we view it as an oracle, which serves as an upper bound on performance for ReAct.
Using True Statistics
Unit Activation
Using False Statistics
Unit Indices
Figure 4: The distribution of per-unit activations in the penultimate layer for OOD data (iNaturalist) by using frue (top) vs. mismatched (bottom) BatchNorm statistics for OOD data.
Method
INaturalist 99.59
Places
SUN
Textures
Oracle (batch OOD for estimating BN statisties)
99.09
98.32
91.43
ReAct (single OOD)
96.22
94.20
91.58
89.80
No ReAct [33]
89.95
85.89
82.86
85.99
Table 4: Comparison with oracle using OOD BN statistics. Model is trained on ImageNet (see Section 4.1 Values are AUROC.
8

---

In particular, results in Table suggest that our method favorably matches the oracle performance using the ground truth BN stanstics. This is encouraging as our method does not impose any batch assumption and can be feasible for single-input testing scenarios.
What about networks trained with different normalization mechanisms? Going beyond batch normalization [22], we further investigate (1) whether networks trained with alternative normalization approaches exhibit similar activation patterns, and (2) whether ReAct is helpful there. To answer this question, we additionally evaluate networks trained with WeightNorm [43] and GroupNorm [54]- two other well-known normalization methods. As shown in Figure S the unit activations also display highly distinctive signature patterns between ID and OOD data, with more chaos on OOD data. In all cases, the networks are trained to adapt to the ID data, resulting in abnormal activation signatures on OOD data testing. Unlike BatchNorm, there no easy oracle (e.g., re-estimating the statistics on OOD data) to counteract the ill-fated normalizations.
We apply ReAct on models trained with WeightNorm and GroupNorm, and report results in Table S Our results suggest that ReAct consistently effective under various normalization schemes. For example, ReAct reduces the average FPR95 by 23.54% and 14.7% respectively. Overall, ReAct has shown broad efficacy and compatibility with different OOD scoring functions (Section
ID (ImageNet)
OOD (INaturalist)
Unit Activation
Unit Indices
Unit Indices
Figure 5: The distribution of per-unit activations in the penultimate layer for ID (ImageNet
and OOD
(Natuurliss [I6] on model trained with weight normalization
OOD Datasets
Methods
iNaturalist
SUN
Places
Textures
Average
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
GroupNorm
W.O. ReAct
63.38
88.45
65.11
85.52
63.46
84.34
69.17
83.22
06.28
83.38
ReAct
39.45
92.95
51.57
87.90
52.78
87.32
62.50
81.76
51.58
87.48
WeightNorm
w.o. ReAct
40.71
92.52
48.07
89.39
50.92
87.87
61.65
80.71
50.34
87.62
ReAct
19.73
93.76
Table 5: Effectiveness of ReAct different normalization methods. ReAct consistently improves OOD detection performance for the model trained with GroupNorm and WeightNorm. In-distribution is ImageNet- dataset. Bold numbers are superior results.
7 Related Work
OOD uncertainty estimation with discriminative models. The problem of classification with rejection can date back to early works on abstention [4] [0, which considered simple model families
such SVMs 6 comprehensive survey OOD detection can be found in 58 The phenomenon
of neural networks' overconfidence in out-of-distribttion data is first revealed by Nguyen et al.
[38
Early works attempted to improve the OOD uncertainty estimation by proposing the ODIN score [17
B1. OpenMax score [3), and Mahalanobis distance-based score [29]. Recent work by Liu et al. 3
proposed using energy score for OOD uncertainty estimation, which can be casily derived from a discriminative classifier and demonstrated advantages over the softmax confidence score both empirically and theoretically. [Wang et al. [5I] further showed an energy-based approach can improve OOD uncertainty estimation for lalti-label classification networks. [Huang and 18 revealed that approaches developed for common CIFAR benchmarks might not translate effectively into large-scale ImageNet benchmark, highlighting the need to evaluate OOD uncertainty estimation in large-scale real-world setting. Previous methods primarily derive OOD scores using original activations. In contrast, our work is motivated by a novel analysis of internal activations, and shows that rectifying activations is a surprisingly effective approach for OOD detection.
Neural network activation analysis. the activation of individual layers 221
Neural networks have been studied at the granularity of
34
59 63 or individual networks [30]. In particular, [i
al. (0) studied the similarity of activation space between two independently trained neural networks.

---

Previously, Hein et al. [33 showed that neural networks with ReLU activation can lead to arbitrary high activation for imputs far away from the training data. We show that using ReAct could efficiently alleviate this undesirable phenomenon. ReAct does not rely on auxiliary data and can be conveniently used for pre-trained models. The idea of rectifying unit activation 26),, which is known as Relu6, was used to facilitate the learning of sparse features. In this paper, we first show that rectifying activation can drastically alleviate the overconfidence issue for OOD data, and result, improve OOD detection.
OOD uncertainty estimation with generative models. OOD inputs resort to generative models that directly estimate density [ 20
Alternative approaches for detecting
50 An
input is deemed as OOD if it lies in the low-likelihock regions. A plethora of literature has emerged
to utilize generative models for OOD detection 2 40
52 [56.. Interestingly, Nalisnick B3 showed that deep generative models can assign high likelihood to OOD data. Moreover,
et al. generative models can be prohibitively challenging to train and optimize, and the performance can often lag behind the discriminative counterpart. In contrast, our method relies on a discriminative classifier, which is easier optimize and achieves stronger performance.
Distributional shifts have attracted increasing research interests 25 is important to recognize and differentiate various types of distributional shift problems. Literature in OOD detection is commonly concerned about model reliability and detection of label-space shifts, where the OOD inputs have disjoint labels w.r... ID data and therefore should not be predicted by the model. Meanwhile, some works considered label distribution shift 22 32 42
Distributional shifts.
17l 53 where
the label space is common between ID and OOD but the marginal label distribution changes, as well as covariate shift in the input space [4] 39 where inputs can be crruuttion-shiftee or domain- shifted 17 48 It is important to note that our work focuses on the detection of shifts where the label space y is different between II) and OOD data and hence the model should not make any prediction, instead of label distribution shift and covariate shift where the model is expected to generalize.
8 Conclusion
This paper provides simple activation rectification strategy termed ReAct, which truncates the high activations during test time for OOD detection. We provide both empirical and theoretical insights characterizing and explaining the mechanism by which ReAct improves OOD uncertainty estimation. By rectifying the activations, the outsized contribution of hidden units on OOD output can be attenuated, resulting in a stronger separability from ID data. Extensive experiments show ReAct can significantly improve the performance of OOD detection on both common benchmarks and large-scale image classification models. Applications of multi-class classification can benefit from our method, and we anticipate further research in OOD detection to extend this work to tasks beyond image classification. We hope that our insights inspire future research to further examine the internal mechanisms of neural networks for OOD detection.
Societal Impact
Our project aims to improve the dependability and trustworthiness of modern machine learning models. This stands to benefit wide range of fields and societal activities. We believe out-of- distribution uncertainty estimation is an increasingly critical component of systems that range from consumer and business applications (e.g., digital content understanding) transportation (e.g., driver assistance systems and autonomous vehicles), and to health care (e.g., unseen disease identification). Many of these applications require classification models in operation. Through this work and by releasing our code, we hope to provide machine learning researchers with a new methodological perspective and offer machine learning practitioners plug-and-pla tool that renders safety against OOD data in the real world. While we do not anticipate any negative consequences to our work, we hope to continue to build on our method in future work.
Acknowledgement
YS and YL supported by the Office of the Vice Chancellor for Research and Graduate Education (OVCRGE) with funding from the Wisconsin Alumni Research Foundation (WARF).
10

---

References
[1] Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias- corrected calibration is hard-to-beat at label shift adaptation. In International Conference Machine Learning, pages 222-232. PMLR, 2020.
[2] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang. and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. International Conference on Learning Representations, 2019.
[3] Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. Proceedings of the IEEE conference computer vision and pattern recognition, pages 1563-1572, 2016.
[4] CK Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46, 1970.
[5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[6] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3): 273-297, 1995.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference computer vision and pattern recognition, pages 248-255 2009.
[8] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[9] Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt distribution shifts? In International Conference on Machine Learning, pages 3 3145-3153, PMLR, 2020.
[10] Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In International Workshop on Support Vector Machines, pages 68-82. Springer, 2002.
[11] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pages 330-645. Springer, 2016.
[13] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[14] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 19t::::993.22000 2019.
[15] Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and -iitt--iittiittion examples in neural networks. Proceedings of International Conference on Learning Represen- tations, 2017.
[16] Grant Horn, Oisin Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist Species Classification and Detection Dataset. 06 2018.
[17] Yen- Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out-of out-of-distribution data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10951-10960. 2020.

---

[18] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space, In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8710-8719, June 2021.
[19] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Neural Information Processing Systems, 2021.
[20] Xun Huang, Yixuan Li, Omid Poursaeed. John E Hopcroft, and Serge J Belongie. Stacked generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, volume 2, page 3, 2017.
[21] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Leurning, volume 37 Proceedings of Machine Leurning Research, pages 448-456, Lille, France, 07-09 Jul 2015. PMLR.
[22] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International conference on machine learning, pages 448-456. PMLR, 2015.
[23] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, 2014.
[24] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distribution data. Advances in Neural Information Processing Systems, 33, 2020.
[25] Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[26] Alex Krizhevsky. Convolutional deep belief networks on cifar- 10, 2010.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] Balaji Lakshminarayanaan Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pages 6402-6413, 2017.
[29] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribttion samples and adversarial attacks, In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[30] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hopcroft. Convergent learning: Do different neural networks learn the same representations? In International Conference on Learning Representations (ICLR), 2016.
[31] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. In 6th International Conference on Learning Representutions, ICLR 2018, 2018.
[32] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR, 2018
[33] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based detection. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[34] Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018. URL
[35] Govind Mudholkar and Alan D Hutson. The spsilon-skew-noree distribution for analyzing near-normal data. Journal of statistical planning and inference, 83(2):291-309) 2000.
12

---

[36] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019.
[37] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[38] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015.
[39] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32:1 13991 1-0022, 2019.
[40] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pages 14680-14691, 2019.
[41] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 40022, 2014.
[42] Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new priori probabilities: a simple procedure. Neural Computation, 14(1):21 2002.
[43] Tim Salimans and Diederik P Kingma. Weight normalization: simple reparameterization to accelerate training of deep neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 901-909, 2016.
[44] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiee Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[45] Robin Tibor Schirmeisstrr, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. arXiv preprint arXiv:2006. 10848, 2020.
[46] Joan Serrå, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-disrriution detection with likelihood-based generative models. In International Conference on Learning Representations, 2020.
[47] Avanti Shrikumar and Anshul Kundaje. Calibration with bias-corrected temperature scaling im- proves domain adaptation under label shift in modern neural networks. Preprint at https://arxiv. org/abs/1901.06855 vl, 2019.
[48] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, pages 9229-9248. PMLR, 2020.
[49] Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algo- rithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
[50] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, pages 4790. 2016.
[51] Haoran Wang, Weitang Liu, Alex Bocchieri. and Yixuan Li. Can multi-label classification networks know what they don't 't know? Advances in Newral Information Processing Systems, 2021.
[52] Ziyu Wang, Bin Dai, David Wipf, and Jun Zhu. Further analysis of outlier detection with deep generative models. Advances in Neural Information Processing Systems, 33, 2020.
13

---

[53] Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. arXiv preprint arXiv:2107.04522 2021.
[54] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[55] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, pages 3485-3492. IEEE Computer Society, 2010.
[56] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in Neural Information Processing Systems, 33, 2020.
[57] Pingmei Xu, Krista A Ehinger. Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint 15X1::1504.06755, 2015.
[58] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized dustr--dist detection: A survey. arXiv preprint arXiv:21 10.1133.. 11334, 2021.
[59] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems, volume 27, 2014.
[60] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.03355, 2015.
[61] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
[62] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places:
10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1466 2017.
[63] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. arXiv preprint 0830:,11806.0289,22011
14


Reviews:
[{"review_id": "y1csmLg5SBf", "reviewer": "Reviewer_QB7D", "summary": "This paper identifies that out-of-distribution (OOD) data tend to cause overconfident activations much more often than in-distribution data. Based on this observation, a simple approach called ReAct (Rectified Activations) is proposed, which essentially applies a threshold to the activations of the penultimate layer of a CNN classifier. Extensive experiments demonstrate that ReAct works with different network architectures and OOD detection algorithms, and the improvements over previous SOTA methods are significant.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n\n- ReAct is well motivated. Though the design is simple, it works very well with different network architectures and is compatible with several existing OOD detection methods.\n- A theoretical analysis is provided to justify ReAct's effect on reducing both the activations and output for OOD data.\n- The experiments are extensive and the results are strong.\n- The analysis part is insightful, especially the discussions that connect the overconfidence issue to BatchNorm. I think the findings and results could be useful to research beyond OOD detection, such as OOD generalization and domain adaptation.\n- The paper is well written and easy to follow.\n\n**Weakness**\n\nI do not find major issues in this paper. The following comments are mainly related to the presentation.\n\n- By just reading the abstract and the introduction, it is unclear how and why ReAct helps OOD detection. It would be clearer to mention that the OOD detection part is based on existing methods such as softmax scores or the energy-based scores. It would also be better to highlight that the idea of ReAct generalizes to different network architectures and works with different OOD detection methods.\n- Though the motivation for trimming the activations is intuitive, the reason why ReAct improves the OOD scoring functions is a bit vague. Taking the energy function (Liu et al., 2020) as example, which essentially associates the sum of logits to the marginal distribution (the likelihood). It is less intuitive why lowering the confidence on OOD samples helps make the OOD scores more distinguishable.\n- Line199: Fig.2 shows that the left tail has a higher frequency than the right tail. Why the right tail has a higher density? Is this a typo or is there something wrong with my understanding? Please clarify on this.\n\n---\nPost-rebuttal update:\n\nI have read the authors' responses and other reviewers' comments.\n\nOverall, I find the proposed approach simple and well-motivated, and the experiments solid and extensive enough for supporting the claim. Hence, I think the paper is worth sharing with the community and keep my rating unchanged.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "iJlTRyaFg2O", "reviewer": "Reviewer_phHd", "summary": "This proposes ReAct to reduce model overconfidence on OOD data.\nThis paper observes the mean activation of OOD data in the penultimate layer has larger variations across units and is biased towards having sharp positive values, which produces overconfident predictions on OOD data.\nReAct rectifies the activations at an upper limit for OOD data and preserves the activation for ID data.\nThe results show that ReAct makes output distributions between ID and OOD data more separable.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall methods is simple and effective to detect OOD data.\nExtensive expermental and theoretical analysis are given.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-- ReAct performs well on the classification networks. Can ReAct help OOD detection in other computer vision tasks, such as detection and segmentation networks?\n\n-- As  OOD data can trigger unit activation patterns that are significantly different from ID data. Why not detection OOD data from such separable unit activation patterns directly?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "NwSlEs_syMC", "reviewer": "Reviewer_68cE", "summary": "The authors propose a simple method for OOD detection, called ReAct, which truncates high activations at test time. Experiments show that ReAct performs better than state-of-the-art OOD detection methods. The authors also theoretically verify the effectiveness of ReAct.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed ReAct is simple, novel, and shows high performance compared to conventional OOD detection methods. In addition, the entire paper is very readable with clear arguments. However, there are some inadequacies in the verification and other aspects as follows.\n\n- I think that the performance of ReAct is greatly affected by the layer at which it is applied, but this is not fully verified in this paper. The Appendix shows the differences in activation for each layer, suggesting that the deeper the layer, the easier it is to distinguish between ID and OOD. However, the authors do not indicate the extent to which these differences actually affect performance. In addition, the Appendix examines the activity values of ResNet-50, but they should verify and show whether the trend is similar for other architectures (such as ResNet-18 used in Sec. 4.2).\n- In the proof, the authors model the distribution of OOD as the epsilon skew-normal distribution, since it is positively skewed. In the proof, the distribution of OOD is positively skewed, so it is modeled by the epsilon skew-normal distribution. Under this assumption, they prove that ReAct is effective. However, it does not explain why the distribution of OOD becomes positively skewed. In addition, it is not clear whether there is a case where the distribution of OOD is not positively skewed (which is directly related to a limitation of the proposed method). The authors should clarify them.\n- According to the setup, the authors use a validation set of Gaussian noise to determine the value of p. However, in order for this Gaussian noise tuned p to be valid for testing, the validation set of Gaussian noise and the test set, i.e., the OOD, should have the same distributional trend. The authors should verify this.\n- Other minor comments\n  - This method is simple and should be easy to implement, but I think it would help the reader to understand it better if the code is included.\n  - There should be a clear formulation of what S is around Equation 3.  I found out later that this is a score function, meaning any existing OOD scoring function, but it should not be introduced into Equation 3 without explaining.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim that the proposed method shows good performance on a variety of data and settings and can be combined with various existing OOD methods, but they are not very explicit about the limitations of the proposed method. For example, this study proves that the proposed method is effective under the hypothesis that the distribution of OOD is positively skewed, but if this does not hold, there should be no guarantee that the proposed method is effective. The authors should include such discussions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KKGFuCCK1zN", "reviewer": "Reviewer_hJUX", "summary": "The paper proposes a very simple way of improving out-of-distribution detection performance in neural network classifiers by rectifying the pre-logit layer's activations.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe approach is very simple to apply. \nIt is good that the authors demonstrate results on ImageNet which is not standard in the field of OOD detection.\n\nWeaknesses:\nThe authors claim to compare to state-of-the-art methods, but the highest performing methods in the literature are variations on outlier exposure [16]. The authors correctly point out that this method uses additional data at train time. I think the authors should include separate evaluations, with their method being applied to outlier-aware approaches (like [16]) in addition to the outlier-unaware approaches that they study.\n\nIn the related work section the authors bring up [14] as an example of using an \"auxiliary OOD dataset\" during training. This seems like a stretch since that paper uses a noise distribution generated from the in-distribution as training out-distribution. By the same logic one could claim that this paper uses an \"auxiliary OOD dataset\" because they use a noise dataset as validation set for hyperparameter selection.\n\nThe technical contribution by the paper is trivial. Also, as the authors mention, the idea of clipping activations has been around for over ten years (clippedReluLayer even having an official implementation in matlab...), so the novelty of the approach is extremely limited.\n\nThe paper should definitely include their OOD performance measures for each test out-distribution (as opposed to just the means) in the appendix.\n\nThe authors show that their method leads to a degradation in test accuracy (which they correctly point out can be mitigated by an additional forward pass). However, it would be interesting to know if this drop in performance still occurs when training or even just fine-tuning with the ReAct activation. In general, I don't see why the paper doesn't investigate if doing this changes their performance at all. \n\n\nClarity:\nThe paper is mostly simple to follow. However, there are a few issues.\nIn line 68 f is defined as a function from X to Y. However, later (e.g. in Eq. 2) it is clearly assumed that f represents logits and that f is a function onto R^|Y|.\n\nIt is not entirely obvious how the percentile is chosen. Line 120 states \"We use a validation set of Gaussian noise\" so presumably this is the OOD dataset that is used to choose the percentile. However, first of all no details of this Gaussian noise are given (mean, variance, clip to image space or not?) and the authors do not even state which metric is being optimized for (AUROC, FPR95, AUPR?).\n\n\nAdditional feedback:\nIn line 243 the authors state that it is surprising that using OOD batch norm statistics normalizes the mean and variance of the activations. I don't see how this is surprising at all. That is the point of batch norm after all.\n\nThe writing of the paper needs much improvement. The style is occasionally too informal (e.g. line 22 \"[...] modern neural networks can, contrary to popular belief, produce overconfident prediction [...]\", who holds this \"popular belief\"?) and there are many typos (e.g. line 84 an->a, line 115 class->classes, line 140 given->give etc...). \n\nThe formatting of paragraphs is somewhat inconsistent. Some end with periods \".\", others do not.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper omits the mandatory discussion of their paper's weaknesses and limitations. No reason for this is given in the checklist.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper observes that OOD samples tend to have overconfident activations and it proposes a simple activation clipping mechanism to reduce model overconfidence on OOD data. Although the proposed method is simple and straightforward, the reviewers found the theoretical analysis and insightful discussion valuable to the research field. Given the strong results and clear presentation of the ideas, I recommend acceptance. 
Decision: Accept (Poster)