RG: OUT-OF-DISTRIBUTION DETECTION WITH REACTIVATE GRADNORM
Authors: Mingyu Xu
Kexin Wang
Zheng Lian
Licai Sun
Bin Liu
Jianhua Tao
Venue: Submitted to ICLR 2023 (2023)
Keywords: OOD detection
Uncertainty Learning

Abstract: Detecting out-of-distribution (OOD) data is critical to building reliable machine learning systems in the open world. Previous works mainly perform OOD detection in feature space or output space. Recently, researchers have achieved promising results using gradient information, which combines the information in both feature and output space for OOD detection. However, existing works still suffer from the problem of overconfidence. To address this problem, we propose a novel method called ``Reactivate Gradnorm (RG)'', which exploits the norm of the clipped feature vector and the energy in the output space for OOD detection. To verify the effectiveness of our method, we conduct experiments on four benchmark datasets. Experimental results demonstrate that our RG outperforms existing state-of-the-art approaches by 2.06\% in average AUROC. Meanwhile, RG is easy to implement and does not require additional OOD data or fine-tuning process. We can realize OOD detection in only one forward pass of any pretrained model.

Under review as a conference paper at ICLR 2023
RG: OUT-OF-DISTRIBUTION DETECTION WITH REACTIVATE GRADNORM
Anonymous authors Paper under double-blind review
ABSTRACT
Detecting fittof--distrbuutionn (OOD) data is critical to building reliable machine learning systems in the open world. Previous works mainly perform OOD detec- tion in feature space or output space. Recently, researchers have achieved promis- ing results using gradient information, which combines the information in both feature and output space for OOD detection. However, existing works still suf- fer from the problem of overconfidence. To address this problem, we propose novel method called "Reactivate Gradnorm (RG)", which exploits the norm of the clipped feature vector and the energy in the output space for OOD detection. To verify the effectiveness of our method, we conduct experiments four bench- mark datasets. Experimental results demonstrate that our RG outperforms existing state-of-the-art approaches by 2.06% in average AUROC. Meanwhile, RG is easy to implement and does not require additional OOD data or fine-tuning process. We can realize OOD detection only one forward pass of any pretrained model.
INTRODUCTION
In addition to the need for the accuracy of predictions, more and more attention has been paid to whether the model can make rejection identification when faced with completely unfamiliar samples. People want models that are not only accurate in their familiar data distribution but also aware of uncertainty outside the training distribution. This gives rise to the importance of (OOD) detection, which determines whether an input in-distribution (ID) or OOD, And OOD detection is widely used in fields with high safety requirements, such as medical diagnosis (Nair
al. 2020 and autonomous driving (Amini et al. 2018).
Deep neural networks can easily make overconfident predictions on OOD inputs, which increases the challenge to separate ID and OOD data Van den Oord et al. (2016); [Chnn et al.] (202T). For instance, model may wrongly but confidently classify an image of crab into the clapping class, even though no crab-related concepts appear the training set. Previous works focused on deriving OOD uncertainty measurements from the output space (Hendryeks & Gimpel 2016 Liu al. 2020)
or feature space (Lee et al.) 2018 Sun et al. 2022). A recent work (Huang et al. 2021 1) based on gradients has intrigued us. Actually, gradient information from feature space and output space, which can be derived from the process of the BP algorithm. However, this method still has room for further improvement on OOD detection, which encourages us to utilize both output space and feature space information for better OOD detection.
often be decomposed into information
In this paper, we perform OOD detection by jointly using information from feature space and output space. Formally, we propose Reactivate Gradnorm (RG), simple and effective method to detect OOD by utilizing the inputs and outputs of the last layer of the neural networks. Specifically, RG directly uses the product of the 1-norm of the clipped input of the last layer of neural network and the logarithm of the exponential sum of the outputs (free energy) as the OOD scoring function. The reason for using the 1-norm of the hidden layer features is that the neurons will be activated for the ID sample. The motivation for cropping it comes from the fact that there will be a few OOD samples with strong features. Appropriate cropping can reduce the I-norm of the features of the OOD samples without excessively affecting the 1-norm of the features of the ID samples. The energy information in the logits space is selected instead of the information in the probability space (like Hennrryksss & Gimpel 2016)) because there is information loss from the logits space to the probability space (the relative size information of the logits will be ignored by the softmax layer).

---

Under review as a conference paper at ICLR 2023
on the other hand, there are good theoretical and practical effects by using the energy as an OOD evaluation score.
Empirically. have established excellent performance the large-scale ImageNet benchmark. RG is vastly superior to previous use of energy after crop ReAct (Sun et al. (02T) 8.9% by AUROC, and our source of inspiration Grodnorm 5.86% by AUROC. Our method also achieves excellent performance compared to the MOS (Huang Li 2021 2.06% by AUROC
Our key results and contributions are summarized follows:
We propose RG, simple and effective OOD uncertainty estimation method, which is label- agnostic (no label required), OOD agnostic (no outlier data required), train data agnostic (Only the pre-trained model is used and no fine-tuning extra training).
We conduct sufficient experiments the combination of information from output space and input space to help us better understand the effectiveness of our OOD detection meth- ods. RG promotes the average AUROC by 2.06% compared to the current best method under the same pre-trained model and dataset. Experiments show that using information from both feature space and output space has gain for OOD detection.
We perform simple theoretical analysis of our method that using information from both feature space and output space at the same time helps to model the distribution of training data, which facilitates ood detection. And we unify several previous approaches under the equation 10 in a new framework.
2 BACKGROUND
In supervised learning, denote by X Rd the input space and Y = {1, C} the output space. A neural network fzz 0) = (f( 0) with the parameter we abbreviate it as f(x), which is mapping from X to R When given dataset D {(x,, y1) )>) the supervised learning
task is to minimize:
R() ((
where leg usually used the cross-entropy loss:
cfo(m)
(CE(J(x), -log Σi f((x)
where y the ground boound-truth label.
Problem statement OOD detection can be formulated as a binary classification problem. The goal to design a discriminator G(x)) which is a mapping from X to R. Given a threshold c, we will decide sample x as OOD sample if and only if G(x)) < The design of the discriminator G is often related to the neural network model fax 0), which will help the neural network model reject Typically, will be set to the fraction of 95% of In-distribution (ID) data that can to be identified as ID. The key challenge to derive a scoring function G(xx) that captures OOD uncertainty. Previous OOD detection approaches primarily rely the output or feature space for deriving OOD scores, and there has been some recent interest in utilizing gradient information for OOD detection. We will reveal that effective gradient-based OOD detection method is a method that combines the information from output space and feature space. And based on it, a more efficient method is proposed in the following section.
the recognition when G(x)
MOTIVATION AND METHOD
In this section, we will first describe the gradient-based OOD detection method and then analyze that the gradient-based OOD detection method is based on the synthesis of the information in the feature space and the information in the output space in section 3.1. The gradient-basee OOD detection method inspired us to design appropriate OOD score which is combination of the norm of the clipped feature vector and the energy in the output space to achieve OOD detection in Section 3.2. In section 3.3, unify several previous approaches under the equation 10 in framework.
2

---

Under review as a conference paper at ICLR 2023
3.1 GRADIENT- BASED OOD DETECTION
We start by introducing the loss function for backpropagation and then describe how to design the gradient norm for OOD uncertainty estimation.
We provide perspective to revisit gradient-basee OOD detection: the idea of the gradient- OOD detection method stems from that for a fully trained neural network, When we continue to feed the neural network with samples of the training set, then use the ground truth to calculate the loss and use the loss to backpropagation, the gradient of the neural network parameters will be small because the neural network is fully trained.
However, when doing OOD detection, the ground-truth is missing. So we can not use the ground truth information to calculate prediction loss. A natural idea is to use uniform distribution substitute for ground-truth and the gradient will be large for an ID sample. (Huang et al. (02T):
G(z)= = 0K (ol (Imax(((a)) (f(x)))
(3)
dw
where u = Jiz the network.
- -i log
fft0 Ei- filay w the parameters of
Another way to replace the ground-truth is that we think the probability that the true label has efu(i)) to be class y. Then we can design the score as Igoe et al. (2022)
probability Py 2i=1 fi(a)
mentioned:
GlogPe ||1
G(x) = Eymp(x)]l
(4)
dw
The probability that each sample has probability of pt belongs to the 7th class, then the expectation of the gradient of the classification loss will be small for ID samples. Note that the negative log- likelihood used when calculating the loss function, so for ID data, G(x) will also be larger than OOD data.
Similarly, to avoid the problem missing ground truth, we can also use such a loss function -1=1 efi(x) and design our own novel score as:
O edile)
G(x)
Under some special settings: only the gradient information of the last layer is used, we will find out that equation is actually:
G(x) UV
where V is the L1 norm of the features of the last layer of neural network input, and U = - ps]. Meanwhile the equation has the form of equation where the L norm of the features
-i=
of the last layer of neural network input, and U = 2 > == P1 p:). And the equation 5 also has the form of equation 6 where V is the L1 norm of the features of the last layer of ncural network input, and U D efi())
From the expression, these scores are still difficult to overcome the problem of overconfidence pre- diction of OOD samples. As the example we mentioned the introduction, model may wrongly but confidently classify an image of a crab into the clapping class, even though no crab-related con- cepts appear in the training set. Suppose the crab has strong feature related to the recognition of the clapping class, which will result in large 1-norm in the feature space. This means that there is room for improvement in both and V.
Summarize Some gradient-based OOD detection methods can be transformed into combination of feature information and output information. But that's 's not surprising, because in the BP algorithm the gradient in the last layer is equal the product of the error propagated and input value where the former only depends on the output space, the latter only depends on the feature space. Previous work inspired to explore suitable U/ and V for OOD detection.
3

---

Under review as a conference paper at ICLR 2023
3.2 THE CHOICE OF U AND V
U' comes from the output space. In OOD detection, using the maximum softmax probability Hendrycks Gimpel (2016) natural choice. However OOD samples may also have very confi- dent predictions, so we use an energy score-based score as U:
U = Tlog
hlx)/T
It has good theoretical and practical significance to use energy score as an indicator for OOD detec- tion. is also a very common strategy to estimate the certainty of predictions by summing the neural network output after activation, in the method of estimating the certainty based on the Dirichlet distribution, using so ffppuus(-) + 1 as the activation and summing (Sensoy et al. 2018).
V comes from the feature space. A common strategy is to model OOD data from standard Gaussian distribution feature space. For ood data, the j element of features vector is 0/2 = maa =t). (0, 1). The score of ood is measured by e e For ID data, use - e- as measure. Based on the assumption that each feature in the feature space is independent, we should multiply all V, as V. But this will cause numerical instability, such as if one of the factors is 0 then the product is O.
So V = EL - e Actually we take an approximation of it: V E min(L,, v²). They are all based on the same idea: to avoid overconfident predictions of OOD samples with few strong features, the contribution of each feature the overall score should be suppressed.
In general, the OOD score we use is:
G(x) Tlog E cfi(E)/T)
Σ
man(v;, Is)
where T is the temperature in the energy function, and default is 1. k is the clipping threshold of each feature in the feature vector, and default 1. this paper, equation used as the OOD detection score by default.
3.3 ADDITION-BASED COMBINATION OF U AND V
Different from the combination method of multiplying and V in the previous section, in this section will use another method to combine U and V and provide a perspective that unifies the previous approaches.
To better explain this, let's look the fully connected layer of the last layer of the neural network.
Suppose for the sample x, the input of the last layer of the neural network is v = {u, {u)} Suppose the joint probability that the feature v belongs to class i P(v, C2) = e + where g(-) is mapping from R. to R, w, Rdd b; E R. The choice of 9(-) cannot be too arbitrary, it will be constrained by probability normalization.
The probability that sample belongs to class i is:
P(u, C.) P(C)lu) Σ. P(r,C))
co(t)+w(otb.
ww/++b
Di=1 ea/
This is exactly what the final fully connected layer of the neural network and the softmax layer are working. We will easily see the combination of the fully connected layer and the softmax layer ignores the relative size g(v). This also implies that if the information of v from feature space can be effectively used, the performance of OOD detection can be better than the information of the prob space alone, like MSP (Hendrycks & Gimpel 2016).
Then,
logP(v) log( Σ C(,, log
totttttt g(v) + log
ci(t)
(10)
el(()) from the output space to charac- So we can use g(v) from the feature space, and log( log E terize the probability of sample appearing. The greater the probability P(v). the more likely the
4

---

Under review as a conference paper at ICLR 2023
( e cn(ntb))) that is equal to MSP (Hendryeks & Gimpel 2016). When g(v) O, that is equal to Energy (Liu et al. 2020 When g represents the inverse of the norm of the residuals of the projection of onto the main subspace, that is equal VIM (Wang al., 2022). When g(v) is quadratic, that GEM (Morteza & Li 2022). We can look at these methods under unified framework.
sample is to belong to the ID sample. When g(t) mary(-log()
For practice, if think that the ID samples are more likely be close to the set D = fo R |u, < k ........ N.. g(v) is used to penalize samples that do not belong to set D. The measure of penalty takes the L1 distance from to the set Dx. So we can design our own novel score as:
G(x))= = Σ (minn(o k) - k) + log
e((())
(11)
Ignore the constant term, and provide balance coefficient for the information from the feature space and the output space. We can use the score:
G(x) log( Σ ciltl + E kk)
(12)
Similar to the method we proposed equation 8 that the combination is changed from multiplication to addition.
they both use the same U and V, the difference is
natural idea is to choose a balance coefficient a that makes the standard deviations of the two terms close. Since the choice of the balance coefficient is intuitive, the experimental part. But we will do experiments to explore the appropriate balance coefficient.
equation [ by default in
4 EXPERIMENT
In this section, we evaluate RG on a large-scale OOD detection benchmark with ImageNet- Ik as an in-distribution dataset. We deseribe the experimental setup in Section 4.1 and demonstrate the superior performance of RG over existing approaches Section 4.2, followed by extensive ablations and analyses that improve the understanding of our approach.
4.1 EXPERIMENTAL SETUP
Dataset We evaluate our method on the large-scald scale ImageNet benchmark proposed by Huang & Li (202I). ImageNet benchmark is not only rich in data sources, but also many categories. OOD de- tection for the ImageNet model is more challenging due to both a larger feature space (dim = 2048) as well as larger label space (C = 1000). particular, the large-scale evaluation can be relevant to real-world applications, where the deployed models often operate on images that have high resolu- tion and contain many class labels. Moreover, as the number of feature dimensions increases, noisy signals may increase accordingly, which can make OOD detection more challenging. We evaluate on four OOD test datasets, which are from subsets of iNaturalist (Van Horn et al. 2018). SUN (Xiao) et al. 2010), Places (Zhou et al. 2017), and Textures (Cimpoi et al. (014), with non-overlapping categories w.r.t. ImageNet-Ik. The OOD datasets include various domains including fine-grained images, scene images, and textural images. The amount of OOD data also very large, with the exception of Textures which has 5640 images, the other datasets have 10000 images each.
Model and hyperparameters We mianly use Google BiT-S models (Kolesnikov et al. (020) pre- trained on ImageNet-Ik with a ResNetv2-101 architecture (He et al. 2016). The BiT-S model is adopted not only for its excellent classification performance on ImageNet-Ik but also for better fair comparison with Gradnorm (Huang et al. (021) methods. In Section 4.2. Additionally, we use clipping threshold the default and explore the effect of other clipping thresholds Section 4.2. The temperature parameter T is set to be unless specified otherwise, and we explore the effect of different temperatures Section 4.2. We also report performance on another architecture, DenseNet121 (Huang et al. 2017). At test time, all images are resized to 480 480.
4.2 RESULTS AND ABLATION STUDIES
Comparison with benchmark methods The results for ImageNet evaluations are shown in where method(RG) demonstrates superior performance. We report OOD detection performance for
5

---

Under review as a conference paper at ICLR 2023
Table 1: Main Results, OOD detection performance comparison between RG and basclines. All methods utilize the standard ResNetv2-100 model trained on ImageNet. The classification model is trained on ID data only. All values are percentages.
NNaturalist
SUN
Places
Textures
Average
Method FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
87.59
76.76
KL-matching
27.36
93.00
67.52
78.72
72.61
76.49
49.70
87.07
54,30
83.82
Energy
64.91
88.48
65.33
85.32
73.02
81.37
80.87
75.79
71.03
82.74
ReAct
49.97
89.80
65,30
87.40
73.12
85.34
80.82
70.53
67.30
83.27
Mahalanobis
96.34
46.33
88.43
65 65.20
89.75
64.46
52.23
72.10
81.69
62.02
MOS
ODIN
62.69
89.36
71.67
81.92
76.27
80.67
81.31
76.30
72.99
82.56
Conor Igoe
91.59
87.20
Gradnorm
50.03
90.33
46.48
89.03
60.86
84,82
61.42
81.07
54,70
86.31
OURS
31.38
94.67
35.36
92.53
49.06
89.19
31,80
92.30
36.90
92.17
each OOD test dataset, as well as the average over the four datasets. For a fair comparison, all the methods the pre-trained backhone, without regularizing with auxiliary outlier data. Since our method inspired by Gradnorm(Hung HHang et al. 2021), the settings of the method compared in that article are also the same it, Such as MSP 2017), Mahalanobis(Lec et al. (2018), as well as Energy (Lu et al. (020 We also compared KL matching (Hendrycks pre-trained model on the same dataset. And for Conor Igoe (Igoe et al. (022), we use the L1 norm of feature and the Energy. In addition, we also compare the ReAct (Sun et al. 202T), which has the same clipping threshold I, and uses the Energy score after clipping the score. We reproduce ReAct. Other methods have been reproduced on mos (Huang Li (02T) or Conor Igoe (Igoe al.) 2022), reported the result from them. For fair comparison, primarily compare with methods utilizing a pre-trained discriminative network without regularizing with auxiliary outlier data.
& Gimpel| 2016). ODIN (Liang al.)
al.] (019) and the methods MOS (Huang
202T which use the same
RG outperforms the best gradient-basee based baseline Gradnorm by 5.86% in AUROC. RG also outper- forms competitive feature- based method, Mahalanobis, by 44.79% in FPR95. RG also outper- forms the method ReAct by 30.4% FPR95. Compared with the current group- based OOD detection method MOS, RG has promoted 2. 2.06%. RG is stable with relatively small differences four OOD) datasets. Besides, OOD detection can be achieved in one forward pass without the need for another back pass like GradNorm. Our method is computationally small and requires no additional storage space, almost the same as MSP or energy methods.
Table 2: Ablation on U and V. OOD detection performance by different U and V. All methods utilize
the standard ResNetv2-101 model trained on ImageNet. The classification model is trained on ID data only. All values are percentages. U and V are combined by multiplication. The meaning of the
first line is to use the and of GradNorm (GN). The meaning of the Second line to use the V
of GradNorm and U of Ours. The meaning of the fourth line is to only use the U of Ours.
Method
INaturllitt
SUN
Places
Textures
Average
AUROC
FPR95
AUROC
GN
GN
50.03
90.33
46,48
89.03
60.86
84.82
61.42
81.07
54.70
86.31
Ours
GN
45.64
91.59
41.67
91.04
56.25
87.20
68.67
81.67
$3.06
87.80 88
39.15
93.20
55.55
86.00
34.52
93.08
90.54
Ours
1
61.91
88.48
65.33
83.32
73.02
81.37
80.87
75.79
71.03
82.74
73.18
70.71
87.73
Ours
Ours
31,38
94.67
38.36
92.53
49.06
89.19
31,80
92,30
36.90
92.17
Ablation on U and V. We conduct experiments the results of U and V alone and using U and in combination. There are two directions of ablation experiments: ablation to separate feature space and output space, and ablation to Gradnorm. As we described in Section 3, the combined use of information from feature space and output space can achieve performance that surpasses either of them in OOD detection. On the other hand, also noticed that the information from the output space plays a leading role in OOD detection, which can also reflect the effectiveness the previous methods such Energy that depend on the output space.

---

Under review as a conference paper at ICLR 2023
Compared to Gradnorm, have increase in replacing V with energy alone. Replacing U alone with the cropped 1-norm also improves. This also shows that our U and are suitable for OOD detection based on the BiT network.
Table 3: The effect of the clipping thresholds. temperature. All values are percentages.
OOD detection based our method. Use default
iNaturalist
SUN
Places
Textures
Average
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
0.1
48.04
91.90
47.14
89.58
$8.49
85.39
35.76
90.47
47.36
86.84
94.31
91.86
26.51
93.11
0.5
30.44
94.78
34,69
92.38
48.41
$8.89
25.66
93.43
34,80
92.37
0.7
30.67
94.82
34.98
92.52
48.86
89.14
27.60
93.15
35.53
92.41
31.38
94.67
35.36
92.53
31.80
92,30
36,90
92.17
35.59
93.78
37.54
92.17
51.84
88.71
42.55
88.80
41.88
90.87
43.30
92.10
40.83
91.29
55.44
87.55
$6.00
83.12
48.89
88.52
00
45.64
91.59
41.67
91.04
87.20
68.67
81.67
53.06
87.88
The effect of the clipping thresholds We evaluate our method RG with different clipping thresholds from = 0.1 to k = As shown in 0.5 or 0.7 optimal, while either increasing or decreasing the clip threshold will degrade the performance. The appropriate clipping threshold is related to the distribution of ID samples in the feature space.
If the clipping threshold is relatively large, OOD samples with few strong features will be identified as ID samples. the clipping thresholds are relatively small, the information loss of the samples in the feature space will be serious, which will lead ID samples being identified as OOD samples. As the clipping threshold increases, the results converge to the unclipped result, which is the result of the last row. When the clipping small, the performance will be worse than clipping. This shows that a suitable clipping threshold is beneficial to filter out those OOD samples that make overconfident predictions because of extremely strong features.
Table 4: The effect of temperature. OOD detection based our method. Use default clipping thresh- old. All values are percentages.
INaturalist
SUN
Places
Textures
Average
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
0.5
32.84
95.56
41.26
91.27
54.35
87.80
39.50
90.60
42.04
91.31
31.38
94.67
35.36
92.53
31.80
92.30
92.17
62.63
85.40
52.53
85.09
66.97
80.55
37.94
90.98
55.02
85.51
72.13
77.74
60.98
78.01
74.57
72.06
42.45
88.06
63.03
78.97
73.04
76.83
62.01
77.07
75.46
70.96
42.87
87.77
63.35
78.16
16
73.14
76.70
62.07
76.90
75.51
70.77
42.93
87.74
63.41
78.03
32
73,18
76.67
62.14
76.87
75.54
70.72
42.93
87.73
63.45
77,80
64
73.18
76.06
62.15
76.86
75.54
70.71
42.93
87.73
63.55
77.99
The effect of temperature. We evaluate our method RG with different temperatures T. As shown in 4 T= is optimal, while either increasing decreasing the temperature will degrade the per- formance. This can be explained as the temperature balances the information in the feature space and the input space. The higher the temperature, the stronger the dependence on the feature space, and the lower the temperature, the stronger the dependence on the output space. As the temperature increases, the results converge the penultimate row in which is result that depends entirely on the feature space.
plus V exploration. As described in Section 3.3, OOD detection can also be performed using an additive combination of U and V. we test the OOD detection performance based on equation [2 Then the key to the experiment is the choice of the hyperparameter a, this balances the effect of feature space and output space OOD. A heuristic is to make the standard deviation the two terms close, which we denote by Q.,. a, need be obtained from the training data of the network. To calculate it quickly, we randomly select 1k values from the training set to calculate. The experimental results are shown in Table 5 All of the average AUROC are better than the MOS which achieves average AUROC of 0.901. On datasets where the output space benefits OOD detection more, the optimal alpha is slightly less than On datasets where the feature space benefit OOD detection more, the optimal alpha is slightly greater than 1.
parameter that

---

Under review as a conference paper at ICLR 2023
Table 5: The effect of balance factors. OOD detection based equation 12
All values are percentages.
iNaturalist
SUN
Places
Textures
Average
a FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
0,3cx
49.68
87.81
40.92
90.61
0,4c
29.20
94.95
34,82
92.41
48.46
88.73
36.17
89.89
37.16
91.50
o.5g
92.01
0.6rt
30.27
94,88
35,03
92.60
49.03
89.32
31.12
92.04
36.36
92.21
0.8
35.03
94.20
37.20
92.05
51.78
88.85
30.23
92.97
38.56
92.02
39.39
91.20
54.31
87.91
30.20
93.32
40.77
91.39
1,2a,
42.91
92.32
41.62
90.25
36.50
86.81
30.76
93.37
42.95
90.69
Table 6: OOD detection performance comparison on different architecture, CenseNet-121. Model trained on ImageNet-Ik lk the ID dataset. All methods are post hoc and can be directly used for pre-traine pre-trained models
INaturalist
SUN
Piaces
Textures
Average
Method
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
MSP
48.55
89.16
69.39
80.46
71.42
80.11
68.51
78.69
64.47
82.11
Energy
84.29
ReAct
44.25
91.14
63.48
85.48
68.52
82.93
69.15
78.17
61.35
84.43
Mahalanobis
97.36
42.24
98.24
41.17
97.32
47.27
62.78
56.53
88.93
46,80
ODIN
37.00
93.29
57.30
86.12
61.91
84.14
56.49
84.62
53.18
87.04
Gradnorm
23.87
93.97
43.04
87.79
53.92
83.04
43.16
87.48
41.00
88.07
Conor Tgoe
19,05
95.93
37.46
90.73
47.65
87.37
37.59
89.08
35.44
90.78
OURS
14.91
96.93
34,80
91.45
45.59
88.32
27.52
92.77
30.71
92,36
OURStclips=0.7
14.48
97.01
34.48
91.36
45.14
88.33
25.90
93.01
30.00
92.43
OURS(clipeDDS)
14.46
97.01
35.16
91.17
45.28
88,24
25.53
93.07
30.11
92.37
RG is effective on alternative neural network architecture. We evaluate RG on different archi- tecture DenseNet- 121 and report performance in Table 6 For a fair comparison, we reproduction
ReAc(Sun et al. (2021) and Connor Igoe Igoe et al. (002)), and other numbers were reported in source of inspiration Gradnorm by 10.29% in FPR95 and 4.29% in AUROC. If we use the optimal clipping threshold 0.5 shown in FPR95 can drop 0.6% compared to the default clipping threshold 1. This shows that the appropriate clipping threshold is related in different network structures. Additionally, we also compare with state-of-the-art nonparametric feature space methods KNN (Sun al. 2022). Because that method requires relatively high storage space, We decided to compare on the same ResNet-50 model trained on ImageNet. The results are shown in the table 7 we report KNN-based results from Sun et al. 2022). This also shows that our method is effective on another pre-trained model.
Huang et al. (021)) RG is consistently effective, outperforming
RELATED WORKS
OOD detection by Output-basee Methods The earliest OOD detection method is based on MSP, which uses the maximum softmax probability the indicator
of II) data (Hendrycks & Gim- pel 2016). The researchers' interest then turned to study OOD) scores in the output space (Sastry & Oore 2020; Dong al./ (2022). ODIN (Liang al., [2017] Hsu et al., 2020) an output-based method that uses temperature scaling and input perturbation to increase the separability of ID and OOD, After that, researchers' interest shifted from softmax space to logit space. (Liu et al. 2020) proposed using an energy score for OOD detection, which enjoys theoretical interpretation from likelihood perspective (Morteza Li 2022). JointEnergy score (Wang et al., (02TI) is then pro- posed to perform OOD detection for multi-label classification networks. Some recent studies have shown that one of the reasons for the overconfidence OOD the abnormally high activation of few neurons, appropriate inhibition of activated neurons is beneficial for OOD detection, which ReAct (Sun et al.) 202I). Then (Sun Li 2022) proposes a weight sparsification-bun based OOD detection framework termed DICE. These methods have the advantage of being easy without modifying the training procedure and objective.
OOD detection by Feature-based Methods. OOD detection based on feature space is often based on the assumption that after modeling the density function of ID data, OOD data is often in low- density regions, OOD data is far from the center of ID samples Xiao et al. (2010); Zong et al.
8

---

Under review as a conference paper at ICLR 2023
Table 7: OOD detection performance comparison with KNN-based OOD detection. All methods utilize the ResNet-50 model trained on ImageNet. The classification model is trained on ID data only, All values are percentages.
iNaturaiitt
SUN
Places
Textures
Average
Methed
AURROC
FPROS
ALRROC
AUROC
ATROC
FPR95
AUROC
KNN
59,00
86,47
65.82
80.72
76.28
73.76
11.77
97.07
53.97
R5.0]
With Contrasiive Learning
30.15
94.89
48.99
88.62
59.15
84.71
13.55
95.40
38.47
90.91
OURS
96.02
42.83
26.03
92.88
OURS(clipen0:77
18.89
96.28
30,66
92.74
42.22
23.74
93.34
OURS(clip--..)
18.12
96.43
30.58
92.70
42.35
89.34
22.29
94.00
28.34
93.12
2018); Ren et al. (2019) 2021); Ming et al. (2022). A simple density assumption is that the features tollow cassitonniii Gaussian distribution (Lee et al. 2018). For more complex distributions, flow technology can be used (Zisselman & Tamar 2020). Nonparametric methods for estimating density have also recently emerged fiokkCook et al. (2020). And also OOD detection using k-nearest- neighbor has shown good performance (Sun et al. 2022p, but it is based on large amount of known ID data and finding K-nearest neighbors in practical applications is not an easy task in terms of storage and computation.
OOD detection by Fusing information from feature space and output space. Recently, there have been some methods that directly or indirectly mix the feature space information and output space information for OOD detection. (Huang et al., 2021) uses the feature space and output space information implicitly. VIM (Wang et al.] (022) uses the reconstruction error in the feature space and the energy the output space. Igoe et al. (2022) also use the information
feature space and output space for OOD detection. But experiments show that our method is better in performance because we use a more appropriate distance in the feature space.
6 DISCUSSION
Gradient- based OOD detection can often be transformed into combination of information about the feature space and output space, in section 3. the process of the BP algorithm also indirectly reflects this. An important reason why our method outperforms the baseline is that it uses infor- mation from both feature space and output space. Using the proper fusion method makes sense for OOD detection. is good choice to fuse the information of energy and feature space.
Our method benefits lot from the network structure. Because the BN layer (Toffe & Szegedy 2013) is widely used in image recognition, the training data does not shift too much from 0. this is an important reason why reactivation strategy can improve performance. The reactivation strategy in our method is similar to Sun et al. 2021), ). but don't ton't the reactivated feature vector get the energy score. In Section 3.3 we revisit the reactivation method based on distance. From this view, we penalized the deviation from the set D get g(-). VIM (Wang al. 2022] considers the set D to be subspace for training data. OOD-based KNN (Sun et al.] (022) considers the set D to be subset of feature vectors for training data. However, the choice of g()) in Section 3.3 is only heuristic, a strict g(-) should also satisfy normalization equation: |' (f(t)+ffu+bb du 1. In the future, might make some assumptions about 9(.) and then learn the parameters of g(.) from the training data.
7 CONCLUSION
In this paper, we propose RG, novel OOD uncertainty estimation approach utilizing information extracted from the feature space and output space. And we propose framework for combining metrics in the feature space and energy in the output space for OOD detection. Experimental results show that our gradient-based method can improve the performance of OOD detection by up to 2.06% in AUROC, establishing superior performance. Extensive ablations provide further understanding of our approach. We believe that considering both feature space and output space information can improve the performance of OOD detection. At the same time, hope work draws attention to the strong promise the OOD detection methods that combine information from feature space and output space.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexander Amini, Ava Soleimany, Sertac Karaman, and Daniela Rus. Spatial uncertainty sam- pling for end-to-end control. CoRR, abs/1805.000 00020, 2018. URL BKLLtt////axxi... www.iiio...gaa......oo//e//
18C5. 04829
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 430-445. Springer, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Matthew Cook, Alina Zare, and Paul Gader. Outlier detection through null space analysis of neural networks. arXiv preprint arXiv:2007.01260 2020.
Xin Dong, Junfeng Guo, Ang Li, Wei-Te Ting, Cong Liu, and HT Kung. Neural mean discrep- ancy for efficient Computer Vision and Pattern Recognition, pp. 19217-19227, 2022.
detection. In Proceedings of the IEEE/CVF Conference on
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of offtiiibibtttionn examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. A benchmark for anomaly segmentation. 2019.
Yen- Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from diist--iistiiuttinn data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4700 2017.
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distributio detection for large semantic space. In Proceedings the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1770-8719, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distribu- tional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-689, 2021.
Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprint 20X5:22005.10439 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 1448-46. PMLR, 2015.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European confer- ence on computer vision, pp. 491-507. Springer, 2020.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:/706.02690, 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based tion. Advances in Neural Information Processing Systems, 044:11464-1177,, 2020.
detec-
10

---

Under review as a conference paper at ICLR 2023
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. Cider: Exploiting hyperspherical embeddings for disttr-diitiiuuttin detection. arXiv preprint arXiv:2203.044450 2022.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detec- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 8, 2022.
Tanya Nair, Doina Precup, Douglas Arnold, and Tal Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical image analysis,
59:101557, 2020.
Jie Ren, Peter Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022, 2021.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pp. 8491-8501. PMLR, 2020.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. Advances in neural information processing systems, 31, 2018.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In European Conference on Computer Vision, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activa- tions. Advances in Neural Information Processing Systems, 34:144 157444,,, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pp. 20827-20840, 2022.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8779-8778, 2018.
Haoqi Wang, Zhizhong Li, Litong Feng. and Wayne Zhang. Vim: cat-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4921 1931, 2022.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? Advances in Neural Information Processing Systems, 34:29074- 29087, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEFF, 2010.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6): 155--1444,,, 2017.
Ev Zisselman and Aviv Tamar. Deep residual flow for out of distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2020.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national conference learning representations, 2018


Reviews:
[{"review_id": "vkcbgCKLLP", "reviewer": "Reviewer_diYs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors revisit gradient-based OOD detection from the perspective of backpropagation and extend the decomposition G(x)=UV of GradNorm to more loss functions. Here G is a gradient-based detection score, V is the feature norm and U represents output information. According to this decomposition, the authors suggest exploiting suitable U and V for OOD detection. They take U as the energy-based score and derive a V measurement by assuming OOD features follow a standard Gaussian distribution. Experiments on four ImageNet benchmarks demonstrate the effect of the proposed method and discuss the impact of hyperparameters.", "summary_of_the_review": "I list my main questions in this section.\n\n1. In the abstract, the motivation is 'However, existing works still suffer from the problem of overconfidence'. Why the proposed score in (8) can overcome the problem? Could you provide more analysis and comparisons to GradNorm and other OOD detection scores?\n2. In Section 3.1, does \"ground truth\" mean ground truth for classification or ground truth for OOD detection? Can you point out a loss function that corresponds to your proposed score in (8)?\n3. Can we understand the proposed score as an enhancement for the energy-based score? Why did you name it \"Reactivate Gradnorm\"? Is it because the proposed score follows the decomposition G(x)=UV?\n4. What is the purpose of introducing Section 3.3?\n5. The main result with ResNetv2-101 (Table 1) is not as good as the result with ResNet 50 (Table 7). Why use different pre-trained models in this section? In Table 7, is it unfair to compare Reactivate GradNorm to KNN? Would it be more convincing to compare Reactivate GradNorm with KNN+ReAct? \n6. The OOD detection task in this work is a one-sample hypothesis testing problem, i.e., only ID data is accessible. Therefore, the hyperparameters in your score should be determined by the ID data. Table 3 and Table 4 only consider one ID data.\n", "strength_and_weaknesses": "Strength:\n\n- This work proposes a new idea to use the decomposition of gradient-based OOD detection.\n- The proposed method is simple and easy to implement.\n- The empirical result in Table 7 is good.\n\nWeakness:\n\n- The proposed method is not well-motivated.\n- The novelty of the proposed method is unclear.\n- The content is poorly organized.\n- This work lacks a hyperparameter selection method based on ID data.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qY3R0RI5dpY", "reviewer": "Reviewer_TTD6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the gradient-based out-of-distribution (G-OOD) detection problem. By revisiting the current methods, the authors indicate that G-OOD algorithms can be transformed to G(x)=UV, where U is a term representing the output-based score and V indicates the feature-norm-based OOD score. To this end, this work adapts these terms by (1) setting U to the energy score; (2) setting V to the trimmed feature norm. The authors also explore an addition-based mechanism to fuse them. Experimental results show that the proposed methods largely improve the OOD detection performance. ", "summary_of_the_review": "The writing of this work is terrible, and far from the bar of a top-tier AI conference. The novelty is limited.", "strength_and_weaknesses": "Pros:\n\n1. The experiments are thorough and the proposed methods achieved promising results.\n\n2. The addition-based formulation is new to the OOD regime, but its effectiveness is not very surprising. \n\nCons:\n\n1. This idea is not surprising to me and the novelty is limited. First, the relationship between the gradient norm and the output/feature norm has been thoroughly analyzed by [Huang et al. 2021]. The main idea of this work is to replace the U and V terms with two widely-used OOD scores, i.e. energy scores [1] and ReAct [2]. In other words, the proposed methods are simple combinations of existing OOD scores. In effect, an ensemble of two OOD measures is quite straightforward for performance improvement. \nMoreover, the authors did not properly cite these works when adapting them to their own methods. \n\n3. I didn't find much connection between the proposed method and the gradient norm except for the U-V terms. But the title of this work is 'OOD with Reactivated Gradnorm'.\n\n3. The writing is really, really terrible, and I recommend the authors carefully proofread and polish it. And there are also some grammar mistakes and typos:\n\n    (a) 'ImageNet benchmark is not only rich in data sources, but also many categories.' => 'in categories'\n\n    (b) Both 'OOD' and 'ood' were used to represent the terminology 'out-of-distribution'", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m1ESFyQODfS", "reviewer": "Reviewer_7gKP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Detecting OOD data is critical to build reliable machine learning systems, where the models should make reliable predictions for ID data meanwhile detecting OOD data without further predictions. It motivates the recent studies in OOD detection, which has attracted significant attentions recently. The authors propose a novel method named Reactivate Gradnorm, which exploits the norm of clipped feature vector and the energy in the output space for OOD detection. The authors conducted experiments on ImageNet, and the results demonstrate their superiority over the state-of-the-art approaches. ", "summary_of_the_review": "Using gradient information in OOD detection is an interesting direction. However, to improve the quality of the paper, I think the authors can address my concerns. ", "strength_and_weaknesses": "- The authors adopt the gradient information in discerning ID and OOD data. I also believe it is a very important line of research in OOD detection. However, to me, it is not crystal clear *why the proposed method can be superior over previous works in using gradient information*. Especially, the authors claim that the existing works still suffer from the problem of overconfidence. So, two natural questions are 1) why previous works are suffer from overconfident issue and 2) why the proposed method can mitigate such overconfidence. I think the related discussion can be critical, but I am afraid that I cannot find much useful information throughout the paper. \n\n- *The novelty of this paper is limited*. The authors followed previous works [1,2] in calculating the gradient norm of the last layer without back propagation (Eq. 6), and their choices of U and V also follow the previous works in energy scoring [3] and ReAct [4]. So, I think the authors directly combine the several advanced works in previous studies (actually, there are advanced works try to combine different scoring strategies for the improved detection capability of the model [5]), and I am not sure what makes such combination indispensable (for example, one can also change the ReAct by the L2 norm of logit features, so why the authors prefer the ReAct over other advanced scoring strategies). \n\n[1] Conor Igoe, et al. How Useful are Gradients for OOD detection Really? 2022. \n\n[2] Rui Huang, et al. One the Importance of Gradients for Detecting Distribution Shifts in the Wild. NeurIPS, 2022. \n\n[3] Weitang Liu, et al. Energy-based Out-of-distribution Detection. NeurIPS, 2020. \n\n[4] Yiyou Sun, et al. ReAct: Out-of-distribution Detection with Rectified Activations. NeurIPS, 2021. \n\n[5] Hihoon Tack, et al. CSI: Novelty Detection via Contrastive Learning on DIstributionally Shifted Instances. NeurIPS, 2020. \n\n- The paper is motivated by the GradNorm, but *the finally adopted method does not involve much about the gradient information*. So, I am not sure if the addition-based combination can benefit from the gradient information of the model. It will be great if the authors could formally connect the addition-based combination and the GradNorm in math language. \n\n- The authors conduct experiments on ImageNet benchmark, which is a challenging setup with large semantic space and very complex data features. However, *the comparison with some advanced methods (e.g., [6,7]) and the experiments on Hard OOD detection (e.g., CIFAR-10 vs. CIFAR-100, cf., [6]) are missing*. To fully justify the effectiveness of the proposal, I am afraid the authors should conduct more experiments. \n\n[6] Yiyou Sun, et al., Out-of-distribution Detection with Deep Nearest Neighbors. ICML, 2022. \n\n[7] Xuefeng Du, et al. VOS: Learning What You Don't Know by Virtual Outlier Synthesis. ICLR, 2022. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A5xA6OH6tO", "reviewer": "Reviewer_b6vP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose, Reactivate Gradnorm (RG)  in feature space as well as output space for OOD.  Specifically, they multiply the 1-norm of the features and log of the exponential sum of the output.   Their motivation is from a gradient-based method.   For a trained network, if the target is uniform distribution, an in-distribution (ID) instance x will have a large gradient G(x).  If G(x) < c, some threshold, the instance is rejected as OOD .   G(x) can be decomposed into UV, where V depends on the features and U depends on the output.  They propose using an energy based score for U and clipped 1-norm for V.  They further propose a variant that uses U+V, instead of UV.  \n\nThe compared their approach (RG, the UV version) with 9 existing approaches with 4 datasets.  Their results indicate that their approach compare favorably.  For ablation study, they varied the choice of U and V and found their proposed U and V are generally more desirable.  They also perform sensitivity analysis on parameters T (temperature), k (clipping threshold).", "summary_of_the_review": "The decomposition of G into UV is interesting.  Their results indicate their approach compares favorably.  However, the proposed choices of U and V seem ad hoc.  Also, parts of the approach could be explained further.", "strength_and_weaknesses": "Strengths:\n\n1.  The decomposition of G, gradient, into U and V is interesting\n\n2.  Their results indicate that their approach compares favorably against existing approaches.\n\nWeaknesses:\n\n1.  Parts of the proposed approach could be explained further (details in the next section)\n\n2.  What are the principles/properties for choosing U and V besides outputs and features?  The proposed choices seem ad hoc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject