On the Importance of Gradients for Detecting Distributional Shifts in the Wild
Authors: Rui Huang
Andrew Geng
Yixuan Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: out-of-distribution detection
uncertainty estimation
gradient

Abstract: Detecting out-of-distribution (OOD) data has become a critical component in ensuring the safe deployment of machine learning models in the real world. Existing OOD detection approaches primarily rely on the output or feature space for deriving OOD scores, while largely overlooking information from the gradient space. In this paper, we present GradNorm, a simple and effective approach for detecting OOD inputs by utilizing information extracted from the gradient space. GradNorm directly employs the vector norm of gradients, backpropagated from the KL divergence between the softmax output and a uniform probability distribution. Our key idea is that the magnitude of gradients is higher for in-distribution (ID) data than that for OOD data, making it informative for OOD detection. GradNorm demonstrates superior performance, reducing the average FPR95 by up to 16.33% compared to the previous best method.

On the Importance of Gradients for Detecting Distributional Shifts in the Wild
Rui Huang
Andrew Geng
Department of Computer Sciences University of Wisconsin-Madison huangrui@cs aicc..cc.... wisc.edu
Department of Computer Sciences* University of Wisconsin-Madison agengOwisc. edu
Yixuan Li
Department Computer Sciences University of Wisconsin-Madison sharonli@cs wisc. edu
Abstract
Detecting out-of-distribution (OOD) data has become a critical component in ensuring the safe deployment of machine learning models in the real world. Ex- isting OOD detection approaches primarily rely on the output or feature space for deriving OOD scores, while largely overlooking information from the gradi- ent space. In this paper, we present GradNorm, simple and effective approach for detecting OOD inputs by utilizing information extracted from the gradient space. GradNorm directly employs the vector norm of gradients, backpropugated from the KL divergence between the softmax output and uniform probability distribution. Our key idea is that the magnitude of gradients is higher for in- distribution (ID) data than that for OOD data, making it informative for OOD detection. GradNorm demonstrates superior performance, reducing the average FPR95 by up 16.33% compared the previous best method. Code and data available: https://gg/ //t::://bbbtt.....// thub. cacluu..commeeee com http://eeplaannnnniiieee wwsc/ haa.a//mmnnniiisaaararnooooooooo gradnorm_oodd
Introduction
When deploying machine learning models in the real world, there is an increasingly important question to ask: "Is the model making a faithful prediction for something was trained on, or is the model making an unreliable prediction for something it has not been exposed to during training?" ?" We desire models that are not only accurate on their familiar data distribution, but also aware of uncertainty outside the training distribution. This gives rise to the importance of out-of-distribution (OOD) detection, which determines whether an input in-distribution (ID) or OOD. As of recently plethora of literature has emerged to address the problem of OOD uncertainty estimation
26
The main challenge in OOD uncertainty estimation stems from the fact that modern deep neural networks can easily produce overcontident predictions on OOD inputs [ This phenomenon makes the separation between ID and OOD data non-trivial task. Much of the prior work focused on deriving OOD uncertainty measurements from the activation space of the neural network, e-g., using 24
27 29 or feature representations 226]. Yet, this leaves an alternative model parameter and gradient space space-largely unexplored. Will model react ID and OOD inputs differently in its gradient space, and if so, can we discover distinctive signatures to separate and OOD data from gradients?
model output 13
14
space sauee-moeel
Work done while A.G was working an undergraduate research assistant with Li's lab.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we tackle this key question by exploring and exploiting the richness of the gradient space, ultimately showing that gradients carry surprisingly useful signals for OOD detection. Formally, we present GradNorm, simple and effective approach for detecting OOD inputs by utilizing gradient extracted from a pre-trained neural network. Specifically, GradNorm employs the vector norm of gradients directly as an OOD scoring function. Gradients are backpropagated from the Kullback- Leibler (KL) divergence [23] between the softmax output and a uniform distribution. ID data is expected to have larger KL divergence because the prediction tends to concentrate
one of the ground-truth classes and is therefore less uniformly distributed. As depicted Figure [ our key idea that the gradient norm of the KL divergence is higher for ID data than that for OOD data, making it informative for OOD uncertainty estimation.
We provide both empirical and theoretical insights. demonstrating the superiority of GradNorm over both output-based and ceature-based methods. Empirically, we establish superior performance on large-scale ImageNet benchmark, well as suite of common OOD detection benchmarks. GradNorm outperforms the previous best method by a large margin, with up to 16.33% reduction in cesse-poittive rate (FPR95). Theoretically, we show that GradNorm captures the joint information between the feature and the output space. The joint information results in an overall stronger separability than using either feature or output space alone.
Our key results and contributions are summarized as follows.
We propose GradNorm, a simple and effective gradient- based OOD uncertainty estimation method, which is both label-agnostic (no label required for backpropagation) and OOD- agnostic (no outlier data required). GradNorm reduces the average FPR95 by 16.33% compared to the current best method.
We perform comprehensive analyses that improve understandings of the gradient-based method under (1) different network architectures, (2) gradient norms extracted varying depths, (3) different loss functions for backpropagation, and (4) different vector norms for aggregating gradients.
We perform a mathematical analysis of GradNorm and show that it can be decomposed into terms, jointly characterizing information from both feature and output space, which demonstrates superiority.
Preliminaries
We start by recalling the general setting of the supervised learning problem. We denote by X' Rd the input space and y = {1, 2, C'} the output space. A learner is given access to a set of training data D {(x,, yi) ] N drawn from an unknown joint data distribution P defined on X' y. A
neural network f(x 0): X + RC minimizes the empirical risk:
Rc(f) = (DCE 9))
where 9 is the parameters of the network, and Les is the commonly used cross-entrope loss:
Ler(s(x), log Ee el-(x)/T
Specifically, (x) denotes the y-th element of f(x) corresponding to the ground-truth label and T' is the temperature.
Problem statement Out-of-distribution (OOD) detection can be formulated as a binary classi- fication problem. In practice, OOD is often defined by distribution that simulates unknowns encountered during deployment time, such as samples from an irrelevant distribution whose label set has no intersection with and therefore should not be predicted by the model. Given a classifier learned on training samples from in-distribution P, the goal is to design a binary function estimator,
ii,,
if 5(x) 2n if S(x) <Y
g(x)
out,
that classifies whether sample x E X' is from P or not. 7 is commonly chosen so that a high fraction (e.g., 95%) of ID data correctly classified. The key challenge is to derive a scoring function S(x) that captures OOD uncertainty. Previous approaches have primarily relied the model's 's output or features for OOD uncertainty estimation. Instead, our approach seeks to compute S(x based on the information extracted from the gradient space, which we describe in the next section.

---

Gradient-based OOD Detection
In this section, we describe our method GradNorm. We start by introducing the loss function for backpropagation and then describe how to leverage the gradient norm for OOD uncertainty estimation.
We calculate gradients w.E.. each parameter by backpropagating the Kullback-Leible (KL) diver- gence | between the softmax output and uniform distribution. Formally, KL divergence quantifies how close preedctee distribution q = {q:} is reference probability distribution p = {pt}.
(e 11
Σ
Pi log
Ps qi
Σ pi log
E pi logp: H( (p).
(2)
In particular, we set reference distribution to uniform u 1/C, 1/CJ E RC, The predictive probability distribution is the softmax output. Our KL divergence for backpropagation can be written as:
el(x)/T
DxL softmax(f(x)) = - Σ log Σj=1 ebx)/T H(u).
(3)
where the first term is the cross-entropy loss between the softmax output and a uniform vector u. and the second term (u) is constant. The KL divergence measures how much the predictive distribution away from the uniform distribution. Intuitively, data is expected to have larger KL divergence because the prediction tends to concentrate on the ground-truth class and is thus distributed less uniformly.
GradNorm as OOD score For given parameter the gradient of the above KI. divergence
@D softmax
(fcx(f(x).
Ow
C
Ow
21
where w is component of network parameter 0. Notice that the gradient of the entropy term is 0, i.e. (H = 0. In other words, the gradient of KL divergence equivalent to averaging the derivative of the categorical cross-entropp loss for all labels.
Figure An example of two-dimensional input space. Input data is depicted the ry-plane, while gradient norm for each input is depicted in the dimension. The magnitude of gradients is higher ID data (light green) than that OOD data (deep blue).
We define the OOD score via vector norm of gradients of the selected parameters:
S(x) = xDxx
1 softmax (oftmax(f(x))) (f()) lpp
(5)
Ow
where ||-1lp denotes Lo-norm and w is the set of parameters in vector form We term our method GradNorm, short for gradient norm. In practice, GradNorm can be conveniently imple- mented by calculating the cross-entropy loss between the predicted softmax probability and a uniform vector as the target. We will discuss the choices and impacts of the selected parameter set in Section
Rationale of GradNorm Our operating hypothesis is that using the KL divergence for backpropa- gation, the gradient norm is higher for ID data than that for OOD data. As we show in Section 4.2 using the gradient norm of the KL divergence is more effective than using the KL divergence directly. Moreover, GradNorm derived from the KL divergence with a uniform target offers two advantages over gradient norms derived from the standard cross-entrope loss.
First, our method is label-agnostic and does not require any ground-truth label. It can be flexibly used during inference time when the label is unavailable for either ID or OOD data.
Second, captures the uncertainty across all categories, providing more information for OOD detection. We will provide empirical evidence to show the importance of utilizing all labels Section ?
We concatenate all selected parameters into single vector regardless of the original shapes of the parameters.

---

Method
iNaturalist
SUN
Places
Textures
Averuge
Space
Method
FPRUS
AUROC
FPROS
AUROC
AUROC
FPRRS
AURRC
AURROT
MSP
63.69
87.59
79.98
78.34
81.44
76.76
82.73
74.45
76.96
79.29
Ouiput
ODIN XR
62.69
89.36
71.67
83.92
76.27
80.67
81.31
76.30
72,99
82.56
Energy
65.33
85.32
73.02
80.87
75.79
71.03
Feature
Mahalanobis
Mi
96.34
46.33
88.43
65.20
89.75
64.46
52.23
72. 10
81.69
62.02
Gradient
GradNorm (ours)
50,03
90.33
46.48
89.03
60.86
84,82
61.42
81.07
54.70
86.31
Table 1: Main Results. OOD detection performance comparison between GradNorm and baselines. All methods utilize the standard ResNetv2- 1011 model trained on ImageNet [ The classification model trained on data only. f indicates larger values are better, while 4 indicates smaller values are better. All values are percentages. All methods are post hac and can be directly used for pre-trained models.
4 Experiments
In this section, we evaluate GradNorm large-scale OOD detection benchmark with ImageNet-Il Ik as in-distribution dataset [16]. We describe experimental setup in Section 4.1 and demonstrate the superior performance of GradNorm over existing approaches in Section 4.2 followed by extensive ablations and analyses that improve the understandings of our approach.
4.1 Experimental Setup
Dataset We use the large-scale ImageNet OOD detection benchmark proposed by Huang and Li 16 ImageNet benchmark is not only more realistic (with higher resolution images) but also more challenging (with a larger label space of 1,000 categories). We evaluate on four OOD test datasets,
which are from subsets of iNaturalist + SUN 4 Places
5l and Textures [ with
non-overlapping categories w.r.f. ImageNet-Ik Ik (see Appendix B. for detail). The evaluations span diverse range of domains including fine-grained images, scene images, and textural images. We further evaluate on CIFAR benchmarks that are routinely used literature (see Appendix (A)
Model and hyperparameters We use Google BİT-S models ResNetv2-11 architecture ([]] We report performance on an alternative architecture, DenseNet- 121 [15], in Section 4.2 Additionally, we use L Lj-norm-basee based OOD scores as the default and explore the effect other norms in Section 4.2 The temperature parameter is set to be unless specified otherwise, and we explore the effect of different temperatures in Section 42 At test time, all images are resized to 480 480.
2
pre-trained on ImageNet-Ik 1k with
4.2 Results and Ablation Studies
Comparison with output- and feature-based methods The results for ImageNet evaluations shown in Table D where GradNorm demonstrates superior performance. We report OOD detection performance for each OOD test dataset, as well the average over the four datasets. For fair comparison, all the methods use the same pre- pre-trained backbone, without regularizing with auxiliary outlier data. In particular, Energy Details and hyperparameters of baseline methods can be found in Appendix B.2
compare with MSP [1] ODIN 271, Mahalanobis (26). well as
GradNorm outperforms the best output-based baseline, Energy score 229 by 16.33% in FPR95.
GradÑorm also outperforms competitive feature-based method, Mahalanobis (6), by 26.99% in FPR95. We hypothesize that the increased size of label space makes the class-conditional Gaussian density estimation less viable. It is also worth noting that significant overheads can be introduced by some methods. For instance, Mahalanobis [26] requires collecting feature representations from intermediate layers over the entire training set, which is expensive for large-scale datasets such as ImageNet. In contrast, GradNorm can be conveniently used through simple gradient calculation without seppe--parametr tuning or additional training.
Gradients from the last layer is sufficiently informative In this ablation, we investigate several variants of GradNorm where the gradients are extracted from different network depths. Specifically, we consider gradients of (1) block n: all trainable parameters in the n-th block, (2) all parameters: all trainable parameters from all layers of the network, and (3) last layer parameters: weight parameters from the last fully connected (FC) layer.
https://gttaab..oo/ggooleeeeeese transfer

---

Imageteer Maauralist
ImageWet
Inagethet Places
Inagehet Textures
SUN
Dematt
Dumalty
Dunalty
L1 Norm of Last Layer Grodlents
Li Norm Last Layer Groalents
L1 Nomm Last Layer Grodients
L Wi Last Layer drodients
(a) Gradient norms using KT. divergence between the softmax prediction and the uniform target.
Imegches
Imugshet
mapeNat
Imegshes
Maturall slst
SUN
Maces
Tenturas
Danaty
2 Deraty
Dendty
11 horm cast Layer Gradients
Nor Last Layer Gracieric
Narm of Last Layer Grudients
Nomm Last Layar Craciants
(b) Gradient using KL divergence between the softmax prediction and the one-hot target.
Figure 2: Comparison of Lı-norm distrihutions of last layer gradients hetween KI. divergence with uniform target and KI. divergence with one-hot target. We show in-distribution data in green and OOD data in gray.
Table contrasts the OOD detection performance using different gradient space. For each setting, we report the FPR95 and AUROC averaged across four OOD datasets, We observe that gradients from deeper layers tend to yield significantly better performance than shallower layers. This is desirable since gradients w.r.f deeper layers are computationally more efficient than shallower layers. In- terestingly, Graiingyy,yraalllle GradNorm obtained from the last linear layer yield the best results among all variants. Practically, one only needs to perform backpropagation w.r.f. the last linear layer, which incurs negligible computations. Therefore, main results are based on the norm of gradients tracted from weight parameters the last FC layer of the
Gradient Space
FPR95
AUROC
+
t
Block
73.52
76.41
Block
74.34
76.63
Block
71.73
78.11
Block
65.07
85.11
All params 69.35
81.14
Last layer params
54.70
86.31
Table 2: Effect of GradNorm using different
subset of gradients. Gradient norm derived from deeper layers yield better OOD detec- tion performance.
neural network.
GradNorm with one-hot uniform targets In this ablation, we contrast GradNorm derived using uniform targets (ours) v.s. one-hot targets. Specifically, our scoring function Equation 5 is equivalent
1 (Ler(f(x), S(x) 1
Σ 1=1
(6)
8w
which captures the gradient of cross-entropy loss across all labels. In contrast, compare against an alternative scoring function that utilizes only one dominant class label:
Sane hos (x) = (J)A( lI.
0w
where ğ the predicted class with the largest output.
We first analyze the score distributions using uniform targets (top) and one-hot targets (bottom) for ID and OOD data in Figure 2 There two salient observations
draw: (1) using uniform target (ours), gradients of ID data indeed have larger magnitudes than those OOD data, as the softmax prediction tends to be less uniformly distributed (and therefore results in larger KI. divergence). In contrast, the gradient norm using one-hot targets shows the opposite trend, with ID data having lower magnitudes. This also expected since the training objective explicitly minimizes the cross-entropy loss, which results in smaller gradients for the majority of ID data. (2) The score distribution using one-hot targets displays a strong overlapping between ID (green) and OOD (gray) data, with large variances. In contrast, our method GradNorm can significantly improve the separability between ID and OOD data, resulting in better OOD detection performance.

---

Gradform from one-hot targe: GradNomm from uniform target
Figure reports the OOD detection performance
using uniform targets (ours) v.s. one-hot targets. We use 11-norm in both cases. For one-hot tar- gets, we use the negative norm, i.e. -Soee hot (x), to align with the convention that IID data has higher scores. GradNorm with uniform targets outper- forms its counterpart with one-hot targets by a large margin. For instance, GradNorm reduces FPR95 by 48.35% when evaluated the SUN dataset. Our analysis signifies the importance measuring OOD uncertainty using all label infor- mation.
100
$4.83
93.92
50.03
46.48
40
iNaturalist
SUN
Places
Textures
Figure 3: OOD detection performance (FPR95) com- parison between uniform (ours) v.s. one-hol target.
GradNorm is effective alternative neural network architecture We evaluate GradNorm a different architecture DenseNet- 121 [. and report performance in Table GradNorm is consis-
tently effective, outperforming the best baseline, Energy 22 by 10.29% in FPR95.
INaturalist
SUN
Places
Textures
Average
Method
Space
Method
FPRUS
AUROC
FPRUS
AUROC
FPROS
AUROC
HPR0S
AUROC
AUROC
MSP ODIN
48.55
89.16
69.39
80.46
71,42
80.11
68.51
78.09
64.47
82.1
Output
37.00
93.29
57.30
86.12
61.91
84.14
56.49
84.62
53.18
87.04
Energy
36.39
93.29
54.91
86.53
$9.98
84.29
53.87
85.07
51.29
87.30
Feature
41.17
Gradient
GradNorm (ours)
23.87
93.97
43.04
87.79
$3.92
83.04
43.16
87.48
41.00
88.07
Table 3: OOD detection performance comparison on different architecture, DenseNet-121 (15). Model
trained on ImageNet-1k f as the ID) dataset. All methods are post hoc and can be directly used for pre-trained models.
L1-norm is the most effective How does the choice of Lp Lo-mor in Equation 5 affect the OOD detection performance? To understand this, we show in Figure 4 the comparison using -nomm Loe as well as the fraction norm (with 0.3). Compared with higher-order norms, -nomo achieves the best OOD detection performance on all four datasets. We hypothesize that -nom is better suited since it captures information equally from all dimensions in the gradient space, whereas higher-order norms will unfairly highlight larger elements rather than smaller elements (due to the effect of the exponent p). In the extreme case, L -orm only considers the largest element (in absolute value) and results the worst OOD detection performance among all norms. On the other hand, the fraction norm overall does not outperform L norm. We additionally provide results for more norms in Appendix
C with (0.3, 0.5, 0.8, 1, 2, 3, 4, 5, 6, oo).
Laa
95
L:
90
FPR95
75
60
50
40
60
INaturalist
SUN
Places
Textures
iNaturalist
SUN
Places
Textures
Figure 4: OOD detection performance comparison under different norms.. show FPR95 (leff) and AUROC (right).
Effect of temperature scaling We evaluate our method GradNorm with different temperatures T from T' 0.5 to T' 1024. As shown Figure T 1 is optimal, while either increasing or decreasing the temperature will degrade the performance. This can be explained mathematically via the term in Equation g Specifically, using large temperature will result smoother softmax distribution, with C'
e/j/r
D e5yr closer to (and V - 0). This leads to less distinguishable

---

distribution between ID and OOD. Our method can be hyperparameter-fee free by setting T 1. For completeness, we have included numerical results under wider range of in Appendix
iNaturalist
Places
AUROC
FPR95
Textures
100
10a
10a
10
10f
10a
102
Temperature
Temperature
Figure 5: OOD detection performance of GradNorm with varying temperature parameter T. We show AUROC (leff) and FPR95 (right).
Divergence
GradNorm of KL Divergance
90
GradNorm is more effective than directly using KL divergence We provide an ablation, contrast- ing the performance of using GradNorm v.s. using the KL divergence derived from Equation 3 OOD scoring function. The results are show in Figure where GradNorm yields significantly better perfor- mance than the KI. divergence directly extracted from the output space, demonstrating the superiority of gradient space for OOD detection.
80
80.87
73.02
70
FPRo5
60
00. 05.85
61.42
50
50.03
30
INaturalist
Places
Textures
Figure 6: Comparison between GradNorm v.s. di-
rectly using the KL divergence as scoring function.
Effect of model capacity In this ablation, explore the OOD detection performance of GradNorm with vary- ing model capacities. For the ease of experiments, we di- rectly use Google BiT-S models pre-trained on ImageNet- 1k [7. We compare the performance of the following model family (in increasing size): BiT-S-R50x1, BiT- S-R101x1, BiT-S-R50x3, B1T-S-R152X2, BiT-S-RI01X3, All models are ResNetv2 architectures with varying depths and width factors. The average performance 4 OOD datasets is reported in Table OOD detection perfor- mance is optimal when the model size is relatively small (ResNetv2-1014) 101xxl), while further increasing model capacity will degrade the performance. Our experiments suggest that overparameterization can make gradients less distinguishable between ID and OOD data and that GradNorm is more suitable under mild model capacity.
Model Size (depth x width)
FPR95
AUROC
f
50x1
56.91
84,17
101x1
55.84
84.63
50x3
61.74
81.89
152x2
61.76
81.33
101x3
66.20
78.89
Table 4: OOD detection performance the model capacity increases.
5 Analysis of Gradient-based Method
In this section, we analyze the best variant of GradNorr., Li-norm of the last layer gradients (sce Section 4.2). and provide insights on the mathematical interpretations. Specifically, we denote the last FC layer in neural network by:
f(x) T x b,
(8)
where /= Je.. C is the logit output, x [21, #2, Fm R the input feature vector, W |mxC is the weight matrix, and b IR is the bias vector.

---

magelett kanuraikst
Imageevet
Imagehet Places
ImageNet Textunes
SUN
Density
Deootty
Danalty
First summation (
Firt simmation
First summarion
First summation 1)
(a) Distribution of U
magelett Ruturattt
Imagehie
Imagehet Placss
Imagelet Texcures
SUN
Densry
Secord summation
Second summation
Second summmation
Secord summrol on
(b) Distribution of V
We show the distributions of the two summations decomposed from the Inomorm of the last layer gradient, for both in-distribution data (blue) and out-of-distribution data (gray).
Figure 7:
GradNorm captures joint information between feature and output First rewrite the KL. divergence between the softmax prediction and the uniform target as:
SelT
DRL (wll/s/otmaxx(((x)))
log
H(u)
C
Dj=1
e//T
1
C.
.lg jml
H(u)
Then we consider the derivative of DKL w.r.f. each output logit fa:
① (log
0DRl
1
1 - CT
0fe
CT
Of.
1
elofT
1-C.
CT
Next the derivative of DKL W.E.t the weight matrix can be written as:
0Da. oW
0Dr..
efi/T
elo/T
[11, F2. ... Fmm] I [
C.
C.
Σ E
0f
CT
Finally, the -orm of gradients of the weight matrix is simply the sum of absolute values of all elements in the gradient matrix:
m
0D OW
Σe (eeeb1))) 1)
ebIT
S(x)
ΣΣ
Σ
Σ
C.
CT
ij
c/IT
(9)
1 C-C.
CT
CT
where the first multiplicative term -= =m= [xa]] second term V/ characterizes information in the output space.
is the Ln-norm of the feature vector x, and the

---

Ablation on U and V In Figure 7 we plot distribution densities of U and V. for both ID and OOD data. It important to note that and V measure statistical distributions in the feature space and the output space, respectively. Therefore, GradNorm captures the joint information between the feature and the output space. The multiplication of both U and V results in an overall stronger separability between ID and OOD, as seen in Figure 2a We report the OOD detection performance using U and V individually as scoring functions in Table 5 both of which are less competitive than GradNorm.
INaturalist
SUN AUROC
Places
Textures
Average
Method
FPR95
AUROC
FPRU5
FPRUS
AUROC
FPR55
AUROC
FPR95
AUROC
1
U (feature space)
77.84
74.33
61,90
78.74
76.42
72.75
67.84
72.77
71.00
74.65
V (output space)
66.14
88.45
69.49
83.
75.95
78.98
81.13
76.06
73.18
81.66
(joint space)
50.05
90.33
46.48
89.03
60.86
84.82
81.07
54.70
86.31
Table 5: OOD detection performance using the decomposed U (feature space) and V/ (output space) as scoring functions. Model is ResNetv2-101 trained on ImageNet-1k 77
6 Discussion
To the best of our knowledge, there is very limited prior work studying how to use gradients for OOD detection. In this section, we discuss connections and differences between GradNorm and previous OOD detection approaches that utilize gradient information, in particular ODIN (Section 6. and
Lee and AlRegib's ss approach (Section
6.1 Comparison with ODIN
Our work is inspired by ODIN 2 which first explored using gradient information for OOD detection. In particular, ODIN proposed using input pre-processing by adding small perturbations obtained from the input gradients. The goal of ODIN perturbations is to increase the softmax score of any given input by reinforcing the model's belief in the predicted label. Ultimately the perturbations have been found to create a greater gap between the softmax scores of ID and OOD inputs, thus making them more separable and improving the performance of OOD detection.
is important to note that ODIN only uses gradients implicitly through input perturbation, and OOD scores are still derived from the output space of the perturbed inputs. Different from ODIN, GradNorm utilizes information solely obtained from the gradient space. The effectiveness of GradNorm beckons revisiting of combining information obtainable from the gradient space and the output space, which could provide a stronger method. We leave this question for future exploration.
6.2 Comparison with Lee and AlRegib
Lee and AlRegib 25 proposed to train an auxiliary binary classifier using gradient information from ID) and OOD data. Importantly, they do not directly use gradient norms for OOD detection, but instead, use them the input for training separate binary classifier. Furthermore, the binary classifier is trained on the OOD datasets, which can unfairly overfit the test data and does not suit OOD-agnostic settings in the real world. In contrast, our methodology mitigates the shortcomings in that GradNorm (1) does not require any new model training. (2) is hyperparameter-fieee free, and (3) is suitable for OOD -gD-atoottic settings. For these reasons, these two methods not directly comparable. However, for completeness, we also reproduce Lee and AlRegib's s method using random noise as a surrogate of OOD data and compare it with GradNorm in Appendix GradNorm outperforms their approach by 15.45% in FPR95 in this fair comparison.
Moreover, we provide comprehensive ablation studies and analyses on different design choices in using gradient- based methods for OOD detection (network architectures, gradients at different layers, loss functions for backpropagation, different Ln-norms, diverse evaluation datasets, and different temperatures, etc.), which were previously not studied in 255. In particular, Lee and AlRegib utilize L2 L2-norm gradients without comparing them with other norms. Our ablation study leads to the new finding that L1 norm works best among all variants with GradNorm, and outperforms Lz-norm by up to 22.31% in FPR95. Furthermore, Lee and AlRegib utilize gradients from all layers to train separate binary classifier, which can cause the computational cost to become intractable for deeper and larger models. In contrast, with GradNorm we show that the last layer gradient will always yield the best performance among all gradient set selections. Consequently, GradNorm incurs negligible computational cost. We believe such thorough understandings will be valuable for the field.

---

7 Related Work
OOD uncertainty estimation with discriminative models The problem of classification with re- jection can date back to early works on abstention B 19,, which considered simple model families such as SVMs [6. A comprehensive survey on OOD detection can be found in [49]. We highlight representative works for post hoc detection, The phenomenon of neural networks' overconfidence to
OOD data is revealed by Nguyen et al. [4 Early works attempted to improve the OOD uncertainty
estimation by proposing the ODIN score 27 OpenMax [1] and Mahalanobis distance (26]. Recent
work by Liu et al. 22 proposed using an energy score for OOD uncertainty estimation, which can be easily derived from discriminative classifier and demonstrated advantages over the softmax confidence score both empirically and theoretically. [aang et al. 44 further showed an energy-based approach can improve OOD uncertainty estimation for multi-label classification networks. [Huang and Li 16 revealed that approaches developed for common CIFAR benchmarks might not translate effectively into a large-scale ImageNet benchmark, highlighting the need to evaluate OOD uncer- tainty estimation in a large-scale real-workds setting. Existing approaches derive OOD scores from either output or feature space, In contrast, we show that gradient space carries surprisingly useful information for OOD uncertainty estimation, which was underexplored in the literature.
OOD uncertainty estimation with generative models Alternative approaches for detecting OOD inputs resort to generative models that directly estimate density [
18 35 38 42) An input is deemed as OOD if it lies in the low-likelihood regions. A plethora of literature has emerged utilize generative models for OOD detection [19 37 39
48). Interestingly, Nalisnick et al. 3 showed that deep generative models can assign high likelihood OOD data. Moreover, generative models can be prohibitively challenging to train and optimize, and the performance can often lag behind the discriminative counterpart. contrast, our method relies on discriminative classifier, which is casier to optimize and achieves stronger performance.
46
Distributional shifts Distributional shifts have attracted increasing research interests important to recognize and differentiate various types of distributional shift problems. Literature in OOD detection is commonly concerned about model reliability and detection of label-space shifts [13]
20 It is
27 29,, where the OOD inputs have disjoint labels w. w.r.f. ID data and therefore should not be predicted
by the model. Meanwhile, some works considered covariate shifts in the input space (30) 36). where inputs can be corruption-shifted or domain-shifted [1]]. However, covariate shifts are commonly used evaluate model robustness 122 and domain generalization performance [52]. where the label
space y' remains the same during test time. It is important to note that our work focuses on the
detection of shifts where the model should not make any prediction, instead of covariate shifts where the model is expected to generalize.
8 Conclusion
In this paper, we propose GradNorm, a novel OOD uncertainty estimation approach utilizing informa- tion extracted from the gradient space. Experimental results show that our gradient-based method can improve the performance of OOD detection by up to 16.33% in FPR95, establishing superior performance. Extensive ablations provide further understandings of our approach. We hope that our research brings to light the informativeness of gradient space, and inspires future work to utilize gradient space for OOD uncertainty estimation.
Societal Impact
Our project aims to improve the reliability and safety of modern machine learning models. This stands to benefit a wide range of fields and societal activities. We believe cut-of-distributte uncertainty estimation is an increasingly critical component of systems that range from consumer and business applications (e.g., digital content understanding) to transportation (e.g., driver assistance systems and autonomous vehicles), and to health care (e.g., unseen disease identification). Through this work and by releasing our code, we hope to provide machine learning researchers with new methodological perspective and offer machine learning practitioners an easy-to-use tool that renders safety against OOD data in the real world. While we do not anticipate any negative consequences to our work, we hope to continue to build on our framework in future work.
Acknowledgement
Research supported by the Office of the Vice Chancellor for Research and Graduate Education (OVCRGE) with funding from the Wisconsin Alumni Research Foundation (WARF).
10

---

References
[1] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, 2016.
[2] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), 2021.
[3] CK Chow. On optimum recognition error and reject tradeoff. IEEE Transactions information theory, 16(1):41-44 1970.
[4] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
[5] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[6] Corinna Cortes and Vladimir Vapnik. Support- networks. Machine learning, 20(3): 273-297, 1995.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. lece, 2009.
[8] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[9] Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In International Workshop on Support Vector Machines, pages 68-82. Springer, 2002.
[10] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing ad- versarial examples. In International Conference Learning Representations, ICLR 2015, 2015.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
[12] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
[13] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of -uttof-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[14] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from nut-of-distribution data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10951-10960, 2020.
[15] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[16] Rui Huang and Yixuan Li. Towards scaling out-of distribbitiiito detection for large semantic space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[17] Xun Huang, Yixuan Li, Omid Poursaeed, John E Hoperoft, and Serge J Belongie. Stacked generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, volume 2, page 3, 2017.
[18] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, 2014.
11

---

[19] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distribution data. Advances in Neural Information Processing Systems, 33, 2020.
[20] Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[21] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV 2020, 2020.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79--3) 1951.
[24] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pages 6402-6413, 2017.
[25] Jinsol Lee and Ghassan AlRegib. Gradients as a measure of uncertainty in neural networks, In 2020 IEEE International Conference on Image Processing (ICIP), pages 2416-2420. IEEE, 2020.
[26] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[27] Shiyu Liang. Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, 2018
[28] Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[29] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.
[30] Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark Gales, Alexey Noskov, An- drey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint 10X.v:207.00755: 007.07455 2021.
[31] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable outtof-distributiinn detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5216-5223, 2020.
[32] Eric Nalisnick, Akihiro Matsukawa, Yee whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference Learning Representattonss, 2018
[33] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[34] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015.
[35] Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. Proceedings of the 30th International Conference Neural Information Processing Systems, pages 4797-4805, 2016.
12

---

[36] Yaniv Ovadia, Emily Fertig, Jie Ren. Zachary Nado, D Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 12:119911114441 -4002 2019.
[37] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribun detection. In Advances in Neural Information Processing Systems, pages 14680-1469) 2019.
[38] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286 PMLR, 2014.
[39] Robin Tibor Schimmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. arXiv preprint arXiv: 10848, 2020.
[40] Joan Serrå, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-based generative models. In International Conference on Learning Representations, 2020.
[41] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In Advances in Neural Information Processing Systems, 2021. 2021.
[42] Esteban G Tabak and Cristina V Turner. family of nonparametric density estimation algo- rithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
[43] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 8769-8778, 2018.
[44] Haoran Wang. Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't 't know? Advances in Neural Information Processing Systems, 2021.
[45] Ziyu Wang, Bin Dai, David Wipf, and Jun Zhu. Further analysis of outlier detection with deep generative models. Advances in Neural Information Processing Systems, 33, 2020.
[46] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Led- sam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Con- trastive training for improved -uttof-distribution detection. arXiv preprint arXiv:2007.05566 2020.
[47] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and patlern recognition, pages 3485-3492. IEEE, 2010.
[48] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in Neural Information Processing Systems, 33, 2020.
[49] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distributiio detection: A survey. arXiv preprint arXiv:21 10. 1334, 2021.
[50] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365, 2015.
[51] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6)):452-1466 2017.
[52] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization:
A survey, arXiv preprint arXh:2103.002503, 2021.
13


Reviews:
[{"review_id": "xnq47R1AL0f", "reviewer": "Reviewer_7r4h", "summary": "This work presents a new method for detecting out-of-distribution inputs that relies on the gradient norm of the KL divergence between the model’s softmax output and a uniform probability distribution.  The method is easy to implement (using the norm of the gradient of a standard cross entropy loss with a uniform label vector), can be used at inference time without access to true labels, and does not require training an additional model.\n\n\nThe authors present empirical evidence that their method outperforms prior work by a significant margin, reducing the false positive rate (FPR95) for out-of-distribution detection by 10.89% and 14.47% on ImageNet and CIFAR-10 respectively. Additionally, the authors perform an ablation study that explores how their method performs as they vary layer parameters, type of target distribution used (uniform versus real), temperature, type of gradient norm, and using only individual terms that form a decomposition of the gradient norm.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Major Comments**\n\n1. *Evaluation on covariate shifts.* The authors only evaluate on datasets where the class hierarchy is distinct from that of the in-distribution dataset.  It is not clear that GradNorm would still perform well on covariate shifts. The authors claim that the GradNorm technique is effective since the in-distribution predictions tend to concentrate on the ground truth class and are thus distributed less uniformly.  It’s possible that covariate-shifted data would similarly have a higher GradNorm because the predictions would concentrate on the ground truth class. For ImageNet, the authors could have evaluated on ImageNetV2, ObjectNet, ImageNet-R, ImageNet-C, ImageNet-Vid-Robust, or YouTube-Bounding-Boxes. For CIFAR, the authors could have evaluated on CINIC-10, CIFAR-10.1, or CIFAR-10C. \n\n2. *Hyperparameter to set the gradient norm threshold.*  The authors claim that their work is hyperparameter free if one sets the temperature to T=1.  However, isn’t the threshold on the gradient norm for separating in-distribution versus out-of-distribution data ($\\gamma$) an important hyperparameter?  Clarifying how this parameter is set for different datasets would improve the paper.\n\n**Minor comments**\n\n3. *Calibration*. From Figure 7, it looks like the authors only tried one temperature value that was smaller than T=1. The authors also do not give an explanation for why temperatures smaller than T=1 would work. It would also be interesting to see if the optimal temperature for calibration would fare any better than T=1.\n\n\n4. *Explanation of prior work.* It would be helpful if the authors gave a brief description of prior work they compare against, e.g. MSP, ODIN, Generalized ODIN, Energy or Mahalanobis. \n\n5. *Effect of label noise.* Mislabeled images might force the in-distribution predictions to be more uniformly distributed. Have the authors looked at whether GradNorm can be used to detect label noise? Would the technique be able to distinguish between out of distribution and label noise?\n\n6. *Organization.* The section “Effect of temperature scaling” is an ablation study and should be included in Section 4.\n\n\n**Originality**: The method is novel and the authors describe how it is distinct from prior work.\n\n**Quality**: See comments above.\n\n**Clarity**: Paper is clearly written, but could be improved by clearer descriptions of related work.\n\n**Significance**: Problem is important, and proposed algorithm is easy to implement and deploy.\n\nIf major comments are addressed, I would consider raising my score to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author should discuss potential negative societal impacts of their work in the main paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "UUodirRk2gC", "reviewer": "Reviewer_UMiA", "summary": "The manuscript proposes using the gradient norm of a trained classifier for out-of-distribution detection. Concretely, the norm of the gradient of the KL divergence between the actual prediction and the uniform prediction with regard to the network parameters is used to distinguish in-distribution and out-of-distribution data. Results on several benchmarks show this method has better OOD detection performance than some other methods.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Update**: Increase score to 6 due to added experiment SVHN vs CIFAR10\n\nI found the manuscript fairly easy to read and understand, the method seems sensible and the results are interesting.\n\nI see some weaknesses in the discussion of related work and in the evaluation.\n\nI would suggest to more clearly and extensively discuss in one place exactly the relation of this work to [28]. And refer to that more extensive discussion in the introduction already, as it seems very crucial in order to determine what is novel about this work. Like it seems to me that that [28] use may be the same gradients as in this manuscript? From their “To generate gradients, we utilize a confounding label where all classes are positive (i.e. a vector of all 1’s).” Or is there a difference still between the two? Somehow it seems the motivation in this manuscript and theirs is opposite, judging from ““If the model is unfamiliar with the given input, however, the amount of updates required to 1) learn new features to properly represent the new input, and 2) associate the new features with the confounding label, would be larger” [28] and “Our operating hypothesis is that using the KL divergence for backpropagation, gradient norm is higher for ID data than that for OOD data“ (this manuscript). Do you have any comments on that? Or maybe I have some misunderstanding?\n\nRegarding empirical evaluation, where do the numbers for [28] come from? Comparing Table 4 in this manuscript to Table 1 in [28] the numbers seem completely different. Also, shouldn’t your method perform worse since their method even trains an extra classifier that sees OOD data? It would be good to also clearly show what is the difference in method that causes your results to be better, if they are indeed so.\n\nAlso I think it would be good if you could clearly delineate what methods you are including and excluding in your comparison. There are so many variants for OOD detection methods, generative, discriminative, with outliers, without outlines etc. and it would be good to know what you consider fair comparisons and what not.\n\nIt would also be important to report values not only for CIFAR10 in-dist vs SVHN OOD but also reverse case, SVHN in-dist vs CIFAR10 OOD as some methods degrade the performance in the latter “easier” setting.\n\nHow do the results in 4.2 table 3 relate to the results in table 2? Seems even the best results from table 3 do not reach the results from table 2?\n\nYou should include https://arxiv.org/abs/1911.07421 and https://arxiv.org/abs/2006.10848 in related work generative models and your supplementary comparison for generative models. https://arxiv.org/abs/2007.05566 could be another interesting related work.\n\nAnalysis and Ablation studies seem interesting to me.\n\nWhat was the motivation for training your own networks and not using pretrained networks on CIFAR10 etc.?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I only saw a very minimal discussion of this at the beginning of the introduction.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AsbGasNeUIO", "reviewer": "Reviewer_3Auv", "summary": "The work proposes a novel method for OOD detection. The method itself uses as scoring function the norm of the gradient of the KL divergence of the softmax vector to a uniform distribution. Importantly, this method does not require ground-truth OOD data and does not require hyperparameter tuning. The main contribution of the work is empirical demonstration that the method achieves state-of-the-art performance on a variety of datasets, as well as ablations to further understanding into the effectiveness of the proposed method. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is overall a solid contribution and would recommend acceptance for the following reasons. \n- The method is relatively simple and practical. The only operation is taking the norm of the gradient of the KL divergence, and does not require ground-truth label information. This latter fact is especially important as it might obviate the need for excessive human intervention in practice. Methods that use ground-truth label information almost seem to be “cheating”, and seem to have a scaling problem by requiring samples from the OOD dataset to begin with. \n- The writing and organization were exceptionally clear throughout. I do not actively do work in OOD detection, yet I found the work easy to follow. \n- The experimental methodology is high-quality and demonstrates a significant improvement over baselines. \n    - The baselines were tuned, which is especially important because the paper has many experiments on imagenet, but some of the baselines were originally only tested on the cifar datasets. \n    - There is a large variety of datasets considered as well, which helps to give a balanced sense of the performance of the proposed method. Indeed, the method performs worse than some baselines on some datasets, yet performs the best on average across all datasets. \n    - The ablations (to KL divergence, different L_p norms, different temperatures, gradnorm of different layers, one-hot vs. all labels, feature and output space components) were extensive and interesting. I especially found comparison to using just the norm of the KL divergence interesting; based on the motivation on line 35, I would have expected KL divergence to do better than the proposed method (more on this point below). \n- I liked the mathematical analysis in section 5 to decompose the gradnorm into a feature space quantity and an output space quantity, and especially appreciated how it led to another ablation experiment. Given that the main contribution of the paper is empirical, I saw this section as a nice bonus to further understanding of the proposed method, and in future work would look forward to seeing more extensive mathematical analysis. \n\nThere did remain limitations of the paper, but these were not enough to outweigh its advantages above. I think addressing these limitations would make the paper stronger.\n- I found the intuition for the gradnorm to be lacking. The main intuition is on lines 35 and 94, where I get the impression the ID data should have a higher KL divergence. However, it’s not clear to me that the gradient norms for ID data should be higher. The experiments certainly demonstrate that the gradient norm is a good discriminator of ID and OOD data, but I would have liked better intuition for this result. \n- There was no discussion of limitations or negative potential impact, which is a problem considering the explicit NeurIPS requirement this year. As a suggestion to the authors, I think the derivation of the gradnorm decomposition in section 5 could be left to the appendix so that more room is left in the main body for a discussion of limitations and negative potential impact of the work. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "copied from above: \n\nThere was no discussion of limitations or negative potential impact, which is a problem considering the explicit NeurIPS requirement this year. As a suggestion to the authors, I think the derivation of the gradnorm decomposition in section 5 could be left to the appendix so that more room is left in the main body for a discussion of limitations and negative potential impact of the work. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "A7HcuOqOuZW", "reviewer": "Reviewer_p1oz", "summary": "The work presents GradNorm which uses the norm of gradient, back-propagated from the KL divergence between the softmax output and a uniform probability distribution, as the uncertainty score for OOD detection. The experimental results show that the proposed method outperforms the baselines on ImageNet1k vs OOD and CIFAR vs OOD benchmarks. Ablation studies are presented to help better understand the proposed method. The paper is written well and the experiments are comprehensive .\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Originality: The proposed method GradNorm is novel, simple and efficient, as demonstrated by the experimental results. Most existing work rely on the output (MSP, ODIN, SNGP) or feature space (Mahalanobis) for deriving OOD scores. This work suggests making use of the information from the gradient space for OOD detection. \n\n- Quality: The results are presented clearly. Comprehensive ablation studies were conducted and that did help me better understand the method.  It would be great to include the benchmark dataset CIFAR10 vs CIFAR100 as one of the near-OOD tasks.\n\n- Clarity: The work is well written and the content is well organized. One comment I have is regarding one of the baseline methods, called Lee and AlRegib [28]. It is not clear how the experiment for this method was conducted exactly. As mentioned in line 151-152, Lee and AlRegib requires “a binary classifier trained on the OOD datasets, which can unfairly overfit to the test data and does not suit OOD-agnostic settings in the real world”. But it is also stated in line 139-140 that “for a fair comparison, all the methods use the same pre-trained backbone, without regularizing with auxiliary outlier data”.\n\n- Significance: the results are important. This work inspires the future work to use the information from the gradient space for OOD detection. Combining the information from gradient space with that from the output space and feature space, would potentially result in a better OOD score.\n\n\nUpdate: Thank the authors for responding to my questions. I keep my score. I would suggest the authors to incorporate the reviewers' comments to the final version. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations were not addressed. It is unknown if the method works well on discrete data like text. That can be left as a future work.\n\nSocietal Impact was not discussed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using the KL divergence between the softmax output and the uniform distribution to detect OOD examples (OOD examples are expected to have lower KL divergence). Authors further show extensive empirical results in support of their method.

All reviewers agree that this work is novel, well-written and is a significant contribution to the community. Reviewers have provided detailed comments to improve this work and increase its impact and I highly recommend authors to take all these suggestions into account. In particular, I suggest further explanation and investigation about the effectiveness of this technique when the class label sets are distinct as well as when there is covariate shift.

Given reviewers' consensus, I recommend accepting the paper.
Decision: Accept (Poster)