On the Importance of Gradients for Detecting Distributional Shifts in the Wild

1 Introduction . When deploying machine learning models in the real world , there is an increasingly important question to ask : “ Is the model making a faithful prediction for something it was trained on , or is the model making an unreliable prediction for something it has not been exposed to during training ? ” We desire models that are not only accurate on their familiar data distribution , but also aware of uncertainty outside the training distribution . This gives rise to the importance of out-of-distribution ( OOD ) detection , which determines whether an input is in-distribution ( ID ) or OOD . As of recently a plethora of literature has emerged to address the problem of OOD uncertainty estimation [ 2 , 13 , 14 , 16 , 24 , 26–29 , 31 , 32 , 41 ] . The main challenge in OOD uncertainty estimation stems from the fact that modern deep neural networks can easily produce overconfident predictions on OOD inputs [ 34 ] . This phenomenon makes the separation between ID and OOD data a non-trivial task . Much of the prior work focused on deriving OOD uncertainty measurements from the activation space of the neural network , e.g. , using model output [ 13 , 14 , 24 , 27 , 29 ] or feature representations [ 26 ] . Yet , this leaves an alternative space—model parameter and its gradient space—largely unexplored . Will a model react to ID and OOD inputs differently in its gradient space , and if so , can we discover distinctive signatures to separate ID and OOD data from gradients ? ⇤Work done while A.G was working as an undergraduate research assistant with Li ’ s lab . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we tackle this key question by exploring and exploiting the richness of the gradient space , ultimately showing that gradients carry surprisingly useful signals for OOD detection . Formally , we present GradNorm , a simple and effective approach for detecting OOD inputs by utilizing gradient extracted from a pre-trained neural network . Specifically , GradNorm employs the vector norm of gradients directly as an OOD scoring function . Gradients are backpropagated from the KullbackLeibler ( KL ) divergence [ 23 ] between the softmax output and a uniform distribution . ID data is expected to have larger KL divergence because the prediction tends to concentrate on one of the ground-truth classes and is therefore less uniformly distributed . As depicted in Figure 1 , our key idea is that the gradient norm of the KL divergence is higher for ID data than that for OOD data , making it informative for OOD uncertainty estimation . We provide both empirical and theoretical insights , demonstrating the superiority of GradNorm over both output-based and feature-based methods . Empirically , we establish superior performance on a large-scale ImageNet benchmark , as well as a suite of common OOD detection benchmarks . GradNorm outperforms the previous best method by a large margin , with up to 16.33 % reduction in false-positive rate ( FPR95 ) . Theoretically , we show that GradNorm captures the joint information between the feature and the output space . The joint information results in an overall stronger separability than using either feature or output space alone . Our key results and contributions are summarized as follows . • We propose GradNorm , a simple and effective gradient-based OOD uncertainty estimation method , which is both label-agnostic ( no label required for backpropagation ) and OODagnostic ( no outlier data required ) . GradNorm reduces the average FPR95 by 16.33 % compared to the current best method . • We perform comprehensive analyses that improve understandings of the gradient-based method under ( 1 ) different network architectures , ( 2 ) gradient norms extracted at varying depths , ( 3 ) different loss functions for backpropagation , and ( 4 ) different vector norms for aggregating gradients . • We perform a mathematical analysis of GradNorm and show that it can be decomposed into two terms , jointly characterizing information from both feature and output space , which demonstrates superiority . 2 Preliminaries . We start by recalling the general setting of the supervised learning problem . We denote by X = Rd the input space and Y = { 1 , 2 , ... , C } the output space . A learner is given access to a set of training data D = { ( xi , yi ) } Ni=1 drawn from an unknown joint data distribution P defined on X ⇥ Y . A neural network f ( x ; ✓ ) : X ! RC minimizes the empirical risk : RL ( f ) = ED ( LCE ( f ( x ; ✓ ) , y ) ) , where ✓ is the parameters of the network , and LCE is the commonly used cross-entropy loss : LCE ( f ( x ) , y ) = log e fy ( x ) /T PC c=1 e fc ( x ) /T . ( 1 ) Specifically , fy ( x ) denotes the y-th element of f ( x ) corresponding to the ground-truth label y , and T is the temperature . Problem statement Out-of-distribution ( OOD ) detection can be formulated as a binary classification problem . In practice , OOD is often defined by a distribution that simulates unknowns encountered during deployment time , such as samples from an irrelevant distribution whose label set has no intersection with Y and therefore should not be predicted by the model . Given a classifier f learned on training samples from in-distribution P , the goal is to design a binary function estimator , g ( x ) = ⇢ in , if S ( x ) out , if S ( x ) < , that classifies whether a sample x 2 X is from P or not . is commonly chosen so that a high fraction ( e.g. , 95 % ) of ID data is correctly classified . The key challenge is to derive a scoring function S ( x ) that captures OOD uncertainty . Previous approaches have primarily relied on the model ’ s output or features for OOD uncertainty estimation . Instead , our approach seeks to compute S ( x ) based on the information extracted from the gradient space , which we describe in the next section . 3 Gradient-based OOD Detection . In this section , we describe our method GradNorm . We start by introducing the loss function for backpropagation and then describe how to leverage the gradient norm for OOD uncertainty estimation . We calculate gradients w.r.t . each parameter by backpropagating the Kullback-Leibler ( KL ) divergence [ 23 ] between the softmax output and a uniform distribution . Formally , KL divergence quantifies how close a model-predicted distribution q = { qi } is to a reference probability distribution p = { pi } , DKL ( p || q ) = X i pi log pi qi = X i pi log qi + X i pi log pi = H ( p , q ) H ( p ) . ( 2 ) In particular , we set the reference distribution to be uniform u = [ 1/C , 1/C , ... , 1/C ] 2 RC . The predictive probability distribution is the softmax output . Our KL divergence for backpropagation can be written as : DKL ( u || softmax ( f ( x ) ) = 1 C CX c=1 log e fc ( x ) /T PC j=1 e fj ( x ) /T H ( u ) , ( 3 ) where the first term is the cross-entropy loss between the softmax output and a uniform vector u , and the second term H ( u ) is a constant . The KL divergence measures how much the predictive distribution is away from the uniform distribution . Intuitively , ID data is expected to have larger KL divergence because the prediction tends to concentrate on the ground-truth class and is thus distributed less uniformly . GradNorm as OOD score For a given parameter w , the gradient of the above KL divergence is : @ DKL ( u || softmax ( f ( x ) ) @ w = 1 C CX i=1 @ LCE ( f ( x ) , i ) @ w , ( 4 ) where w is a component of network parameter ✓ . Notice that the gradient of the entropy term is 0 , i.e . @ H ( u ) / @ w = 0 . In other words , the gradient of KL divergence is equivalent to averaging the derivative of the categorical cross-entropy loss for all labels . We now define the OOD score via a vector norm of gradients of the selected parameters : S ( x ) = k @ DKL ( u k softmax ( f ( x ) ) @ w kp , ( 5 ) where k·kp denotes Lp-norm and w is the set of parameters in vector form2 . We term our method GradNorm , short for gradient norm . In practice , GradNorm can be conveniently implemented by calculating the cross-entropy loss between the predicted softmax probability and a uniform vector as the target . We will discuss the choices and impacts of the selected parameter set w in Section 4.2 . Rationale of GradNorm Our operating hypothesis is that using the KL divergence for backpropagation , the gradient norm is higher for ID data than that for OOD data . As we show in Section 4.2 , using the gradient norm of the KL divergence is more effective than using the KL divergence directly . Moreover , GradNorm derived from the KL divergence with a uniform target offers two advantages over gradient norms derived from the standard cross-entropy loss . • First , our method is label-agnostic and does not require any ground-truth label . It can be flexibly used during inference time when the label is unavailable for either ID or OOD data . • Second , it captures the uncertainty across all categories , providing more information for OOD detection . We will provide empirical evidence to show the importance of utilizing all labels in Section 4.2 . 2We concatenate all selected parameters into a single vector regardless of the original shapes of the parameters . 4 Experiments . In this section , we evaluate GradNorm on a large-scale OOD detection benchmark with ImageNet-1k as in-distribution dataset [ 16 ] . We describe experimental setup in Section 4.1 and demonstrate the superior performance of GradNorm over existing approaches in Section 4.2 , followed by extensive ablations and analyses that improve the understandings of our approach . 4.1 Experimental Setup . Dataset We use the large-scale ImageNet OOD detection benchmark proposed by Huang and Li [ 16 ] . ImageNet benchmark is not only more realistic ( with higher resolution images ) but also more challenging ( with a larger label space of 1,000 categories ) . We evaluate on four OOD test datasets , which are from subsets of iNaturalist [ 43 ] , SUN [ 47 ] , Places [ 51 ] , and Textures [ 5 ] , with non-overlapping categories w.r.t . ImageNet-1k ( see Appendix B.1 for detail ) . The evaluations span a diverse range of domains including fine-grained images , scene images , and textural images . We further evaluate on CIFAR benchmarks that are routinely used in literature ( see Appendix A ) . Model and hyperparameters We use Google BiT-S models3 [ 21 ] pre-trained on ImageNet-1k with a ResNetv2-101 architecture [ 11 ] . We report performance on an alternative architecture , DenseNet121 [ 15 ] , in Section 4.2 . Additionally , we use L1-norm-based OOD scores as the default and explore the effect of other Lp-norms in Section 4.2 . The temperature parameter T is set to be 1 unless specified otherwise , and we explore the effect of different temperatures in Section 4.2 . At test time , all images are resized to 480 ⇥ 480 .

Summary: This work presents a new method for detecting out-of-distribution inputs that relies on the gradient norm of the KL divergence between the model’s softmax output and a uniform probability distribution.  The method is easy to implement (using the norm of the gradient of a standard cross entropy loss with a uniform label vector), can be used at inference time without access to true labels, and does not require training an additional model.   The authors present empirical evidence that their method outperforms prior work by a significant margin, reducing the false positive rate (FPR95) for out-of-distribution detection by 10.89% and 14.47% on ImageNet and CIFAR-10 respectively. Additionally, the authors perform an ablation study that explores how their method performs as they vary layer parameters, type of target distribution used (uniform versus real), temperature, type of gradient norm, and using only individual terms that form a decomposition of the gradient norm.