On the Importance of Gradients for Detecting Distributional Shifts in the Wild

1 Introduction . When deploying machine learning models in the real world , there is an increasingly important question to ask : “ Is the model making a faithful prediction for something it was trained on , or is the model making an unreliable prediction for something it has not been exposed to during training ? ” We desire models that are not only accurate on their familiar data distribution , but also aware of uncertainty outside the training distribution . This gives rise to the importance of out-of-distribution ( OOD ) detection , which determines whether an input is in-distribution ( ID ) or OOD . As of recently a plethora of literature has emerged to address the problem of OOD uncertainty estimation [ 2 , 13 , 14 , 16 , 24 , 26–29 , 31 , 32 , 41 ] . The main challenge in OOD uncertainty estimation stems from the fact that modern deep neural networks can easily produce overconfident predictions on OOD inputs [ 34 ] . This phenomenon makes the separation between ID and OOD data a non-trivial task . Much of the prior work focused on deriving OOD uncertainty measurements from the activation space of the neural network , e.g. , using model output [ 13 , 14 , 24 , 27 , 29 ] or feature representations [ 26 ] . Yet , this leaves an alternative space—model parameter and its gradient space—largely unexplored . Will a model react to ID and OOD inputs differently in its gradient space , and if so , can we discover distinctive signatures to separate ID and OOD data from gradients ? ⇤Work done while A.G was working as an undergraduate research assistant with Li ’ s lab . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we tackle this key question by exploring and exploiting the richness of the gradient space , ultimately showing that gradients carry surprisingly useful signals for OOD detection . Formally , we present GradNorm , a simple and effective approach for detecting OOD inputs by utilizing gradient extracted from a pre-trained neural network . Specifically , GradNorm employs the vector norm of gradients directly as an OOD scoring function . Gradients are backpropagated from the KullbackLeibler ( KL ) divergence [ 23 ] between the softmax output and a uniform distribution . ID data is expected to have larger KL divergence because the prediction tends to concentrate on one of the ground-truth classes and is therefore less uniformly distributed . As depicted in Figure 1 , our key idea is that the gradient norm of the KL divergence is higher for ID data than that for OOD data , making it informative for OOD uncertainty estimation . We provide both empirical and theoretical insights , demonstrating the superiority of GradNorm over both output-based and feature-based methods . Empirically , we establish superior performance on a large-scale ImageNet benchmark , as well as a suite of common OOD detection benchmarks . GradNorm outperforms the previous best method by a large margin , with up to 16.33 % reduction in false-positive rate ( FPR95 ) . Theoretically , we show that GradNorm captures the joint information between the feature and the output space . The joint information results in an overall stronger separability than using either feature or output space alone . Our key results and contributions are summarized as follows . • We propose GradNorm , a simple and effective gradient-based OOD uncertainty estimation method , which is both label-agnostic ( no label required for backpropagation ) and OODagnostic ( no outlier data required ) . GradNorm reduces the average FPR95 by 16.33 % compared to the current best method . • We perform comprehensive analyses that improve understandings of the gradient-based method under ( 1 ) different network architectures , ( 2 ) gradient norms extracted at varying depths , ( 3 ) different loss functions for backpropagation , and ( 4 ) different vector norms for aggregating gradients . • We perform a mathematical analysis of GradNorm and show that it can be decomposed into two terms , jointly characterizing information from both feature and output space , which demonstrates superiority . 2 Preliminaries . We start by recalling the general setting of the supervised learning problem . We denote by X = Rd the input space and Y = { 1 , 2 , ... , C } the output space . A learner is given access to a set of training data D = { ( xi , yi ) } Ni=1 drawn from an unknown joint data distribution P defined on X ⇥ Y . A neural network f ( x ; ✓ ) : X ! RC minimizes the empirical risk : RL ( f ) = ED ( LCE ( f ( x ; ✓ ) , y ) ) , where ✓ is the parameters of the network , and LCE is the commonly used cross-entropy loss : LCE ( f ( x ) , y ) = log e fy ( x ) /T PC c=1 e fc ( x ) /T . ( 1 ) Specifically , fy ( x ) denotes the y-th element of f ( x ) corresponding to the ground-truth label y , and T is the temperature . Problem statement Out-of-distribution ( OOD ) detection can be formulated as a binary classification problem . In practice , OOD is often defined by a distribution that simulates unknowns encountered during deployment time , such as samples from an irrelevant distribution whose label set has no intersection with Y and therefore should not be predicted by the model . Given a classifier f learned on training samples from in-distribution P , the goal is to design a binary function estimator , g ( x ) = ⇢ in , if S ( x ) out , if S ( x ) < , that classifies whether a sample x 2 X is from P or not . is commonly chosen so that a high fraction ( e.g. , 95 % ) of ID data is correctly classified . The key challenge is to derive a scoring function S ( x ) that captures OOD uncertainty . Previous approaches have primarily relied on the model ’ s output or features for OOD uncertainty estimation . Instead , our approach seeks to compute S ( x ) based on the information extracted from the gradient space , which we describe in the next section . 3 Gradient-based OOD Detection . In this section , we describe our method GradNorm . We start by introducing the loss function for backpropagation and then describe how to leverage the gradient norm for OOD uncertainty estimation . We calculate gradients w.r.t . each parameter by backpropagating the Kullback-Leibler ( KL ) divergence [ 23 ] between the softmax output and a uniform distribution . Formally , KL divergence quantifies how close a model-predicted distribution q = { qi } is to a reference probability distribution p = { pi } , DKL ( p || q ) = X i pi log pi qi = X i pi log qi + X i pi log pi = H ( p , q ) H ( p ) . ( 2 ) In particular , we set the reference distribution to be uniform u = [ 1/C , 1/C , ... , 1/C ] 2 RC . The predictive probability distribution is the softmax output . Our KL divergence for backpropagation can be written as : DKL ( u || softmax ( f ( x ) ) = 1 C CX c=1 log e fc ( x ) /T PC j=1 e fj ( x ) /T H ( u ) , ( 3 ) where the first term is the cross-entropy loss between the softmax output and a uniform vector u , and the second term H ( u ) is a constant . The KL divergence measures how much the predictive distribution is away from the uniform distribution . Intuitively , ID data is expected to have larger KL divergence because the prediction tends to concentrate on the ground-truth class and is thus distributed less uniformly . GradNorm as OOD score For a given parameter w , the gradient of the above KL divergence is : @ DKL ( u || softmax ( f ( x ) ) @ w = 1 C CX i=1 @ LCE ( f ( x ) , i ) @ w , ( 4 ) where w is a component of network parameter ✓ . Notice that the gradient of the entropy term is 0 , i.e . @ H ( u ) / @ w = 0 . In other words , the gradient of KL divergence is equivalent to averaging the derivative of the categorical cross-entropy loss for all labels . We now define the OOD score via a vector norm of gradients of the selected parameters : S ( x ) = k @ DKL ( u k softmax ( f ( x ) ) @ w kp , ( 5 ) where k·kp denotes Lp-norm and w is the set of parameters in vector form2 . We term our method GradNorm , short for gradient norm . In practice , GradNorm can be conveniently implemented by calculating the cross-entropy loss between the predicted softmax probability and a uniform vector as the target . We will discuss the choices and impacts of the selected parameter set w in Section 4.2 . Rationale of GradNorm Our operating hypothesis is that using the KL divergence for backpropagation , the gradient norm is higher for ID data than that for OOD data . As we show in Section 4.2 , using the gradient norm of the KL divergence is more effective than using the KL divergence directly . Moreover , GradNorm derived from the KL divergence with a uniform target offers two advantages over gradient norms derived from the standard cross-entropy loss . • First , our method is label-agnostic and does not require any ground-truth label . It can be flexibly used during inference time when the label is unavailable for either ID or OOD data . • Second , it captures the uncertainty across all categories , providing more information for OOD detection . We will provide empirical evidence to show the importance of utilizing all labels in Section 4.2 . 2We concatenate all selected parameters into a single vector regardless of the original shapes of the parameters . 4 Experiments . In this section , we evaluate GradNorm on a large-scale OOD detection benchmark with ImageNet-1k as in-distribution dataset [ 16 ] . We describe experimental setup in Section 4.1 and demonstrate the superior performance of GradNorm over existing approaches in Section 4.2 , followed by extensive ablations and analyses that improve the understandings of our approach . 4.1 Experimental Setup . Dataset We use the large-scale ImageNet OOD detection benchmark proposed by Huang and Li [ 16 ] . ImageNet benchmark is not only more realistic ( with higher resolution images ) but also more challenging ( with a larger label space of 1,000 categories ) . We evaluate on four OOD test datasets , which are from subsets of iNaturalist [ 43 ] , SUN [ 47 ] , Places [ 51 ] , and Textures [ 5 ] , with non-overlapping categories w.r.t . ImageNet-1k ( see Appendix B.1 for detail ) . The evaluations span a diverse range of domains including fine-grained images , scene images , and textural images . We further evaluate on CIFAR benchmarks that are routinely used in literature ( see Appendix A ) . Model and hyperparameters We use Google BiT-S models3 [ 21 ] pre-trained on ImageNet-1k with a ResNetv2-101 architecture [ 11 ] . We report performance on an alternative architecture , DenseNet121 [ 15 ] , in Section 4.2 . Additionally , we use L1-norm-based OOD scores as the default and explore the effect of other Lp-norms in Section 4.2 . The temperature parameter T is set to be 1 unless specified otherwise , and we explore the effect of different temperatures in Section 4.2 . At test time , all images are resized to 480 ⇥ 480 .

Summary: The work proposes a novel method for OOD detection. The method itself uses as scoring function the norm of the gradient of the KL divergence of the softmax vector to a uniform distribution. Importantly, this method does not require ground-truth OOD data and does not require hyperparameter tuning. The main contribution of the work is empirical demonstration that the method achieves state-of-the-art performance on a variety of datasets, as well as ablations to further understanding into the effectiveness of the proposed method.