Year: 2021

Abstract:
Uncertainty estimation and out-of-domain (OOD) input detection are critical for improving the safety and robustness of machine learning. Unfortunately, most methods for detecting OOD examples have been evaluated on small tasks while typical methods are computationally expensive. In this paper we propose a new gradient-based method called GRODIN. The proposed method is conceptually simple, computationally cheaper than ensemble methods and can be directly applied to any existing and deployed model without re-training. We evaluate GRODIN on models trained on CIFAR-10 and ImageNet datasets, and show its strong performance on various OOD ImageNet datasets such as ImageNet-O, ImageNetA, ImageNet-R, ImageNet-C.

heading: 1 INTRODUCTION
text: Many problems are solved with machine learning techniques today. Prominent examples of supervised learning power could be found in many domains: image classification, speech recognition, medical surveys, self-driving cars, text translation (6; 19; 28; 24; 1), etc.. In some applications, it is very important to be confident on the prediction of a trained model: self-driving cars, medicine, stock market trading, etc.. Usually, the prediction mechanism remains uncovered and it requires additional efforts to ensure the model quality when working with real-world data. We search for a new way to automatically understand that prediction on this particular sample is not reliable with data uncertainty estimation or knowledge uncertainty estimation (2).
One of the most important tasks in this field is out-of-domain detection. It allows us to separate the data suitable for inference from the examples that are not intended for model usage. Due to the importance of the task out-of-domain methods evolved from ‘naive‘ approaches to sophisticated algorithms. These algorithms either analyse the prediction model results to understand if we could rely on them, or build a new dedicated model for OOD. The first type of the methods we call “lightweight”, as they usually could be used for on-the-fly verification of prediction. The other class could deliver better detection quality, but requires substantive computational resources and separate training. We refer to this one as “heavy”.
The “lightweight” models are usually built on top of certain prediction tasks, such as classification, and are based on the properties of loss functions used in the training process. Softmax probability is an example of such property, it is directly used for many practical implementations (10; 7). The prediction quality of the softmax probability thresholding could be extended with input preprocessing and probability calibration as shown in ODIN(20) algorithm. This method provides the state-of-theart results on most available benchmarks (tiny-ImageNet (26), CIFAR-10 (16), CIFAR-100 (16), MNIST (3), SVHN (25), etc.). However, there is room for improvement. There is a bunch of more challenging problems constructed on top of ImageNet(26) image classification task: ImageNet-A (12), ImageNet-O (12), ImageNet-R (8), ImageNet-C (9).
The alternative “heavy” methods were developed: outlier exposure (11); mahalanobis distance-based confidence score (18); class of strong and theoretically justified out-of-distribution detectors called ensembles (2; 23; 22). All these methods usually provide better results, but are hard to use in practice: some of them require changing the training loss function and therefore neural network retraining (11); or require solving auxiliary logistic regression problem and heavy computation of mahalanobis distance in (18); sometimes researcher need additional training and inference of several neural networks simultaneously for ensemble methods (22).
Recent studies in deep learning suggest that the gradient of the loss function taken for a particular example carries a lot of information other than model prediction. Influence functions (14) use information about first and second derivatives with respect to network weights to estimate what is the impact of a single precedent on test set quality. Neural tangent kernel (13) gives us another point of view on the training process and generalization from the position of gradient analysis. To the best of our knowledge, there is only one work in out-of-distribution detection domain that utilize the gradient of the model with respect to weights (17). This approach uses information from the gradient of different layers as a building blocks for their separate DNN model. The approach requires a lot of out-of-distribution data to train the model and could not be assigned to “lightweight” models.
In this paper, we focus on developing a new simple and computationally lightweight approach for detecting out-of-distribution examples based on gradient analysis. We investigate the derivative of cross-entropy loss function and decompose it into two parts, showing the difference between them. The proposed method is being tested on several challenging benchmarks and compared with various methods. Also, we analyse the impact of highly popular input preprocessing and give intuition why this technique helps OOD detectors.

heading: 2 METHOD
text: Model mistakes come from different sources: irreducible errors, low model capacity, wrong modeling assumptions, sample distribution shift, and so on. For each type of errors, we define a formal problem setup and develop methods to deal with it. Those setups are not mutually exclusive and their methods could complement each other.
This paper focuses on a particular problem: estimation of the probability that a particular sample comes from a different population. To formally define this problem we use notation from Tab. 1. Given this notation, there exist a statistical population Γ. Elements of this population are pairs (x, y), where x ∈ Rn is a sample, y ∈ Y is a label. We assume that there exists a deterministic correspondence between population precedent and corresponding label. Statistically representative training set X is used to train prediction model by minimization of empirical loss function L. With enough data and mild assumptions on loss function, the resulting model with parameters θ̂ should replicate correspondence between x and y. As a result, our model should give a correct prediction to any x from the general population. Thus, given assumptions, there is a natural way to define xi as the out-of-domain sample:
xi is out-of-domain↔ ∀(x, y) ∈ Γ : xi 6= x (1)
Our task is to estimate, if the particular sample xi is out-of-domain, given only the estimated model parameters θ̂.
The core idea of our method is quite simple. We train prediction model by minimisation of some additive loss function L = 1m ∑ (x,y)∈X `(θ,x, y). When minimisation is performed by stochastic gradient descent or similar methods, the expectation of the model parameters gradient norm reduces on training set elements with the number of optimization steps. The expectation aims to zero for flexible enough decision functions. In tangent kernel space (NTK for neural models) the optimization process looks like seeking for optimization parameter θ that pulls representation of training set points ∇`(θ,x, y) to zero vector. We can use this property and determine the model behavior for an example by the distance of its representation in the trained tangent kernel space (defined by θ̂) from zero.
Using this information we can predict many practical properties of the particular prediction: expected accuracy, model uncertainty, out-of-domain points detection, etc.. In this paper, we focus on the latter case and build a method for out-of-domain detection on top of this principle.
The modern definition of the domain depends on the application and we will show this difference in the experimental section. From a theoretical perspective, we equate domain and the population. This way we allow a researcher to define their notion of the domain with the selection of sampling technique. The ideally trained model brings all points of the population to a stationary point in model parameters space. In this paper we make an assumption that kernel of this transformation is equal to the population and the observed transformation differs from ideal one with Gaussian error: ∇θ`(θ,x, y)|θ̂ = ∇θ`(θ,x, y)|θ∗ + δ, δ ∼ N (0, E). Now we can define a log probability of population Γ membership:
g(x, θ̂) = logP (x|Γ) ∼ ∥∥∇θ`(θ,x, y)|θ̂∥∥22
The main problem for practical application of the method is to define a loss function for examples, that come to the model during its exploitation and have no label. In this case, it is natural to use the class assigned by the model ĉ as a label and check the uncertainty of the model in the decision. We focus, without loss of generality, on image classification problem over k classes with neural models. Negative log likelihood over softmax probability modeling is one of the most commonly used loss functions for this problem:
`(θ,x, y) = − log e hy(x,θ)∑k
c=1 e hc(x,θ)
where hc are outputs of the neural model for each class c. Using this model and the label assumption we can formulate the result log probability of population membership g(x, θ̂) for arbitrary point x:
g(x, θ̂) = ∥∥∥∥∥∇θ log ehĉ(x,θ)∑k c=1 e hc(x,θ) ∣∣∣∣∣ θ̂ ∥∥∥∥∥ 2 2
Now we can choose significance level α and filter out points that belong to the population with probability less than this significance level: P (x|Γ) < α. In practice we don’t use significance level directly, but define a threshold δ(α) for g(x, θ̂). If we have information on out-of-domain examples we choose the threshold that delivers the selected significance level on cross-validation. We use this setup as a baseline variant of our method in the experimental section.
In the literature there are at least two ways how to enhance the result of the baseline model: temperature scaling (5), and input perturbation (20). In our framework these enhancements are easily explained and compliment the baseline model. To show this we need to unfold components of the gradient vector:
∇θ`(x, θ̂) = −∇θ log maxc′ e h c′ (x,θ)∑k
c=1 e hc(x,θ) ∣∣∣ θ̂
= S(x, θ̂)G(x, θ̂), where S(x, θ̂) = 1
1+ ∑ c 6=ĉ e (hc(x,θ̂)−hĉ(x,θ̂))
G(x, θ̂) = − ∑ c 6=ĉ e (hc(x,θ̂)−hĉ(x,θ̂)) ∂(hĉ(x,θ)−hc(x,θ)) ∂θ ∣∣∣ θ̂
In the last expression there are two multiplicative parts (S and G). S is a scalar part, that reflects the probability of the predicted class. The G is cumulative gradient vector for all classes output nodes of the network weighted by their preference over predicted class ehc(x,θ̂)/ehĉ(x,θ̂). The first part is equal to the softmax score from ODIN up to the notations used. The second part is the information from NTK space.
As one can see both S and G heavily depend on class weights hc, and it is important to calibrate these values for the task the model is used for. One of the most popular ways of model calibration is
temperature parameter injection (5). In this method we scale class weights hc with the temperature factor 1T . The optimal temperature is then optimized using some calibration quality metric like Expected Calibration Error (5) for the target dataset. The solution of this optimization problem depends on the data provided, see the experimental section for details.
The other way of out-of-domain detection model improvement is altering the source signal, e.g. the image in case of the image classification task. In our framework we assume that during training process the optimization pulls domain space to a stationary point in model parameters space. As only training examples are observed, we pull the entire domain using the train set as mount points. In this process the rest of the domain slacks and this cause well-known difference in gradients between training set points and validation points. To reduce this slack we can move point towards training points. We follow the alteration procedure introduced in the ODIN paper:
xp = xo − ε sign ( −∇x log maxc′ e hc′ (x,θ)/T∑k
c=1 e hc(x,θ)/T ∣∣∣∣∣ xo )
During this procedure, source image is altered to reduce the gradient and compensate the slack between training points and the rest of the domain.
To sum it up, the proposed method is a composition of ε-input perturbation, temperature calibration by scaling outputs by 1/T factor and decomposition of ∇θ`(x, θ̂) into two multiplicative parts: S(x, θ̂) and G(x, θ̂). These parts can be used as a single estimator: g(x, θ̂) = ‖S(x, θ̂)G(x, θ̂)‖22, or alone: g(x, θ̂) = ‖G(x, θ̂)‖22, g(x, θ̂) = S(x, θ̂). The last variant gives us exactly ODIN.

heading: 3 EXPERIMENTS
text: We performed a set of experiments to study the properties of our method, its effectiveness and its competitiveness with other approaches. Both versions of the proposed gradient method were evaluated: G and S ∗G to show that there is a significant difference in their performance. We used popular image collections as a source of out-of-domain examples. Benchmark is standard: out-of-distribution detector is evaluated as a binary classifier that should separate in-domain samples from out-of-domain. The test dataset is a mixture of the in-domain test set and the out-of-domain dataset. The tiny-ImageNet dataset was used as an out-of-domain for all experiments with networks, trained on CIFAR-10. For neural networks, trained on ImageNet-1K, we used four recently proposed out-of-distribution collections, that represent different distribution shifts: ImageNet-O, ImageNet-R, ImageNet-A, and part of ImageNet-C. Image databases statistics are shown in Tab. 2.
• ImageNet-O (12) is a subset of ImageNet-22k that do not intersect with ImageNet-1K. It consists of images that fool Maximum Softmax Probability out-of-distribution detector (10) based on ResNet-50 neural network and represents distribution shift that brings new unobserved classes. Popular out-of-distribution detection methods work poorly on this setup as they utilize logits or softmax probabilities that are heavily biased by the design of the dataset.
• ImageNet-R (8) contains images of ImageNet objects with different textures and styles: paintings, sculptures, origami, cartoons, toys, etc. Classes are a subset of original ImageNet classes. This dataset represents distribution shift that does not change the semantic of objects but change their shape or texture.
• ImageNet-A (12) contains images of ImageNet objects that are extremely hard to be classified correctly by any neural network trained on ImageNet. It represents natural adversarial examples: it is obvious for humans what is depicted on an image, but trained model struggles to give the correct answer.
• ImageNet-C (9) contains copies of the ImageNet validation set that were corrupted with various noises. Original dataset consists of fifteen noise types, each noise type has five levels of severity. We selected the most challenging for the ResNet-50 model noise type according to the original paper: level five ‘Frosted Glass Blur‘. ImageNet-C dataset represent a distribution shift that occurs when for example a camera or transmission channel is broken.
In our experimental study out-of-distribution detectors are applied to popular neural network architectures trained on CIFAR-10 or ImageNet-1K collections, such as ResNet-18 and ResNet-50. The detailed information about the models used for each setup could be found in Tab. 3. All experiments were performed with the PyTorch framework, due to a blind-review we attached source code to supplementary materials and will provide the link to the repository with the camera-ready paper version.
The following out-of-distribution detectors were used as competitors to our method:
1. Maximum Softmax Probability (MSP), also referred as a baseline, described in (10). This method uses negative maximum softmax probability as an out-of-distribution score.
2. ODIN described in (20). This method is a modification of MSP. It combines input preprocessing and MSP together using negative maximum softmax probability of preprocessed image as an out-of-distribution score.
3. Maximum Logit (ML) described in (7). This method uses negative maximum logit (neural network output without applying softmax activation function) as an out-of-distribution score.
4. Ensemble method(2; 23), trained with different random seeds. The ensemble consists of 6 different uncertainty measures that can be used as a scoring function (confidence, entropy of expected, expected entropy, mutual information, EPKL, MKL) and we choose the one that brings the best separation in terms of ROC-AUC as the final score. Here we need to train several models from scratch, thus the access to a training set is mandatory. As a result, the method can’t be used on an arbitrary pretrained model so we evaluate it only for the ResNet-50 model, as we train it from scratch.
ROC-AUC is our quality measure for out-of-distribution detection. All table cells express experimental results in µ± σ pairs, where µ is the average value of the metric and σ is its bootstrap deviation computed on 100 resamplings.
The rest of the experimental section consists of two parts: the competitive comparison in Sec. 3.1 and ablation study for our method in Sec. 3.2.

heading: 3.1 COMPARISON WITH OTHER METHODS
text: The following methodology is used to compare out-of-domain detectors for each collection from Tab. 2:
• The classification model is trained on a base dataset (CIFAR-10 or ImageNet-1K)
• The OOD model parameters are tuned on the validation subset of the collection to maximize the ROC-AUC score
• The resulting method is then applied to test part of the collection and reported in the result table
Parameters tuning is required for the proposed gradient method and for ODIN. We used grid-search to estimate optimal ε and T . The grid for ε was 21 evenly spaced points from 0 to 0.004. We select among {1, 2, 5, 10, 20, 50, 100, 200, 500, 1000} to find optimal temperature T . The same grid for ε and T was used in ODIN paper (20).
Our experiments can be divided into two groups:
1. Ones that use pretrained weights from PyTorch collection. A researcher can skip the training procedure and just evaluate methods on downloaded data. These experiments were performed with the ResNet-18 model trained on the ImageNet classification task.
2. Experiments with models trained from scratch. In this work we train WideResNet-28-10 for CIFAR-10 classification and ResNet-50 for ImageNet classification. All parameters for the training procedure were taken from original papers for these models. For ResNet50 experiments, we trained four (as proposed gradient methods require two forward and two backward passes) different models from different random seeds to evaluate ensemble methods.
Results of out-of-distribution detections are presented in tables 4, 5. One can see that both gradient methods: G and SG outperform all baselines on CIFAR-10 vs tiny ImageNet benchmark with a significant margin. Confidence intervals for G and SG parts intersect, so we can say that both methods show nearly the same performance.
Results for ImageNet-like datasets are more various. Let’s inspect results for every dataset separately (both architectures ResNet-18 and ResNet-50 behave nearly the same):
• ImageNet-O: It is the only dataset on which SG-part is better than G-part. Still, both gradient methods outperform all baselines with giant margin.
• ImageNet-R, ImageNet-C: G-part outperforms all baselines on these two datasets. SG-part showed bad performance giving up to ODIN and ML.
• ImageNet-A: On this dataset G-part outperforms ODIN and ML but failed to overtake ensemble baseline.
To sum it up, G-part can significantly outperform all “lightweight” baselines (ones that do not require training several models from scratch) on both benchmarks. And only in the case of ImageNet-A with ResNet-50 as a predictor the ensemble model was able to outperform the proposed method.

heading: 3.2 ABLATION STUDY
text: The proposed method consists of three parts: input perturbation, temperature scaling, and scoring based on a gradient with respect to neural network weights. Our initial intuition suggested that a gradient-based scoring scheme is self-sufficient and could be used as a stand-alone discrepancy
measure. Turns out this is not the case, and both preprocessing steps are also essential to achieve competitive results.
We evaluated this impact on ImageNet-like datasets. This stand-alone discrepancy measure is a specific case of our final methods, thus it is enough to set free parameters for particular values: T = 1 and/or ε = 0. Thus we compare the final model with 3 simplified variants: T -scaling only model, where ε was fixed to zero and temperature T was tuned with grid-search; ε-perturbation only, with temperature T fixed to 1 and ε tuned on the grid; the pure gradient-scored method with ε equal 0 and T equal 1. The presented results are obtained on the test set, while hyperparameters were chosen depending on the best performance on gridsearch set.
The results are presented in Tab. 6, 7. As one can see for both G and SG parts vanilla and only ε tuning are nearly the same and both are weaker than only T tuning or full method. Only T tuning in some cases match the full scheme, but to achieve outstanding results it is necessary to use both preprocessing steps.

heading: 4 DISCUSSION
text: The task for out-of-domain detection is usually related to sensitive information and it is extremely hard to find real-world data on this topic. To resolve this issue there are joined academic and industry initiatives such as NeurIPS Challenge (21). The data from this event is not yet publicly available and we have to use state of the art datasets, that are artificially designed. In this section, we want to analyse this bias and its influence on the methods from the experimental section.

heading: 4.1 MODEL CALIBRATION
text: In theoretical Sec. 2 we noted, that calibration procedure is mandatory because the datasets used for out-of-domain detection could differ from the training set in many ways. We use Expected Calibration Error (5) as a measure of temperature fit for our data:
ECE(T,X, θ̂) = B∑ b=1 |Xb| |X| ∣∣∣acc(Xb|θ̂, T )− conf(Xb|θ̂, T )∣∣∣
where dataset X is ordered by model confidence `(x, θ̂, T ) and then split into bins Xb. For each bin we use two statistics: acc(Xb|θ̂, T ) — mean accuracy for this bin of the model with weights θ̂, conf(Xb|θ̂, T ) — mean model confidence score e`(x,θ̂,T ). When the labels from out-of-domain dataset match those from the training set, this parameter can be calculated and optimized directly. During this optimization we have found, that the higher temperature T is used, the better ECE score becomes. This result matches conclusion from the ODIN paper, but looks counter-intuitive at the first glance. In the Fig. 1 we presented a dependence graph between conf(Xb|θ̂, T ) and acc(Xb|θ̂, T ) for ImageNet-A dataset. From this graph, it is clear, that with the growth of model confidence its accuracy becomes lower. Calibration assumes these variables are correlated which is not the case. The problem comes from dataset construction where we choose the examples, that are hard to detect (12). This way we bias the dataset towards the examples that give high model confidence but a wrong label. In this case calibration procedure become irrelevant and will choose the highest possible temperature T .
From our experiments, we conclude that it is practical to use higher temperatures when we use S part only. Unlike ODIN, GRODIN uses information from G part which is influenced by the temperature as well. With the grows of the temperature the contribution of gradient vectors of hc from all classes become even. Irrelevant classes give almost random gradients and this signal becomes too noisy. That is why we need to seek for optimal temperature. On the other hand, the signal from G is based not only on class weights but on fundamental properties of the image, that are detected on different layers of the model (4). That is why our model works better on the biased towards hard examples datasets, such as ImageNet-O.

heading: 4.2 GRADIENT COMPONENTS
text: Our method is based on assumption that classification model parameters gradient contains information on point resemblance to the domain the model is trained for. But it is unclear how this information is distributed among S and G parts of the gradient. In this section we want to confirm the assumption that G part is necessary for the model.
To understand if the G part compliments information from S (that is already used in ODIN) we take the in-domain and out-of-domain examples mixture from ImageNet-R that is hard to split by S and use information from G part for this task. In the left part of the Fig. 2 reader can find the histograms of two distributions grouped by levels of S: red for out-of-domain examples and blue for those in-domain. From these examples we chose those bins that are placed between two black lines. This area is the hardest for separation and with S signal alone we can achieve 0.708± 0.002 ROC-AUC. The right part of the Fig. 2 contains the histogram for this same subset of “hard” examples but from G perspective. As one can see, these points are still hard to separate, but the G part contains a bit different signal and allows to get 0.735± 0.002 ROC-AUC. This fact demonstrates, that information
from G part contains additional information to S and explains the difference of GRODIN results the from original ODIN algorithm.

heading: 4.3 METHOD COMPLEXITY
text: Our method is computationally quite lightweight. Input perturbation requires one forward pass through the network, and one backward pass to compute a gradient. After this, we need one more forward and backward pass to compute gradient l2-norm at the perturbed point. The total complexity is O(|F |+ |B|+ |Θ|), where O(|F |) is forward pass complexity, O(|B|) is backward pass complexity, and |Θ| is a number of parameters in neural network. In practice, the computation time is approximately two times slower than single training step (forward + backward pass), and approximately 4− 6 times slower, than a single inference step.

heading: 5 CONCLUSIONS
text: In this paper, we have found a way to utilize a back propagation procedure for out-of-distribution detection. We think this solution is both practical and theoretically found. From a practical point of view modern DL inference accelerators are suitable for back propagation step by their design and our approach could be easily implemented in practice. The other practical advantage of the proposed method is that it uses only prediction model properties and there is no need for training an additional model for out-of-distribution detection. The theoretical basis of the proposed method comes from recent theoretical results in neural tangent kernels and allows us to say that the method will be relevant for bigger models that emerge in recent papers.
We have found that interaction of the model and the argument item depends severely on the gradient with respect to model parameters at this item. This naive and straightforward assumption allows us to predict out-of-distribution items. In this paper, we have shown that a method based on this property is able to outperform the state-of-the-art methods. We used several modern collections (CIFAR-10/100, ImageNet-O, ImageNet-R, ImageNet-A, part of ImageNet-C, tiny-ImageNet) to explore the properties of our method and to compare it with state-of-the-art results. We found that in all cases our approach gives comparable results and in eight of nine experiment cases performs significantly better.
All modern out-of-distribution detectors use parameters tuning for better detection on different collections and we found that parameters could dramatically change the performance of the method. In a situation when the environment is unknown during the training process (e.g. adversarial attacks) this property makes such methods impractical. We found that proposed method has an extremely stable set of optimal parameters that could be used in a wide variety of situations.
In this work we used a very simple approach to gradient analysis and one of the possible directions of the future work is to apply more sophisticated instruments such as gradient normalization, newton step instead of gradient, gradient component distribution analysis on per layer/block/individual basis. We believe that back propagation has a wider field of applications than out-of-domain detection and going to expand this field in the future.

Review summaries:
This paper proposes an anomaly detection score based on the derivative of the log-likelihood: $ \mid\mid \nabla_{\theta} \log p_{\theta}(\hat{c} | \mathbf{x}) \mid \mid_{2}^{2} $ where $\hat{c}$ is the predicted class (as we want the method to be applicable to test points).  The intuition is that if $\mathbf{x}$ is from training set, then the model should already be fairly well fit to the point, meaning that the derivative should be near zero.  This metric along with some variants derived from sub-terms are compared on image data sets against baselines such as ODIN, maximum softmax probability, maximum logit, and ensembles.  

This paper proposes to use the norm of gradients as the score to detect OOD data. The experiments use CIFAR-10 and ImageNet as the distribution datasets and use the unseen classes or perturbed images for the OOD data. The result shows the norm of gradients improves the OOD detection.

This paper proposed a new method called GRODIIN to detect whether a new sample is similar with training set or from out of distribution. The basic assumption is that the expectation of gradient descent should be close to zero vector on the in-domain training set. Then this paper can compute the gradient on a new example, and compare its distance in the tangent kernel space.