GRODIN: Improved Large-Scale Out-of-Domain detection via Back-propagation
Authors: Gleb Yengalych
Igor E. Kuralenok
Vasily A Ershov
Venue: ICLR 2022 Submitted (2022)
Keywords: Out of distribution
deep learning
gradient
backpropagation

Abstract: Uncertainty estimation and out-of-doman (OOD) input detection are critical for improving the safety and robustness of machine learning. Unfortunately, most methods for detecting OOD examples have been evaluated on small tasks while typical methods are computationally expensive. In this paper we propose a new gradient-based method called GRODIN for OOD detection. The proposed method is conceptually simple, computationally cheaper than ensemble methods and can be directly applied to any existing and deployed model without re-training. We evaluate GRODIN on models trained on CIFAR-10 and ImageNet datasets, and show it's strong performance on various OOD ImageNet datasets such as ImageNet-O, ImageNet-A, ImageNet-R, ImageNet-C.

Under review as a conference paper at ICLR 2022
GRODIN: IMPROVED LARGE-SCALE DOMAIN DETECTION VIA BACK-PROPAGATION
OUT-OF-
Anonymous authors Paper under double-blind review
ABSTRACT
Uncertainty estimation and out-of-domain (OOD) input detection critical for improving the safety and robustness of machine learning. Unfortunately. most methods for detecting OOD examples have been evaluated small tasks while typical methods are computationally expensive. In this paper we propose a new gradient-basee method called GRODIN. The proposed method is conceptually simple, computationally cheaper than ensemble methods and can be directly applied to any existing and deployed model without re-training. We evaluate GRODIN on models trained on CIFAR-10 10 and ImageNet datasets, and show its strong performance on various OOD ImageNet datasets such as ImageNet-O, ImageNet- A, ImageNet-R, ImageNet-C.
INTRODUCTION
Many problems are solved with machine learning techniques today. Prominent examples of supervised learning power could be found in many domains: image classification, speech recognition, medical surveys, self-driving cars, text translation 66 19 important be confident on the prediction of trained model: self-driving cars, medicine, stock market trading, etc.. Usually, the prediction mechanism remains uncovered and it requires additional efforts to ensure the model quality when working with automatically understand that prediction on this particular sample is not reliable with data uncertainty estimation or knowledge uncertainty estimation (2).
28 24 D. etc.. In some applications, it is very
data. We search for a new way to
One of the most important tasks in this field is out-of-domain detection. It allows separate the data suitable for inference from the examples that are not intended for model usage. Due to the importance of the task out-of-domain methods evolved from "naive" approaches to sophisticated algorithms. These algorithms either analyse the prediction model results to understand if we could rely them, or build new dedicated model for OOD. The first type of the methods we call "lightweight". as they usually could be used for on-the-fly verification of prediction. The other class could deliver better detection quality, but requires substantive computational resources and separate training. We refer to this one as "heavy".
The "lightweight" models usually built on top of certain prediction tasks, such classification, and are based on the properties of loss functions used the training process. Softmax probability an example of such property, it is directly used for many practical implementations (1077). The prediction quality of the softmax probability thresholding could be extended with input preprocessing and probability calibration shown in ODIN algorithm. This method provides the state-of-the- art results on most available benchmarks Liny-ImageNNe (26). CIFAR-10 (6)), CIFAR- 100 (I6). MNIST 33, SVHN (25), etc.). However, there room for improvement. There is a bunch of more challenging problems constructed on top of ImageNet(26) image classification task: ImageNet-A
112, ImageNet-O (2), ImageNet-R (8, ImageNet-C. (9).
The alternative "heavy" methods were developed: outlier exposure (1I); mahalanobis distance-based confidence score (8); class of strong and theoretically justified out-of-distribution detectors called ensembles (2) 23 22). All these methods usually provide better results, but are hard to use in practice: some them require changing the training loss function and therefore neural network retraining (II); require solving auxiliary logistic regression problem and heavy computation of mahalanobis distance in (18): sometimes researcher need additional training and inference several neural networks simultaneously for ensemble methods (2

---

Under review as a conference paper at ICLR 2022
Recent studies in deep learning suggest that the gradient of the loss function taken for particular example carries a lot of information other than model prediction. Influence functions (14) use information about first and second derivatives with respect network weights to estimate what is the impact of a single precedent on test set quality. Neural tangent kernel (13) gives us another point of view on the training process and generalization from the position of gradient analysis. To the best of our knowledge, there is only one work in out-of-distribution detection domain that utilize the gradient of the model with respect to weights (1). This approach uses information from the gradient of different layers as a building blocks for their separate DNN model. The approach requires a lot of out-of of-distribution data train the model and could not be assigned to "lightweigh" models.
In this paper, we focus on developing a new simple and computationally lightweight approach for detecting out-of-distribution examples based on gradient analysis. We investigate the derivative of cross-entropy loss function and decompose it into two parts, showing the difference between them. The proposed method is being tested on several challenging benchmarks and compared with various methods. Also, we analyse the impact of highly popular input preprocessing and give intuition why this technique helps OOD detectors.
METHOD
r
is statistical population. training set, consisting of m i.i.d. precedents from population i-th precedent features vector and corresponding label. f is particular element of model parameters space e. loss function.
X = (xx Hm
x; R Vi E V
0 E θ
6(0, x, x,
L = Σ (x,)ex
((0, x, v)
emperical loss function used in training process. output of neural network class before softmax layer. (2 norm.
1x 0)
11-112
Table 1: Notation
Model mistakes come from different sources: irreducible errors, low model capacity, wrong modeling assumptions, sample distribution shift, and For each type of errors, define formal problem setup and develop methods to deal with it. Those setups are not mutually exclusive and their methods could complement each other.
This paper focuses on a particular problem: estimation of the probability that a particular sample comes from different population. To formally define this problem we use notation from Tab. T Given this notation, there exist a statistical population r. Elements of this population are pairs (x, 3). where E R" is a sample, y E is label. We assume that there exists deterministic correspondence between population precedent and corresponding label. Statistically representative training set X used train prediction model by minimization of empirical loss function L. With enough data and mild assumptions on loss function, the resulting model with parameters i should replicate correspondence between x and y. As a result, our model should give a correct prediction to any x from the general population. Thus, given assumptions, there is a natural way to define x; as the oft-of-domain sample:
x; oft-oi- <> v(x, y) E T.: :T:x
Our task is to estimate, if the particular sample x; is out-of-domain, given only the estimated model parameters
The core idea of our method is quite simple. We train prediction model by minimisation of some additive loss function L = + Σxx,))xx É(O, x, y). When minimisation is performed by stochastic gradient descent or similar methods, the expectation of the model parameters gradient norm reduces on training set elements with the number of optimization steps. The expectation aims to zero for flexible enough decision functions. In tangent kernel space (NTK for neural models) the optimization process looks like seeking for optimization parameter that pulls representation of training set points (e(0, x, y) to zero vector. We can use this property and determine the model behavior for example by the distance of its representation in the trained tangent kernel space (defined by 0) from zero
2

---

Under review as a conference paper at ICLR 2022
Using this information accuracy, model uncertainty, out-of-domain points detection, etc.. In this paper, we focus on the latter case and build method for out-of-domain detection top of this principle.
predict many practical properties of the particular prediction: expected
The modern definition of the domain depends on the application and we will show this difference in the experimental section. From theoretical perspective, we equate domain and the population. This way we allow researcher to define their notion of the domain with the selection of sampling technique. The ideally trained model brings all points of the population to stationary point in model parameters space. In this paper we make an assumption that kernel of this transformation equal the population and the observed transformation differs from ideal one with Gaussian error: Vol(O, x. v Vol(O, x, y)le- + 6, 8 ~ (00, E). Now we can define a log probability of population P membership:
( 0) = LxgP( Voe(O, x. 4>1
The main problem for practical application of the method is define a loss function for examples, that come to the model during its exploitation and have no label. In this case, it is natural to use the class assigned by the model é label and check the uncertainty of the model in the decision. We focus, without loss of generality, on image classification problem over & classes with neural models. Negative log likelihood over softmax probability modeling is one of the most commonly used loss functions for this problem:
(xx,0)
E(O, x, v) = - log
(x,6))
where hr are outputs of the neural model for each class Using this model and the label assumption we can formulate the result log probability population membership g(x, 0) for arbitrary point x:
Me(x,0)
g(x, 0) =
V log
Σ (x.
Now we can choose significance level a and filter out points that belong to the population with probability less than this significance level: P(x[T) a. In practice we don't 't't use significance level directly, but define a threshold 8(a:) for g(x, 0). If we have information on out- of-main examples we choose the threshold that delivers the selected significance level on cross-validatio We use this setup as a baseline variant of our method in the experimental section.
In the literature there are at least two ways how to enhance the result of the baseline model: tem- perature scaling (5), and input perturbation (20). In our framework these enhancements are easily explained and compliment the baseline model. To show this we need to unfold components of the gradient vector:
Ve log max (or.00))
v)l(x,0)
Σ etecx,0 = =S(x, 0) 8), where
5(x,0) = 1+2ise -1
G(x,0) -eza (loc(x (ho -..(x))) (x.0
38
In the last expression there probability of the predicted class. The G is cumulative gradient vector for all classes output nodes of the network weighted by their preference predicted class (x.0)) ha(.0)) The first part is equal to the softmax score from ODIN up to the notations used. The second part is the information from NTK space.
multiplicative parts (S and G). S is scalar part, that reflects the
As one can see both S and G heavily depend on class weights h.s and it is important to calibrate these values for the task the model is used for. One of the most popular ways of model calibration is
3

---

Under review as a conference paper at ICLR 2022
temperature parameter injection ( In this method scale class weights he with the temperature factor f. The optimal temperature then optimized using some calibration quality metric like Expected Calibration Error (5) for the target dataset. The solution of this optimization problem depends on the data provided, see the experimental section for details.
The other way of out-of-domain detection model improvement is altering the source signal, e.g. the image in case of the image classification task. In our framework we assume that during training process the optimization pulls domain space stationary point in model parameters space. As only training examples are observed, we pull the entire domain using the train set as mount points. In this process the rest of the domain slacks and this cause well-known difference in gradients between training set points and validation points. To reduce this slack we can move point towards training points. We follow the alteration procedure introduced the ODIN paper:
Vx log max, (xx00/
Xp X. - ssiign
Et che
During this procedure, source image altered to reduce the gradient and compensate the slack between training points and the rest of the domain.
To sum it up, the proposed method a composition of E-input perturbation, temperature calibration by scaling outputs by I/T factor and decomposition Vol(x, 0) into two multiplicative parts: S(x, 0) and G(x, 0). These parts can be used as single estimator: g(x, 6) = | O)G 0)l12. or alone: g(x, 0) ||G 0) g(x, 0) 5(x,0). The last variant gives exactly ODIN.
EXPERIMENTS
We performed set of experiments to study the properties of our method, its effectiveness and its competitiveness with other approaches. Both versions of the proposed gradient method were evaluated: G' and S * G' to show that there significant difference in their performance.
used popular image collections as source of out-of-domain examples. Benchmark is standard: out-of-distribution detector is evaluated as binary classifier that should separate in-domain samples from
The test dataset is a mixture of the in-domain test set and the out-of-dommin dataset. The Imageet dataset used out-of-domain for all experiments with networks, trained on CIFAR-10. 10. For neural networks, trained on ImageNet- IK, we used four recently proposed out-of-distribution collections, that represent different distribution shifts: ImageNet-O. O, ImageNet-R, ImageNet-A, and part of ImageNet-C. Image databases statistics are shown in Tab.
ImageNet-O (12) is subset of ImageNet-224 -22k that do not intersect with ImageNet-1K 1. It consists of images that fool Maximum Softmax Probability out-of-distribution detector (IO) based on ResNet-50 neural network and represents distribution shift that brings new observed classes. Popular they utilize logits or softmax probabilities that are heavily biased by the design of the dataset.
detection methods work poorly on this setup
ImageNet-R ( contains images of ImageNet objects with different textures and styles: paintings, sculptures, origami, cartoons, toys, etc. Classes are subset of original ImageNet classes. This dataset represents distribution shift that does not change the semantic of objects but change their shape or texture.
contains images of ImageNet objects that extremely hard to be classi- fied correctly by any neural network trained on ImageNet. It represents natural adversarial examples: it is obvious for humans what is depicted on image, but trained model struggles to give the correct answer.
ImageNet-A (1)
ImageNet-C ( contains copies of the ImageNet validation set that were corrupted with various noises. Original dataset consists fifteen noise types, each noise type has five levels of severity. We selected the most challenging for the ResNet-50 model noise type according to the original paper: level five FFrosted Glass Blur'. ImageNet-C dataset represent a distribution shift that occurs when for example camera transmission channel is broken.
4

---

Under review as a conference paper at ICLR 2022
Table 2: List of used datasets
Dataset
Train size
Validation size
Test size
Outlier for
Reference
CIFAR-IO
50000
500
9500
(15)
ImageNet- IK
22 1.28 10"
3000
47000
(27
tiny ImageNet
500
9500
CIFAR-10 10
27
ImageNet-O
600
1400
ImageNet IK
(2)
ImageNet-R
600
29400
ImageNet IK
8
ImageNet-A
600
6900
ImageNet-11
12
ImageNet-C
3000
47000
ImageNet-IK
In our experimental study out-of-distribution detectors are applied to popular neural network archi- tectures trained CIFAR-10 or ImageNet-IIe IK collections, such ResNet- 18 and ResNet-50. The detailed information about the models used for each setup could be found in Tab. 3 All experiments were performed with the PyTorch framework, due blind-review attached source code to supplementary materials and will provide the link to the repository with the camera-ready paper version.
Table 3: List of used neural networks
Architecture Wide-ResNet-28-10
Training set
Number parameters
Accuracy (test set)
Reference
CIFAR-IO 10
36.5M
96.2
(2
ResNet- 18
ImageNet-Il 1K
23 11.5M
69.7
(6)
ResNet-50
ImageNet- 1K
2V 25.5M
76.1
The following out-of-distribution detectors were used as competitors to our method:
Maximum Softmax Probability (MSP), also referred as a baseline, described in (10). This method uses negative maximum softmax probability as an out-of-distribution score.
2. ODIN described in (20). This method is modification of MSP. It combines input prepro- cessing and MSP together using negative maximum softmax probability of preprocessed image as dutttf-distriiution score.
3. Maximum Logit (ML. described in (7). This method negative maximum logit (neural network output without applying softmax activation function) as an out-of-distribution score.
4. Ensemble method 23). trained with different random seeds. The ensemble consists of different uncertainty measures that can be used scoring function (confidence, entropy of expected, expected entropy, mutual information, EPKI., MKL) and we choose the one that brings the best separation terms of ROC-AUC as the final score. Here we need to train several models from scratch, thus the access to training set is mandatory. As result, the method can't be used on an arbitrary pretrained model so we evaluate it only for ResNet-50 model,
train it from scratch.
ROC-AUC is our quality measure for disttr-iiistiittionn detection. All table cells express experimen- tal results in μ + a pairs, where y is the average value of the metric and a is its bootstrap deviation computed on 100 resamplings.
The rest of the experimental section consists two parts: the competitive comparison in Sec, 3.1 and ablation study for our method in Sec.
3.1 COMPARISON WITH OTHER METHODS
The following methodology is used compare out-of-domain detectors for each collection from Tab.
The classification model is trained on a base dataset (CIFAR-IP 10 or ImageNet-
The OOD model parameters are tuned on the validation subset of the collection to maximize the ROC-AUC score
5

---

Under review as a conference paper at ICLR 2022
The resulting method is then applied to test part of the collection and reported in the result table
Parameters tuning required for the proposed gradient method and for ODIN. We used gearch to estimate optimal and T. The grid for E was 21 evenly spaced points from 0 to 0.004. We select among (1, 5, 10, 20, 50, 100, 200, 500, 1000) to find optimal temperature T. The same grid for and 7 was used in ODIN paper (20).
Our experiments can be divided into two groups:
1. Ones that pretrained weights from Py PyTorch collection. A researcher can skip the training procedure and just just evaluate methods on downloaded data. These experiments were performed with the ResNet-18 model trained on the ImageNet classification task.
2. Experiments with models trained from scratch. In this work we train WideResNet-28-10 for CIFAR-10 classification and ResNet-50 for ImageNet classification. All parameters for the training procedure were taken from original papers for these models. For ResNet- 50 experiments, we trained four (as proposed gradient methods require two forward and two backward passes) different models from different random seeds evaluate ensemble methods.
Results of out-of-distribution detections are presented in tables 4 methods: G and SG outperform all baselines on CIFAR-10 10 vs tiny ImageNet benchmark with significant margin. Confidence intervals for G and SG parts intersect, so we can say that both methods show nearly the same performance.
One can see that both gradient
Results for ImageNet-like datasets are more various. Let's inspect results for every dataset separately (both architectures ResNet- 18 and ResNet-50 behave nearly the same):
ImageNet-O: It is the only dataset which SG-part is better than G-part. Still, both gradient methods outperform all baselines with giant margin.
ImageNet-R, ImageNet-C:: G-part outperforms all baselines on these two datasets. SG-part showed bad performance giving up to ODIN and ML.
ImageNet-A: On this dataset G-part outperforms ODIN and ML but failed to overtake ensemble baseline.
To sum it up, G-part can significantly outperform all "lightweight" baselines (ones that do not require training several models from scratch) on both benchmarks. And only in the case of IeageNet-A with ResNet-50 as a predictor the ensemble model was able to outperform the proposed method.
Table 4: Wide-Resnet-28-11 CIFAR-I0 vs tiny- Imageen. ROC-AUC values.
MSP
ODIN
ML
G-part
SG-part 0.981 0.001
0.954 + 0.001
0.97 + 0.001
0.965 = 0.001
0.983 + 0.001
Table 5: ImageNet- like datasets, ROC-AUC values.
Donases
Ensemole
G-part
3G
ImageNet-O
0.451 1.511000 0.000
0.228 :: 0.007
0.399 1 0.006
0.76 1 0.006
0.84 1 0.006
ImageNetRR
0.776 0.000
0.8 1: 0.001
0.838 : 0.001
0.852 :: 0..01
0.833 1 0.002
ResNet-18
ImageNai-A
0.18 0.0033
0.002
0.002
0.833 0.002
0.377 0.002
ImageNat-C
0943 1.430000.001
0,974
0.0001
0.974
0.0001
0,985 0.0001
0.95 1 0.001
ImageNar-D
0.47 4 0.006
0.599 = 0.007
0.564 = 0.008
0.612 0.006
0.735 = 0.006
0.769 4 0.006
ResNei-50
ImagaNer-R
IaageNer-A
0,839 +:
+: 0.002
0.001
0.002
0.001
0,85
0,8%
0.001
0,877
0.001
0,899 0.002
0.001
ImagoNer-C
0537 +: 0.001
0.969
0.0001
0.969
0.0001
0.976 + 0.0001
0.982 + 0.0001
0,988 + 0.001
3.2 ABLATION STUDY
The proposed method consists of three parts: input perturbation, temperature scaling, and scoring based on a gradient with respect to neural network weights. Our initial intuition suggested that gradient-based scoring scheme is self-sufficient and could be used as stand-alone discrepancy

---

Under review as a conference paper at ICLR 2022
measure. Turns out this is not the case, and both preprocessing steps also essential to achieve competitive results.
We evaluated this impact ImageNet-like datasets. This stand-alone discrepancy measure is specific case our final methods, thus it is enough to set free parameters for particular values: = 1 and/or 0. Thus compare the final model with simplified variants: T-scaling only model, where E was fixed to zero and temperature was tuned with grid-search; s-perturbation only, with temperature 7 fixed to and tuned the grid; the pure gradient-scored method with equal 0 and T equal I. The presented results are obtained on the test set, while hyperparameters were chosen depending on the best performance on gridsearch set.
The results are presented in Tab. tuning are nearly the same and both are weaker than only T tuning or full method. Only T tuning in some cases match the full scheme, but to achieve outstanding results it is necessary to use both preprocessing steps.
As one can see for both G and SG parts vanilla and only
Table 6: Ablation study for G-part. ROC-AUC values.
Dataset
G-vanilla
G-(only E)
G-(only T)
G
ImageNet-O
0.517 = 0.007
0,517 = 0.007
0.678 E 0.008
0.76 0.006
ImageNet-R
0.786 = 0.001
0.786 = 0.001
0.839 + 0.001
0.852 +
ResNet-18
ImageNet-A
0.826 = 0.003
0.826 = 0.003
0.852 + 0.002
0.853 0.002
ImageNet-C
0.951
0.951 =
0.985 +
0.985 =
ImageNet-O
0.502 = 0.007
0.502 = 0.007
0.657 + 0,008
0.735 + 0.006
ResNet-50
ImageNet-R
0.813 = 0.002
0.811 = 0.002
0.863 + 0.001
0.877
0.001
ImageNet-A
0.849 = 0.002
0.849 + 0.002
0.877
+ 0.002
0.877 + 0.002
ImageNet-C
0.948 0.001
0.948 + 0.001
0.982
0.0001
0.982
0.0001
Table 7: Ablation study for SG-part. ROC-AUC values.
Dataset
SG-vanilla
SG-(only
SG-(only T)
SG
ImageNet-O
0.518 + 0.007
0.538 + 0.007
0.691 + 0.006
0.804 + 0.006
ImageNet-R
0.786 + 0.001
0.79 + 0.002
0.787 + 0.002
0.833 +
ResNet-18
ImageNet-A
0,826 + 0.002
0.82 + 0.002
0.82 + 0.002
0.837 - 0.002
ImageNet-C
0.931 + 0.001
0.937 + 0.001
0.937 0.001
0.95 + 0.001
ImageNet-O
0.503 + 0.007
0.503 + 0.007
0.659 + 0.007
0.769 = 0.006
ImageNet-R
0.813 + 0.002
0.811 + 0.002
0.811 0.001
0.856 + 0.001
ResNet-50
ImageNet-A
0.849 = 0.002
0.849 + 0.002
0.846 + 0.002
0.859 =
ImageNet-C
0.948 + 0.001
0.948 + 0.001
0.939 0.001
0.958 = 0.001
DISCUSSION
The task for out-of-domain detection is usually related to sensitive information and it is extremely hard to find real-world data on this topic. To resolve this issue there are joined academic and industry initiatives such as NeurIPS Challenge (2I). The data from this event is not yet publicly available and we have to use state of the art datasets, that are artificially designed. In this section, we want to analyse this bias and its influence on the methods from the experimental section.
4.1 MODEL CALIBRATION
In theoretical Sec. 2 we noted, that calibration procedure is mandatory because the datasets used for out-of-domain detection could differ from the training set in many ways. We use Expected Calibration Error (5) as a measure of temperature fit for our data:
B
X1 X
ECE X.
Σ
acc aco( (o,( co,
7

---

Under review as a conference paper at ICLR 2022
0.12
0.28
1,60,1000 T) and acc( X610, 00, T) for ImageNet-A dataset. Red Figure I: Dependence graph between (X,l0, the trend line.
where dataset X is ordered by model confidence ((x, 0, T) and then split into bins X For each bin we use two statistics: aco(Xx]8, T) mean accuracy for this bin of the model with weights A, cx,,((Xx)0, 2) mean model confidence score c((x,0,T))
When the labels from out-of-domain dataset match those from the training set, this parameter can be calculated and optimized directly. During this optimization we have found, that the higher temperature T' is used, the better ECE score becomes. This result matches conclusion from the ODIN paper, but looks counter-intuitive at the first glance. In the Fig. I we presented dependence graph between co,f(X,,0, 00 T) and (ec(XX,)e T) for ImageNet-A dataset. From this graph, it clear, that with the growth of model confidence its accuracy becomes lower. Calibration assumes these variables are correlated which not the case. The problem comes from dataset construction where we choose the examples, that are hard to detect (12). This way we bias the dataset towards the examples that give high model confidence but a wrong label. In this case calibration procedure become irrelevant and will choose the highest possible temperature T.
From our experiments, we conclude that it is practical to use higher temperatures when we use S part only. Unlike ODIN, GRODIN uses information from G part which is influenced by the temperature as well. With the grows of the temperature the contribution of gradient vectors of ht from all classes become even. Irrelevant classes give almost random gradients and this signal becomes too noisy. That is why we need to seek for optimal temperature. On the other hand, the signal from G is based only on class weights but on fundamental properties of the image, that are detected on different layers of the model (4). That is why our model works better on the biased towards hard examples datasets, such as ImageNet-O.
4.2 GRADIENT COMPONENTS
Our method is based on assumption that classification model parameters gradient contains information on point resemblance the domain the model is trained for. But it is unclear how this information is distributed among S and G parts of the gradient. In this section we want to confirm the assumption that G part is necessary for the model.
To understand if the G part compliments information from S (that is already used in ODIN) we take the in-domain and out-of-domain examples mixture from ImageNet-R that hard to split by S and information from G part for this task. In the left part of the Fig. 2 reader can find the histograms of two distributions grouped by levels of S: red for out-of-domain examples and blue for those in-domain. From these examples chose those bins that are placed between two black lines. This area is the hardest for separation and with S signal alone we can achieve 0. + 0.002 ROC-AUC.
The right part of the Fig. 2 contains the histogram for this same subset of "hard" examples but from G perspective. As one can see, these points are still hard to separate, but the G part contains a bit different signal and allows to get 0.735 + 0.002 ROC-AUC. This fact demonstrates, that information
8

---

Under review as a conference paper at ICLR 2022
Figure 2: Histograms ImageNet-R scoring functions. S for full dataset to the left and G for only "hard" examples to the right
from G part contains additional information S and explains the difference of GRODIN results the from original ODIN algorithm.
4.3 METHOD COMPLEXITY
Our method is computationally quite lightweight. Input perturbation requires one forward pass through the network, and one backward pass to compute a gradient. After this, we need one more forward and backward pass to compute gradient (2-norm at the perturbed point. The total complexity is O(LF) + B + [0]), where O(FFl) is forward pass complexity, O( is backward pass complexity, and | is number of parameters in neural network. In practice, the computation time is approximately two times slower than single training step (forward + backward pass), and approximately 4 times slower, than single inference step.
5 CONCLUSIONS
In this paper, we have found way to utilize back propagation procedure for out-of-distributin detection. We think this solution both practical and theoretically found. From a practical point of view modern DL inference accelerators are suitable for back propagation step by their design and our approach could proposed method is that it uses only prediction model properties and there is no need for training an additional model for out-of-distribution detection. The thcoretical basis of the proposed method comes from recent theoretical results in neural tangent kernels and allows to say that the method will be relevant for bigger models that emerge in recent papers.
easily implemented in practice. The other practical advantage of the
We have found that interaction of the model and the argument item depends severely the gradient with respect to model parameters at this item. This naive and straightforward assumption allows predict diitr--disttibutionn items. In this paper, have shown that method based on this property is able to outperform the state-of-the-art methods. We used several modern collections 101AR- ImageNet-O,, ImageNet-R, ImageNet-A, part of ImageNet-C, tiny-ImageNee) to explore the properties of our method and to compare it with state-of-the-ar results. We found that in all cases our approach gives comparable results and in eight of nine experiment cases performs significantly better.
All modern collections and we found that parameters could dramatically change the performance of the method. In situation when the environment is unknown during the training process (e.g. adversarial attacks) this property makes such methods impractical. We found that proposed method has an extremely stable set of optimal parameters that could be used in a wide variety of situations.
detectors use parameters tuning for better detection on different
In this work we used very simple approach to gradient analysis and one of the possible directions of the future work is to apply more sophisticated instruments such as gradient normalization, newton step instead of gradient, gradient component distribution analysis per layer/block/individd basis. We believe that back propagation has a wider field of applications than out-of-domann detection and going to expand this field in the future.

---

Under review as a conference paper at ICLR 2022
REFERENCES
[1] Agnihotri Akhil, Saraf Prathamesh, Bapnad Kriti Rajesh. A Convolutional Neural Network Approach Towards Self-Driving Cars. 2019.
[2] Andrey Malinin. Uncertainty Estimation in Deep Learning with application to Spoken Language Assessment. 2019.
[3] Deng Li. The mnist database of handwritten digit images for machine learning research // IEEE Signal Processing Magazine. 2012. 29, 141-142.
[4] Gu Jiuxiang, Wang Zhenhua, Kuen Jason, Ma Lianyang, Shahroudy Amir, Shuai Bing, Liu Ting, Wang Xingxing, Wang Li, Wang Gang, Cai Jianfei, Chen Tsuhan. Recent Advances in Convolutional Neural Networks. 2017.
[5] Guo Chuan, Pleiss Geoff. Sun Yu, Weinberger Kilian Q. On Calibration of Modern Neural Networks. 2017.
[6] He Kaiming. Zhang Xiangyu, Ren Shaoqing, Sun Jian. Deep Residual Learning for Image Recognition. 2015.
[7] Hendrycks Dan, Basart Steven, Mazeika Mantas, Mostajabi Mohanmadreza, Steinhardt Jacob, Song Dawn. Scaling Out-of-Distribution Detection for Real-World Settings. 2020.
[8] Hendrycks Dan, Basart Steven, Mu Norman, Kadavath Saurav, Wang Frank, Dorundo Evan, De- sai Rahul, Zhu Tyler, Parajuli Samyak, Guo Mike, Song Dawn, Steinhardt Jacob, Gilmer Justin. The Many Faces of Robustness: Critical Analysis of ouist--Distriuution Generalization. 2020.
[9] Hendrycks Dan, Dietterich Thomas. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. 2019.
[10] Hendrycks Dan, Gimpel Kevin. Baseline for Detecting Misclassified and Examples in Neural Networks. 2018.
[11] Hendrycks Dan, Mazeika Mantas, Dietterich Thomas. Deep Anomaly Detection with Outlier Exposure. 2019.
[12] Hendrycks Dan, Zhao Kevin, Basart Steven, Steinhardt Jacob, Song Dawn. Natural Adversarial Examples. 2021.
[13] Jacot Arthur, Gabriel Franck, Hongler Clément. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. 2020.
[14] Koh Pang Wei, Liang Percy. Understanding Black-bor Predictions via Influence Functions. 2020.
[15] Krizhevsky A., Hinton G. Learning multiple layers of features from tiny images // Master's thesis, Department of Computer Science, University of Toronto. 2009.
[16] Krizhevsky Alex. Learning multiple layers of features from tiny images. 2009.
[17] Lee Jinsol, AlRegib Ghassan. Gradients as a Measure Uncertainty in Neural Networks. 2020.
[18] Lee Kimin, Lee Kibok, Lee Honglak, Shin Jinwoo. A Simple Unified Framework for Detecting Out-of- Diitri--iittittt Samples and Adversarial Attacks. 2018.
[19] Li Jason, Lavrukhin Vitaly, Ginsburg Boris, Leary Ryan, Kuchaiev Oleksii, Cohen Jonathan M.. Nguyen Huyen, Gadde Ravi Teja. Jasper: An End-to-End Convolutional Neural Acoustic Model. 2019.
[20] Liang Shiyu, Li Yixuan, Srikant R. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. 2020.
10

---

Under review as a conference paper at ICLR 2022
[21] Malinin Andrey, Band Neil, Ganshin Alexander, Chesnokov German, Gal Yarin, Gales Mark J. F., Noskov Alexey, Ploskonosov Andrey, Prokhorenkova Liudmila. Provilkov Ivan, Raina Vatsal, Raina Vyas, Roginskiy, Denis, Shmatova Mariya, Tigas Panos, Yangel Boris. Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks. 2021.
[22] Malinin Andrey, Gales Mark. Uncertainty Estimation in Autoregressive Structured Prediction. 2021.
[23] Malinin Andrey, Mlodozeniec Bruno, Gales Mark. Ensemble Distribution Distillation. 2019.
[24] Narayanan Barath Narayanan, Silva Manawaduge Supun De, Hardie Russell C., Kueterman Nathan K., Ali Redha. Understanding Deep Neural Network Predictions for Medical Imaging Applications. 2019.
[25] Netzer Yuval. Wang Tao, Coates Adam, Bissacco Alessandro, Wi Bo, Ng Andrew Y. Reading Digits in Natural Images with Unsupervised Feature Learning. 2011.
[26] Russakovsky Olga, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhiheng, Karpathy Andrej, Khosla Aditya, Bernstein Michael, Berg Alexander C., Fei-Fei Li. ImageNet Large Scale Visual Recognition Challenge. 2015.
[27] Russakovsky Olga, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhiheng, Karpathy Andrej. Khosla Aditya, Bernstein Michael, Berg Alexander C., Fei-Fei Li. ImageNet Large Scale Visual Recognition Challenge // International Journal of Computer Vision (IICV). 2015. 115, 211-225..
[28] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Lukasz, Polosukhin Illia. Attention Is All You Need. 2017.
[29] Zagoruyko Sergey, Komodakis Nikos. Wide Residual Networks. 2017.


Reviews:
[{"review_id": "eLoOG56QtbQ", "reviewer": "Reviewer_CsE6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using gradients for OOD detection is interesting but not new. This work does not cite any related works in this direction. Several papers show up quickly with a simple search; none of these previous works are discussed in this paper. ([3] is relatively new, so it is ok to not include it in this submission)\n\n[1] https://ieeexplore.ieee.org/abstract/document/9190679\n[2] https://arxiv.org/pdf/2007.09507.pdf\n[3] https://arxiv.org/abs/2110.00218\n\nThis work has several issues in comparison with previous works and the justifications of the proposed method:\n1. There is no explanation for why the norm of gradients can be used to estimate the logP(x|gamma) and how good the estimation is. The first equation on page 3 has no proper justification.\n2. How is the proposed method compared to previous methods that also use gradients (see [1][2][3])? Besides, the text misses a proper citation in many places. For example, at the top of page 3, there should be citations when saying the gradients can predict expected accuracy, model uncertainty, and out-of-domain detection.\n3. The paper regards ODIN as the SOTA and uses it as the primary baseline for the comparison. But it is not true, and there are several light methods potentially better than it. For example, Mahalanobis [4], Gram matrix [5], GODIN [6] are also light by the definition of the paper. A reasonable experiment section should at least compare to one of the above methods.\n4. Would you please discuss or analyze why high-temperature T helps to generate the gradients that comprise a better OOD score?\n5. Section 2 reads not organized. Please consider dividing it into sub-sections and adding indices to the equations.\n6. The customed citation format is confusing. Please consider using the ICLR official format.\n\n[4] https://arxiv.org/abs/1912.12510\n[5] https://arxiv.org/abs/1807.03888\n[6] https://arxiv.org/abs/2002.11297\n", "summary_of_the_paper": "This paper proposes to use the norm of gradients as the score to detect OOD data. The experiments use CIFAR-10 and ImageNet as the distribution datasets and use the unseen classes or perturbed images for the OOD data. The result shows the norm of gradients improves the OOD detection.", "summary_of_the_review": "This submission lacks a proper justification for the designed method and lacks an appropriate comparison to related works. These issues signal a rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "TPekJtCItAi", "reviewer": "Reviewer_zSjc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper discussed an important problem and proposed a simple yet interesting solution. Please see detailed comments below.\n\nMinor issues:\n- better to add numbers to equations \n- better to use \"[]\" to cite references, and keep \"()\" for equation indexes. \n- page 2, last paragraph, \"with the number of optimization steps\": not clear. Maybe you want to say \"when the optimization converges\"?\n\n\nMajor issues:\n - Page 2, \"This expectation aims to zero for flexible enough decision functions\": this is a very strong assumption. Could you define what are \"flexible enough decision functions\"?\n- Page 2, last paragraph, \"zero vector\": do you mean the norm of the vector is close to zero? or every dimension of the vector is zero? Did you assume any batch normalization or layer normalization?\n- Page 3, 2nd paragraph, \"the observed transformation differs from ideal one with Gaussian error\": I am not sure whether \"the Gaussian distribution\" is the case. Could you give some empirical analysis? Or add some references? Especially I don't know if it holds in high dimensional space.  \n- Page 3, the 2nd equation: what if the new sample comes from a \"non-of-above\" category?\n- Page 4, the 1st equation: Please add references to the ODIN paper. Also it will help the audience to understand if you could summarize the differences between your method and ODIN. \n- Experiments are interesting, but I am curious whether the proposed method can work for binary classification, like \"human vs non-human\", where the out of domain example can be a dog image which category never appears in the training sets. ", "summary_of_the_paper": "This paper proposed a new method called GRODIIN to detect whether a new sample is similar with training set or from out of distribution. The basic assumption is that the expectation of gradient descent should be close to zero vector on the in-domain training set. Then this paper can compute the gradient on a new example, and compare its distance in the tangent kernel space. ", "summary_of_the_review": "This is an interesting topic and I hope to see more discussions of the proposed method. Currently I am not 100% convinced by the assumption and I wonder whether there are even better solutions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DgDlOgmxu7m", "reviewer": "Reviewer_cmXa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty and technical rigor of this paper are rather low.\n\nRegarding novelty, there have been several score-function-based OOD detection procedures already proposed, and this work does not refer to any of them.  For instance, [Grathwohl et al. [ICLR 2020]](https://arxiv.org/abs/1912.03263) propose $- \\mid\\mid \\nabla_{\\mathbf{x}} \\log p(\\mathbf{x}; \\boldsymbol{\\theta}) \\mid\\mid_{2}^{2}$, with the primary difference being the derivative formulation.  More related still is the work of [Sharma et al. [UAI 2021]](https://arxiv.org/abs/2102.12567).  They use the Fisher information for OOD detection: $$E_{y} \\left[ (\\nabla_{\\theta} \\log p_{\\theta}(\\mathbf{y} | \\mathbf{x})) (\\nabla_{\\theta} \\log p_{\\theta}(\\mathbf{y} | \\mathbf{x}))^{T} \\right].$$\n\nThis paper needs to justify why the proposed variant is superior to these alternatives, both conceptually and empirically.", "summary_of_the_paper": "This paper proposes an anomaly detection score based on the derivative of the log-likelihood: $ \\mid\\mid \\nabla_{\\theta} \\log p_{\\theta}(\\hat{c} | \\mathbf{x}) \\mid \\mid_{2}^{2} $ where $\\hat{c}$ is the predicted class (as we want the method to be applicable to test points).  The intuition is that if $\\mathbf{x}$ is from training set, then the model should already be fairly well fit to the point, meaning that the derivative should be near zero.  This metric along with some variants derived from sub-terms are compared on image data sets against baselines such as ODIN, maximum softmax probability, maximum logit, and ensembles.  ", "summary_of_the_review": "The paper does not properly discuss, compare to, and improve upon related work that also uses OOD scores of the form $\\frac{\\partial}{\\partial \\ \\cdot} \\log p_{\\theta}(\\mathbf{y} | \\mathbf{x})$.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CeIbZH1e_Zp", "reviewer": "Reviewer_gLsy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A positive aspect of this paper is that it evaluates OOD detection performance on the difficult task of separating ImageNet-1k from several other subsets of ImageNet. \n\nHowever, overall the evaluation is severely lacking. The authors compare to only four fairly weak baselines (and one of those only partially). As far as I understand, the authors try to justify this by creating a vaguely defined dichotomy between \"light-weight\" and \"heavy\" OOD detection methods. These methods are dismissed on the basis of being \"hard to use in practice\" for different reasons. I would argue that the proposed GRODIN method is far less practical than, say, outlier exposure [11] or Mahalanobis distance [18] (without preprocessing steps) because they require two full forward and two full backward passes through the network, making inference far more costly. Pre-computing a Mahalanobis distance or training an outlier aware network from scratch would be far cheaper in many cases when comparing to slowing down inference by a factor of 5x. There is no good argument to not even compare to these methods. I also disagree with the authors claim in Section 4.3 that their \"method is computationally quite lightweight\".\n\nFurthermore, the authors did not properly their CIFAR-10 performance even against the baselines that they did include. They only compared the ROC-AUCs against tiny-ImageNet. They should also be testing against CIFAR-100, SVHN, noise distributions etc... In addition to this, using tiny-ImageNet as a test out-distribution may be flawed anyway because there is a clear overlap in classes (frogs, dogs etc. are part of both CIFAR-10 and tiny-ImageNet). It is quite possible that their OOD methods are not picking up semantic differences but rather detect downscaling artifacts on tiny-ImageNet. This needs to at the very least be discussed in the paper.\n\nAnother issue is that the authors' method crucially relies on hyperparameter selection on the test out-distribution. This means that their methods will not perform well simultaneously on different out-distributions. Comparing the vanilla results in Table 6 and Table 7 to the MSP results in Table 5, we see that without these steps, their method barely even outperforms the trivial baseline. While similar criticisms can be made for several older papers on OOD detection (including ODIN [20]), this issue has been circumvented by many more recent works, that do not select separate hyperparameters for each test out-distribution (e.g. [30,31,32]).\n\nOn top of these issues in the evaluation, the paper is unfortunately also quite hard to read. The paper would benefit greatly from having a native English speaker carefully edit the writing. Because of this, I found it somewhat hard to follow the authors' motivation for their method.\n\nMinor comments:\n-Table 1 caption should be above the table\n- all equations are missing punctuation\n- for formatting \"log\" or \"acc\" in math mode, one should use \\mathrm{}\n- Table 5 is quite hard to read because for some reason the error bars appear to use a larger font than the actual values\n- I do not understand why the authors only used the ensemble baseline in a single setting. Their argument on page 5 that they can't use the pre-trained model seems very weak. Why not simply train the models yourself?\n- It seems odd that they use a much larger model on CIFAR-10 than on ImageNet\n- Figure 1 should at least have axis labels\n\n[30] Towards neural networks that provably know when they don't know, Alexander Meinke, Matthias Hein, at ICLR20\n[31] Outlier exposure with confidence control for out-of-distribution detection, Papadopoulos, Aristotelis-Angelos and Rajati, Mohammad Reza and Shaikh, Nazim and Wang, Jiamian, in Neurocomputing21\n[32] ATOM: Robustifying Out-of-Distribution Detection Using Outlier Mining, Chen, Jiefeng and Li, Yixuan and Wu, Xi and Liang, Yingyu and Jha, Somesh, at Joint European Conference on Machine Learning and Knowledge Discovery in Databases 2021 ", "summary_of_the_paper": "The paper proposes a method for detecting out-of-distribution examples by applying the ideas of the popular ODIN method to different scoring functions. They ablate the effects of different parameters and compare their methods to several baseline methods in OOD detection.", "summary_of_the_review": "I recommend rejecting this paper, mainly because there are severe flaws in its evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a gradient-based method for OOD detection. While the paper has some interesting contributions, all the reviewers felt that the current version falls below the ICLR acceptance threshold. I encourage the authors to revise and resubmit to a different venue.
Decision: Reject