Estimating Example Difficulty using Variance of Gradients
Authors: Chirag Agarwal
Sara Hooker
Venue:  (2021)
Keywords: interpretability
human in the loop learning
atypical examples

Abstract: In machine learning, a question of great interest is understanding what examples are challenging for a model to classify. Identifying atypical examples helps inform safe deployment of models, isolates examples that require further human inspection, and provides interpretability into model behavior. In this work, we propose the Variance of Gradients (VOG) as a valuable and efficient proxy metric for detecting outliers in the data distribution. We provide quantitative and qualitative support that VOG is a meaningful way to rank data by difficulty and to surface a tractable subset of the most challenging examples for human-in-the-loop auditing. Data points with high VOG scores are more difficult for the model to learn and over-index on examples that require memorization.

Under review as a conference paper at ICLR 2021
ESTIMATING EXAMPLE DIFFICULTY USING VARIANCE OF GRADIENTS
A Anonymous authors
Paper under double-bling review
ABSTRACT
In machine learning, a question of great interest is understanding what examples are challenging for model to classify. Identifying atypical examples helps inform safe deployment of models, isolates examples that require further human inspection, and provides interpretability into model behavior. In this work, propose Variance of Gradients (VoG) as a valuable and efficient proxy metric for detecting outliers in the data distribution. We provide quantitative and qualitative support that VoG is meaningful way to rank data by difficulty and to surface tractable subset of the most challenging examples for human- human-in-the-- in-the-loopp auditing. Data points with high VoG scores are far more difficult for the model to learn and over-index on corrupted or memorized examples.
INTRODUCTION
Reasoning about model behavior is often easier when presented with subset of data points that are relatively more difficult for a trained model to learn. This not only aids interpretability through case based reasoning (Kim et al. 2016 Caruana 2000) Hooker et al.) 2019), but also be used mechanism to surface a tractable subset of atypical examples for further human auditing (Leibig et al. 2017 Zhang 1992: Hooker et al., 2019), for active learning to inform model improvements,
choose not to classify certain examples when the model is uncertain (Bartlett & Wegkamp) 2008
Cortes et al. 2016).
One of the biggest bottlenecks for human auditing is the large scale size of modern datasets and the cost of annotating each feature (Veale & Binns 2017). Methods which automatically surface subset of relatively more challenging examples for human inspection help prioritize limited human annotation and auditing time. Despite the urgency this use-case, ranking examples by difficulty has had limited treatment in the context of deep neural networks due to the computational cost of ranking a high dimensional feature space. Recent work in this direction has either been limited to small scale datasets or features computational cost which is infeasible for most practitioners (Hooker et al. 2019 Carlini et al. 2019 Koh & Liang 2017).
In this work, we start with simple hypothesis examples that a model has difficulty learning will exhibit higher variance in gradient updates over the course of training. On the other hand, we expect the backpropagated gradients of the samples that relatively easier to learn will have lower variance because performance on that example does not consistently dominate the loss over the course of training. The gradient updates for the relatively easier examples are expected stabilize early in training and converge to a narrow range of values. We term this class normalized ranking mechanism Variance of Gradients VoG, and demonstrate across a variety of large scale datasets that efficiently ranks the difficulty of both training and test examples. VoG can be computed using either the predicted or true label, making it valuable unsupervised auditing tool test time when the true label is unknown.
Validating the behavior of VoG on artificial data. To begin, we illustrate the principle and effectiveness of VoG in a contrived toy example setting. The data was generated using two separate isotropic Gaussian clusters with a total of 500 data points. In such a simple low dimensional problem, closer the decision boundary. In
the most challenging examples for the model to classify Fig. Ia we visualize the trained decision boundary of a multiple layer perceptron (MLP) with a single hidden layer trained for 15 epochs. VoG computed at relative intervals for each training data point.

---

Under review as a conference paper at ICLR 2021
In Fig. lb plot the final VoG score (Sec. against the distance to the trained boundary. As expected. VoG allocates the highest scores to the most challenging examples that are closest to the decision boundary which exhibit the greatest variance in gradient updates over the course of the training process.
16
Glass-0
Perpendcuuer distanco
12
Feature-2
10-
-5
-10
Class-0
Class-1
-15
-5.0
0.025 0.050 0.075 0.100 0.150 0.175
Feature-1
(a) Toy dataset trained decision boundary
(b) Distance VOG
Figure We compute the variance of gradients (VoG) for each training data point in this two dimensional toy problem. On the right, we show that VoG accords higher scores to the most challenging examples closest to the decision boundary (as measured by the perpendicular distance).
Contributions We scale this toy experiment and demonstrate consistent results across two differ- architectures and three datasets Cifar-10, Cifar- 100 Krizhevsky al. (2009) and ImageNet. (Russakovsky et al., 2015). Our contributions can be enumerated as follows:
1. We propose Variance of Gradienst (VoG) class-normalized variance gradient score for determining the relative ease of learning data samples within a given class (Sec.2). Bec...).
2. We show that VoG is an effective auditing tool for ranking high dimensional datasets by difficulty. VoG assigns higher scores to test-set examples that are more challenging for the model to classify. Restricting evaluation to the test-set examples with the lowest VoG greatly improves generalization performance. (Sec.3) 33
3. VoG identifies clusters of images with clearly distinct semantic properties. As seen in Fig.4p. Low scores feature images with far less cluttered backgrounds and more prototypical vantage points of the object. In contrast, images with high scores over-index on images with cluttered backgrounds and atypical vantage points of the object of interest (zoomed in on part of the object, side profile of the object, shot from above).
4. VoG effectively surfaces OOD and memorized examples We empirically show that VoG allocates higher scores to examples that require memorization (Sec.55) and duitr--distriuution examples from curated benchmarks like ImageNet-O (Hendrycks et al. 2011).. We use VoG to explore how learning differs at different stages training and show that VoG rankings are sensitive to the stage of training and provide insight into the learning process in deep neural networks.
Implications of this work It is becoming increasingly important for deep neural networks (DNNs) to make decisions that are interpretable to both researchers and -usess sensitive domains such health care diagnostics (Xie et al. 2019 Gruetzemmccher al., 2018 Badgeley al. 2019
Oakden-Rayner et al. 2019), self-driving cars [HHTSA] 2017) and hiring (Dastin) 2018 Harwell 2019) providing tools for domain experts audit models of upmost importance. Our work offers an efficient method to rank the global difficulty of examples and automatically surface a possible subset aid human interpretability. VoG can be computed using checkpoints stored over the course of training and is model agnostic. Critically, VoG can be computed using the predicted label, which makes it unsupervised auditing tool at test time.
METHODOLOGY
We consider supervised classification problem where DNN is trained to approximate the function F that maps an input variable X to an output variable Y, formally F X + Y. E Y is a discrete label vector associated with each input Each label y corresponds to of C categories classes.
2

---

Under review as a conference paper at ICLR 2021
Lowest VoG
Highest VoG
Lowest VoG
Highest VoG
Cifar- 100; apple
Cifar- 10; plane
(a) Early Training
(b) Late-stage Training
Figure 2: The 5x5 grid shows the top-25 Cifar-10 and Cifar- 100 training-set images with the lowest and highest VoG scores in the Early (a) and Late (b) training stage respectively of two randomly chosen classes. Lower VoG images evidence uncluttered backgrounds (for both apple and plane) in the Late training stage. VoG also appears to capture a color bias present during the Early training stage for both apple (red). The VoG images in Late training stage present unusual vantage points, with images where the frame is zoomed in on the object of interest.
given input image X can be decomposed into set of pixels n,, where (1, (1,...,N) N) and N is the total number pixels in the image. For a given image, we compute the gradient of the activation A with respect each pixel z4. Here, designates the pre-softmax layer of the network and p the index of either the true or predicted class probability. We consider S as a matrix that represents the gradient of A with respect individual pixels Fe, i.e., for an image of (say) 3 x x 32 size, S will be a 3 x 32 x 32 gradient matrix.
0A
S Or;
This formulation may feel familiar it is often computed based upon the weights of trained model and visualized as image heatmap for interpretability purposes (Baehrens et al. 20100: Simonyan al. (2013). Here, instead intend compute the average variance of the input gradients for the same image across training to arrive at a scalar score that is a proxy measure of how challenging the example to learn. Without loss of generality, we take the sum across the color channels to arrive at a gradient matrix S where S E 3232x32 For a given set of K checkpoints, we generate the above gradient matrix S for all individual checkpoints, i.e., (SI,..., SK). We then calculate the mean gradient y by taking the average of the K gradient matrices. Note, is the mean across different checkpoints and the same size as the gradient matrix S. We then calculate the variance of gradients across each pixel using the equations:
K
S.
(2)
K
K
VoG
S: 1u)2
(3)
Here, VoGpizel is matrix representing the variance of gradients of each pixel in the image. We average the pixel-wise variance of gradients to compute scalar VoG score for the given input image:
1 N
VoG
sum (VoG
where, N is the total number of pixels a given image. Hence, for every data sample X we compute scalar value indicating the variance of gradients score. To calculate the class-normalized VoG
3

---

Under review as a conference paper at ICLR 2021
score, we calculate the mean and deviation of all the VoG scores belonging to each class where c E {1.... C}. from the dataset. In order to account for inherent differences in variance between classes, normalize the absolute ranking of the VoG score by class-level mean and standard deviation. This amounts to asking: What is the variance of gradients for this image relative to all other exemplars for this class category?
2.1 EXPERIMENTAL SETUP
Datasets: We evaluate methodology Cifar-10 and Cifar-100 Krizhevsky et al. (2009) and ImageNet (Russakovsky et al. 2015).
ResNet-18 network (He al.) 2016) for both Cifar-10 and Cifar- 100. For each dataset, we train for 350 epochs using stochastic gradient descent (SGD) and compute the input gradients for each sample every 10 epochs. We implemented standard data augmentation by applying cropping and horizontal flips of input images. We use a base learning rate schedule of 0.1 and adaptively change to 0.01 at 150 and 0.001 250 training epochs. The top-1 test-set accuracy for Cifar-10 10 and Cifar- 100 were 89.57% and 66.86% respectively.
Cifar Training: We
ImageNet Training: We use ResNet-50 (He al. 2015) trained on ImageNet. The network was trained with batch normalization (loffe& & Szegedy 2015), weight decay, decreasing learning rate schedules, and augmented training data. We train for 32, 000 steps (approximately
epochs) on ImageNet with batch size of 1024 images. We store 32 checkpoints over the course of training, practice observe that VoG ranking very stable computed with as few as checkpoints. Our model achieves top-1 accuracy of 76.68% and top-5 accuracy of 93.29%.
Lowest VoG
Highest VoG
Lowest VoG
Highest VoG
OKB
nagpie
pop bottle
Figure 3: Each 5x5 grid shows the top-25 ImageNet training-sel set images with the lowest and highest VoG scores for the class magpie and pop bottle with their predicted labels below the image. Training set images with higher VoG scores (b) tend to feature zoomed-in images with atypical color schemes and vantage points.
3 EVALUATING VOG
For all datasets considered, we compute VoG for both training and eval sets. In this section, we evaluate the utility of VoG as an auditing tool. We evaluate the stability of the VoG ranking, measure how discriminative it is at separating easy examples from difficult, and comment on the qualitative properties of images at either end of the VoG spectrum.
Qualitative inspection of ranking A qualitative inspection of examples with high and low VoG scores shows that there are distinct properties to the images either end of the ranking. We visualize 25 images ranked lowest and highest according to VoG for both the entire dataset (visualized for ImageNet in Fig. (5) and for specific classes (visualized for ImageNet Fig. and for Cifar- and C00 in Fig. 2). We observe that ranking by VoG produces clusters with clearly distinguished semantic features. Images with low VoG score tend to have uncluttered and often white backgrounds with the object of interest centered clearly in the frame. Images with the highest VoG scores have cluttered backgrounds and the object of interest not casily distinguishable from the background. We also note that images with high VoG score tend feature atypical vantage points of the objects
4

---

Under review as a conference paper at ICLR 2021
such highly zoomed frames, side profiles of the object shots taken from above. Other, the object of interest is partially occluded or there are image corruptions present such as heavy blur.
Test-set error and VoG A valuable property of an auditing tool is to be able to effectively discriminate between easy and challenging examples. Here, we measure whether VoG is able to do so. In Fig. 4 plot the test-set error of examples bucketed by VoG decile. For this and the remainder of the experiments, we compute VoG using checkpoints stored from the first and last 3 epochs. Thus, at each point of the x-axis, we are computing the test-set error on the 10% of data whose VoG score falls between each decile. Note that we plot error, so lower is better. We show that examples at the lowest percentiles of VoG have far lower error rates. clasiiiassiiiattion increases with an increase in VoG scores, Our results are consistent across all datasets, yet more pronounced for the more complex datasets Cifar- 100 and ImageNet. We ascribe this to differences in underlying model complexity. In Fig. 11 we observe that test-set accuracy on the lowest VoG scored images improves beyond baseline test-set performance. Our models show improved generalization when restricted to low VoG images.
20
40
86 top test sot cmor
3.8
36
34
tost
24
32
32
10
ye 30
22
0 10
20 30
40 50 60 70 80
90 100
10 20
90 100
20
so 60 100
VOG percentile range
percentile range
VOG percentile range
(a) Cifar-10
(b) Cifar-100
(c) ImageNet
Figure 4: The mean top-1 I test error (y-axis) for the exemplars thresholded by VoG score percentile (x-axis). Across 10, Cifar-100 100 and ImageNet, we observe that misclassification increases with an increase in VoG scores. Across all datasets observe that the group of samples in the top- 10 percentile VoG scores have the highest error rate, iee, contains most number of misclassified samples. For all datasets, model generalization improves the bottom 10th percentile relative to the entire dataset.
VoG as an unsupervised auditing tool Many auditing tools to evaluate and understand possible model bias require the presence of labels for protected attributes and underlying variables. However, this is highly infeasible in real- world settings (Veale & Binns 2017). For image and language datasets, the high dimensionality of the problem makes it hard to identify a priori what are underlying variables to be aware of. Even acquiring the labels for limited number of attributes protected by law (gender, race) expensive and/or may be perceived intrusive leading to noisy or incomplete labels. One key advantage of VoG is that we show it continues to produce a useful ranking even when the gradients are computed with respect to the predicted label. In Fig. we include the top and bottom 25 VoG ImageNet test images using predictions. We also computed the mean test-error for the predicted VoG distribution, and find that it also effectively discriminates between top-10 10 and bottom- 10 examples with 73.6% and 77.5% accuracy respectively (Fig. 12).
Stability of VoG ranking To build trust with the end-user, a key desirable property of any auditing tool is consistency in performance. We would expect consistent method to produce ranking with a closely bounded distribution of scores across independently trained runs of the same model architecture and dataset. To measure the consistency of the VoG ranking, we train 5 Cifar- 10 networks from random initialization using the training methodology explained in Sec. 2.1 Empirically, Fig, 6aa shows that VoG rankings evidence consistent distribution of test-error each percentile given the same model and dataset.
4 LEVERAGING VOG TO UNDERSTAND EARLY AND LATE TRAINING DYNAMICS
Recent work has shown that there are distinct stages to training in deep neural networks (Achille ct al., 2017; Tiang et al. 2020 Mangalam & Prabhu 2019 Faghri et al. 2020). In our second set of experiments, explore whether rankings according to VoG are sensitive the stage of the
5

---

Under review as a conference paper at ICLR 2021
PT
PT
PT Gee mosia
ST worthe
pte ga minae
PTh DAN
5.531 oven
OO Drgint Phoedda
GT aP33
OR cade
conna
OT bes co
2D achicca
00060 EE
1,003 ER Paber
2000 OU ARRI Miter
teom
5,000 QL
Bi Roctsh
0.0044
3 0234
a 130s
0000
COOCS
DC
57500
5,00
5,003
5.309
LGO of
0,006
GoCCs
5,771
6.5004
11443
6,223
(a) Lowest VoG
(b) Highest VoG
Figure 5: Each 5x5 grid shows the top-25 ImageNet test-set images with the lowest and highest VoG scores for the top-1 predicted class. Test set images with higher VoG scores tend to feature zoomed-in images and misclassified more as compared to the lower VoG images which tend to feature more prototypical vantage points of objects.
+
ImageNet-O ImageNet-val
Ran
Percentage Images
Run
2.
top-1 test set
26
24
50 70 80 90 100
25
75
100
VOG percentile range
VOG percentile range
(a)
Figure 6: Left: Consistency in ranking is an important attribute of any auditing tool. Here, we plot the VoG top-1 test set emror for ResNet- 18 networks independently trained on Cifar- 10 from random initialization. The plots show that VoG produces stable ranking with similar distribution of error in each percentile across all images. Right: We measure the distribution of ImageNet-O) images across percentiles. We find that higher percentiles of VoG over-index these out of distribution images.
training process. Hence, we compute VoG separately for two different stages of the training process, which we term (1) the Early stage (first three epochs), and (2) the Late stage (last three epochs). Test-set accuracy the Early stage is 44.65%, 14.16% and 51.87% for Cifar- 10, Cifar- 100 and ImageNet respectively. In the Late stage it is 89.57%, 66.86% and 76.69% for Cifar- 10, Cifar-100 100 and ImageNet respectively.
We find that there a noticeable visual difference between the image ranking computed for Early and Late stages of training. As seen in Fig. 2 for some classes such apple it appears that VoG scores also capture network color bias present during the Early training stages. For these classes, the lowest VoG scores over-index red colored apples.

---

Under review as a conference paper at ICLR 2021
For ImageNet, compute the distribution of error in early and late stage and find a remarkable flipping point (Fig. 7 During the early training stage, samples having higher VoG score tend have lower rate the gradient updates center easy examples. This phenomenon is reversed during the late stage of the training where most easy example have been learnt, and updates to the harder examples dominate the the computation of variance. Hence, samples having high VoG results in higher error rate.
% top-1 test set erro
top-1 error
27.5
60
25.0
55
22.5
20.0
50
17.5
45
15.0
10
20 30
50 60 70 80 90 100
10
20
30 40 50 60 70 80 90 100 VOG percentile range
VOG percentile range
(a) Early stage
(b) Late stage
Figure 7: The mean top-1 test set error (y-axis) for the exemplars thresholded by VoG score percentile (x-axis) in ImageNet validation set. The Early (a) and Late (b) stage VoG analysis shows inverse behavior where the role of VoG flips the training progresses.
VOG score
VOG score
2
correct
shuffled
correct
shuffled
Cifar-10
(b) Cifar-
Figure 8: Box-plot of subset the VoG distribution of all examples with correct labels against the 20% of the dataset with shuffled labels. It visible that the distribution VoG scores, both the mean (red line in the plot) and spread, for shuffled data is higher than that of the correct samples for both C0 (right plot) and Cifar-100 (left plot).
5 RELATIONSHIP BETWEEN VoG SCORES AND MEMORIZED/OOD EXAMPLES
Recent work has highlighted that deep neural networks produce output probabilities that are uncali- brated (Guo al.) 2017 Kendall & Gal 2017: Lakshminarayanan et al. 2017 Hendrycks & Gimpel 2016) and thus cannot be interpreted as a measure of certainty. If VoG is a useful auditing tool, we expect capture model uncertainty even when this is not reflected in the end probabilities.
To this end, we consider VoG rankings on two tasks where the network produces highly certain predictions for incorrect
dittrf-distributioon inputs.
Surfacing examples that require memorization Overparameterized networks have been shown to achieve training error by memorizing examples (Zhang et al. 2016 Feldman, 2020). We explore whether VoG is able to identify examples that require memorization and the rest of the dataset. To do this, replicate the general experiment setup of Zhang al. (2016) and replace 20% of all labels in
7

---

Under review as a conference paper at ICLR 2021
the training set with random shuffled labels. We re-train the model from random initialization and compute VoG scores at relative intervals across training for all examples in the training set. Our network achieves 0% training error which would only be possible given successful memorization of the noisy examples with shuffled labels. Is VoG able to discriminate between these memorized examples and the rest of the dataset? In Fig. g we plot the box plot distribution of VoG scores for the subset of the data with shuffled labels that required memorization beside correct labels. We find that the mean and spread of the examples with the shuffled labels are higher when compared the rest of the dataset and that this difference is statistically significant. We perform two-sample t-test with unequal variances. At a p-value of 0.01, we reject the null hypothesis that the mean of both samples are the same and conclude that the difference in VoG means is statistically significant. Shuffled labels have a different VoG distribution than the non-shuffled dataset. We include more details about the statistical testing in the appendix.
0.225
V060
0.200
Training loss
0.150
0.125
0.100
0.050
0.025
0
Epochs
Figure 9: We consider using VoG as a ranking mechanism to accelerate training. VoGo upweights examples using the VoG score during training. Models trained using VoGo (green) converges faster and to lower training loss as compared to SGD (blue). The VoG scores were calculated using K = number of checkpoints and observe sharp drop in the training once start scaling the gradient updates of the batches with their respect VoG scores after the third epoch.
ImageNet-O experiments We consider ImageNet-O (Hendrycks et al. 2019), an open source curated out-of of-dissributtin (OOD) dataset designed to fool classifiers. Image Net-O consists of images that not included in the original 1000 ImageNet classes. These images were selected with the goal of producing high confidence incorrect ImageNet- K predictions of labels from within the training distribution. We are interesting in understanding if VoG being atypical or out of distribution. We would expect to observe that ImageNet-O examples would be over-represented in top percentiles of VoG scores vs the lowest scores. In Fig. 6b we see that this is indeed the case. We plot the count of ImageNet-O image in each percentile as a fraction of the total count of ImageNet-O images (2000 images in total). ImageNet-O images are relatively over-represented at high levels of VoG, with 30% of all images in the top-25th percentile vs 24% in the bottom 25th percentile.
correctly rank ImageNet-O examples as
CAN WE USE VOG TO ACCELERATE AND IMPROVE TRAINING PERFORMANCE?
In this section, we consider whether the VoG score can be using to improve the optimization process by re-weighting examples considered to be atypical examples on the fly during the training process. As toy experiment, we trained single layer (32 neurons) feed-forward network on the handwritten MNIST digit classification task. As a preliminary exploration of VoG, we weigh each mini-batch gradient update with their respective VoG scores. We compare this VoGo variant with standard mini-batch batih Stochastic Gradient Descent (SGD) and Variance Of Gradients Optimizer (VoGo) for 10 epochs using batch size of 256 (algorithm details of VoGo in the appendix Algorithm []. As preliminary implementation of VoGo, we weigh each mini- batch gradient update with their respective VoG scores.
8

---

Under review as a conference paper at ICLR 2021
The initial weights of the architectures were set to the same seed for both SGD and VoG optimizers. In Fig. 9 we observe that VoG achieves a lower training loss as compared to SGD. The training difference between the optimizers is reflected in the model's testing performance. Across the different runs, models trained using VoGo achieve a testing accuracy of 91.77 = 0.10% as compared Notably, the performance deviation across different runs is also smaller for VoGo as compared to its counterpart. Training with VoGo accelerates training, achieves higher test-set accuracy and reduces the stochasticity of the training process.
SGD which achieves almost 20% lower performance 68.50++3.0999
7 RELATED WORK
Our work proposes a method to rank training and test examples by estimated difficulty. Given the size of modern day datasets (Hooker (220)), this can be a powerful interpretability tool isolate tractable subset of examples for human-in-the-logg auditing and also aid in curriculum learning (Bengio al. 2009). Prior work has proposed different notions of what subset merits surfacing. Early work by (Zhang 1992; Bien & Tibshirani, 2012: Kim et al.] 2015 Kim et al., (016) that introduced the notion of prototypes. quintessential examples in the dataset, but did not focus on deep neural networks. Kim et al. (2016) also requires assumptions about the statistics of the input distribution. Gal & Ghahramani (2016) showed how we can use dropouts as a Bayesian approximation method for representing model uncertainty in deep learning. Work by Li et al. (2017) requires modifying the architecture to prefix an autoencoder in order to surface a set of prototypes. Koh & Liang (2017) proposes influence functions identify training points most influential on given prediction.
Unlike previous works, we propose a measure that can be extended to rank the entire dataset by estimated difficulty (rather than surfacing prototypical subset). Additionally, ranking individual samples using methods like Koh & Liang (2017) would be extremely computationally extensive. Our method does not require modifying the architecture making any any assumptions about the statistics of the input distribution. In that sense, our work is complementary to recent work by Diang al. (2020) which proposes a c-score rank each example by alignment with the training instances, Hooker et al. (2019) which classify examples as outliers according to sensitivity to varying model capacity and Carlini et al.] (2019) which consider several different measures to isolate prototypes that could conceivably be extended to rank the entire dataset. We note that the c-score method proposed by Jiang et al. (2020) is considerably computationally intensive to compute than VoG as it requires training up 20, 000 network replications per data set. Several of the propotype methods considered by Carlini et al. (2019) require training ensembles of models, as does the compression sensitivity measure proposed by Hooker al. (2019). Our method is both different in formulation and can be leveraged using a small number of existing checkpoints saved over the course of training.
8 CONCLUSION AND FUTURE WORK
Our methodology offers one way for humans to better understand the relative difficulty of different examples. One of our key findings is that VoG is far more challenging classify for the algorithm and surfaces clusters of images with distinct visual properties. VoG is straight-forward to compute and can take advantage of current best practices of storing multiple checkpoints over the course of training. In practice, domain expert may choose to compute VoG for class of particular interest which would further reduce the computational cost.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. ArXiv, 0tb//1111.08855 2017.
Marcus Badgeley, John Zech, Luke Dakden-Rayner, Benjamin Glicksberg, Manway Liu, William Gale, Michael McConnell, Bethany Percha, and Thomas Snyder. Deep learning predicts hip fracture using confounding patient and healthcare variables. npj Digital Medicine, 2:31, 04 2019. doi: 00008///17766019-011111
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert MÅžller. How to explain individual classification decisions. Journal of Machine Learning Research, 1 1((Ju):1803-1881, 2010.
Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. J. Mach. Learn. Res., 9:1823-1840, June 2008 ISSN 1532-4435. URL http:///llll acm. org/
Settation.cfmii-111331 1442799
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009,
Jacob Bien and Robert Tibshirani. Prototype selection for interpretable classification, arXiv e-prints, art. arXiv: 202.5933, February 2012.
Nicholas Carlini, Úlfar Erlingsson, and Nicolas Papernot. Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications. arXiv e-prints, 1rXiv:1910.13422 October 2019.
Rich Caruana. Case-based explanation for artificial neural nets. In Helge Malmgren, Magnus Borga, and Lars Niklasson (eds.), Artificial Neural Networks in Medicine and Biology, pp. 303-308, London, 2000. Springer London. ISBN 978-1-4471-0513-8.
Corinna Cortes, Giulia DeSalvo, and Mchryar Mohri. Boosting with abstention. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1660-1668 Curran Associates, Inc., 2016. URL //aperss www.aaaastttnnn pdf
htips.cc/paper//See http:///ppppeee66-ooooooooooo.
Jeffrey Dastin. Amazon scraps secret ai recruiting tool that showed bias against women. Reuters, 2018. URL htttts::/reet.rs/2220/
Fartash Faghri, David Duvenaud, David Fleet, and Jimmy Ba. A Study of Gradient Variance in Deep Learning. arXiv e-prints, art. arXiv:2007.04532 July 2020.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. Proceedings of the 52nd Anmual ACM SIGACT Symposium on Theory of Computing, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Ross Gruetzemacher, Ashish Gupta, and David B. Paradice. 3d deep learning for detecting pulmonary nodules in et scans. Journal of the American Medicul Informatics Association: JAMIA, 25 10: 1301-1310, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. arXiv e-prints, art. arXiv: 74699499, Jun 2017.
Drew Harwell. A face-scanning algorithm increasingly decides whether you deserve the job. The Washington Post, 2019. URL https::/t///at hwtp:/awpp.ppp sttapo.st/23330 st/2X3bup0
K. He, X. Zhang, S. Ren, and Sun. Deep Residual Learning for Image Recognition. ArXiv e-prints, December 2015.
10

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv e-prints, art. arXiv:1610.02 111..6211...66.. Oct 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint FrXivv/907.07111 2019.
Sara Hooker. The hardware lottery, 2020.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What Do Compressed Deep Neural Networks Forget? arXiv e-prints, art. mtXiv1911.05245. November 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Ziheng Jiang, Chiyuan Zhang. Kunal Talwar, and Michael C. Mozer. Characterizing Structural Regularities of Labeled Data in Overparameterized Models, arXiv e-prints, art. arXiv:2002.03206, February 2020.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael Mozer. Exploring the memorization- generalization continuum in deep learning, arXiv preprint arXiv:2002.03220 2020.
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5574-5584. Curran Associates, Inc., 2017.
Been Kim, Cynthia Rudin, and Julie Shah. The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification. arXiv e-prints, art. 10Xiv:1503.01161, March 2015.
Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples are not enough, learn to criticize! criticism for interpretability. In D. D. Lee, M. Sugiyama, U. V. Luxburg. Guyon, and Garnett in Neural Information Processing Systems 29, pp. 2280-2288. Curran Associates, Inc., 2016.
Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. arXiv e-prints, art,
March 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Confer- ence on Neural Information Processing Systems, NIPS' 17, pp. 6405-6416, USA, 2017. Curran Associates Inc. ISBN 978-1-5108-6096 URL /ttt:://dl.. acm. org/ citatttttttt ion. cfm?
1d=3295222. 3295387
Christian Leibig, Vaneeda Allken, Murat Seckin Ayhan, Philipp Berens, and Siegfried Wahl. Lever- aging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7, 12 2017. doi: 10.108//41198--1 199--117-777777-22
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. CORR, abs/1710.00406, 2017. htRp:ttt://axxi org/ abs/ / 1710. 04806
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first. 2019.
NHTSA. Technical report, U.S. Department of Transportation. National Highway Traffic, Tesla Crash Preliminary Evaluation Report Safety Administration. PE 16-007, Jan 2017.

---

Under review as a conference paper at ICLR 2021
Luke Oakden- Rayner,, Jared Dunnmon, Gustavo Cameiro, and Christopher Ré. Hidden Stratification Causes Clinically Mcaningful Failures in Machine Learning for Medical Imaging. arXiv e-prints, art. 1XXi:1909.112755 Sep 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, Dec 2015. ISSN 1573-1405. doi: 10.10077511265--115008166; URL 5- 5-081-
https://doi: .orii.orr www.rorgg.0.............. 10000
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.603: 2013.
Michael Veale and Reuben Binns. Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society, 472:2053951117745530, 2017. doi:
10.117/2059511174333
Hongtao Xie, Dongbao Yang, Nannan Sun, Zhineng Chen, and Yongdong Zhang. Automated pulmonary nodule detection in ct images using deep convolutional neural networks. Pat- tern Recognition, 85:109
119, 2019. ISSN 0031-3203, doi: https///oo..rg/1000011/// patcog 201c0o.2018.0.0033 URL http:// / /wwww www.ssccieeeectte...cc..ce..eeee httpe/ccccoocccceee///tttilll/l p11/8003152011800771
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 16Xi::16/1.035330 2016.
Jianping Zhang. Selecting typical instances in instance-based learning. In Derek Sleeman and Peter Edwards (eds.), Machine Learning Proceedings 1992, pp. 470 479. Morgan Kauf- doi: https://doi.org/10..00 URL http:::/www.cciiittt wwwpwwwwwwwwwwwwwwwwwwwwww...... rect. commcm//i arricllessA//S778855882222255553
mann, San Francisco (CA), 1992. ISBN 978-1-55860-247-22
1978-1-558600224-22500666
12

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
Toy Experiment We generate the clusters for classification using scikit-learn library. We used 90-10% split for dividing the dataset into the training and testing category. linear Multiple Layer Perceptron (MLP) network with single hidden layer of 10 neurons was trained using Stochastic Gradient Descent (SGD) optimizer for epochs. We divided the training process into three stages: (1) Early [0, 5) epochs. (2) M iddlle [5, 10), and (3) Late stage [10, 15). Our trained model achieves a 0% testing error using a linear boundary (Fig. lap.
Class Level Error Metrics and VoG Here, explore whether VoG is able to capture class level differences in difficulty. We compute VoG scores for each image in the test-set of Cifar- 10 and Cifar-100 (both test-sets have 10, 000 images). In Fig. TO plot the average absolute VoG score for each class against the false negative rate for each class. We find that there is a positive, albeit weak, correlation between the two, classes with higher VoG scores have higher mis-classification The correlation between these metrics is 0.65 and 0.59 for Cifar- 10 and Cifar- 100 respectively. Given that VoG computed per-example level, we find interesting that the aggregate average of VoG is able to capture class level differences in difficulty.
rate.
r=0.55 p-val=0.04
r=0.59 p-val=8.39e
60
Error rate (in %))
Error rate (in %)
20
40
15
10
20
-0.2 Normalized class-wise VOG scores
-0.8
-0.6
-1.0
008
0.6
-0.4
-02 0.0 0.2
Normalized class-wise VOG scores
(a) Cifar-10
(b) Cifar-100
Figure 10: Plot of class false negative rate (y-axis) against average class VoG score for all classes (x-axis). Left: Cifar- Right: Cifar- 100. There is statistically significant positive correlation between class level emror metrics and average VoG score (alpha at 0.05).
50
50
50-
%6 top test set error
40
40.
40
set
tast set
30
30
30
20
20
20
top
10
100
bottom-10
top-10
top-10
bottom-11
all-tast
Cifar-10
Cifar-100
(c) ImageNet
Figure 11: Bar plots showing the mean top-1 emror rate (in %) for three group of samples from (1) the subset the test-set with the bottom 10th percentile of VoG scores, (2) the complete testing dataset, and (3) the subset of the test-set with the top 10th percentile of VoG scores.
Statistical Significance of Memorization Experiments Is VoG able discriminate between these memorized examples and the rest of the dataset? In Fig. g we plot the box plot distribution of VoG scores for the subset of the data with shuffled labels that required memorization beside correct labels. We find that the mean and spread of the examples with the shuffled labels are higher when compared to the rest of the dataset and that this difference is statistically significant. The groups
13

---

Under review as a conference paper at ICLR 2021
% top-1 test set error
26
25
0 10 20 30 40 50 60 70 80 90 100
VOG percentile range
Figure 12: The mean top-1 test error (y-axis) for the exemplars thresholded by VoG score percentile (x-axis) calculated using the predicted labels. We observe that misclassification increases with an increase VoG scores. Across ImageNet we observe that VoG calculated for the predicted labels follows the general trend as in Fig. where the top-10 percentile VoG scores have the highest error rate.
of population in the test consists of the VoG scores for the unaltered training set and the subset of the training set with shuffled labels. We conduct this t-test for both Cifar- 10 and Cifar-10M datasets. The two-sample I-test produces p-value that can be used to decide whether there is evidence of a significant difference between the two distributions probability that the difference between the sample means is large, i.., the smaller the p-value, the stronger is the evidence that the two populations have different means.
VoG scores. The p-value represents the
Null Hypothesis: μ1 μ12
Alternative Hypothesis: μ1 * μ12
If the p-value is less than your significance level (a 0.05 in this experiment), can reject the null hypothesis, i.e., the difference between the two means is statistically significant. The details for the individual t-tests for Cifar-10 and Cifar- 100 given below:
Cifar-10: The statistics for the samples in the correct and shuffled labels are:
Corrected labels: μ1 0.62; 01 0.54; N₁ 40000
Shuffled labels: μ12 = 0.85; d2 = 0.75; N₂ = 10000
p-value is < 0.001
Result: Reject Null Hypothesis (the two populations have different VoG means)
Cifar- The statistics for the samples in the correct and shuffled labels
Corrected labels: = 0.54; σı 0.46; N₁ 40000
Shuffled labels: μ2 0.82; 02 0.71; N₂ 10000
p-value is < 0.001
Result: Reject Null Hypothesis (the two populations have different VoG means)
B VARIANCE OF GRADIENTS OPTIMIZER (VoGo)
Let us consider deep neural network F that is be trained supervised classification problem. The function F' maps an input variable X to an output variable Y, formally F X + Each y E Y corresponds to one of C' categories classes. We compute the gradient of the activation A with respect to each input in the training set. As before, I can represent the pre- post-softmax layer of the network and p the index of either the true the predicted class probability.
0AL S
ax
In Algorithm I provide brief overview of the Variance Of Gradients Optimizer (VoGo). For simplicity, we have not mentioned the update steps for the bias parameters. As an initial test, we weight each mini-batch gradient update with their respective VoG scores. Batches having higher
14

---

Under review as a conference paper at ICLR 2021
VoG score are relatively harder for the model run and hence up weight their updates in the direction of the gradient descent.
Algorithm 1 Variance of Gradient optimizer
1: Initialization: Choose initial value of w; step size y for storing the number of snapshots (set to 3 for the MNIST experiment); learning rate ny; VoG regularization coefficient μ..
2: for < total epochs do
3: S, = dict()
4:
for i, (x. y) in enumerate(X, Y) do
5:
vog;
6:
Vpred F(x)
7:
1= LosS(Spredd !)
AA
8:
Stemp
dz
9:
len( S.) < y then
10:
S (.l..apeend((emp)
11:
else
12:
S S. :] [Stemp]
13:
end
14:
if / 7 then
15:
tog. = VOG(S,
end if
17:
Wt+1 = Wt 1 x vog: X 1) V[(w))
end for end for
19:
15


Reviews:
[{"review_id": "z5TPERrNIul", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose Variance of Gradients (VoG) as a quantifiable metric to identify examples that are difficult to classify. This is motivated by the intuition that examples that are easy to classify do not contribute much to the loss beyond early stages of training, hence don't contribute much to the gradient. The gradient $S_{it}$ w.r.t. every pixel $i$ is tracked across $t=1..K$ snapshots through the training process. The mean $\\mu_i$ of $S_{it}$ is computed across snapshots. and finally the VOG of each example $j$ is computed as the average over all $N$ pixels and $K$ snapshots of the squared difference between the gradient $S_{it}$ and $\\mu_i$.\nQualitative and quantitative proof of correlation of VoG with diffuculty of examples is provided:\nQualitative: Authors illustrate that examples with high VoG tend to have cluttered backgrounds or odd angles. \nQuantitative: High correlation betwee number of erroneous examples in test set and associated VoG, especially on the harder datasets like ImageNet.\nVoG is used to study phases of training a neural network. Interestingly, if we focus only on early epochs of training, the difficult examples have low VoG while easy examples have high VoG. As in early phases of training, models concentrate on learning the easy examples, and the error rate on these easy examples is still high.\nVoG is also used to identify out of distribution examples that high capacity models tend to get right only by memorization.\n\nStrengths:\n- A new metric, VoG, is proposed that is easy to calculate and shown to be associated with examples that are difficult to classify. Having such a powerful tool will be useful for a variety of purposes, from identifying atypical examples for human auditing, to aided interpretability of models.\n- The metric is easy to compute as compared to competing methods. It can easily be adopted by practitioners as they using checkpoints that are often computed and saved anyway.\n- Empirical results are convincing for the most part. There is a clear increase in error-rate as VoG increases. And this relationship is shown to hold across various network initializations. VoG is shown to vary throughout training. As the network begins to converge, the difficult examples show consistently high VoG. \n\nWeaknesses:\n- Fig. 4 is not clear. For each value of decile on the x-axis, the error rate is computed on that 10% of data. And the error rate is shown to be higher for larger values of VoG. However, the maximum error rate even for the maximum value of VoG is 20-40%. Therefore, there are clearly upto 80% of examples with high VoG that are still correctly classified. Authors don't explain why this is the case.\n- Fig. 7: It is not clear why the error rate associated with the difficult examples that have low VoG early in training is low. Shouldn't the errors associated with difficult examples remain the same throughout training i.e., the network never learns to classify these examples correctly. Why would the error rate degrade? Or are the authors reporting percentage of total errors on the y-axis. In which case while the asbolute number of errors associated with difficult examples remains the same, their relative ratio as compared to the overall number of errors increases as training proceeds. Please clarify.\n- Fig. 8: Out of distribution examples e.g., deliberately shuffled labels are shown to be associated with slightly higher VoG score values. Can the authors include a significance test to show this is a material difference. There is a high variance in VoG values for shuffled examples. Why would these examples exhibit lower VoG? Can the authors provide some intuition behind this.  \n\nConclusion:\nOverall, my decision is to accept the paper because this is a powerful proposal that deserves to be investigated further. However, I have some reservations about the empirical results as described above. If authors can explain/clarify these aspects it would be a much stronger submission.\n\nIn addition to those listed above, the authors should address the questions below in future work:\n- How come not all or even a majority of examples that are misclassified by a network have high VoG?\n- Will results hold across various types of models? What is the relationship of VoG with capacity of models? \n- Will results hold across domains?", "extra_scores": {}, "extra_text": {}}, {"review_id": "rORb0u5dCC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThe authors propose to use the variance of the gradient (VOG) values measured during neural network training as an estimate of the difficulty of examples. For each example, the gradient of the loss function is computed with respect to each pixel during training. The VOG is the mean over all the pixels of the variance of the gradient over different training steps. The author show that VOG is correlated to train and test set error (highest VOG sample have highest error rate). VOG also shows that easy example (with lower error rate) are leant first during training. Out of distribution sample or sample with incorrect label also show a higher VOG mean and a larger variance. VOG can therefore be used to detect incorrectly labeled samples.\n\n\n##########################################################################\n\nReasons for score: \n \n\nThe idea proposed in this paper is simple and seems to be effective. However, the authors should demonstrate that in practice VOG can be used to increase the accuracy : \n* is it possible to use VOG as a confidence score and classify samples only when their VOG is above a given threshold ? at threshold 0, this is the classical setup. At threshold 0.8 (if voG can be normalized between 0 and 1), only sample with VOG >0.8 are classified. This should lead to higher accuracy. Is it the case ? To what extend ?\n* what is the computational complexity of computing VOG on test set to detect outliers or difficult examples ? How many training step s on test samples have to be done ?\n\n\nPage 3, A^l_n is not defined ;  in the equation, S should be S_{ti}\n\nThe gradient is computed over all the pixel of the image. If the image is large, this could be too computational expensive. Is it possible to sample the pixels and still have a good estimate of the VOG ?\n\n\nPage 4, Figure 3 : the sample generated by data augmentation seem to be the most difficult sample. The same inspection should be conducted on the original samples only. On the other hand, could VOG be used to guide the data augmentation process by selecting the \"difficult\" samples ?\n\n\nPage 5, figure 4. The y axis is different for the different figure, which is misleading. \nPage 6, figure 6 a : add the legend, it is not obvious at the first sight that there are 5 curves.\nPage 7, figure 7 : could you explain how is computed VOG on test set sample ? are they added to the training set without updating the gradient ? for the curves, is the VOG computed on all the training steps or only on the same number of steps at the beginning or end of the training ?\n\nRelated works : papers on curriculum learning  and bayesian approximation could be cited\n- Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML '09). Association for Computing Machinery, New York, NY, USA, 41–48. DOI:https://doi.org/10.1145/1553374.1553380\n- Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML'16). JMLR.org, 1050–1059.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XExNbpXK_7U", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to use a scalar measures called Variance of Gradients to discover challenging-to-learn examples on which the model is more likely to make an error. They illustrate that low VoG examples are \"prototypical\" and more easily understood, while high VoG examples typical exhibit occlusions, strange angles, zooms, cluttered backgrounds and other characteristics which make they visually also challenging to classify. Furthermore, high VoG examples are also the ones which the model finds more challenging to correctly classify. \n\nIn terms of dataset interpretability and analysis, I think this work provides interesting empirical insights, especially using such a simple method, which is always good. \n\nThe work is clearly written (with a notable exception discussed below), easy to read, conceptually straightforward and information. However, this work has a number of issues, which (at the moment) prevent me from recommending acceptance.\n\n1. The authors do not actually provide a clear definition of VoG - it is not clear WHAT we are taking the gradient off with respect to an input pixel - the predicted class probability? S is a matrix, but the average of S (mu_i) is a scalar. The notation is opaque and needs further clarification. Furthermore, it is not clear how class normalisation by mean-std is done. Given that the VoG is a central aspect of this work, it MUST be clarified to remove any and all ambiguities. (NECESSARY)\n\n2 In their synthetic analysis the authors show that high VoG example typically lie on the decision boundary between classes and therefore exhibit high data uncertainty (or high aleatoric uncertainty). In practice, judging from the picture, this seems to be also the case in practice on C10/C100/Imagenet. However, the authors do not answer *WHY* examples on decision boundaries should have a high VoG score. (Theoretical interpretation necessary)\n\n3. While I understand that this is an empirical work, I think it requires discussion of the relationships of the proposed measure to other uncertainty/difficulty/confidence estimation approaches is necessary. Of particular importance is the relationship to ensembles of models. Note note the VoG,  which examines the average per-pixel variance of the gradient across an ensemble of models from different checkpoints (stages of training), relates to other measures of ensemble diversity. (Discussion necessary) \n\nFinally, it would be interesting if the authors examined the effects of filtering out training data based on VoG. (Nice to have, but not necessary).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8GOsHpMuRDn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe authors propose *variance-of-gradients* (VOG) as a metric for assessing the difficulty of an example for a model.  For each example, the gradients of the model loss are backpropagated to the input features (pixels for images) for a sample of checkpoints, the per-pixel standard deviation is calculated over the checkpoints, and then the standard deviation is averaged over pixels. The authors present evidence that the VOG metric can differentiate difficult and easy examples, both for labeled examples, and also for unlabeled examples, where a label is imputed as the model prediction. The metric also discriminates visually distinct image types, and shows some correlation with examples that had to be memorized by the model.\n\nPros:\n\n- VOG is an intuitively appealing metric.  If the gradients for an example vary wildly over training, it would seem to be an indicator that the model's representation of that example is unstable, and conflicting signals are present in the example.  Exploration of gradient statistics is a fruitful area for understanding model behavior and training dynamics.\n\n- The authors examine VOG from several interesting viewpoints.  It can be used as a trustworthiness indicator (correlates with accuracy), but also seems to correlate with out-of-distribution examples and example memorization.\n\n- The correlation pattern with accuracy reverses from early- to late-stage training, which gives an interesting viewpoint into training dynamics: early on, gradient updates have high variance for easy examples, and later on, gradient variance is high for the examples that remain difficult for the classifier.\n\nCons:\n\nOverall, while VOG clearly shows some useful properties, there could be more clear benchmarking and comparison to other methods for detection of example difficulty.  The paper currently reads as thorough data analysis showing intriguing results, but without yet clear enough metrics to judge VOG in the context of the many methods for model trustworthiness / OOD detection.  The paper does cite and describe relevant prior research, but more quantitative comparison is called for.\n\nOne of the most important figures is 6b, wherein it is shown that out-of-distribution data is concentrated at the higher quantiles of VOG.  However, the authors should present some baselines here:\n*    The actual probability of the highest confidence class.  The authors state that \"DNNs produce output probabilities that are uncalibrated and thus cannot be interpreted as a measure of certainty\".  While it true that the output probabilities can be problematic, I would assert that they are still widely used, and there is some onus on the authors to demonstrate the superiority of VOG empirically.\n*   There are numerous methods for classify-with-abstain, and one-class classification which could be benchmarked against.  Such methods have been extended to neural architectures.  A few examples that come to mind:\n    *    Liu, Z., Wang, Z., Liang, P. P., Salakhutdinov, R. R., Morency, L. P., & Ueda, M. (2019). Deep gamblers: Learning to abstain with portfolio theory. In Advances in Neural Information Processing Systems (pp. 10623-10633).\n    *    Jiang, H., Kim, B., Guan, M., & Gupta, M. (2018). To trust or not to trust a classifier. In Advances in neural information processing systems (pp. 5541-5552).\n    *    Malinin, A., & Gales, M. (2018). Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems (pp. 7047-7058).\n    *    One-class classification is related, and could be used to detect out-of-distribution data.  Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S. A., Binder, A., ... & Kloft, M. (2018, July). Deep one-class classification. In International conference on machine learning (pp. 4393-4402).\n    *    Hendrycks, D., Mazeika, M., & Dietterich, T. (2018). Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606.\n\nOf course, not all of these methods necessarily need to be benchmarked, but the most representative or competitive methods should have comparisons.\n\nOverall the VOG distributions show quite a bit of overlap between in and out-of-distribution examples on ImageNet-O, and correct and shuffled examples on the Cifar-10 and Cifar-100 datasets.  Hence it is not a 100% reliable indicator and needs to be more firmly placed in context.\n\nA few specific comments:\n*   In the \"Contributions\" section, the authors state \"Restricting evaluation to the test-set examples with the lowest VOG greatly improves generalization performance”.  This confused me, as the authors show that lowest VOG examples have higher accuracy; however this seems to be different from generalization performance being improved, which is a statement about the model quality.\n*    VOG shows vastly different dynamics early and late in training, which is an interesting result.  Hence, in practice, should we use late-VOG for difficulty estimation?  The authors should present clearer guidance here.\n*    In Figure 6a, the authors show that the relationship between VOG and test set error is stable between model retrains.  However, I would argue that it would be a much stronger result to show that VOG is a stable property of the image, as this would show example difficulty for a model class.\n\nA few proof-reading issues:\n*    In \"Methodology\", the authors reference $A^l_n$ without defining $A$.  From the context it seems this is activation, but it causes a bit of a confusion when reading.\n*    Equation (2) contains a subscript $i$ on the left-hand-side, whereas the right hand side sums over $i$.  I suspect the LHS should have a distinct index for the example.\n*   Hendrycks et al., \"Natural adversarial examples\", has two different entries in \"References\".\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4B8BM8J3jND", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new metric for measuring example difficulty: variance of gradients. Given a neural network for image classification, it measures the variability in the gradient of the output of the penultimate layer of the network with respect to each pixel, as observed during the training process. This quantity is averaged over all pixels to measure the difficulty of an image. Illustrative examples, observed correlation with error rate on test data, and successful identification of out-of-distribution examples indicate that the proposed metric is useful.\n\nThe paper is very nicely written and a pleasure to read.\n\nThe primary shortcoming of the paper is that it does not compare to any other measure of example difficulty. An obvious metric is the entropy of the estimated class probability distribution for an example (computed either with or without some form of calibration of the probability estimates). It is unclear whether the proposed metric provides a significant amount of additional information.\n\nAlthough intuitively plausible, and backed up by experimental results, the metric is not justified by providing some form of theory.\n\nData augmentation is used in the experiments but the paper does not explain how this is accounted for in the calculation of the metric. Is the metric computed for the unmodified training examples only?\n\nFigure 6 (left): Why not compute the consistency of the rankings directly? The difference in observed error is an indirect measure.\n\nTypos:\n\n\"p is the index of either the true or the predicted class probability.\"\n\n\"ImaageNet-O\"\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a new uncertainty measure of examples called "Variance of Gradients" (VoGs); it demonstrates that VoGs are correlated with mistakes, and can be useful for guiding optimization. 

On the positive side, the reviewers generally think that the ideas of this paper is nice and contribute to the research thrust in gradient-based uncertainty. In addition, the paper provides valuable empirical insights. 

However, the reviewers also pointed out a few important limitations:
- A more thorough comparison to prior methods is needed to convince the readers for actual usage. There are many other methods (e.g. predicted entropy) for example difficulty estimation / classifier trustworthiness that need to be compared to.
- The stability of individual VoG scores needs to be investigated further

The authors are encouraged to address these limitations in the next iteration.
Decision: Reject