Deep Learning Through the Lens of Example Difficulty

Abstract: Existing work on understanding deep learning often employs measures that compress all data-dependent information into a few numbers. In this work, we adopt a perspective based on the role of individual examples. We introduce a measure of the computational difficulty of making a prediction for a given input: the (effective) prediction depth. Our extensive investigation reveals surprising yet simple relationships between the prediction depth of a given input and the model’s uncertainty, confidence, accuracy and speed of learning for that data point. We further categorize difficult examples into three interpretable groups, demonstrate how these groups are processed differently inside deep models and showcase how this understanding allows us to improve prediction accuracy. Insights from our study lead to a coherent view of a number of separately reported phenomena in the literature: early layers generalize while later layers memorize; early layers converge faster and networks learn easy data and simple functions first.


Reviews and author responses:

 Thank you for reading our response. Below, we respond to your remaining concerns:

- Larger datasets: We agree there is always some value in more experiments and in particular having experiments on larger datasets. However, we want to emphasize that we have presented extensive empirical evidence on image classification datasets that are most commonly used in the field and we think we have provided enough evidence for our analysis. This detailed investigation was not possible for us on large datasets due to computational costs and limitations.

- Label Noise: We want to emphasize again that the empirical observations from prior work shows that the effect of label noise is NOT the same as limited data (eg. see [1] but also any training curve on well-known image classification datasets such as MNIST, CIFAR-10, CIFAR-100, SVHN, etc.). Therefore, we disagree with your statement.


Given that you find our paper to be novel and significant, and that we have addressed your major concerns, we respectfully believe that classifying our paper under "Marginally below the acceptance threshold" may not be really reflective of your current review. Therefore, we kindly hope that you would consider increasing your score and recommend accepting the paper.

[1] The Implicit Bias of Gradient Descent on Separable Data. D Soudry, E Hoffer, MS Nacson, S Gunasekar, N Srebro. The Journal of Machine Learning Research (JMLR) 2018.



 Thanks you for your comment, which indeed clarify some of my concerns. I am happy to raise my note to 5.

I still believe that more experimental evidence (esp. on larger datasets) would considerably strengthen your work.

About your comment on overfitting and label noise, I would suggest that it only applies to the asymptotic case (i.e. very large samples). When training from a small number of examples, sampling error (i.e. the "noise" resulting from the fact that your empirical data allow for a limited approximation of the underlying distribution) will act exaxctly as the label noise in the asymptotic case, and result in overfit. In particular, it will favor the (empirical) distribution of examples in the training set over that of the validation set (and in your case, this will be aggravated by the fact that your knn supports are the training examples). If it weren't the case, sample size would have no (or little) effect on generalization, which is not what we observe in practice. 


This paper explores how  deep feed-forward or convolutional networks perform image classification tasks. The authors consider the layers of a trained network as a sequence of embeddings, from which nearest-neighbour predictors can be computed from the training examples and their predicted class. This allows one to compute successive class predictions for a given example, as it gets processed from the input layer to the final softmax. They introduce the Prediction Depth of an example as the earliest layer where its final classification is reached and stays without change. 

Experimenting with three network architectures and four image datasets, the authors suggest that the Prediction Depth of an example is a good estimator of its accuracy, learning difficulty, and confidence (in term of distance both from the class separator and closest adversarial examples). They also propose that it is consistent, for a given data point, over different trained models and (to a lower degree) architectures.

Finally, they show how their findings relate to known properties of neural networks, and define a new typology of difficult examples in image classification.     **Originality** 

The paper relies on existing techniques and architectures. The authors provide references. Prediction depth is a new indicator, but related metrics have been proposed by others. The main original contribution is the use of prediction depth as an estimator of example difficulty, and the suggestion that it can summarize several important concepts, like prediction accuracy, model perplexity and learning speed. 

**Clarity**

The paper is complete, well organized, and provides a detailed bibliography. It could be made clearer by reorganizing the figures, so that they are closer to the sections they illustrate. Since the setting and use of kNN predictors over a trained model is central to this work, it would deserve a paragraph at the beginning of paper. Right now, the information is scattered  in section 2 and appendix A, which makes the paper difficult to understand on first read. Appendix A.3 features hyperparameter optimization, but more importantly describes the training schedules that were actually used on all datasets and architectures. The name of the section should reflect this, and learning curves for all datasets should be provided. In figure 5, there are epochs, how are they defined? 

The word "consistency" is used in figure 2 and 3 with different meanings. The paragraph starting on line 135 could be made clearer by clarifying between 'accuracy' and 'generalization'. Part C of the appendix is made of very short paragraphs referencing additional graphs: for instance, C.5 is just a reference to figures C31 to C34. It might be better to directly refer to the figures in the main. 

**Quality** 

The ideas and intuitions proposed in the paper are very interesting, and definitely worth pursuing. However, the supporting experiments are not always convincing, and several questions about the experiments, the definition of prediction depth, and the experimental support of claims, need to be clarified. I group them into four concerns that, I think, should be addressed.  

**Concern 1: Overfit:** 
All the datasets feature about 50 000 examples. Appendix A.3 states that training used about 10 000 steps, over batches of 256 examples. This means each example in the dataset was seen about 50 times. This, together with the use of large learning rates and no weight decay, might raise concerns about overfitting. Also, figure 5 displays a learning curve (actually, an average of five experiments) which shows a test set accuracy going up until epoch 20, and then falling down to an asymptote, while training set accuracy goes on improving. This, again, might signal overfit. Finally, in several cases, computing prediction depths on training or validation samples result in different conclusions. This might be an important finding, but it is also a telltale sign of an overfitting model. Can overfit be ruled out in these experiments? This can be done in a number of ways: learning curves, cross-validation, switching to larger datasets such as imageNet, training models on less steps and seeing if the results still hold...

Making sure that the model does not overfit is all the more important as the training examples serve as the basis for the kNN predictors, and memorisation of training data will influence prediction depth computation. This could be mitigated by using held-out examples as the basis for the kNN - i.e. compute the nearest neighbours of examples not seen at training, e.g. from the validation set, and see whether this results in different prediction depths (more on this in the next paragraph).

**Concern 2: Robustness of the definition of prediction depth:** 
Prediction depth is defined by two criteria: "first correct prediction" and "no change until output", how important is the second? In other words, if an example is correctly classified from layer 2 to 10, except on 9, should it really be depth 10? An analysis of whether such cases exist, and whether the conclusions are robust to such changes in the definition of prediction depth, would be very interesting, and make prediction depth a more robust idea.

kNN are computed from the training examples. This will not scale to larger training sets. Besides, the use of training data might raise concerns with overfitting, or the relation between prediction depth and the distribution of training data. How robust is the estimation of prediction depth with respect to downsampling the kNN base (e.g. from 50k to 10k), or to using different examples (e.g. the validation set) as the kNN base? Also, a discussion about the practical aspects of computing prediction depth would be very interesting.

**Concern 3: Experimental support for claims:** 
The first claim in 2.2 seems supported by 8 examples, and 4 datasets. This is very weak. Can other examples of intuitively hard or easy examples and datasets be constructed? Maybe by using predictions from a very simple baseline classifier as a measure of example difficulty, and adding increasing amount of noise on one sample (as was done in CIFAR 10) and seeing how this impacts prediction depth?

For consistency over architecture (fig. 2), predictions depths are averaged over an ensemble of different models. These averages are then shown to be correlated over different architectures. This is an important result, but unless the variance of prediction depth for different models with the same architecture is very small, it does not prove consistency. Showing that prediction depth is robust to changes in the model and the architecture is required to support the claim that prediction depth measures example difficulty, independently of the model used to compute it. 

The visual analysis of correlation between accuracy or learning speed and average prediction depths is complicated by the fact that many distributions tend to concentrate around a few (usually extreme) values. Since many distributions are bimodal, it might be advantageous to cluster the results into high and low values, and compute simpler, binary, statistics. This would be easier to follow, but not less relevant.

**Concern 4: Example difficulty vs nearest neighbours:** 
Prediction depth of an example is a function of the predictions of its nearest neighbours, according to the embeddings defined by  each layers. When a low prediction depth is observed it might be because, after each layer, the (potentially new) neighbours of the examples always predict the same class. I believe this is the original intuition for prediction depth. But it could also be because the nearest neighbours just remain the same from layer to layer, because the point sits in a less densely sampled part of feature space (with respect to the base vectors of kNN predictors). Classifying such examples as 'low difficulty' is counter-intuitive, and this might be a weakness in the definition of prediction depth. Is it possible to measure how frequently this happens? Intuitively, the high dimension of layer embedding should make this situation frequent, but the fact that training vectors often lie close to a low dimension manifold mitigates this. 

If such a problem occurred, solutions could be to vary the kNN base vectors from layer to layer, or to compute the predicted class from a random sample of the k nearest neighbors (e.g. take the 30 nearest, sample 15 randomly, and compute the class from them).

**Significance**

Despite the experimental concerns, this is an important paper, which potentially impacts several fields of research, from semi-supervised learning to interpretability. Improving the experiments will result in a very good paper, that definitely fits in a major conference.

**Overall**

The ideas proposed in this paper are important and interesting, and the suggestion that prediction depth in a trained model can serve as a useful proxy for interpreting data complexity is very thought-provoking. Unfortunately, the definition of the prediction depth and some of the authors' claims should be better supported by the experiments. Also, some of the results might be tainted by overfitting over a small dataset and use of training data as the basis of nearest neighbour calculations.

I believe more experimental confirmation is needed before this paper can be published. Hence my rating of 4, which I would be **very** happy to increase if better experimental support of claims and definitions is provided.  

**Typos**

Bibliographical references do no seem to follow the NeurIPS formatting

l. 19 : behaviour

l. 67: NLU (Natural Language Understanding?)

l. 163: of in a particular

l. 156: the the

l.198: ref should be C4 ? Limitations are discussed. It might be worth stressing the fact that the approach is specific to feed-forward and convolutional networks performing classification tasks, over supervised datasets.  

 We would like to thank you for your careful reading of our work and providing valuable and encouraging feedback! Below we address all your concerns and in that light, we hope you would consider increasing your score:

1. **Prediction depth for training data**: Prediction depth for training split is plotted nearly as often as the validation split. Figure 1 shows results from the training split. Figure 2 (left) shows both splits, Figure 2 (right) is reproduced for both splits in all data sets in Appendix C.2. Figures 3 and 4 naturally address the validation split since they are focused on understanding the relationship between prediction depth and generalization. Figure 5 (left) is replotted as individual 2d histograms in Appendix C.4 for both data splits in all combinations of data set and architecture. Figure 5 (middle and right) shows both splits. Figure 6 (left and middle) is reproduced in Appendix C.5 for both splits in all data sets and architectures, and Figure 6 (right) describes both splits. Figure 7 explicitly compares prediction depth in both splits. Figure 8 shows only the validation split. In most cases the plots for the two splits are rather similar and for lack of space we did not describe them both. In the final version, we will make the pointers to plots of the training split clearer, and describe those plots in greater depth, particularly those occurring in the appendix.

2. **Too much information in Fig 3**: Thank you for highlighting this. One of the main challenges we faced during writing this paper was presenting as much of the insightful information that we found in our study as possible given the limitations of the conference format. Consequently, the paper and figures are dense. We will improve the captions to describe the density in the histogram and the marker sizes. as suggested. Please let us know if you have suggestions for how to improve the figures further. 

3. **Example difficulty and the prediction depth**: In the paper we show how prediction depth is connected to multiple other notions of example difficulty and (in Figure 1) that it corresponds to intuitive notions of example difficulty. It is interpretable as a notion of difficulty: “Networks use more layers to determine outputs for harder examples.” In Figure 7 we connect prediction depth to a more detailed view of example difficulty. We believe that this is sufficient to justify describing prediction depth as a notion of example difficulty.

4. **Choosing appropriate titles**: Thank you for this valuable feedback! In Section 2 we introduce the prediction depth as one notion of example difficulty. In Section 3 we compare the prediction depth to other notions of example difficulty. This is the distinction between sections 2 and 3, which we are happy to clarify in the text. We will additionally improve the subsection headings in Section 3, as follows: 3.1-->Depth of a prediction gives a linear lower bound on its consistency. 3.2-->The prediction depth of an input is correlated with its learning difficulty. 3.3-->Deep models exhibit larger margins for inputs with lower prediction depth.


**Re open question**: We absolutely agree with your intuitions! We believe that understanding how intermediate layers transform difficult examples can play an important role in discovering the reasons behind the success of deep learning. We are excited to see how future work in this direction would build on the techniques and notions defined in this work.

**Final note**: We hope that you are satisfied with our response, and the commitment to 1. enhance pointers to plots of the prediction depth in the training split and also their written descriptions, 2. Clarifying the structure of the paper in sections 2 and 3, and improving the titles of subsections in Section 3. If you are satisfied with our response we hope you consider increasing your score.

 Thank you for careful reading of our paper and providing detailed feedback! We are glad that you find our paper complete, well-organized, novel and significant! We will take your comments on the clarity of the paper into account for the final version of the paper. Below, we address all your concerns regarding the experiments:

1. **Overfitting**: 

We completely agree that overfitting could potentially affect the results in such experiments. However, we want to reassure the reviewer that when running experiments we have checked the learning curves to make sure such overfitting does not happen. Furthermore, we are providing the learning curves for some architectures and data sets for you to get a sense of how they look: https://github.com/neuripsfileshare/Files. These learning curves demonstrate that overfitting did not occur (the final test accuracy is the highest one). Please let us know if you’d be interested in seeing more such curves for any specific experiment. 

We also want to briefly discuss the reasons you mentioned for your concern about overfitting:

- Large number of epochs, large learning rate, no weight decay: There is increasing empirical and theoretical evidence that in settings with no or small label noise, there is no overfitting due to implicit regularization of (stochastic) gradient descent. For example, learning curves in Fig 5 in [1] show no overfitting when the experiments are run for a long time with no weight decay and no label noise (similar to our case). Moreover, theoretical and empirical results in [2] suggest that running gradient descent with no weight decay for a long time would improve the margin and hence the performance.

- Overfitting in Fig 5: Fig 5 is the only experiment in our paper with label noise. As we discussed above, the overfitting is expected in the case of label noise but the empirical evidence suggests that the datasets we used in our experiments do show overfitting when the label noise is not added explicitly. 

- Different behavior of prediction depth for training and validation points: This is expected as the training and test performance differ. However, that does not indicate that the test acc has necessarily degraded toward the end of the training. This is also consistent with the learning curves. 


 
2. **Robustness of the definition of prediction depth**: 

We agree with you about the importance of verifying the robustness of prediction depth. That is why we discussed its robustness in Section 2.2 and showed that it is robust to the choice of random seed and similar architectures. We next discuss your questions about some of the choices we have made in our definition. Please also see Appendix E where we discuss several alternative definitions and considerations that were taken when developing the definition of prediction depth.

- No change until output: We do not expect this choice to be important but we chose this to be consistent with other measures of example difficulty. For example, “iteration learned” is defined in a similar way [3]. Furthermore, our results in Section 2.2 shows that this definition is robust to the choice of random seed.

- Using validation data for KNN base: This is a very interesting question and a root of one of the key design choices that we made when defining the prediction depth, which allows us to make the analysis presented there. In the paper we stress the distinction between the ground truth class and the consensus class of an input. For this analysis it is important to use the training split as the support set, which unifies the ground truth and predicted classes of the support set. Using the validation split as a support set separates the predicted and ground truth classes of the support set, which significantly complicates the analysis. This was one of the considerations taken in devising the prediction depth. We are happy to clarify this point, adding it to the discussion in Appendix E.

- Subsampling of the support set for KNN: This is a very interesting point. Note that the impact of more training data appears mostly in better representation learning in different layers and one can fix the size of the support set of KNN (say 10000) even when the training data is large. Therefore, subsampling of the support set ought not to change the overall conclusions of the study. We are happy to also clarify this in the Appendix.


3. **Experimental support for claims**:

Below we address all your three concerns about experimental support:

- Easy and difficult examples:  We are happy to add more evidence for the statement that the easiest and hardest examples identified by the prediction depth (examples shown in Fig. 1) correspond to intuitively easy and hard examples. At https://github.com/neuripsfileshare/Files we show examples (not hand-picked) from 2 randomly chosen classes for each of the four data sets for ResNet18. We are happy to add these to the appendix to strengthen the paper. We already report the large majority of our results for every combination of 4 data sets and 3 architectures in the Appendix, demonstrating repeatability of our observations for different architectures and data sets. The four data sets used are popular benchmarks in image classification. 

- Consistency over architecture: The results presented in Fig. 2 (left) are not averaged over ensemble of models but it shows the correlation between two different architectures trained with different random seeds. Please also note that diagonal comparisons between an architecture and itself show the correlation for the same architecture trained with different random seeds.

- Visual analysis: While we believe our plots are more informative, we agree that your suggestion can be useful in terms of simplifying the presentation and would make the message more understandable. We are happy to also present the results as you suggested in the final paper. 


4. **Example difficulty vs nearest neighbors**: 
Thanks for engaging so deeply and asking such good questions! While this scenario is possible, we believe it is very unlikely. The reason is that in high dimensions it is very unlikely to have a set of points in less populated areas that are significantly closer to each other than other points. Moreover, such points are less likely to have similar labels. The likely scenario is that points in less populated areas have non-robust neighbors with different labels. We also observed this in our visual inspections of easy and hard examples.

**Final Note**

We want to thank you for such exemplary and detailed comments. Given your understanding about the novelty and significance of our work and that we have addressed all your concerns, we hope that you increase your score and strongly support accepting our paper.


**References**

[1] Deep double descent: Where bigger models and more data hurt. P Nakkiran, G Kaplun, Y Bansal, T Yang, B Barak, I Sutskever. International Conference on Learning Representations (ICLR) 2019.

[2] The Implicit Bias of Gradient Descent on Separable Data. D Soudry, E Hoffer, MS Nacson, S Gunasekar, N Srebro. The Journal of Machine Learning Research (JMLR) 2018.

[3] An Empirical Study of Example Forgetting during Deep Neural Network Learning. M Toneva, A Sordoni, R Tachet des Combes, A Trischler, Y Bengio, Geoffrey J Gordon.  International Conference on Learning Representations (ICLR) 2019.

 We would like to thank you for your thorough reading of our manuscript including the appendix. We are excited that you find the prediction depth to be novel, interesting and useful! 

We further agree with your understanding of the main contributions of the paper and that there are many applications stemming from the prediction depth metric including those discussed in the paper. We look forward to seeing how the prediction depth is used to develop new insights and algorithms in the future!

Intuitively, some examples are "harder" to learn than others.  Currently, there are two ways of measuring "hardness": the "statistical view", in which an example is hard if its classification is sensitive to small changes to the training set; and the "learning view", in which an example is hard if its label is learned only at the end of the optimization process.  This paper proposes a new "computational view," in which an example is considered hard if many layers of processing are required to produce the prediction.   Computational hardness is formalized through a metric called "prediction depth," which is the depth at which the prediction of a KNN classifier operating on the intermediate hidden representations starts to agree with the network's ultimate prediction.  For instance, consider a 20-layer trained network and a training example for which the network predicts "airplane."  Suppose that a KNN classifier operating on the first-layer hidden representations predicts this example as "dog," and a KNN  classifier operating on the second-layer hidden representations predicts this example as "cat", while KNN classifiers on layers 3 through 20 predict this example as "airplane."  Then the prediction depth is 3.  The intent is: the higher the prediction depth, the harder the example.

The paper first confirms that prediction depth is robust across random seeds and relatively robust across architectures, and that visually 'easy' images have low prediction depths and visually "hard" images have high prediction depths.  Then the paper demonstrates that the "computational hardness" of prediction depth is correlated with the "statistical hardness" of sensitivity: examples with low prediction depths tend to be classified the same regardless of perturbations to the training dataset (e.g. bootstrap resampling), whereas examples with high prediction depths are not robust to changes in the training dataset.  Then the paper demonstrates that the "computational hardness" of prediction depth is correlated with the pre-existing notion of "learning speed" hardness: examples with low prediction depths tend to be learned earlier in training, whereas examples with high prediction depths are not learned until late in training.  Then the paper demonstrates that examples with low prediction depths have lower 'output margins' (the difference in logit between the predicted class and the other classes) as well as lower 'input margins' (the L2 distance in input space to the nearest adversarial perturbation).  

  Prediction depth seems like an interesting metric for measuring hardness of examples that could become a nice tool in the deep learning literature.  The method is original (to my knowledge), and the paper is very clearly written.  Appendix D lists some potential applications of this metric, and it's not hard to imagine others. Yes

Authors proposed a measurement named prediction depth for each data point, that is defined as the index of a layer after which a KNN classifier gives the same predictions as the final prediction of the network. Experimental evidence shows that the prediction depth is correlated to the example learning difficulty and visualizes examples from four extremes, classified by the predict depth in training and the one in validation.  The proposed method is novel and proved to be capable of estimating the example difficulty. It also has the potentials to explain some phenomena in deep learning, as discussed in Section 5. But there are several points that I didn't understand and found strange:
- There is no analysis on prediction depth for training data. Each figure is based on the random splits of training and validation sets, and the prediction depths are averaged when the data point appears in the validation set. As authors stated, there are "two notions of computational difficulty", but only one notion is analyzed. 
- I found it very difficult to understand Figure 3 and related figures, which contain too much information. The captions of figures are quite long and difficult to understand. Moreover, the histograms (the color bars), representing the frequency of correct predictions, have not been much discussed. As well, the size of markers in Figure 4, showing the fraction of the data set, has not been discussed either. It takes me much time to understand these presentations but they were not very useful. 
- It seems like the example difficulty is just one intuitive notion. Authors may need a better way to show its agreement to the proposed prediction depth.
- All Section 3 seem to talk about the example difficulty, but from different views. Some more concise subtitles, or a better organization may help readers to better understand the content. The end of Section 2 is of the same subject as Section 3, but is in another section. 

Overall, this paper introduces a novel notion, which relates to the example difficulty and shows the potentials to connecting some phenomena in deep learning, while some details and the clarity need to be improved.  

One open question inspired by this paper: If the first layer can be quite discriminative for classifying the objects, what do the rest of layers do? Imagine this, some easy examples are already clustered together in the early layer while some difficult ones are not. At the late layer, these easy examples are still clustered but the difficult ones join them. This may be very interesting to analyze the transformations done by intermediate layers for the difficult examples. yes

Meta-review: This paper proposes a new metric of example "hardness" with a computational flavor---the number of layers of representation processing before which a KNN classifiers begins to agree with the correct label. The reviewers all agree that the paper is interesting, clear, and introduces a suitably novel and important idea. While the reviewers disagree about the sufficiency of the experiments, they all see positive aspects to the papers and none argue for rejection. The authors have received considerable feedback throughout this process and I hope that they will strengthen the paper accordingly for the camera ready version. 

Decision: accept