Deep Learning Through the Lens of Example Difficulty

1 Introduction . Much of the existing work on understanding deep learning “ integrates out ” the data , viewing the inductive bias of the model , or the properties of the optimizer as central to the success of the approach . Examples of such work include studies of eigenvalues of the Hessian and the geometry of the loss landscape ( Ghorbani et al. , 2019 ; Yao et al. , 2020 ; Sagun et al. , 2016 ; Li et al. , 2018 ; Pennington and Bahri , 2017 ; Sagun et al. , 2018 ) , studies of margin and effective generalization measures ( Long and Sedghi , 2019 ; Unterthiner et al. , 2020 ; Jiang et al. , 2020 , 2018 ; Kawaguchi et al. , 2017 ) and mean-field studies of stochastic optimization ( Smith et al. , 2021 ; Stephan et al. , 2017 ; Smith and Le , 2018 ) . However , in practice , we are rarely concerned with only the average behavior of a model . One pathway to understanding the principles that govern how deep models process data is to study the properties of deep models for data points with different “ amounts ” or “ types ” of example difficulty . There are a number of definitions of example difficulty in the literature ( E.g . see Carlini et al . ( 2019 ) ; Hooker et al . ( 2019 ) ; Lalor et al . ( 2018 ) ; Agarwal and Hooker ( 2020 ) ) . Two are particularly relevant to this work . Firstly , the probability of predicting the ground truth label for an example , when that example is omitted from the training set ( Jiang et al. , 2021 ) , which represents a statistical view of example difficulty . Secondly , the difficulty of learning an example , parameterized by the earliest training iteration after which the model predicts the ground truth class for that example in all subsequent iterations ( Toneva et al. , 2019 ) . This measure represents a learning view of example difficulty 2 . ∗Work completed as part of the Google AI Residency Program 2We expand on other notions of example difficulty in Appendix B . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . These notions suffer from two fundamental limitations . While early-exit strategies in computer vision ( Teerapittayanon et al. , 2016 ; Huang et al. , 2018 ) and NLP ( Dehghani et al. , 2018 ; Liu et al. , 2020b ; Schwartz et al. , 2020 ; Xin et al. , 2020 ) suggest predictions for easier examples require less computation , the above example difficulty notions do not encapsulate the processing of data inside a given converged model . Moreover , existing notions of example difficulty ( E.g . Carlini et al . ( 2019 ) ) provide a one-dimensional view of difficulty which can not distinguish between examples that are difficult for different reasons . In this paper , we take a significant step towards resolving the above shortcomings . To take the processing of the data into account we propose a new measure of example difficulty , the prediction depth , which is determined from the hidden embeddings . To escape the one-dimensional view of difficulty , we introduce three distinct difficulty types by relating the hidden embeddings for an input to high-level concepts about example difficulty : “ Does this example look mislabeled ? ” ; “ Is classifying this example only easy if the label is given ? ” ; “ Is this example ambiguous both with and without its label ? ” . Furthermore , we show how this enhanced notion of example difficulty can unify our understanding of several seemingly unrelated phenomena in deep learning . We hope that the results presented in this work will aid the development of models that capture heteroscedastic uncertainty , our understanding of how deep networks respond to distributional shift , and the advancement of curriculum learning approaches and machine learning fairness . These connections are discussed in Section 5 . Contributions Our main contributions are as follows : • We introduce a measure of computational example difficulty : the prediction depth ( PD ) . The prediction depth , illustrated in Figure 1 , represents the number of hidden layers after which the network ’ s final prediction is already ( effectively ) determined ( Section 2 ) . • We show that the prediction depth is larger for examples that visually appear to be more difficult , and that prediction depth is consistent between architectures and random seeds ( Section 2.2 ) . • Our empirical investigation reveals that prediction depth appears to establish a linear lower bound on the consistency of a prediction . We further show that predictions are on average more accurate for validation points with small prediction depths ( Section 3.1 ) . • We demonstrate that final predictions for data points that converge earlier during training are typically determined in earlier layers which establishes a correspondence between the training history of the network and the processing of data in the hidden layers ( Section 3.2 ) . • We show that both the adversarial input margin and the output margin are larger for examples with smaller prediction depths . We further design an intervention to reduce the output margin of a network and show that this leads to predictions being made only in the latest hidden layers ( Section 3.3 ) . • We identify three extreme forms of example difficulty by considering the prediction depth in the training and validation splits independently and demonstrate how a simple algorithm that uses the hidden embeddings in one middle layer to make predictions can lead to dramatic improvements in accuracy for inputs that strongly exhibit a specific form of example difficulty ( Section 4 ) . • We use our results to present a coherent picture of deep learning that unifies four seemingly unrelated deep learning phenomena : early layers generalize while later layers memorize ; networks converge from input layer towards output layer ; easy examples are learned first and networks present simpler functions earlier in training ( Section 5 ) . Experimental Setup : To ensure that our results are robust to the choice of architectures and datasets , we report empirical findings for ResNet18 ( He et al. , 2016 ) , VGG16 ( Simonyan and Zisserman , 2015 ) and MLP architectures trained on CIFAR10 , CIFAR100 ( Krizhevsky et al. , 2009 ) , Fashion MNIST ( FMNIST ) ( Xiao et al. , 2017 ) and SVHN ( Netzer et al. , 2011 ) datasets . All models were trained using SGD with momentum . Our MLP comprises 7 hidden layers of width 2048 with ReLU activations . Details of the datasets , architectures , and hyperparameters used can be found in Appendix A . Related Work : Our work uses hidden layer probes to determine example difficulty . We have discussed how our study relates to prior work on example difficulty . Hidden layer probes have also been used to study deep learning . Deep k-NN methods ( Papernot and McDaniel , 2018 ) determine their predictions and estimate their own uncertainties by comparing the hidden embeddings of an input to those of the training set . Cohen et al . ( 2018 ) showed that SVM , k-Nearest Neighbors ( k-NN ) and logistic regression probes achieve similar accuracies . However , they did not study the processing of individual data points nor did they relate the k-NN accuracy to notions of example difficulty . Alain and Bengio ( 2017 ) used linear classifier probes in the hidden layers to interrogate deep models and demonstrated that linear separability of the embeddings increases monotonically with depth . We provide a more detailed discussion of related work in Appendix B . 2 Prediction Depth : a Computational View of Example Difficulty . We discussed the statistical and learning views of example difficulty in Section 1 . In this section , we introduce a computational view of example difficulty parametrized by the prediction depth as defined in Section 2.1 . This computational view asserts that , for “ easy ” examples , a deep model ’ s final prediction is effectively made after only a few layers , while more layers are used for “ difficult ” examples . 2.1 Definition . Asserting that the final prediction is effectively determined in earlier layers of a model , before the output , we estimate the depth at which a prediction is made for a given input as follows 3 : 1 . We construct k-NN classifier probes from the embeddings of the training set after particular layers of the network , including the input and the final softmax . The placement of k-NN probes is described in Appendix A.5 . We use k = 30 in the k-NN probes . Appendix A.4 establishes that the k-NN accuracies we report are insensitive to k over a wide range . 2 . A prediction is defined to be made at a depth L = l if the k-NN classification after layer L = l − 1 is different from the network ’ s final classification , but the classifications of k-NN probes after every layer L ≥ l are all equal to the final classification of the network . Data points consistently classified by all k-NN probes are determined to be ( effectively ) predicted in layer 0 ( the input ) 4 . It is worth noting that the prediction depth can be calculated for all data points : both in the training and validation splits . This leads to two notions of computational difficulty : • The difficulty of predicting the ( given ) class for an input ( in the training split ) • The difficulty of making a prediction for an input , unseen in advance ( from the validation split ) We examine both notions of computational difficulty in this paper and use the distinction between them to describe different forms of example difficulty in Section 4 . 2.2 Prediction depth is a meaningful and robust notion of example difficulty . In this section we show that prediction depth agrees with intuitive notions of example difficulty and that it is consistent between different training runs and similar architectures . Prediction depth is higher for examples and datasets that seem more difficult If prediction depth is a sensible measure of example difficulty then we would expect the following sanity checks to be observed : 1 . Individual data points that are visually confusing or mislabeled should have larger prediction depths as compared to images that are clear examples of their class . 2 . Data points from tasks that are intuitively simpler should have lower prediction depths on average . Figure 1 shows that the prediction depth passes both of these sanity checks . Appendix C.1 presents additional images , providing further evidence for this claim . Prediction depth is consistent across random seeds and similar architectures Figure 2 shows that the prediction depth is highly consistent between different architectures and random seeds for all datasets . Perfect agreement is not expected as different deep learning algorithms have different inductive biases which affects the perceived difficulty of examples . We observe stronger correlation 3In the process of arriving at this definition of the prediction depth we considered several alternatives , including using the ground truth class in place of the predicted class and using logistic regression probes in place of k-NN probes . See Appendix E for a discussion on the choices we made in our definition . 4Implementation details can be found in Appendix A.6.1 . between prediction depth for ResNet18 and VGG16 , than between VGG16 and MLP . This may be explained by the fact that ResNet18 and VGG16 are both convolutional networks and we expect their inductive biases to be more similar to one another than to MLP .

Summary: This paper explores how  deep feed-forward or convolutional networks perform image classification tasks. The authors consider the layers of a trained network as a sequence of embeddings, from which nearest-neighbour predictors can be computed from the training examples and their predicted class. This allows one to compute successive class predictions for a given example, as it gets processed from the input layer to the final softmax. They introduce the Prediction Depth of an example as the earliest layer where its final classification is reached and stays without change.   Experimenting with three network architectures and four image datasets, the authors suggest that the Prediction Depth of an example is a good estimator of its accuracy, learning difficulty, and confidence (in term of distance both from the class separator and closest adversarial examples). They also propose that it is consistent, for a given data point, over different trained models and (to a lower degree) architectures.  Finally, they show how their findings relate to known properties of neural networks, and define a new typology of difficult examples in image classification.