Title: Sample Importance in Training Deep Neural Networks
Authors: Tianxiang Gao
Vladimir Jojic
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning

Abstract: The contribution of each sample during model training varies across training iterations and the model's parameters. We define the concept of sample importance as the change in parameters induced by a sample. In this paper, we explored the sample importance in training deep neural networks using stochastic gradient descent. We found that "easy" samples -- samples that are correctly and confidently classified at the end of the training -- shape parameters closer to the output, while the "hard" samples impact parameters closer to the input to the network. Further, "easy" samples are relevant in the early training stages, and "hard" in the late training stage. Further, we show that constructing batches which contain samples of comparable difficulties tends to be a poor strategy compared to maintaining a mix of both hard and easy samples in all of the batches. Interestingly, this contradicts some of the results on curriculum learning which suggest that ordering training examples in terms of difficulty can lead to better performance.


Review discussion:
Can the authors convince me that the overall importance is a sensible measure ? If g_i^t > g_j^t, then clearly example i has a greater impact on \theta_{t+1} then x_j (ignoring issues of Euclidian vs Fisher metric). The magnitude of gradients will change significantly during learning however, and I am not sure what conclusions one can draw from \sum_t g_i^t vs \sum_t g_j^t.
For example, gradients tend to have higher norms early in training than at convergence, in which case weighting each gradient equally is problematic.
Furthermore, a high learning may not even convergence and instead oscillate between local minima, in which case the "total/overall importance" would be meaningless.
The overall sample importance is not inflated by the oscillation between local minima.
Firstly, we observed that the training objective is continuously dropping even at the last epoch. There is no oscillation in the training objective.
Secondly, we tracked the gradients across the iterations for each sample. The gradients computed on easy samples converge to zero quickly.
Hence, easy samples’ importance drops in later epochs. The gradients computed on hard samples keep the same sign until the end of the training. As we are using early stopping for training, the objective is far from reaching a local minimum where we could see an oscillation effect.
In Figure 4, we show that overall sample importance is correlated with (but not the same as) the training negative log likelihood.
Specifically, change in the parameter due to a sample’s presence is not equal to its negative-log-likelihood.
Overall sample importance is a coarse measurement of the total change in parameters contributed by a sample. In section 3.6, we found that ordering batches based on overall sample importance do not reduce the test error comparing to regular (random) batch ordering.
However, the key observation in our paper is that epoch-specific sample importance to each layer.
Specifically, the hard samples have the biggest impact on shaping the parameters in lower layers -- layers close to the input. This epoch-layer-specific sample importance is a direct measurement of the sum of squares of the change in parameters in each epoch. The clusters in Figure 5 and 6 are clustered based on epoch-layer-specific sample importance.
Can you first confirm that graphs with solid colors (such as the bottom-right graph of figure 2) are cumulative.
in other words,  the right interpretation of the bottom right graph of figure 2 is (i) that  importances are approximately equal for every class (The legend seems to contradict this) and not (ii) that trucks are 8 times more important than airplanes. Do all solid-fill graphs follow this convention ?
It seems intuitively normal that output layers be trained first since the gradient flows backwards.
Also it is to be expected that easy samples are learned before that hard ones.
Do you think this would be a reasonable explication of the results you observe ? If not why ? Putting aside the last results (which tend to show that curriculum learning is probably a bad idea, something we can agree on), would you give a different explication for your results ? (what explication and why of course)
The graphs with solid colors are cumulative and it is a decomposition of the overall importance. The right interpretation of the bottom right graph of figure 2 is (i) importances are approximately equal for every class. Trucks and cars have approximately equal sample importance. All the solid-fill graphs follow this convention.
Your explication is correct. The output layers are trained first since the gradient flows backward.
It is true that easy samples are learned before the hard ones. This makes our main observation from the result: hard samples are the most important factor in shaping the parameters in low-level layers.
With the standard cross-entropy loss for discriminative classification, the log-loss is q*log(p) and the gradient is p-q, where p is predicted and q is the target.
so when NLL or log-loss is high, the gradient or SI should also be high.
Therefore, can the authors clarify why this is observed in practice:  "NLL is not predictive of the SI".
To establish terminology we just go over some details of back-prop. We also include a specific example where NLL and SI are very different. Please forgive the verbosity.
The negative log-loss is -L = -q*log(f(x,theta)), in classification task, q=1 for the true label, and f(x,theta) is the predicted probability of the true class, where x is the input, theta is the parameter. The gradient, in this case, can be written as: g = -1/f *df/dtheta. Using back-propagation, we know that the gradient for a particular layer l is df/dw^l = C^L * w^L * C^{L-1} * w^{L-1} * … * C^{l} * h^{l}, where C^l is the derivative of the activation function in the lth layer, and h^l is the input to the lth layer.
Hence, the gradient for a particular parameter can be decomposed into the product of negative inverse probability -1/f, activation derivatives C^L … C^{l+1} before lth layer and parameters w^L … w^{l+1} before lth layer and the input to the lth layer, h^l.
This indicates that the gradient is not only related to NLL but also related to 1) the parameter weight that scales w in each layer in the current network, 2) the activation pattern (C and h) for particular samples. Here we present two examples for these two situations:
1) NLL can be different from SI in different epochs as parameters are changing. An example is Figure 5 “easy samples” cluster. In the very early epochs, NLL is large, but all the weights are initialized to be small.
Hence, the gradients, which contains the product of the parameters, are small. As training goes on, NLL becomes smaller, but parameters are becoming large.
Hence, there is an increase in SI in the middle early stage.
When the NLL is converging close to zero, and there is no room for gradient to improve the objective, f becomes very close to 1, and output softmax function is saturating (for example, a binary softmax is equivalent to sigmoid function, which has derivative df/dx = f(1-f), and it will be close to 0 if f is close 1), and SI will be small again. This example illustrated that NLL is not predictive of SI when parameters are different.
2) NLL is not predictive of SI when the sample input is different. As we noted, the activation pattern and input will also affect the gradient magnitude even when parameters are fixed.
Hence, a sample with larger NLL might have smaller SI comparing to another sample, even if the parameters remain the same. Here is a simple synthetic example to illustrate this: Let us consider a simple binary linear logistic regression, where f(x,w) = sigmoid(x*w). The gradient for w is: g = dNLL/dw = -1/f * f(1-f) * x. (suppose this is a positive class sample). Let’s consider a setting, where w = [-1000,-0.1], and two samples x_a = [0.01,0], x_b = [0,10].
Hence, NLL_a  = - log sigmoid(x_a*w) = -log sigmoid(-10) ≈ 10, which is a very large value, while the SI_a = (g_a)^2 = (-1/sigmoid(-10) * (sigmoid(-10)*(1-sigmoid(-10)))*0.01)^2  ≈ (-0.001)^2 = 1e-6, which is a very small value.
However, for x_b, NLL_b =  - log sigmoid(x_b*w) = -log sigmoid(-1)  ≈ 1.31, and SI_b = (g_b)^2 = (-1/sigmoid(-1) * (sigmoid(-1)*(1-sigmoid(-1)))*10)^2  ≈ (-7.3)^2 approximately 53, which is very large. This example shows that sample with high NLL can have small SI, and sample with small NLL can have large SI.
(paper summary) The authors introduce the notion of “sample importance”, meant to measure the influence of a particular training example on the training of a deep neural network. This quantity is closely related to the squared L2 norm of the gradient, where the summation is performed over (i) parameters of a given layer or (ii) across all parameters. Summing this quantity across time gives the “overall importance”, used to tease apart easy from hard examples. From this quantity, the authors illustrate the impact of [easy,hard] example during training, and their impact on layer depth.
(detailed review)
I have several objections to this paper.
First and foremost, I am not convinced of the “sample importance” as a meaningful metric. As previously mentioned, the magnitude of gradients will change significantly during learning, and I am not sure what conclusions one can draw from \sum_t g_i^t vs \sum_t g_j^t.
For example, gradients tend to have higher norms early in training than at convergence, in which case weighting each gradient equally seems problematic. I tried illustrating the above with a small thought experiment during the question period: “if” the learning rate were too high, training may not even converge in which case sample importance would be ill-defined.  Having a measure which depends on the learning rate seems problematic to me, as does the use of the L2 norm. The “input Fisher” norm, \mathbb{E} \frac{\partial \log p} {\partial x} (for a given time-step) may be better suited, as it speaks directly to the sensitivity of the classifier to the input x (and is insensitive to changes in the mean gradient norm).
But again summing Fisher norms across time may not be meaningful.
The experimental analysis also seems problematic. The authors claim from Fig. 2 that output layers are primarily learnt in the early stage of training.
However, this is definitely not the case for CIFAR-10 and is debatable for MNIST: sample importance remains high for all layers during training, despite a small spike early on the output layer. Fig 2. (lower, middle) and Fig. 6 also seems to highlight an issue with the SI measure: the SI is dominated by the input layer which has the most parameters, and can thus more readily impact the gradient norm. Different model architectures may have yielded different conclusions. Had the authors managed to use the SI to craft a better curriculum, this would have given significant weight to the measure. Unfortunately, these results are negative.
PROS:
+ extensive experiments
CONS:
- sample importance is a heuristic, not entirely well justified
- SI yields limited insight into training of neural nets
- SI does not inform curriculum learning
Sample importance is a direct measurement of parameter changes that contributed from a sample at a specific iteration during training. The reviewer claims that “gradients tend to have higher norms early in training than at convergence”, which is true for “easy” samples, but not for “hard” samples. Summing the gradients norm across time is just one of several the measurements we use and it is not the focus of our paper. To study the different contribution from samples to different parameters at different training stages, we measured the stage-wise, layer-wise sample importance.
For example, one of the most important findings is that the parameters in lower-level layers are changed more by “hard” samples during late stage.
The “input Fisher” norm suggested by the reviewer, is related to the saliency [1] in computer vision. It speaks directly to the sensitivity of the classifier to the input features rather than the existence of an input. In our paper, we care about how the change in the network parameters depends on the contribution of input during training.
However, the sensitivity of the network output with regard to the input reveals only the current state of the network, but nothing about the network training.
The average sample importance of output layer reach maximum during early stage and keeps dropping as training goes on (Fig 2, left figures).
Hence, we have the argument of parameters in output layers are primarily learned in the early stage of training. We can see that the later in the training stage, the smaller the average sample importance in output layer parameters. For SI measure in Fig.6, we care more about the layer-wise, stage-wise specific sample importance rather than overall sample importance. The most important message from the Fig.6 is that the sample importance has different stage-wise patterns in different sample groups.
[1] Simonyan, Karen, Andrea Vedaldi, and Andrew Zisserman. "Deep inside convolutional networks: Visualising image classification models and saliency maps." arXiv preprint arXiv:1312.6034 (2013).
This paper examines the so called "Sample Importance" of each sample of a training data set, and its effect to the overall learning process.
The paper shows empirical results that shows different training cases induces bigger gradients at different stages of learning and different layers.
The paper shows some interesting results contrary to the common curriculum learning ideas of using easy training samples first.
However, it is unclear how one should define "Easy" training cases.
In addition, the experiments demonstrating ordering either NLL or SI is worse than mixed or random batch construction to be insightful.
Possible Improvements:
It would be nice to factor out the magnitudes of the gradients to the contribution of "sample importance". Higher gradient (as a function of a particular weight vector) can be affected weight/initialization, thereby introducing noise to the model.
It would also be interesting if improvements based on "Sample importance" could be made to the batch selection algorithm to beat the baseline of random batch selection.
Overall this paper is a good paper with various experiments examining how various samples in SGD influences the various aspect of training.
The “Easy” vs “Hard” samples have been discussed in Curriculum Learning (CL) [1] and Self-paced Learning (SPL) [2]. In CL, “easy” and “hard” samples are defined manually, or by the classification error on a pre-trained classifier. In SPL, “easy” and “hard” samples are defined based on their training error at each training step. Most of the previous methods defined the “easy” and “hard” based on an error-based metric. In our paper, we made a comparison between the samples with similar sample importance pattern over the training and their training error. We treat the samples with small training NLL as “easy” samples and large training NLL as “hard” samples. We found that “easy” samples contributes more during early stages and in higher layers while “hard” samples contribute more during late stages and in lower layers.
Sample importance is the square of the parameter change contributes from a particular sample. It directly measures how much the parameters are changed due to the existence of a sample at a particular iteration. In FIg.1 we showed that Sample importance is preserved between initializations of the network. The noise introduced to the model from different initialization is very small.
[1] Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48. ACM, 2009.
[2] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pp. 1189–1197, 2010.
The paper proposes a new criterion (sample importance) to study the impact of samples during the training of deep neural networks. This criterion is not clearly defined (the term \phi^t_{i,j} is never defined, only \phi^t_i is defined;
Despite the unclear definition, it is understood that sample importance is the squared l2 norm of the gradient for a sample i and at time t strangely scaled by the squared learning rate (the learning rate should have nothing to do with the importance of a sample in this context).
The paper presents experiments on the well known MNIST and CIFAR datasets with correspondingly appropriate network architectures and choice of hyper-parameters and initialisations. The size of the hidden layers is a bit small for Mnist and very small for CIFAR (this could explain the very poor performance in figure 6: 50% error on CIFAR)
The study of the evolution of sample importance during training depending on layers seems to lead to trivial conclusions
- “the overall sample importance is different under different epochs” => yes the norm of the gradient is expected to vary
- “Output layer always has the largest average sample importance per parameter, and its contribution reaches the maximum in the early training stage and then drops” => 1. yes since the gradient flows backwards, the gradient is expected to be stronger for the output layer and it is expected to become more diffuse as it propagates to lower layers which are not stable. As learning progresses one would expect the output layer to have progressively smaller gradients. 2. the norm of the gradient depends on the scaling of the variables
The question of Figure 4 is absurd “Is Sample Importance the same as Negative log-likelihood of a sample?”.
Of course not.
The results are very bad on CIFAR which discredits the applicability of those results.
On Mnist performance is not readable (figure 7): Error rate should only be presented between 0 and 10 or 20%
Despite these important issues (there are others), the paper manages to raises some interesting things: the so-called easy samples and hard samples do seem to correspond (although the study is very preliminary in this regard) to what would intuitively be considered easy (the most representative/canonical samples) and hard (edge cases) samples.
Also the experiments are very well presented.
\phi^t_{i} is a vector of parameter affectability, which is the contribution of ith sample to the change to all the parameters in iteration t.
Specifically, \phi^t_{i,j} is the parameter affectibility to the jth parameter in the network from sample i. We are sorry for this unclear definition and we have revised this in the new version.
Q_d has already been defined in the original text in page 3 line 6, “where Q_d is a set consists of the indexes of all parameters in layer d.”
We defined the sample importance as the l2-norm of the change in the parameters contributed from a sample at a specific iteration, of course, the sample importance is affected by learning rate as it reflects that change in parameters at a particular iteration from the sample. In order to make the SI between different training stages comparable, we used fixed learning rate.
Hence, the different behaviors of SI in different training stages are not affected by learning rate.
In order to make the analysis comparable between MNIST and CIFAR, we only use feed forward neural network for analysis, as feed forward neural network is the most basic and general structure for deep learning. We will generalize our results and findings further for other architectures like convolutional networks.
These are two direct observations in our paper.
However, they are not the focus of our paper. We focused more on the samples that contribute more to different layers and different training stages.
For example, in Fig. 5 and Fig. 6 we found that “easy” samples contributes more to the early stage of training and “hard” samples contributes more to the late stage of the training. These observations are not easily obtained without our experiments.
The sample importance is not the same as the NLL of a sample, but it becomes more correlated during late stage as we shown in Figure 4. Intuitively, if the NLL of a sample is large, there is much room for the network to adjust its parameters to reduce the NLL of this sample, while if the NLL is small or close to zero, there is a very tiny room for network to fit this sample as it is already perfectly trained.
Hence, such question is not intuitively obvious, like the first question raised by AnonReviewer1, which demands an example for the case where SI can be not related to NLL.
Thanks for your suggestion. We have revised the Fig.7 to the new scale in the new version.
Following your comment, I consider the issue of clarity of the definition resolved.
On the question of the dependance to the learning rate: Your criterion depends on the learning rate which does not seem to make sense especially since it could very easily not depend on it. The fact that the learning rate is fixed in your experiments is not the point. It seems to me that the criterion is (unnecessarily) flawed for all future studies and that the solution is very simple. Do you agree that the learning rate has nothing to do with what you are trying to measure ? If you do, why would you want to keep it ?
On the question of the poor results: You suggest exploring other architectures such as convolutional neural networks. This is not the point. With simple feed forward networks the performance you have on MNIST and CIFAR should be higher. The fact that it is so low suggests that you did something wrong during training and raises serious questions about the validity and applicability of your results. To be clear, your results on MNIST should be at the very least 97% correct and on CIFAR 65% correct (the results of the first study on CIFAR dating back to 2009, i.e. 7 years ago, should not be that hard to beat)
On the question "Is Sample Importance the same as Negative log-likelihood of a sample?". The fact that both are correlated is not the point. If your want to study the relation between NLL and SI that is perfectly fine.
However it is absurd to ask whether both quantities are the same which they very obviously are not. Asking such an absurd question discredits your work and this conference if you are allowed to publish in it.
We believe that we have miscommunicated the intent of the experiments in Figure 4, through careless summarization.
Specifically, the word "same" is jarring.
We agree that the question of whether the NLL and SI are equal can be trivially answered. They are definitionally distinct mathematical objects. One is a function and the other is a sum of the squares of the gradients of that function.
What we aimed to answer is whether the NLL is highly predictive of sample importance.
In the experiments, NLL becomes correlated with the SI in the late stage of training, but not so in the early ones. We believe this is an interesting observation.
To address the concern of the reviewer, we changed the manuscript to say "correlated" rather than "same". This reflects the intent of the experiment.

Meta-review:
role: PC
data: comment: The reviewers provided detailed, confident reviews and there was significant discussion between the parties. 
 
 Reviewer 2 and 3 felt quite strongly that the paper was a clear reject. Reviewer 1 thought the paper should be accepted.
 
 I was concerned with the two points raised by R3 and don't feel they were adequately addressed by the author's comments:
 
 - Dependence of the criteria on the learning rate (this does not make sense to me); and
 - Really really poor results on CIFAR-10 (and this is not being too picky, like asking them to be state-of-the-art; they are just way off)
 
 I engaged R1 to see how they felt about this. In reflection, I side with the majority opinion that the paper needs re-work to meet the ICLR acceptance bar.

Decision: Reject