An Empirical Study of Example Forgetting during Deep Neural Network Learning

1 INTRODUCTION . Many machine learning models , in particular neural networks , can not perform continual learning . They have a tendency to forget previously learnt information when trained on new tasks , a phenomenon usually called catastrophic forgetting ( Kirkpatrick et al. , 2017 ; Ritter et al. , 2018 ) . One of the hypothesized causes of catastrophic forgetting in neural networks is the shift in the input distribution across different tasks—e.g. , a lack of common factors or structure in the inputs of different tasks might lead standard optimization techniques to converge to radically different solutions each time a new task is presented . In this paper , we draw inspiration from this phenomenon and investigate the extent to which a related forgetting process occurs as a model learns examples traditionally considered to belong to the same task . Similarly to the continual learning setting , in stochastic gradient descent ( SGD ) optimization , each mini-batch can be considered as a mini- “ task ” presented to the network sequentially . In this context , we are interested in characterizing the learning dynamics of neural networks by analyzing ( catastrophic ) example forgetting events . These occur when examples that have been “ learnt ” ( i.e. , correctly classified ) at some time t in the optimization process are subsequently misclassified — or in other terms forgotten — at a time t′ > t. We thus switch the focus from studying interactions between sequentially presented tasks to studying interactions between sequentially presented dataset examples during SGD optimization . Our starting point is to understand whether there exist examples that are consistently forgotten across subsequent training presentations and , conversely , examples that are never forgotten . We will call the latter unforgettable examples . We hypothesize that specific examples consistently forgotten between subsequent presentations , if they exist , must ∗Equal contribution . Correspondence : MT : mariya @ cmu.edu , AS : alsordon @ microsoft.com †Work done while interning at Microsoft Research Montreal Code available at https : //github.com/mtoneva/example forgetting not share commonalities with other examples from the same task . We therefore analyze the proportion of forgettable/unforgettable examples for a given task and what effects these examples have on a model ’ s decision boundary and generalization error . The goal of our investigation is two-fold . First , we attempt to gain insight into the optimization process by analyzing interactions among examples during learning and their influence on the final decision boundary . We are particularly interested in whether we can glean insight on the compressibility of a dataset , and thereby increase data efficiency without compromising generalization accuracy . It is a timely problem that has been the recent focus of few-shot learning approaches via meta-learning ( Finn et al. , 2017 ; Ravi & Larochelle , 2017 ) . Second , we aim to characterize whether forgetting statistics can be used to identify “ important ” samples and detect outliers and examples with noisy labels ( John , 1995 ; Brodley & Friedl , 1999 ; Sukhbaatar et al. , 2014 ; Jiang et al. , 2018 ) . Identifying important , or most informative examples is an important line of work and was extensively studied in the literature . Techniques of note — among others — are predefined curricula of examples ( Bengio & LeCun , 2007 ) , self-paced learning ( Kumar et al. , 2010 ) , and more recently meta-learning ( Fan et al. , 2017 ) . These research directions usually define “ hardness ” or “ commonality ” of an example as a function of the loss on that particular example at some point during training ( or possibly at convergence ) . They do not consider whether some examples are consistently forgotten throughout learning . Very recently , Chang et al . ( 2017 ) consider re-weighting examples by accounting for the variance of their predictive distribution . This is related to our definition of forgetting events , but the authors provide little analysis of the extent to which the phenomenon occurs in their proposed tasks . Our purpose is to study this phenomenon from an empirical standpoint and characterize its prevalence in different datasets and across different model architectures . Our experimental findings suggest that : a ) there exist a large number of unforgettable examples , i.e. , examples that are never forgotten once learnt , those examples are stable across seeds and strongly correlated from one neural architecture to another ; b ) examples with noisy labels are among the most forgotten examples , along with images with “ uncommon ” features , visually complicated to classify ; c ) training a neural network on a dataset where a very large fraction of the least forgotten examples have been removed still results in extremely competitive performance on the test set . 2 RELATED WORK . Curriculum Learning and Sample Weighting Curriculum learning is a paradigm that favors learning along a curriculum of examples of increasing difficulty ( Bengio et al. , 2009 ) . This general idea has found success in a variety of areas since its introduction ( Kumar et al. , 2010 ; Lee & Grauman , 2011 ; Schaul et al. , 2015 ) . Kumar et al . ( 2010 ) implemented their curriculum by considering easy the examples with a small loss . In our experiments , we empirically validate that unforgettable examples can be safely removed without compromising generalization . Zhao & Zhang ( 2015 ) ; Katharopoulos & Fleuret ( 2018 ) relate sample importance to the norm of its loss gradient with respect to the parameters of the network . Fan et al . ( 2017 ) ; Kim & Choi ( 2018 ) ; Jiang et al . ( 2018 ) learn a curriculum directly from data in order to minimize the task loss . Jiang et al . ( 2018 ) also study the robustness of their method in the context of noisy examples . This relates to a rich literature on outlier detection and removal of examples with noisy labels ( John , 1995 ; Brodley & Friedl , 1999 ; Sukhbaatar et al. , 2014 ; Jiang et al. , 2018 ) . We will provide evidence that noisy examples rank higher in terms of number of forgetting events . Koh & Liang ( 2017 ) borrow influence functions from robust statistics to evaluate the impact of the training examples on a model ’ s predictions . Deep Generalization The study of the generalization properties of deep neural networks when trained by stochastic gradient descent has been the focus of several recent publications ( Zhang et al. , 2016 ; Keskar et al. , 2016 ; Chaudhari et al. , 2016 ; Advani & Saxe , 2017 ) . These studies suggest that the generalization error does not depend solely on the complexity of the hypothesis space . For instance , it has been demonstrated that over-parameterized models with many more parameters than training points can still achieve low test error ( Huang et al. , 2017 ; Wang et al. , 2018 ) while being complex enough to fit a dataset with completely random labels ( Zhang et al. , 2016 ) . A possible explanation for this phenomenon is a form of implicit regularization performed by stochastic gradient descent : deep neural networks trained with SGD have been recently shown to converge to the maximum margin solution in the linearly separable case ( Soudry et al. , 2017 ; Xu et al. , 2018 ) . In our work , we provide empirical evidence that generalization can be maintained when removing a substantial portion of the training examples and without restricting the complexity of the hypothesis class . This goes along the support vector interpretation provided by Soudry et al . ( 2017 ) . 3 DEFINING AND COMPUTING EXAMPLE FORGETTING . Our general case study for example forgetting is a standard classification setting . Given a dataset D = ( xi , yi ) i of observation/label pairs , we wish to learn the conditional probability distribution p ( y|x ; θ ) using a deep neural network with parameters θ . The network is trained to minimize the empirical risk R = 1|D| ∑ i L ( p ( yi|xi ; θ ) , yi ) , where L denotes the cross-entropy loss and yi ∈ 1 , . . . k. The minimization is performed using variations of stochastic gradient descent , starting from initial random parameters θ0 , and by sampling examples at random from the dataset D. Forgetting and learning events We denote by ŷti = argmaxk p ( yik|xi ; θt ) the predicted label for example xi obtained after t steps of SGD . We also let accti = 1ŷti=yi be a binary variable indicating whether the example is correctly classified at time step t. Example i undergoes a forgetting event when accti decreases between two consecutive updates : acc t i > acc t+1 i . In other words , example i is misclassified at step t + 1 after having been correctly classified at step t. Conversely , a learning event has occurred if accti < acc t+1 i . Statistics that will be of interest in the next sections include the distribution of forgetting events across examples and the first time a learning event occurs . Classification margin We will also be interested in analyzing the classification margin . Our predictors have the form p ( yi|xi ; θ ) = σ ( β ( xi ) ) , where σ is a sigmoid ( softmax ) activation function in the case of binary ( categorical ) classification . The classification margin m is defined as the difference between the logit of the correct class and the largest logit among the other classes , i.e . m = βk − argmaxk′ 6=k βk′ , where k is the index corresponding to the correct class . Unforgettable examples We qualify examples as unforgettable if they are learnt at some point and experience no forgetting events during the whole course of training : example i is unforgettable if the first time it is learnt t∗ verifies t∗ < ∞ and for all k ≥ t∗ , accki = 1 . Note that , according to this definition , examples that are never learnt during training do not qualify as unforgettable . We refer to examples that have been forgotten at least once as forgettable . 3.1 PROCEDURAL DESCRIPTION AND EXPERIMENTAL SETTING . Following the previous definitions , monitoring forgetting events entails computing the prediction for all examples in the dataset at each model update , which would be prohibitively expensive . In practice , for each example , we subsample the full sequence of forgetting events by computing forgetting statistics only when the example is included in the current mini-batch ; that is , we compute forgetting across presentations of the same example in subsequent mini-batches . This gives a lower bound on the number of forgetting events an example undergoes during training . Algorithm 1 Computing forgetting statistics . initialize prev acci = 0 , i ∈ D initialize forgetting T [ i ] = 0 , i ∈ D while not training done do B ∼ D # sample a minibatch for example i ∈ B do compute acci if prev acci > acci then T [ i ] = T [ i ] + 1 prev acci = acci gradient update classifier on B return T We train a classifier on a given dataset and record the forgetting events for each example when they are sampled in the current mini-batch . For the purposes of further analysis , we then sort the dataset ’ s examples based on the number of forgetting events they undergo . Ties are broken at random when sampling from the ordered data . Samples that are never learnt are considered forgotten an infinite number of times for sorting purposes . Note that this estimate of example forgetting is computationally expensive ; see Sec . 6 for a discussion of a cheaper method . We perform our experimental evaluation on three datasets of increasing complexity : MNIST ( LeCun et al. , 1999 ) , permuted MNIST – a version of MNIST that has the same fixed permutation applied to the pixels of all examples , and CIFAR-10 ( Krizhevsky , 2009 ) . We use various model architectures and training schemes that yield test errors comparable with the current state-of-the-art on the respective datasets . In particular , the MNIST-based experiments use a network comprised of two convolutional layers followed by a fully connected one , trained using SGD with momentum and dropout . This network achieves 0.8 % test error . For CIFAR10 , we use a ResNet with cutout ( DeVries & Taylor , 2017 ) trained using SGD and momentum with a particular learning rate schedule . This network achieves a competitive 3.99 % test error . For full experimentation details , see the Supplementary .

Summary: This paper studies the forgetting behavior of the training examples during SGD. Empirically it shows there are forgettable and unforgettable examples, unforgettable examples are like "support examples", one can achieve similar performance by training only on these "support examples". The paper also shows this phenomenon is consistent across different network architectures.