Title: EMPIRICAL STUDY OF EASY AND HARD EXAMPLES IN CNN TRAINING Anonymous

Abstract: Deep Neural Networks (DNNs) generalize well despite their massive size and capability of memorizing all examples. There is a hypothesis that DNNs start learning from simple patterns based on the observations that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite the importance of understanding the learning dynamics of DNNs, properties of easy and hard examples are not fully investigated. In this paper, we study the similarities of easy and hard examples respectively among different CNNs, assessing those examples contributions to generalization. Our results show that most easy examples are identical among different CNNs, as they share similar dataset-dependent patterns (e.g., colors, structures, and superficial cues in high-frequency). Moreover, while hard examples tend to contribute more to generalization than easy examples, removing a large number of easy examples leads to poor generalization, and we find that most misclassified examples in validation dataset are hard examples. By analyzing intriguing properties of easy and hard examples, we discover that the reason why easy and hard examples have such properties can be explained by biases in a dataset and Stochastic Gradient Descent (SGD).

1. INTRODUCTION:

From a traditional perspective of generalization, overly expressive models can memorize all examples and result in poor generalization. However, deep neural networks (DNNs) achieve an excellent generalization performance even if models are over-parameterized (Zhang et al., 2017) . The reason for this phenomenon remains unclear. (Arpit et al., 2017) show that brute-force memorization does not happen in DNNs training, and propose a hypothesis that DNNs start learning from simple patterns. Their hypothesis is based on observations of examples that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite their importance in understanding how DNNs start learning, easy and hard examples are not thoroughly examined.

In this paper, we study easy and hard examples, and their intriguing properties are shown. First, we investigate whether easy and hard examples differ on various CNNs architectures. For our experiments, we introduce easiness as a metric to measure how early examples are classified. According to easiness, we calculate the matching rates of easy and hard examples between different CNN architectures. Surprisingly, CNNs start learning from identical easy examples and they are visually similar to each other. It indicates that different CNNs start learning from similar patterns.

Second, we investigate the patterns that make examples easy. We study CNNs with examples from which some patterns are intentionally removed. We find that the patterns CNNs start learning from are dataset dependent (e.g., colors, structures, and high-frequency information).

Lastly, we conduct ablation and corrupted-label experiment to determine how easy and hard examples contribute differently to generalization. We find that hard examples contribute more to generalization than easy examples, however, removing a large number of easy examples leads to poor generalization. This phenomenon can be explained by biases in a dataset. In addition, we discover that misclassified examples in validation dataset are consistent and mostly hard examples.

Those properties can not be explained by a hypothesis that CNNs start learning from simple patterns (Arpit et al., 2017) . From observations, we hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes. There are two reasons why there are easy and hard examples: biases in a dataset and Stochastic Gradient Descent (SGD).

A dataset naturally contains various biases leading some patterns to appear as a majority or a minority. The majority and minority patterns are called frequent patterns and rare patterns, respectively. Since SGD randomly picks samples to train a model, frequent patterns tend to be used more than rare patterns in a batch. It means that the gradients of frequent patterns dominate the direction of the update, and some examples that are discriminative by frequent patterns are consistently well classified at early epochs.

2. APPROACH:

In this paper, we perform several experiments to investigate unknown properties of easy and hard examples. We introduce easiness as a metric of how early examples are classified. Based on easiness, we reveal several properties of easy and hard examples.

2.1. EASINESS:

To measure how early an example is classified, we introduce easiness e t as a criteria, where t denotes the index of an example. We repeat m-epochs training N times and take a record of the number of correct classifications. Let c t be the number of trials in which the example x t is correctly classified. easiness e t is computed as:

EQUATION

where 

EQUATION

where |•| denotes the size of a set.

3. EXPERIMENTS:



3.1. PREPARATIONS:

We utilize CIFAR-10 (Krizhevsky, 2009) and ImageNet 2012 dataset (Russakovsky et al., 2015) for our experiments.

CIFAR-10. On CIFAR-10, translation by 4 pixels, horizontal flipping, and global contrast normalization are applied onto images with 32 × 32 pixels. We use a list of models to CIFAR-10: three layer (Zagoruyko & Komodakis, 2016) , DenseNet-BC 12-100 (Huang et al., 2017) and ResNeXt 4-64d (Xie et al., 2017) mini ImageNet. We randomly select 10 classes1 from ImageNet 2012 dataset for an interpretable analysis and call them mini ImageNet. On mini ImageNet, resizing images with the scale and aspect ratio augmentation and horizontal flipping are applied onto images. Then, global contrast normalization is applied to randomly cropped images with 224 × 224 pixels. We use a list of models to mini ImageNet: three layer MLPs, AlexNet (Krizhevsky et al., 2012) , VGG11 (Simonyan & Zisserman, 2015) , VGG11 with Batch Normalization (Ioffe & Szegedy, 2015) , ResNet 18 (He et al., 2016) and DenseNet 121 (Huang et al., 2017) As the optimizer, we use AdamW (Loshchilov & Hutter, 2017) and parameters are as follows: α = 0.001, β 1 = 0.9, β 2 = 0.99, w norm = 0.05 and T i = 300. We train a model for 300 epochs and use 128 batch size for CIFAR-10 and 64 batch size for mini ImageNet.

4. EASY EXAMPLES ARE IDENTICAL IN DIFFERENT CNNS:

To determine whether easy and hard examples are shared across different CNN architectures, we calculate matching rates according to easiness. Table 1 (a) and (c Table 1 (b) and (d 

Easy examples. The matching rates of easy examples between CNNs in:



6. WHY CNNS CONSIDER THOSE EXAMPLES AS easy:

To further analyze what properties make examples easy for CNNs, we perform three experiments: color, shuffle and radial masking experiment (Jo & Bengio, 2018) . In these experiments, we calculate easiness based on examples from which some patterns are intentionally removed.

Color experiment. To investigate whether CNNs look at colors at early epochs, we use easy examples that are interpolated to grey. Let us consider that R, G, B, and Grey ∈ R H×W are red, green, blue and grey channels respectively with height H and width W . The interpolations are calculated as follows: where α is in the range of 0 ≤ α ≤ 1, with 0 indicating that the example is the grey picture and with 1 showing that the example is colored. Grey channel is obtained by averaging RGB channels: Shuffle experiment. The work of (Noroozi & Favaro, 2016) inspires the idea of shuffle experiment. This experiment lets us understand whether CNNs look at structures at early epochs. Then, we cut images into pieces with fixed tile size and reconstruct images with randomly shuffled tiled images. The number of correct classification for the example c t is calculated by using shuffled tiled images.

EQUATION

Examples of the shuffled images are depicted in Figure 3 (b). Tile Size means the relative size when compared to the full image size (e.g., if Tile Size is 0.33, the size of a tiled image is the 33% size of the full image.).

Radial masking experiment. (Jo & Bengio, 2018) show that DNNs misclassify examples without high-frequency information that a human can recognize, and claims that DNNs solve problems by looking at superficial cues on examples. The goal of this experiment is to check whether CNNs look at such superficial cues to classify easy examples at early epochs. The work of radial masking experiment is conducted by (Jo & Bengio, 2018) . Radial mask M r ∈ R H×W , where H and W are height and width, is computed as follows:

M r [i, j] = 1 if (i, j) -( W 2 , H 2 ) L2 ≤ r, 0 otherwise,

where i and j are indices of M r . M r is a radial mask in the Fourier domain that removes higherfrequency in examples. The bigger r is, the higher frequency of an example is kept. Radial masked examples are shown in Figure 3 (c) .

Result and analysis.:

To verify what patterns CNNs start looking at, we calculate the matching rate between regular examples and examples from which some patterns are intentionally removed. First, we train a model for m-epochs with regular data augmentation. Then, easiness is calculated by using examples from which some patterns are intentionally removed.

The result are shown in Figure 2 . The result of color experiment is the opposite tendency to the result of shuffle and random masking experiment. In CIFAR-10, CNNs start looking at the structure and high-frequency information. In mini ImageNet, CNNs start looking at the colors. It indicates that initially learned patterns are dataset dependent.

In mini ImageNet, we observe that the accuracy does not drop when r is 1.1 small in radial masking experiment. As radial masked examples are depicted in Figure 3 

8. RELATED WORK:

A dataset naturally contains various biases. For instance, (Ponce et al., 2006) shows some averaged images of Caltech-101 (Li et al., 2004) are not homogeneous and recognizable. They claim that Caltech-101 may have inter-class variability but lacks intra-class variability. (Torralba & Efros, 2011) mentions several biases in a dataset: Selection bias means that examples in a dataset tend to have particular kinds of images (example: there are many examples of a sports car in the car category).

Capture bias represents the manner in which pictures are usually taken (example: a picture of a dog is usually taken from the front with the dog looking at the photographer and occupying most of the picture). In this paper, we find that easy and hard examples are highly associated with such biases.

There are three types of training schemes that start learning from easy examples (i.e., curriculum learning) (Kumar et al., 2010) , uncertain examples (Settles, 2010; Chang et al., 2017) or hard examples (i.e., hard negative mining) (Lin et al., 2017; Simo-Serra et al., 2015; Shrivastava et al., 2016; Loshchilov & Hutter, 2016; Wang & Gupta, 2015) . Those methods rely on the loss value (Arpit et al., 2017) . Consequently, to explain why easy and hard examples have such properties, we hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes.

As future work, by using easiness, dataset compression and mislabel mining can be considered for a large-scale dataset. Besides, we would like to analyze the phenomenon of easy and hard examples on natural language and sound domain. Our work utilize datasets with 10 classes, thus it is necessary to conduct further research on large-scale and more complexed datasets. 

:

Image IDs are n01751748, n02123394, n02169497, n02883205, n03125729, n03954731, n04332243, n04355358, n04553703 and n07717556.

Figures and Tables:

) are much higher than the case of random selection. This result shows that CNNs start learning from the same examples even if CNN architectures are different. In addition, the matching rate between MLPs and CNNs are low in CIFAR-10, but high in mini ImageNet. Hard examples. The matching rates of hard examples are shown in

). Intriguingly, in mini ImageNet, the matching rates among CNNs with batch normalization(Ioffe & Szegedy, 2015) are high. It shows that batch normalization in CNNs architecture affects hard examples. Since easy examples are mostly identical, the difference of hard examples relates to the difference of generalization ability. It requires further analysis. Why there are easy and hard examples? We hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes. It can be explained by biases in a dataset and SGD. As a example, let us consider the classification problem of a ladybird and mantis. If a ladybird and mantis are always onto a leaf in training examples, the pattern of a leaf is harmful to discriminate examples. We call such harmfulness as contradiction. SGD force the model not to use contradicted patterns. If the model uses the pattern of a leaf to discriminate a mantis, ladybird examples stochastically update the model to discriminate a ladybird by the pattern of a leaf, and then the model start misclassifying mantis. On the one hand, classified and misclassified examples by contradicted patterns interrupt to use contradicted patterns each other. On the other hand, non-contradicted patterns across classes are learned without interferences and used to discriminate examples. When examples are classified by non-contradicted patterns, their loss value and the magnitudes of their gradients become smaller. Thus, the model stops learning contradicted patterns. There are biases in a dataset, thus some patterns appear frequently and some patterns appear rarely. Since SGD updates the model by averaged gradients, the gradients of frequent patterns become global directions of the gradient to update the model. Easy examples are discriminative by frequent and non-contradicted patterns, thus those patterns are learned first. As easy examples are learned and get smaller loss values, rare patterns in hard examples start being learned. That's why examples are identical among different CNN architectures since such patterns are dependent on a dataset and not on CNNs architectures. 5 EASY EXAMPLES ARE VISUALLY SIMILAR TO EACH OTHER Visualizations 2 of easy and hard examples are shown in Figure 1. To calculate easiness, WRN 16-4 and ResNet18 are respectively used for CIFAR-10 and mini ImageNet. As can be seen in Figure 1, easy examples are visually similar to each other, and hard examples tend to be visually different from each other. This result implies that different CNNs start learning from similar patterns.

The rest of easy and hard examples are provided in the supplementary material.

Figure 1: Easy and hard examples of CIFAR-10 dataset and mini ImageNet. a)-d) are examples from CIFAR-10. e)-h) are examples from mini ImageNet.

The interpolated examples are shown in Figure3(a).

Figure 2: The result of easy examples in (a) color, (b) shuffle and (c) radial masking experiments in CIFAR-10 and mini ImageNet. The vertical axis is the matching rate between the case with regular examples and examples from which some patterns are intentionally removed. The model is trained for m-epochs with regular data augmentation.

Figure 4: The result of ablation and corrupted label experiment. The vertical axis is the accuracy. The horizontal axis is the drop rate or random label rate. If the drop rate is 0.3, it means that 30% of examples are discarded. If random label rate is 0.3, it means that 30% of examples are randomly labeled. WRN 16-4 and ResNet 18 are used respectively for CIFAR-10 and mini ImageNet. "easy" on figures means that examples with high easiness are mainly removed or randomly labeled. In the same manner, "hard" denotes examples with low easiness are mainly processed and "random" represents processed examples are randomly selected. Error bars represent standard deviation over 5 trials.

(a) Easiest examples of airplane (b) Hardest examples of airplane (c) Easiest examples of car (d) Hardest examples of car (e) Easiest examples of car (f) Hardest examples of car (g) Easiest examples of cat (h) Hardest examples of cat (i) Easiest examples of deer (j) Hardest examples of deer (k) Easiest examples of dog (l) Hardest examples of dog (m) Easiest examples of frog (n) Hardest examples of frog (o) Easiest examples of horse (p) Hardest examples of horse (q) Easiest examples of ship (r) Hardest examples of ship (a) Easiest examples of sea snake (b) Hardest examples of sea snake (c) Easiest examples of persian cat (d) Hardest examples of persian cat (e) Easiest examples of leaf beetle (f) Hardest examples of leaf beetle (g) Easiest examples of bow tie (h) Hardest examples of bow tie (i) Easiest examples of cradle (j) Hardest examples of cradle (k) Easiest examples of woodworking plane (l) Hardest examples of woodworking plane (m) Easiest examples of strainer (n) Hardest examples of strainer (o) Easiest examples of sundial (p) Hardest examples of sundial (q) Easiest examples of washbasin (r) Hardest examples of washbasin

Let us consider two different set of examples X A and X B . The matching rate M AB between X A and X B is calculated as:

Table content: <table><tr><td>In our experiments, τ is calculated so that 10% of examples with the highest and lowest easiness</td></tr><tr><td>belong to easy and hard examples, respectively.</td></tr><tr><td>2.2 MATCHING RATE</td></tr><tr><td>Based on easiness, we use matching rate to measure how similar easy and hard examples are between</td></tr><tr><td>various CNN architectures.</td></tr></table>

The matching rates of easy and hard examples among various CNNs in CIFAR-10 and mini ImageNet. Since the values in the tables are symmetric and redundant, we put "-" on unnecessary table cells for clarity. We use 10% of examples with the highest and lowest easiness to calculate the matching rate between different CNNs architectures. Random means that we randomly select 10% of examples from the datasets.

Table content: <table><tr><td/><td/><td colspan="6">MLPs WRN DenseNet-BC ResNeXt Random</td></tr><tr><td colspan="2">MLPs</td><td>0.70</td><td>0.20</td><td>0.19</td><td>0.21</td><td>0.1</td><td/></tr><tr><td colspan="2">WRN</td><td>-</td><td>0.86</td><td>0.71</td><td>0.49</td><td>0.1</td><td/></tr><tr><td colspan="3">DenseNet-BC -</td><td>-</td><td>0.91</td><td>0.47</td><td>0.1</td><td/></tr><tr><td colspan="2">ResNeXt</td><td>-</td><td>-</td><td>-</td><td>0.82</td><td>0.1</td><td/></tr><tr><td/><td/><td colspan="4">(a) Matching rate of easy examples in CIFAR-10</td><td/><td/></tr><tr><td/><td/><td colspan="6">MLPs WRN DenseNet-BC ResNeXt Random</td></tr><tr><td colspan="2">MLPs</td><td>0.96</td><td>0.15</td><td>0.14</td><td>0.14</td><td>0.1</td><td/></tr><tr><td colspan="2">WRN</td><td>-</td><td>0.90</td><td>0.64</td><td>0.50</td><td>0.1</td><td/></tr><tr><td colspan="3">DenseNet-BC -</td><td>-</td><td>0.83</td><td>0.47</td><td>0.1</td><td/></tr><tr><td colspan="2">ResNeXt</td><td>-</td><td>-</td><td>-</td><td>0.86</td><td>0.1</td><td/></tr><tr><td/><td/><td colspan="4">(b) Matching rate of hard examples in CIFAR-10</td><td/><td/></tr><tr><td/><td colspan="7">MLPs AlexNet VGG VGG-BN ResNet DenseNet Random</td></tr><tr><td>MLPs</td><td>0.63</td><td>0.61</td><td>0.67</td><td>0.31</td><td>0.52</td><td>0.48</td><td>0.1</td></tr><tr><td>AlexNet</td><td>-</td><td>0.87</td><td>0.72</td><td>0.41</td><td>0.50</td><td>0.48</td><td>0.1</td></tr><tr><td>VGG</td><td>-</td><td>-</td><td>0.90</td><td>0.37</td><td>0.59</td><td>0.59</td><td>0.1</td></tr><tr><td colspan="2">VGG-BN -</td><td>-</td><td>-</td><td>0.87</td><td>0.25</td><td>0.31</td><td>0.1</td></tr><tr><td>ResNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.96</td><td>0.65</td><td>0.1</td></tr><tr><td colspan="2">DenseNet -</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.75</td><td>0.1</td></tr><tr><td/><td colspan="5">(c) Matching rate of easy examples in mini ImageNet</td><td/><td/></tr><tr><td/><td colspan="7">MLPs AlexNet VGG VGG-BN ResNet DenseNet Random</td></tr><tr><td>MLPs</td><td>0.0</td><td>0.07</td><td>0.03</td><td>0.0</td><td>0.01</td><td>0.01</td><td>0.1</td></tr><tr><td>AlexNet</td><td>-</td><td>0.94</td><td>0.48</td><td>0.09</td><td>0.26</td><td>0.25</td><td>0.1</td></tr><tr><td>VGG</td><td>-</td><td>-</td><td>0.78</td><td>0.20</td><td>0.45</td><td>0.26</td><td>0.1</td></tr><tr><td colspan="2">VGG-BN -</td><td>-</td><td>-</td><td>0.46</td><td>0.46</td><td>0.42</td><td>0.1</td></tr><tr><td>ResNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.52</td><td>0.44</td><td>0.1</td></tr><tr><td colspan="2">DenseNet -</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.68</td><td>0.1</td></tr><tr><td/><td colspan="5">(c) Matching rate of hard examples in mini ImageNet</td><td/><td/></tr><tr><td colspan="4">multilayer perceptrons (MLPs) WRN 16-4</td><td/><td/><td/><td/></tr></table>

9 CONCLUSION AND FUTURE WORKIn this paper, unknown properties of easy and hard examples are revealed. Our experiments demonstrate that different CNNs start learning from identical examples, and easy and hard examples contribute differently to generalization. However, discovered properties can not be explained by a hypothesis that CNNs start learning from simple patterns

Table content: <table/>

Peer review (iclr2019):
 This paper extends the observation made in Arpit et al (2018) that deep networks prioritize learning simple patterns that are shared across many training examples. This paper further digs in this direction by defining a measure of "easiness" of a sample and making several empirical observations using this measure. This paper finds that the set of examples that can be identified as easy and hard for different architectures have a large intersection across architectures. It is also shown qualitatively that the set of easy examples have visually similar characteristics while hard examples are different from easy examples and also from one another. By artifically alterning the training images by changing color, structure and frequency components in individual examples, it is shown that the training process targets different characteristics that is dataset dependent. The effect of dropping samples from the set of easy and hard examples is studied and it is shown that dropping a small fraction of easy examples does not hurt generalization significantly (due to redundancy in patterns contained in such examples) while dropping even a smaller fraction of hard examples hurts generalization more significantly. Overall i find the above empirical observations and some of the other arguments in this paper interesting but i think there is a lot of scope for improvement in the paper. In its current form, I find the language of this paper quite informal in a number of places, and some of the claims/deductions made in the paper are not well justified and they need to be changed accordingly. If these issues are fixed, I will improve my score. Specifically the issues are: 1. The notion of "easiness" introduced in Eq. (1) is clearly inspired by the experiment in Fig. 1 of Arpit et al (2018), and is also very similar. However, there is no mention of this in the paper. 2. Based on the experiments in table 1, it is mentioned in section 4 that "CNNs start learning from the *same* examples even if CNN examples are different." This statement is simply inaccurate. There is a decent fraction of easy and hard examples that are shared across different architectures in some cases. But this does not imply what the authors have claimed. The claim seems very careless. 3. In the sentence following "Why there are easy and hard examples?", the authors introduce the hypothesis of frequent patterns that are not *contradicted across classes*. I find this terminology unusual. What does it mean by patterns being contradiction? These sentences need to be re-worded. 4. Following this hypothesis, it is mentioned that "SGD force the model not to use contradicted patterns". Grammar aside, the argument of SGD forcing the model sounds very informal. Even beyond that, I am not sure I see the basis for making such a claim. This whole paragraph (and the one that follows) sounds like a hypothesis that the authors have in mind about the observation made in table 1, but it is put forth as if they are facts. 5. In section 5, based on the experiment that shows that easy examples are visually similar, the authors write, "This result *implies* that different CNNs start learning from similar patterns.". This is again bad deduction. The qualitative results at best *suggest* that different CNNs start learning from similar patterns, but do not imply it. 6. The experiment in section 6 was interesting but the text was hard to follow and did not describe the results clearly. 7. In section 7, it is mentioned that "Randomly removing examples consistently produces the best performance." This sentence is misleading. Randomly dropping samples clearly hurts performance compared to baseline. The claim should be that randomly removing examples hurts the performance least compared with dropping easy and hard examples. 8. Finally, the authors mention at multiple instances that the observations made in this paper cannot be explained by the hypothesis set forth by Arpit et al (2018). I am not sure I understand the relevance of this claim. Explaining the observations made in this paper is never mentioned as a goal in Arpit et al (2018). These statements need to be fixed.