Title: EMPIRICAL STUDY OF EASY AND HARD EXAMPLES IN CNN TRAINING Anonymous

Abstract: Deep Neural Networks (DNNs) generalize well despite their massive size and capability of memorizing all examples. There is a hypothesis that DNNs start learning from simple patterns based on the observations that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite the importance of understanding the learning dynamics of DNNs, properties of easy and hard examples are not fully investigated. In this paper, we study the similarities of easy and hard examples respectively among different CNNs, assessing those examples contributions to generalization. Our results show that most easy examples are identical among different CNNs, as they share similar dataset-dependent patterns (e.g., colors, structures, and superficial cues in high-frequency). Moreover, while hard examples tend to contribute more to generalization than easy examples, removing a large number of easy examples leads to poor generalization, and we find that most misclassified examples in validation dataset are hard examples. By analyzing intriguing properties of easy and hard examples, we discover that the reason why easy and hard examples have such properties can be explained by biases in a dataset and Stochastic Gradient Descent (SGD).

1. INTRODUCTION:

From a traditional perspective of generalization, overly expressive models can memorize all examples and result in poor generalization. However, deep neural networks (DNNs) achieve an excellent generalization performance even if models are over-parameterized (Zhang et al., 2017) . The reason for this phenomenon remains unclear. (Arpit et al., 2017) show that brute-force memorization does not happen in DNNs training, and propose a hypothesis that DNNs start learning from simple patterns. Their hypothesis is based on observations of examples that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite their importance in understanding how DNNs start learning, easy and hard examples are not thoroughly examined.

In this paper, we study easy and hard examples, and their intriguing properties are shown. First, we investigate whether easy and hard examples differ on various CNNs architectures. For our experiments, we introduce easiness as a metric to measure how early examples are classified. According to easiness, we calculate the matching rates of easy and hard examples between different CNN architectures. Surprisingly, CNNs start learning from identical easy examples and they are visually similar to each other. It indicates that different CNNs start learning from similar patterns.

Second, we investigate the patterns that make examples easy. We study CNNs with examples from which some patterns are intentionally removed. We find that the patterns CNNs start learning from are dataset dependent (e.g., colors, structures, and high-frequency information).

Lastly, we conduct ablation and corrupted-label experiment to determine how easy and hard examples contribute differently to generalization. We find that hard examples contribute more to generalization than easy examples, however, removing a large number of easy examples leads to poor generalization. This phenomenon can be explained by biases in a dataset. In addition, we discover that misclassified examples in validation dataset are consistent and mostly hard examples.

Those properties can not be explained by a hypothesis that CNNs start learning from simple patterns (Arpit et al., 2017) . From observations, we hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes. There are two reasons why there are easy and hard examples: biases in a dataset and Stochastic Gradient Descent (SGD).

A dataset naturally contains various biases leading some patterns to appear as a majority or a minority. The majority and minority patterns are called frequent patterns and rare patterns, respectively. Since SGD randomly picks samples to train a model, frequent patterns tend to be used more than rare patterns in a batch. It means that the gradients of frequent patterns dominate the direction of the update, and some examples that are discriminative by frequent patterns are consistently well classified at early epochs.

2. APPROACH:

In this paper, we perform several experiments to investigate unknown properties of easy and hard examples. We introduce easiness as a metric of how early examples are classified. Based on easiness, we reveal several properties of easy and hard examples.

2.1. EASINESS:

To measure how early an example is classified, we introduce easiness e t as a criteria, where t denotes the index of an example. We repeat m-epochs training N times and take a record of the number of correct classifications. Let c t be the number of trials in which the example x t is correctly classified. easiness e t is computed as:

EQUATION

where 

EQUATION

where |•| denotes the size of a set.

3. EXPERIMENTS:



3.1. PREPARATIONS:

We utilize CIFAR-10 (Krizhevsky, 2009) and ImageNet 2012 dataset (Russakovsky et al., 2015) for our experiments.

CIFAR-10. On CIFAR-10, translation by 4 pixels, horizontal flipping, and global contrast normalization are applied onto images with 32 × 32 pixels. We use a list of models to CIFAR-10: three layer (Zagoruyko & Komodakis, 2016) , DenseNet-BC 12-100 (Huang et al., 2017) and ResNeXt 4-64d (Xie et al., 2017) mini ImageNet. We randomly select 10 classes1 from ImageNet 2012 dataset for an interpretable analysis and call them mini ImageNet. On mini ImageNet, resizing images with the scale and aspect ratio augmentation and horizontal flipping are applied onto images. Then, global contrast normalization is applied to randomly cropped images with 224 × 224 pixels. We use a list of models to mini ImageNet: three layer MLPs, AlexNet (Krizhevsky et al., 2012) , VGG11 (Simonyan & Zisserman, 2015) , VGG11 with Batch Normalization (Ioffe & Szegedy, 2015) , ResNet 18 (He et al., 2016) and DenseNet 121 (Huang et al., 2017) As the optimizer, we use AdamW (Loshchilov & Hutter, 2017) and parameters are as follows: α = 0.001, β 1 = 0.9, β 2 = 0.99, w norm = 0.05 and T i = 300. We train a model for 300 epochs and use 128 batch size for CIFAR-10 and 64 batch size for mini ImageNet.

4. EASY EXAMPLES ARE IDENTICAL IN DIFFERENT CNNS:

To determine whether easy and hard examples are shared across different CNN architectures, we calculate matching rates according to easiness. Table 1 (a) and (c Table 1 (b) and (d 

Easy examples. The matching rates of easy examples between CNNs in:



6. WHY CNNS CONSIDER THOSE EXAMPLES AS easy:

To further analyze what properties make examples easy for CNNs, we perform three experiments: color, shuffle and radial masking experiment (Jo & Bengio, 2018) . In these experiments, we calculate easiness based on examples from which some patterns are intentionally removed.

Color experiment. To investigate whether CNNs look at colors at early epochs, we use easy examples that are interpolated to grey. Let us consider that R, G, B, and Grey ∈ R H×W are red, green, blue and grey channels respectively with height H and width W . The interpolations are calculated as follows: where α is in the range of 0 ≤ α ≤ 1, with 0 indicating that the example is the grey picture and with 1 showing that the example is colored. Grey channel is obtained by averaging RGB channels: Shuffle experiment. The work of (Noroozi & Favaro, 2016) inspires the idea of shuffle experiment. This experiment lets us understand whether CNNs look at structures at early epochs. Then, we cut images into pieces with fixed tile size and reconstruct images with randomly shuffled tiled images. The number of correct classification for the example c t is calculated by using shuffled tiled images.

EQUATION

Examples of the shuffled images are depicted in Figure 3 (b). Tile Size means the relative size when compared to the full image size (e.g., if Tile Size is 0.33, the size of a tiled image is the 33% size of the full image.).

Radial masking experiment. (Jo & Bengio, 2018) show that DNNs misclassify examples without high-frequency information that a human can recognize, and claims that DNNs solve problems by looking at superficial cues on examples. The goal of this experiment is to check whether CNNs look at such superficial cues to classify easy examples at early epochs. The work of radial masking experiment is conducted by (Jo & Bengio, 2018) . Radial mask M r ∈ R H×W , where H and W are height and width, is computed as follows:

M r [i, j] = 1 if (i, j) -( W 2 , H 2 ) L2 ≤ r, 0 otherwise,

where i and j are indices of M r . M r is a radial mask in the Fourier domain that removes higherfrequency in examples. The bigger r is, the higher frequency of an example is kept. Radial masked examples are shown in Figure 3 (c) .

Result and analysis.:

To verify what patterns CNNs start looking at, we calculate the matching rate between regular examples and examples from which some patterns are intentionally removed. First, we train a model for m-epochs with regular data augmentation. Then, easiness is calculated by using examples from which some patterns are intentionally removed.

The result are shown in Figure 2 . The result of color experiment is the opposite tendency to the result of shuffle and random masking experiment. In CIFAR-10, CNNs start looking at the structure and high-frequency information. In mini ImageNet, CNNs start looking at the colors. It indicates that initially learned patterns are dataset dependent.

In mini ImageNet, we observe that the accuracy does not drop when r is 1.1 small in radial masking experiment. As radial masked examples are depicted in Figure 3 

8. RELATED WORK:

A dataset naturally contains various biases. For instance, (Ponce et al., 2006) shows some averaged images of Caltech-101 (Li et al., 2004) are not homogeneous and recognizable. They claim that Caltech-101 may have inter-class variability but lacks intra-class variability. (Torralba & Efros, 2011) mentions several biases in a dataset: Selection bias means that examples in a dataset tend to have particular kinds of images (example: there are many examples of a sports car in the car category).

Capture bias represents the manner in which pictures are usually taken (example: a picture of a dog is usually taken from the front with the dog looking at the photographer and occupying most of the picture). In this paper, we find that easy and hard examples are highly associated with such biases.

There are three types of training schemes that start learning from easy examples (i.e., curriculum learning) (Kumar et al., 2010) , uncertain examples (Settles, 2010; Chang et al., 2017) or hard examples (i.e., hard negative mining) (Lin et al., 2017; Simo-Serra et al., 2015; Shrivastava et al., 2016; Loshchilov & Hutter, 2016; Wang & Gupta, 2015) . Those methods rely on the loss value (Arpit et al., 2017) . Consequently, to explain why easy and hard examples have such properties, we hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes.

As future work, by using easiness, dataset compression and mislabel mining can be considered for a large-scale dataset. Besides, we would like to analyze the phenomenon of easy and hard examples on natural language and sound domain. Our work utilize datasets with 10 classes, thus it is necessary to conduct further research on large-scale and more complexed datasets. 

:

Image IDs are n01751748, n02123394, n02169497, n02883205, n03125729, n03954731, n04332243, n04355358, n04553703 and n07717556.

Figures and Tables:

) are much higher than the case of random selection. This result shows that CNNs start learning from the same examples even if CNN architectures are different. In addition, the matching rate between MLPs and CNNs are low in CIFAR-10, but high in mini ImageNet. Hard examples. The matching rates of hard examples are shown in

). Intriguingly, in mini ImageNet, the matching rates among CNNs with batch normalization(Ioffe & Szegedy, 2015) are high. It shows that batch normalization in CNNs architecture affects hard examples. Since easy examples are mostly identical, the difference of hard examples relates to the difference of generalization ability. It requires further analysis. Why there are easy and hard examples? We hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes. It can be explained by biases in a dataset and SGD. As a example, let us consider the classification problem of a ladybird and mantis. If a ladybird and mantis are always onto a leaf in training examples, the pattern of a leaf is harmful to discriminate examples. We call such harmfulness as contradiction. SGD force the model not to use contradicted patterns. If the model uses the pattern of a leaf to discriminate a mantis, ladybird examples stochastically update the model to discriminate a ladybird by the pattern of a leaf, and then the model start misclassifying mantis. On the one hand, classified and misclassified examples by contradicted patterns interrupt to use contradicted patterns each other. On the other hand, non-contradicted patterns across classes are learned without interferences and used to discriminate examples. When examples are classified by non-contradicted patterns, their loss value and the magnitudes of their gradients become smaller. Thus, the model stops learning contradicted patterns. There are biases in a dataset, thus some patterns appear frequently and some patterns appear rarely. Since SGD updates the model by averaged gradients, the gradients of frequent patterns become global directions of the gradient to update the model. Easy examples are discriminative by frequent and non-contradicted patterns, thus those patterns are learned first. As easy examples are learned and get smaller loss values, rare patterns in hard examples start being learned. That's why examples are identical among different CNN architectures since such patterns are dependent on a dataset and not on CNNs architectures. 5 EASY EXAMPLES ARE VISUALLY SIMILAR TO EACH OTHER Visualizations 2 of easy and hard examples are shown in Figure 1. To calculate easiness, WRN 16-4 and ResNet18 are respectively used for CIFAR-10 and mini ImageNet. As can be seen in Figure 1, easy examples are visually similar to each other, and hard examples tend to be visually different from each other. This result implies that different CNNs start learning from similar patterns.

The rest of easy and hard examples are provided in the supplementary material.

Figure 1: Easy and hard examples of CIFAR-10 dataset and mini ImageNet. a)-d) are examples from CIFAR-10. e)-h) are examples from mini ImageNet.

The interpolated examples are shown in Figure3(a).

Figure 2: The result of easy examples in (a) color, (b) shuffle and (c) radial masking experiments in CIFAR-10 and mini ImageNet. The vertical axis is the matching rate between the case with regular examples and examples from which some patterns are intentionally removed. The model is trained for m-epochs with regular data augmentation.

Figure 4: The result of ablation and corrupted label experiment. The vertical axis is the accuracy. The horizontal axis is the drop rate or random label rate. If the drop rate is 0.3, it means that 30% of examples are discarded. If random label rate is 0.3, it means that 30% of examples are randomly labeled. WRN 16-4 and ResNet 18 are used respectively for CIFAR-10 and mini ImageNet. "easy" on figures means that examples with high easiness are mainly removed or randomly labeled. In the same manner, "hard" denotes examples with low easiness are mainly processed and "random" represents processed examples are randomly selected. Error bars represent standard deviation over 5 trials.

(a) Easiest examples of airplane (b) Hardest examples of airplane (c) Easiest examples of car (d) Hardest examples of car (e) Easiest examples of car (f) Hardest examples of car (g) Easiest examples of cat (h) Hardest examples of cat (i) Easiest examples of deer (j) Hardest examples of deer (k) Easiest examples of dog (l) Hardest examples of dog (m) Easiest examples of frog (n) Hardest examples of frog (o) Easiest examples of horse (p) Hardest examples of horse (q) Easiest examples of ship (r) Hardest examples of ship (a) Easiest examples of sea snake (b) Hardest examples of sea snake (c) Easiest examples of persian cat (d) Hardest examples of persian cat (e) Easiest examples of leaf beetle (f) Hardest examples of leaf beetle (g) Easiest examples of bow tie (h) Hardest examples of bow tie (i) Easiest examples of cradle (j) Hardest examples of cradle (k) Easiest examples of woodworking plane (l) Hardest examples of woodworking plane (m) Easiest examples of strainer (n) Hardest examples of strainer (o) Easiest examples of sundial (p) Hardest examples of sundial (q) Easiest examples of washbasin (r) Hardest examples of washbasin

Let us consider two different set of examples X A and X B . The matching rate M AB between X A and X B is calculated as:

Table content: <table><tr><td>In our experiments, τ is calculated so that 10% of examples with the highest and lowest easiness</td></tr><tr><td>belong to easy and hard examples, respectively.</td></tr><tr><td>2.2 MATCHING RATE</td></tr><tr><td>Based on easiness, we use matching rate to measure how similar easy and hard examples are between</td></tr><tr><td>various CNN architectures.</td></tr></table>

The matching rates of easy and hard examples among various CNNs in CIFAR-10 and mini ImageNet. Since the values in the tables are symmetric and redundant, we put "-" on unnecessary table cells for clarity. We use 10% of examples with the highest and lowest easiness to calculate the matching rate between different CNNs architectures. Random means that we randomly select 10% of examples from the datasets.

Table content: <table><tr><td/><td/><td colspan="6">MLPs WRN DenseNet-BC ResNeXt Random</td></tr><tr><td colspan="2">MLPs</td><td>0.70</td><td>0.20</td><td>0.19</td><td>0.21</td><td>0.1</td><td/></tr><tr><td colspan="2">WRN</td><td>-</td><td>0.86</td><td>0.71</td><td>0.49</td><td>0.1</td><td/></tr><tr><td colspan="3">DenseNet-BC -</td><td>-</td><td>0.91</td><td>0.47</td><td>0.1</td><td/></tr><tr><td colspan="2">ResNeXt</td><td>-</td><td>-</td><td>-</td><td>0.82</td><td>0.1</td><td/></tr><tr><td/><td/><td colspan="4">(a) Matching rate of easy examples in CIFAR-10</td><td/><td/></tr><tr><td/><td/><td colspan="6">MLPs WRN DenseNet-BC ResNeXt Random</td></tr><tr><td colspan="2">MLPs</td><td>0.96</td><td>0.15</td><td>0.14</td><td>0.14</td><td>0.1</td><td/></tr><tr><td colspan="2">WRN</td><td>-</td><td>0.90</td><td>0.64</td><td>0.50</td><td>0.1</td><td/></tr><tr><td colspan="3">DenseNet-BC -</td><td>-</td><td>0.83</td><td>0.47</td><td>0.1</td><td/></tr><tr><td colspan="2">ResNeXt</td><td>-</td><td>-</td><td>-</td><td>0.86</td><td>0.1</td><td/></tr><tr><td/><td/><td colspan="4">(b) Matching rate of hard examples in CIFAR-10</td><td/><td/></tr><tr><td/><td colspan="7">MLPs AlexNet VGG VGG-BN ResNet DenseNet Random</td></tr><tr><td>MLPs</td><td>0.63</td><td>0.61</td><td>0.67</td><td>0.31</td><td>0.52</td><td>0.48</td><td>0.1</td></tr><tr><td>AlexNet</td><td>-</td><td>0.87</td><td>0.72</td><td>0.41</td><td>0.50</td><td>0.48</td><td>0.1</td></tr><tr><td>VGG</td><td>-</td><td>-</td><td>0.90</td><td>0.37</td><td>0.59</td><td>0.59</td><td>0.1</td></tr><tr><td colspan="2">VGG-BN -</td><td>-</td><td>-</td><td>0.87</td><td>0.25</td><td>0.31</td><td>0.1</td></tr><tr><td>ResNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.96</td><td>0.65</td><td>0.1</td></tr><tr><td colspan="2">DenseNet -</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.75</td><td>0.1</td></tr><tr><td/><td colspan="5">(c) Matching rate of easy examples in mini ImageNet</td><td/><td/></tr><tr><td/><td colspan="7">MLPs AlexNet VGG VGG-BN ResNet DenseNet Random</td></tr><tr><td>MLPs</td><td>0.0</td><td>0.07</td><td>0.03</td><td>0.0</td><td>0.01</td><td>0.01</td><td>0.1</td></tr><tr><td>AlexNet</td><td>-</td><td>0.94</td><td>0.48</td><td>0.09</td><td>0.26</td><td>0.25</td><td>0.1</td></tr><tr><td>VGG</td><td>-</td><td>-</td><td>0.78</td><td>0.20</td><td>0.45</td><td>0.26</td><td>0.1</td></tr><tr><td colspan="2">VGG-BN -</td><td>-</td><td>-</td><td>0.46</td><td>0.46</td><td>0.42</td><td>0.1</td></tr><tr><td>ResNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.52</td><td>0.44</td><td>0.1</td></tr><tr><td colspan="2">DenseNet -</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.68</td><td>0.1</td></tr><tr><td/><td colspan="5">(c) Matching rate of hard examples in mini ImageNet</td><td/><td/></tr><tr><td colspan="4">multilayer perceptrons (MLPs) WRN 16-4</td><td/><td/><td/><td/></tr></table>

9 CONCLUSION AND FUTURE WORKIn this paper, unknown properties of easy and hard examples are revealed. Our experiments demonstrate that different CNNs start learning from identical examples, and easy and hard examples contribute differently to generalization. However, discovered properties can not be explained by a hypothesis that CNNs start learning from simple patterns

Table content: <table/>

Peer review (iclr2019):
 The paper formulates a definition of easy and hard examples and studies the properties and the training implications of such examples. The paper does not attempt to present insights that change training for the better (although suggests this could be future work), so the primary value it claims to add is our understanding of neural networks. I think the paper presents findings that most deep learning practitioners already find intuitive, which is why I think the paper falls short in its primary mission. An exposé like this could be valuable for an introductory text in deep learning, but I do not think the analysis meets the bar for a cutting edge insight and do not think it should be accepted. Strengths: - Quantifying easy and hard and using that as a starting point for further analysis is not a bad starting point at all. - The experiments form a good starting point for interesting analysis. - The paper is easy to follow and understand. Weaknesses: - The biggest weakness is that I just didn*t find any of conclusions from this study to be that surprising or interesting. I think it*s pretty obvious that neural networks start by learning the most immediately discriminative features ("frequent patterns"). The visual examples of easy and hard examples are not surprising at all (I think you get similar clusters if you just show bottom and top of model confidences, which I think many of us have). In section 7.1, the result that most misclassified examples are hard examples is presented as a surprising result. This is confusing, because this exactly what I would have expected given how you define easy/hard. It would be far more surprising if misclassified examples were all considered easy under your definition. - The paper only scratches the surface. In Figure 2, the results for the two different datasets are quite different. This means only two datasets is probably not enough for us to understand what is going on here in general. Just the conclusion that datasets may be different in terms of easy/hard samples does not take the analysis far enough. It*s also unclear what the reader should make of these conclusions. - In Table 1, let*s say the bottom 30% of samples are actually equally easy. This would mean that the "easy" examples are just a random 1/3 of those samples. Basically, I*m worried about the implications of having a hard cut-off at 10% and if there are situations where the bottom 10% actually changed quite a bit, but the broader picture of easy really didn*t change that much. I guess I*m saying that I didn*t quite gain confidence that definitions of easy/hard and matching rate are the correct way to go here and there might be a better metric that can look at the continuum of easy/hard from e = 0 to 1. You could have some kind of distance function where if an example moved from 5th percentile to 12th percentile, it would constitute a distance of 7. This is perhaps not the right thing either, but presenting an alternative metric and showing that the numbers (up to scale) and conclusions are unchanged would be nice. Other comments: - The new terminology of "contradicted pattern" and "non-contradicted pattern" is a bit confusing. Why aren*t you just calling these "non-discriminative" and "discriminative"? If a mantis and a ladybird are both typically on a leaf, the leaf is not discriminative for this task. However, if a mantis and a boat are typically on differently colored backgrounds, the background is discriminative. Minor comments: - page 1, "easy and hard examples differ on various CNNs architectures" -> "CNN" - page 2, "as a criteria" -> "criterion" - page 2, "We then redefine easy and hard" -> don*t you mean just "define"? Or do you mean that the words already have casual meanings, so this is a redefinition? I still think "define" is less confusing here. - page 2, I think it*s confusing that both easy and hard use the threshold .tau, suggesting it is the same. Maybe put a subscript to make it clear that the two .taus are different. - page 6, "accuracy does not drop" -> could use a "does not *even* drop" for clarity - page 7, "7.1 Do misclassified examples in validation dataset are hard examples": "Do"->"Are", remove "are"