Title: EMPIRICAL STUDY OF EASY AND HARD EXAMPLES IN CNN TRAINING Anonymous

Abstract: Deep Neural Networks (DNNs) generalize well despite their massive size and capability of memorizing all examples. There is a hypothesis that DNNs start learning from simple patterns based on the observations that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite the importance of understanding the learning dynamics of DNNs, properties of easy and hard examples are not fully investigated. In this paper, we study the similarities of easy and hard examples respectively among different CNNs, assessing those examples contributions to generalization. Our results show that most easy examples are identical among different CNNs, as they share similar dataset-dependent patterns (e.g., colors, structures, and superficial cues in high-frequency). Moreover, while hard examples tend to contribute more to generalization than easy examples, removing a large number of easy examples leads to poor generalization, and we find that most misclassified examples in validation dataset are hard examples. By analyzing intriguing properties of easy and hard examples, we discover that the reason why easy and hard examples have such properties can be explained by biases in a dataset and Stochastic Gradient Descent (SGD).

1. INTRODUCTION:

From a traditional perspective of generalization, overly expressive models can memorize all examples and result in poor generalization. However, deep neural networks (DNNs) achieve an excellent generalization performance even if models are over-parameterized (Zhang et al., 2017) . The reason for this phenomenon remains unclear. (Arpit et al., 2017) show that brute-force memorization does not happen in DNNs training, and propose a hypothesis that DNNs start learning from simple patterns. Their hypothesis is based on observations of examples that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite their importance in understanding how DNNs start learning, easy and hard examples are not thoroughly examined.

In this paper, we study easy and hard examples, and their intriguing properties are shown. First, we investigate whether easy and hard examples differ on various CNNs architectures. For our experiments, we introduce easiness as a metric to measure how early examples are classified. According to easiness, we calculate the matching rates of easy and hard examples between different CNN architectures. Surprisingly, CNNs start learning from identical easy examples and they are visually similar to each other. It indicates that different CNNs start learning from similar patterns.

Second, we investigate the patterns that make examples easy. We study CNNs with examples from which some patterns are intentionally removed. We find that the patterns CNNs start learning from are dataset dependent (e.g., colors, structures, and high-frequency information).

Lastly, we conduct ablation and corrupted-label experiment to determine how easy and hard examples contribute differently to generalization. We find that hard examples contribute more to generalization than easy examples, however, removing a large number of easy examples leads to poor generalization. This phenomenon can be explained by biases in a dataset. In addition, we discover that misclassified examples in validation dataset are consistent and mostly hard examples.

Those properties can not be explained by a hypothesis that CNNs start learning from simple patterns (Arpit et al., 2017) . From observations, we hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes. There are two reasons why there are easy and hard examples: biases in a dataset and Stochastic Gradient Descent (SGD).

A dataset naturally contains various biases leading some patterns to appear as a majority or a minority. The majority and minority patterns are called frequent patterns and rare patterns, respectively. Since SGD randomly picks samples to train a model, frequent patterns tend to be used more than rare patterns in a batch. It means that the gradients of frequent patterns dominate the direction of the update, and some examples that are discriminative by frequent patterns are consistently well classified at early epochs.

2. APPROACH:

In this paper, we perform several experiments to investigate unknown properties of easy and hard examples. We introduce easiness as a metric of how early examples are classified. Based on easiness, we reveal several properties of easy and hard examples.

2.1. EASINESS:

To measure how early an example is classified, we introduce easiness e t as a criteria, where t denotes the index of an example. We repeat m-epochs training N times and take a record of the number of correct classifications. Let c t be the number of trials in which the example x t is correctly classified. easiness e t is computed as:

EQUATION

where 

EQUATION

where |•| denotes the size of a set.

3. EXPERIMENTS:



3.1. PREPARATIONS:

We utilize CIFAR-10 (Krizhevsky, 2009) and ImageNet 2012 dataset (Russakovsky et al., 2015) for our experiments.

CIFAR-10. On CIFAR-10, translation by 4 pixels, horizontal flipping, and global contrast normalization are applied onto images with 32 × 32 pixels. We use a list of models to CIFAR-10: three layer (Zagoruyko & Komodakis, 2016) , DenseNet-BC 12-100 (Huang et al., 2017) and ResNeXt 4-64d (Xie et al., 2017) mini ImageNet. We randomly select 10 classes1 from ImageNet 2012 dataset for an interpretable analysis and call them mini ImageNet. On mini ImageNet, resizing images with the scale and aspect ratio augmentation and horizontal flipping are applied onto images. Then, global contrast normalization is applied to randomly cropped images with 224 × 224 pixels. We use a list of models to mini ImageNet: three layer MLPs, AlexNet (Krizhevsky et al., 2012) , VGG11 (Simonyan & Zisserman, 2015) , VGG11 with Batch Normalization (Ioffe & Szegedy, 2015) , ResNet 18 (He et al., 2016) and DenseNet 121 (Huang et al., 2017) As the optimizer, we use AdamW (Loshchilov & Hutter, 2017) and parameters are as follows: α = 0.001, β 1 = 0.9, β 2 = 0.99, w norm = 0.05 and T i = 300. We train a model for 300 epochs and use 128 batch size for CIFAR-10 and 64 batch size for mini ImageNet.

4. EASY EXAMPLES ARE IDENTICAL IN DIFFERENT CNNS:

To determine whether easy and hard examples are shared across different CNN architectures, we calculate matching rates according to easiness. Table 1 (a) and (c Table 1 (b) and (d 

Easy examples. The matching rates of easy examples between CNNs in:



6. WHY CNNS CONSIDER THOSE EXAMPLES AS easy:

To further analyze what properties make examples easy for CNNs, we perform three experiments: color, shuffle and radial masking experiment (Jo & Bengio, 2018) . In these experiments, we calculate easiness based on examples from which some patterns are intentionally removed.

Color experiment. To investigate whether CNNs look at colors at early epochs, we use easy examples that are interpolated to grey. Let us consider that R, G, B, and Grey ∈ R H×W are red, green, blue and grey channels respectively with height H and width W . The interpolations are calculated as follows: where α is in the range of 0 ≤ α ≤ 1, with 0 indicating that the example is the grey picture and with 1 showing that the example is colored. Grey channel is obtained by averaging RGB channels: Shuffle experiment. The work of (Noroozi & Favaro, 2016) inspires the idea of shuffle experiment. This experiment lets us understand whether CNNs look at structures at early epochs. Then, we cut images into pieces with fixed tile size and reconstruct images with randomly shuffled tiled images. The number of correct classification for the example c t is calculated by using shuffled tiled images.

EQUATION

Examples of the shuffled images are depicted in Figure 3 (b). Tile Size means the relative size when compared to the full image size (e.g., if Tile Size is 0.33, the size of a tiled image is the 33% size of the full image.).

Radial masking experiment. (Jo & Bengio, 2018) show that DNNs misclassify examples without high-frequency information that a human can recognize, and claims that DNNs solve problems by looking at superficial cues on examples. The goal of this experiment is to check whether CNNs look at such superficial cues to classify easy examples at early epochs. The work of radial masking experiment is conducted by (Jo & Bengio, 2018) . Radial mask M r ∈ R H×W , where H and W are height and width, is computed as follows:

M r [i, j] = 1 if (i, j) -( W 2 , H 2 ) L2 ≤ r, 0 otherwise,

where i and j are indices of M r . M r is a radial mask in the Fourier domain that removes higherfrequency in examples. The bigger r is, the higher frequency of an example is kept. Radial masked examples are shown in Figure 3 (c) .

Result and analysis.:

To verify what patterns CNNs start looking at, we calculate the matching rate between regular examples and examples from which some patterns are intentionally removed. First, we train a model for m-epochs with regular data augmentation. Then, easiness is calculated by using examples from which some patterns are intentionally removed.

The result are shown in Figure 2 . The result of color experiment is the opposite tendency to the result of shuffle and random masking experiment. In CIFAR-10, CNNs start looking at the structure and high-frequency information. In mini ImageNet, CNNs start looking at the colors. It indicates that initially learned patterns are dataset dependent.

In mini ImageNet, we observe that the accuracy does not drop when r is 1.1 small in radial masking experiment. As radial masked examples are depicted in Figure 3 

8. RELATED WORK:

A dataset naturally contains various biases. For instance, (Ponce et al., 2006) shows some averaged images of Caltech-101 (Li et al., 2004) are not homogeneous and recognizable. They claim that Caltech-101 may have inter-class variability but lacks intra-class variability. (Torralba & Efros, 2011) mentions several biases in a dataset: Selection bias means that examples in a dataset tend to have particular kinds of images (example: there are many examples of a sports car in the car category).

Capture bias represents the manner in which pictures are usually taken (example: a picture of a dog is usually taken from the front with the dog looking at the photographer and occupying most of the picture). In this paper, we find that easy and hard examples are highly associated with such biases.

There are three types of training schemes that start learning from easy examples (i.e., curriculum learning) (Kumar et al., 2010) , uncertain examples (Settles, 2010; Chang et al., 2017) or hard examples (i.e., hard negative mining) (Lin et al., 2017; Simo-Serra et al., 2015; Shrivastava et al., 2016; Loshchilov & Hutter, 2016; Wang & Gupta, 2015) . Those methods rely on the loss value (Arpit et al., 2017) . Consequently, to explain why easy and hard examples have such properties, we hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes.

As future work, by using easiness, dataset compression and mislabel mining can be considered for a large-scale dataset. Besides, we would like to analyze the phenomenon of easy and hard examples on natural language and sound domain. Our work utilize datasets with 10 classes, thus it is necessary to conduct further research on large-scale and more complexed datasets. 

:

Image IDs are n01751748, n02123394, n02169497, n02883205, n03125729, n03954731, n04332243, n04355358, n04553703 and n07717556.

Figures and Tables:

) are much higher than the case of random selection. This result shows that CNNs start learning from the same examples even if CNN architectures are different. In addition, the matching rate between MLPs and CNNs are low in CIFAR-10, but high in mini ImageNet. Hard examples. The matching rates of hard examples are shown in

). Intriguingly, in mini ImageNet, the matching rates among CNNs with batch normalization(Ioffe & Szegedy, 2015) are high. It shows that batch normalization in CNNs architecture affects hard examples. Since easy examples are mostly identical, the difference of hard examples relates to the difference of generalization ability. It requires further analysis. Why there are easy and hard examples? We hypothesize that CNNs start learning from frequent patterns that are not contradicted across classes. It can be explained by biases in a dataset and SGD. As a example, let us consider the classification problem of a ladybird and mantis. If a ladybird and mantis are always onto a leaf in training examples, the pattern of a leaf is harmful to discriminate examples. We call such harmfulness as contradiction. SGD force the model not to use contradicted patterns. If the model uses the pattern of a leaf to discriminate a mantis, ladybird examples stochastically update the model to discriminate a ladybird by the pattern of a leaf, and then the model start misclassifying mantis. On the one hand, classified and misclassified examples by contradicted patterns interrupt to use contradicted patterns each other. On the other hand, non-contradicted patterns across classes are learned without interferences and used to discriminate examples. When examples are classified by non-contradicted patterns, their loss value and the magnitudes of their gradients become smaller. Thus, the model stops learning contradicted patterns. There are biases in a dataset, thus some patterns appear frequently and some patterns appear rarely. Since SGD updates the model by averaged gradients, the gradients of frequent patterns become global directions of the gradient to update the model. Easy examples are discriminative by frequent and non-contradicted patterns, thus those patterns are learned first. As easy examples are learned and get smaller loss values, rare patterns in hard examples start being learned. That's why examples are identical among different CNN architectures since such patterns are dependent on a dataset and not on CNNs architectures. 5 EASY EXAMPLES ARE VISUALLY SIMILAR TO EACH OTHER Visualizations 2 of easy and hard examples are shown in Figure 1. To calculate easiness, WRN 16-4 and ResNet18 are respectively used for CIFAR-10 and mini ImageNet. As can be seen in Figure 1, easy examples are visually similar to each other, and hard examples tend to be visually different from each other. This result implies that different CNNs start learning from similar patterns.

The rest of easy and hard examples are provided in the supplementary material.

Figure 1: Easy and hard examples of CIFAR-10 dataset and mini ImageNet. a)-d) are examples from CIFAR-10. e)-h) are examples from mini ImageNet.

The interpolated examples are shown in Figure3(a).

Figure 2: The result of easy examples in (a) color, (b) shuffle and (c) radial masking experiments in CIFAR-10 and mini ImageNet. The vertical axis is the matching rate between the case with regular examples and examples from which some patterns are intentionally removed. The model is trained for m-epochs with regular data augmentation.

Figure 4: The result of ablation and corrupted label experiment. The vertical axis is the accuracy. The horizontal axis is the drop rate or random label rate. If the drop rate is 0.3, it means that 30% of examples are discarded. If random label rate is 0.3, it means that 30% of examples are randomly labeled. WRN 16-4 and ResNet 18 are used respectively for CIFAR-10 and mini ImageNet. "easy" on figures means that examples with high easiness are mainly removed or randomly labeled. In the same manner, "hard" denotes examples with low easiness are mainly processed and "random" represents processed examples are randomly selected. Error bars represent standard deviation over 5 trials.

(a) Easiest examples of airplane (b) Hardest examples of airplane (c) Easiest examples of car (d) Hardest examples of car (e) Easiest examples of car (f) Hardest examples of car (g) Easiest examples of cat (h) Hardest examples of cat (i) Easiest examples of deer (j) Hardest examples of deer (k) Easiest examples of dog (l) Hardest examples of dog (m) Easiest examples of frog (n) Hardest examples of frog (o) Easiest examples of horse (p) Hardest examples of horse (q) Easiest examples of ship (r) Hardest examples of ship (a) Easiest examples of sea snake (b) Hardest examples of sea snake (c) Easiest examples of persian cat (d) Hardest examples of persian cat (e) Easiest examples of leaf beetle (f) Hardest examples of leaf beetle (g) Easiest examples of bow tie (h) Hardest examples of bow tie (i) Easiest examples of cradle (j) Hardest examples of cradle (k) Easiest examples of woodworking plane (l) Hardest examples of woodworking plane (m) Easiest examples of strainer (n) Hardest examples of strainer (o) Easiest examples of sundial (p) Hardest examples of sundial (q) Easiest examples of washbasin (r) Hardest examples of washbasin

Let us consider two different set of examples X A and X B . The matching rate M AB between X A and X B is calculated as:

Table content: <table><tr><td>In our experiments, τ is calculated so that 10% of examples with the highest and lowest easiness</td></tr><tr><td>belong to easy and hard examples, respectively.</td></tr><tr><td>2.2 MATCHING RATE</td></tr><tr><td>Based on easiness, we use matching rate to measure how similar easy and hard examples are between</td></tr><tr><td>various CNN architectures.</td></tr></table>

The matching rates of easy and hard examples among various CNNs in CIFAR-10 and mini ImageNet. Since the values in the tables are symmetric and redundant, we put "-" on unnecessary table cells for clarity. We use 10% of examples with the highest and lowest easiness to calculate the matching rate between different CNNs architectures. Random means that we randomly select 10% of examples from the datasets.

Table content: <table><tr><td/><td/><td colspan="6">MLPs WRN DenseNet-BC ResNeXt Random</td></tr><tr><td colspan="2">MLPs</td><td>0.70</td><td>0.20</td><td>0.19</td><td>0.21</td><td>0.1</td><td/></tr><tr><td colspan="2">WRN</td><td>-</td><td>0.86</td><td>0.71</td><td>0.49</td><td>0.1</td><td/></tr><tr><td colspan="3">DenseNet-BC -</td><td>-</td><td>0.91</td><td>0.47</td><td>0.1</td><td/></tr><tr><td colspan="2">ResNeXt</td><td>-</td><td>-</td><td>-</td><td>0.82</td><td>0.1</td><td/></tr><tr><td/><td/><td colspan="4">(a) Matching rate of easy examples in CIFAR-10</td><td/><td/></tr><tr><td/><td/><td colspan="6">MLPs WRN DenseNet-BC ResNeXt Random</td></tr><tr><td colspan="2">MLPs</td><td>0.96</td><td>0.15</td><td>0.14</td><td>0.14</td><td>0.1</td><td/></tr><tr><td colspan="2">WRN</td><td>-</td><td>0.90</td><td>0.64</td><td>0.50</td><td>0.1</td><td/></tr><tr><td colspan="3">DenseNet-BC -</td><td>-</td><td>0.83</td><td>0.47</td><td>0.1</td><td/></tr><tr><td colspan="2">ResNeXt</td><td>-</td><td>-</td><td>-</td><td>0.86</td><td>0.1</td><td/></tr><tr><td/><td/><td colspan="4">(b) Matching rate of hard examples in CIFAR-10</td><td/><td/></tr><tr><td/><td colspan="7">MLPs AlexNet VGG VGG-BN ResNet DenseNet Random</td></tr><tr><td>MLPs</td><td>0.63</td><td>0.61</td><td>0.67</td><td>0.31</td><td>0.52</td><td>0.48</td><td>0.1</td></tr><tr><td>AlexNet</td><td>-</td><td>0.87</td><td>0.72</td><td>0.41</td><td>0.50</td><td>0.48</td><td>0.1</td></tr><tr><td>VGG</td><td>-</td><td>-</td><td>0.90</td><td>0.37</td><td>0.59</td><td>0.59</td><td>0.1</td></tr><tr><td colspan="2">VGG-BN -</td><td>-</td><td>-</td><td>0.87</td><td>0.25</td><td>0.31</td><td>0.1</td></tr><tr><td>ResNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.96</td><td>0.65</td><td>0.1</td></tr><tr><td colspan="2">DenseNet -</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.75</td><td>0.1</td></tr><tr><td/><td colspan="5">(c) Matching rate of easy examples in mini ImageNet</td><td/><td/></tr><tr><td/><td colspan="7">MLPs AlexNet VGG VGG-BN ResNet DenseNet Random</td></tr><tr><td>MLPs</td><td>0.0</td><td>0.07</td><td>0.03</td><td>0.0</td><td>0.01</td><td>0.01</td><td>0.1</td></tr><tr><td>AlexNet</td><td>-</td><td>0.94</td><td>0.48</td><td>0.09</td><td>0.26</td><td>0.25</td><td>0.1</td></tr><tr><td>VGG</td><td>-</td><td>-</td><td>0.78</td><td>0.20</td><td>0.45</td><td>0.26</td><td>0.1</td></tr><tr><td colspan="2">VGG-BN -</td><td>-</td><td>-</td><td>0.46</td><td>0.46</td><td>0.42</td><td>0.1</td></tr><tr><td>ResNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.52</td><td>0.44</td><td>0.1</td></tr><tr><td colspan="2">DenseNet -</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.68</td><td>0.1</td></tr><tr><td/><td colspan="5">(c) Matching rate of hard examples in mini ImageNet</td><td/><td/></tr><tr><td colspan="4">multilayer perceptrons (MLPs) WRN 16-4</td><td/><td/><td/><td/></tr></table>

9 CONCLUSION AND FUTURE WORKIn this paper, unknown properties of easy and hard examples are revealed. Our experiments demonstrate that different CNNs start learning from identical examples, and easy and hard examples contribute differently to generalization. However, discovered properties can not be explained by a hypothesis that CNNs start learning from simple patterns

Table content: <table/>

Peer review (iclr2019):
 This paper proposes a specific measure of difficulty for training examples called “easiness”. Easiness is based on training the model N times and counting the number of times an example is classified correctly. Based on this measure, they introduce “matching rate” as a measure of similarity of two architectures. Two architectures are suggested to be similar if the set of easy and hard examples is similar. The rest of the paper presents comparisons of architectures. Considering the problems below, I don’t see any reliable contribution in this paper. - Why this specific definition of easiness? Can you compare to simply using “loss” as a measure for the difficulty of an example? - e_t seems to be measuring the variance of training on a single example. If there is only one example that is always classified correctly, the denominator can be simplified to K. It doesn’t tell us how many training iterations it takes to fit that example. - Why this specific formulation for “matching rate”? Why not a more common measure of similarity between sets such as intersection over union (IoU)? Can you suggest any references using a similar similarity score? - Numbers in Table 1 do not seem particularly big to support the claim in section 4 that “...CNNs start learning from the *same* examples even if CNN architectures are different”. 0.20 is definitely bigger than random 0.1 for the matching rate but it still means only a 20% match. - Random 0.1 is redundant in table 1. - In section 4, define “contradicted patterns”. - Are all images in Figure 1 for one model? How does it compare to visualizing examples according to their loss? - The conclusion in section 5 says “... different CNNs start learning from similar patterns”. As mentioned above, “easiness” and consequently “matching rate” do not provide information about the progress of training and only final trained models. Regardless, this conclusion does not seem particularly unexpected or informative. - Section 6 proposes to test a model on data with a different structure from data provided in training. This is a distribution mismatch and the model is not trained to handle.