Year: 2021

Abstract:
Convolutional Neural Networks (CNNs) is one of the most essential architectures that has shown impressive performance in computer vision tasks such as image classification, detection, and segmentation. In general, an arbitrary number of epochs is used to train such neural networks. In a single epoch, the entire training data—divided by batch size—are fed to the network. However, the optimal number of epochs required to train a neural network is not well established. In practice, validation data is used to identify the generalization gap. To avoid overfitting, it is recommended to stop training when the generalization gap increases. However, this is a trial and error based approach. This raises a critical question: Is it possible to estimate when neural networks stop learning based on only the training data? In this research work, we introduce a hypothesis that analyze the data variation in a layer to predict its near optimal learning capacity. Based on this hypothesis, we predict the near optimal epoch number to train a CNN without using any validation data. We experiment our hypothesis on six different CNN models and on three different datasets (CIFIR 10, CIFIR 100, and SVHN). We save on average 58.49% computational time to train a CNN model. Our code is available at https://github.com/PaperUnderReviewDeepLearning/Optimization.

heading: 1 INTRODUCTION
text: “Wider and deeper are better” has become the rule of thumb to design deep neural network architecture (Guo et al., 2020; Huang et al., 2017; He et al., 2016b; Szegedy et al., 2015; Simonyan & Zisserman, 2014b). Deep neural network requires large amount of data to be trained but how much data we should feed to a deep neural network to gain optimum performance is not well established. Deep neural networks behave “double-descent” curve while traditional machine learning models stuck to the “bell-shaped” curve as deep neural networks have larger model complexity (Belkin et al., 2019). However, in deep neural network the data interpolation is reduced as the data are fed into the deeper layers of the network. This raises a critical question: Can we predict whether the deep neural network keeps learning or not based on the training data behavior?
Convolutional Neural Network (CNN) gains impressive performance on computer vision task (He et al., 2016a). Deeper layer based CNN tends to achieve higher accuracy on vision task (e.g., image classification) (Sinha et al., 2020). In terms of computational time saving, light-weighted CNN architectures are introduced which have a trade-off between speed and accuracy. However, when a CNN architecture reaches its’ model capacity and stops significant learning from the training data remains unclear.
In training phase, all training data are fed into the CNN as an epoch. Current practice is to use many epochs (e.g., 200∼500) to train a CNN model. The optimum number of epochs required to train a CNN model is not well researched. To infer whether the model keeps learning or not in each epoch, validation data are used alongside training data. Traditionally, training of the model is stopped when the validation error or generalization gap starts to increase (Goodfellow et al., 2017). Generalization gap indicates model’s capability to predict unseen data. However, this current approach is based on trial-and-error. Our research objective is to replace this trial-and-error based approach by algorithmic approach.
We hypothesize that a layer after convolution reaches its near optimal learning capacity if the produced data have significantly less variation. We use our hypothesis to identify the epoch where all
the layers reach their near optimal learning capacity which also represents the model’s near optimal learning capacity. Thus, our hypothesis predicts the near optimal epoch number to train a CNN model without using any validation data set.
The selection of optimal epoch number to train a deep neural network is not well established. Followings are some of the recent works that use different epoch number for their experiments: Zhang & He (2020) use 186 epochs, Piergiovanni & Ryoo (2020) use 256 epochs, Peng et al. (2020) use 360 epochs, Khalifa & Islam (2020) use 150 epochs. For CIFIR10 and CIFIR100 dataset Li et al. (2020) and Reddy et al. (2020) use 200 epochs. For ResNet and VGG architecture, Kim et al. (2020) use 200 epochs. Dong et al. (2020) and Liu et al. (2020) also use 200 epochs for their experiment. Huang et al. (2020) use 50∼500 epochs as a range. Curry et al. (2020) use 1000 epochs for their custom dataset. In short, most deep neural models adapt a safe epoch number.
As illustrated by Figure 1, our hypothesis can be deployed as a plug-and-play to any CNN architecture. Our hypothesis does not introduce any additional trainable parameter to the network. At the end of each epoch, our hypothesis verifies the models’ learning capacity. The training is terminated when the model reaches its near optimal learning capacity.
The main contributions of this paper can be summarized as:
• We introduce a hypothesis regarding near optimal learning capacity of CNN architecture.
• We examine the data variation across all the layers of a CNN architecture and correlate to its near optimal learning capacity.
• The proposed hypothesis can predict the near optimal epoch number to train a CNN model without using any validation dataset.
• The implementation of the proposed hypothesis can be deployed as plug-and-play to any CNN architecture and does not introduce any additional trainable parameter to the network.
• To test our hypothesis, we conduct image classification experiments on six CNN architectures and three datasets. Adding the hypothesis to the existing CNN architectures save 32% to 78% of the computational time.
• Finally, we provide detail analysis of our hypothesis on different phases of training and how we obtain the near optimal epoch number.
2 RELATED WORK
Validation data are used along side training data to identify generalization gap (Goodfellow et al., 2017). Generalization refers the model’s capability to predict unseen data. Increasing generalization gap indicates that the model is going to overfit. It is recommended to stop training the model at that point. However, this is a trial and error based approach that has been widely used in current training strategy. In order to use this strategy, validation dataset is required.
In terms of model complexity, modern neural networks have more complexity compared to the classical machine learning methods. In terms of biasvarience trade off for generalization of neural networks, traditional machine learning methods behave the “bell-shaped”, and modern neural networks behave the “double descent curve” (Belkin et al., 2019).
To the best of our knowledge, there is no previous work that investigates at what optimal epoch a CNN model stops learning. However, there are CNN archi-
tectures that aim at obtaining best possible accuracy under a limited computational budget based on different hardware and/or applications. This results a series of works towards light-weight
CNN architectures and have speed-accuracy trade-off, including Xception (Chollet, 2017), MobileNet (Howard et al., 2017), ShuffleNet (Zhang et al., 2018), and CondenseNet (Huang et al., 2018). They use FLOP as an indirect metric to compare computational complexity. ShuffleNetV2 (Ma et al., 2018) uses speed as a direct metric while considering memory access cost, and platform characteristics. However, considering epoch number as metric to analyze computation on CNN or at what optimal epoch CNN reaches optimal learning capacity is not well researched.
For a specific dataset and CNN architecture, the usual practice is to adapt a safe epoch number. However, the epoch number selection is random and an arbitrary safe number is picked for most of the experiments. This inspires us to conduct our research work to find out when CNN almost stops learning and to predict near optimal epoch number to train any CNN architecture regardless of dataset.

heading: 3 TRAINING BEHAVIOR ON DEEP NEURAL NETWORK

heading: 3.1 CONVOLUTIONAL NEURAL NETWORK (CNN)
text: In deep learning, a typical CNN is composed of stacked trainable convectional layers (LeCun et al., 1998). In one epoch (e), the entire training data is sent by multiple iteration (t) in batch size (N). The entire training data sent in one epoch is expressed as e = Nt. The input tensor X is organised by batch size N , channel number c, height h, and width w as X(N, c, h, w). A typical CNN convolution operation on n-th layer can be mathematically represented by Equation 1, where θw are the learned weights of the convolutional kernel.
Xn = (θw ~ Xn−1) (1)
3.2 STABILITY VECTOR
During training phase, we examine whether the deep leaning model is learning or not by measuring data variation after convolution operation. We introduce stability vector S to measure data variation. In every epoch, we construct stability vector for each layer of the deep neural network. The stability vector for e-th epoch and n-th layer is denoted by Sne. At each epoch and each layer, we construct stability vector Sne by computing stability value for all the iterations (t) of an epoch. At t-th iteration, after the convolution of n-th layer, we measure stability value (element of a stability vector) αnt by computing the standard deviation value of Xnt as αnt = σ(Xnt). The pro-
cess of constructing the elements of stability vector is shown in Figure 2.
After t iterations at n-th layer and e-th epoch, we have the stability vector Sne = [αn1, αn2, . . . , αnt]. The process of constructing stability vectors for all the layers (i.e., layers 1 to n) after t number of iterations at epoch e is shown in Figure 3. At every epoch, we have n number of stability vectors (i.e., based on number of layers) where each vector has size t.

heading: 3.3 LAYER AND MODEL STABILITY
text: We compute the mean of stability vector, µen, at e-th epoch and n-th layer by the following equation:
µen = 1
t t∑ i=1 αni (2)
We define a function pr that rounds a number to decimal places r. Thus, if µen = 1.23456, p 2(µen) will return 1.23. For each layer n, we compare the mean of stability vector with previous epoch by rounding to decimal places r by using the following equation:
δn = p r(µen)− pr(µe−1n ) (3)
At n-th layer and e-th epoch, if δn equals zero, we consider that the n-th layer is stable on e-th epoch. If all layers shows the stability by ∑n i=1 δi = 0, then it indicates the possibility that the CNN model becomes stable (i.e., reaches its near optimal learning capacity) and it cannot extract significant information from the training data. To make sure that the CNN model reaches its near optimal learning capacity, we verify the ∑n i=1 δi = 0 for two more epochs and if the result remains same, we reach to the conclusion that the model reaches near optimal learning capacity and we terminate the training phase. The trained model is now ready for testing environment.
All the variables we use in our hypothesis are not trained via back-propagation, and do not introduce any trainable parameter to the network.

heading: 3.3.1 RESNET18 STABILITY ON CIFAR100 DATASET
text: On CIFAR100 dataset, the total number of training sample is 50000. We consider 64 as the batch size for training (i.e., N = 64). So, we need 5000064 = 782 iterations (i.e., t = 782) in an epoch (e) to use entire training data.
At epoch e and layer n, the first iteration constructs the first element (i.e., αn1) of stable vector Sne. In ResNet18 architecture, at epoch e, there are 18 layers and for each layer we construct one stability vector, so we have in total 18 stability vectors (i.e., S1e, S2e, . . . , S18e). The length of each stability vector is 782 because each epoch consists of 782 iterations (Figure 3). Table 1 shows the p2(µen) values for epoch 73 to 76. As the δn is 0 for four consecutive epochs, our hypothesis terminates the ResNet18 training on CIFAR100 dataset at epoch 76.

heading: 4 EXPERIMENTS
text: In this section, we empirically evaluate the effectiveness of our hypothesis on six different CNN architectures such as ResNet18 (He et al., 2016a), ResNet18+CBS (Sinha et al., 2020), CNN (LeCun et al., 1998), CNN+CBS (Sinha et al., 2020), VGG (Simonyan & Zisserman, 2014a), and VGG+CBS (Sinha et al., 2020). We test these CNN architectures on three different datasets (CIFIR10, CIFIR100 (Krizhevsky et al.), and SVHN (Netzer et al., 2011)) and analyze the Computational Time Saving (CTS) and Top-1 classification accuracy by using our hypothesis. We further provide an ablation study to analyze the influence of our strategy.

heading: 4.1 DATASETS, TASKS, AND CNN ARCHITECTURES
text: To evaluate our hypothesis, we perform image classification task on two standard vision datasets namely CIFAR10 and CIFAR100 which contain images for 10 and 100 classes, respectively. SVHN, the other dataset, is a digit recognition dataset which consists of natural images of the 10 digits collected from the street view. Table 2 shows more details about these datasets.
CNN demonstrated remarkable performance in computer vision task. Both ResNet and VGG are based on CNN architecture and have different variations based on number of layers. We consider ResNet18 and VGG16 variations in our experiment to compare our hypothesis with Sinha et al. (2020). Curriculum by Smoothing (CBS) gains performance on the top of these architectures. CBS controls the amount of high-frequency information during the training phase. CBS augments the training scheme and increases the amount of information in the feature maps so that the network can learn progressively better representation of the data.

heading: 4.2 COMPUTATIONAL TIME SAVING (CTS)
text: We report the Computational Time Saving (CTS) based on epoch and iteration numbers. We consider 200 epochs as the benchmark number to compare with Sinha et al. (2020) use 200 epochs as default parameter in CBS (Curriculum by Smoothing). Li et al. (2020) use 200 epochs on CIFAR10 and CIFAR100 datasets. Kim et al. (2020) use 200 epochs on VGG and ResNet architecture. As we use CBS, VGG, and ResNet architectures on CIFAR10, CIFAR100 datasets, we compare the complexity based on 200 epochs for all of our experiments (i.e., six CNN architectures, and three datasets). The iteration number is different based on the dataset size but we use fixed batch size number to obtain iteration number for a dataset (Table 2). In Table 2, we show the batch size (N ), iteration number (t), training, and validation data. We keep the batch size constant (i.e., 64) for all datasets. That is, in one iteration, the model uses 64 samples.
As an example, the training and validation iteration for CIFAR100 is 782 and 157, respectively (Table 2). In order to run ResNet18 architecture on CIFIR100 dataset, it requires 200 × 782 + 200 × 157 = 187800 (Table 3) iterations. Our hypothesis predicts 76 as the near optimal epoch number to train ResNet18 on CIFAR10. The updated architecture based on our hypothesis requires 76 × 782 = 59432 iterations which saves 68.35% computation and gains 0.29 top-1 classification accuracy.

heading: 4.3 ABLATION STUDY
text: The ablation study results are summarized in Table 3. To evaluate the Computational Time Saving (CTS) and Top-1 classification accuracy (Acc.), we run 36 experiments, 18 of them are conducted without our hypothesis and the rest 18 are conducted with our hypothesis. Our hypothesis predicts the near optimal epoch numbers, which are significantly less than 200, for all the 18 experiments. On these three datasets and across the six CNN architectures, the 200 epoch number is considered a safe one by the respective researchers.
By using our hypothesis, computational time saving ranges from 32.12% to 79.34%. On average, we save 58.49% computational time based on the 36 experiments. In terms of top-1 classification accuracy, the result varies. Compared to the state-of-the-art frameworks, the maximum accuracy we gain for VGG16+CBS architecture on CIFAR100 dataset is 1.3 and the maximum accuracy drop for CNN on CIFAR10 dataset is 0.94. On average, the accuracy drops 0.106, based on the 36 experiments.

heading: 5 ANALYSIS OF OUR HYPOTHESIS
text: To analyze our hypothesis, we study the data behavior at all the layers of CNN architectures during the training phase. We introduce data stability concept in layers of CNN which identifies a CNN model’s ability to learn from training data. Our hypothesis predicts the near optimal epoch number required to train a CNN model. We perform ablation study which supports our hypothesis.

heading: 5.1 TRAINING BEHAVIOR ANALYSIS
text: Figure 4 shows the Sne values (for layers 1, 5, 9, 13, and 18) for ResNet18 on CIFAR100 datasets across 200 epochs. Each epoch contains 782 data points for each layer (Table 2). To explain the Sne values behavior, we divide the training
phase into the following four different phases to identify near optimal epoch number where a CNN model gets stable and stops learning: initial phase, curved phase, curved phase to stable phase, and stable phase.
In Figure 5, we show the µen value which is the mean of Sne values for an epoch e and layer n. We also show the µen behavior in these phases. Our goal is to identify the ‘stable phase’ to predict the near optimal epoch number required to train a CNN architecture.
Figure 5 contains more detail explanation of our analysis during these four phases. Each data point represents the mean of stability vector (µen). We use subplots for different layers to provide better understanding.
Initial phase refers to the early stage of training. In this phase, Sne values are unstable across all the layers (Figure 4). We also observe a sharp drop of µen values in most of the layers (Figure 5a). For ResNet18 architecture on CIFAR100 dataset, the approximate range of initial phase is from epoch 1 to epoch 25. The range of all the phases can vary based on CNN architecture and dataset.
It is noteworthy that at layer 1, 5, 9, and 13 the µen values decrease but at layer 18 the µ e n values increase with epochs (Figure 5a, 5b, 5c). The reason behind this is that there is an average pool function used in the last layer of ResNet18 architecture.
Curved phase refers to the smooth changes of Sne values in training phase. We observe Sne values gradually increase or decrease (Figure 4) in curved phase. For ResNet18 on CIFAR100 dataset, the
(a) µen values show the instability during the initial phase of training from epoch 1 to 25 for ResNet18 on CIFAR100 dataset. The instability shows for layer 1, 5, 9, 13, and 18. The sharp drop of µen values can be observed in the initial phase.
(b) µen values show the gradual increase or decrease from epoch 26 to 55 for ResNet18 on CIFAR100 dataset. This smooth transition of µen values creates a curved shape across all layers.
(c) µen values show significantly low fluctuation as the model gets closer to its near optimal learning capacity.
(d) As the rate of change of µen values gets significantly low, the probability of getting stable p2(µen) values for consecutive epochs gets higher. At stable phase, δn=0 indicates that the CNN reaches its near optimal learning capacity and terminates the training. Our hypothesis predicts 76 as the near optimal epoch number for ResNet18 on CIFAR100 dataset.
Figure 5: ResNet18’s data stability (µen) on CIFIR100 dataset. Figures 5a, 5b, and 5c show µ e n values on initial phase, curved phase, and curved phase to stable phase. Figure 5d shows p2(µen) values on stable phase.
approximate range of curved phase is from epoch 26 to epoch 55 (Figure 4). Figure 5b also shows that µen values across all layers create a smooth shaped curve.
Curved phase to stable phase in the start of this phase, the µen values fluctuate but as training goes on the fluctuation gradually decreases with epochs. Figure 5c shows the µen values for ResNet18 on CIFAR100 dataset ranging epoch 56 to 72.
Stable phase refers to the range of epochs where the change of µen values are almost insignificant across all the layers. For each layer n, we compare mean of stability vector with previous epoch by rounding to decimal places r using Equation 3 to compute δn. If there is no significant difference between two epochs’ mean of stability vectors for all the layers, it indi-
cates the possibility that the CNN model may get stable. To make sure that the CNN model reaches its near optimal learning capacity, we verify the ∑n i=1 δi = 0 for two more epochs.
Figure 5c shows stable region for ResNet18 architecture on CIFAR100 dataset. In the figure, we can observe that after two decimal points there is no change in µen values from epoch 73 to 76 for all n layers. So, for ResNet18 architecture and on CIFAR100 dataset, the training is terminated at epoch 76.
It is noteworthy that in the stable phase, we compute δn by using function pr and we choose the value of r = 2. Choosing the value of r = 1 causes very early stop of the training, while r = 3 does not guarantee to stop training at the earliest optimal epoch. Choosing r > 3 does not stop training even if the epoch number is large enough1.
We observe data stability (Sne) for six CNN architectures on CIFAR10, CIFAR100, and SVHN datasets. Figure 6 shows these six CNN architectures’ Sne values during training phase on CIFAR100 dataset.

heading: 5.2 GENERALIZATION ABILITY
text: We also analyze the generalization ability of CNN architectures across wide range of epoch numbers in training. Figure 7 shows the testing accuracy (top-1 classification) of ResNet18, VGG16, and CNN on CIFIR10 dataset where 10 to 200 epochs are used for training. It also includes the testing accuracy where the models are trained by the epoch number predicted by our hypothesis (marked by X). From Figure 7, it is clear that all the models testing accuracy reach to a stable stage after certain number of training epoch. Our hypothesis predicts that ResNet18, VGG16, and CNN reach the near optimal learning capacity at epochs 59, 109, and 78, respectively. Figure 7 also supports our hypothesis because all these models’ generalization ability (i.e., the ability to predict on unseen data) does not significantly improve after the near optimal epoch number predicted by the hypothesis.

heading: 6 CONCLUSION
text: In this work, we introduce and analyze the data stability of CNN architectures in layers. We discover similar data stability pattern in six CNN architectures on three datasets. We propose a hypothesis that can identify near optimal epoch number where CNN architecture stops learning. We also show that current practices select random and safe epoch number to conduct experiment, whereas our hypothesis predicts the near optimal epoch number which is significantly less than the safer option. The proposed hypothesis does not require validation dataset as well as does not introduce any trainable parameter to the network. The implementation of the hypothesis can easily be attached to any existing CNN architecture. We also provide ablation study that shows the computational time saving across six CNN architectures on three datasets. Identifying near optimal epoch number as well as computational time saving support the effectiveness of our hypothesis. Future work is to identify other statistical properties of the data in deeper layers of deep learning model. It will help us to better understand the insight about how data change its’ behavior in deeper layers.
1We checked with r = 4 for ResNet18 architecture on CIFAR 100 dataset and for VGG16 architecture on SVHN dataser and the models do not stop training even after 350 epoch

Review summaries:
The paper proposes a data stability measure for early stopping in training various CNN architectures (RestNet18, VGG16), and a shallow CNN. Experiments demonstrate that comparing the means of layer weights across epochs to a certain number of decimal places allows for a significant (~30%-~70%) computational time savings with a relatively small drop in accuracy. Methodology is thoroughly quantitatively evaluation on classification networks, and a detailed ablation study in presented. 

The paper presents a method to measure the stability of vectors obtaining during training of a CNN as proxy measure for convergence of the model. This is used in a early-stopping fashion to allow interruption of the training procedure, instead of training for a fixed number of iterations/epochs. In comparison with training for a fixed number of epochs and using Curriculum by Smoothing (CBS), the proposed method is competitive in terms of final accuracy, while reducing the number of total epochs. Also, stability measured in 3 benchmark datasets behave similarly.

The paper describes a method to define the optimal number of epochs for training process without using validation set.  The work proposes to compute, for each epoch and for each layer, the stability vector defined as the vector containing the standard deviation of the activations for each CNN layer, for all iterations of a particular epoch.  At each epoch, authors compute the mean of the stability vector for each layer (average over the iterations). If the difference between the mean of the stability vectors of two successive epochs for a layer is small, we can assume that the layer is stable for the considered epochs. If this happens for all layers, we can stop the training process.   The proposed approach is of simple implementation and the authors tested it on 6 different CNN based architectures for classification on CIFAR10, CIFAR100 and SVHN datasets.