When do Convolutional Neural Networks Stop Learning?

1 INTRODUCTION . “ Wider and deeper are better ” has become the rule of thumb to design deep neural network architecture ( Guo et al. , 2020 ; Huang et al. , 2017 ; He et al. , 2016b ; Szegedy et al. , 2015 ; Simonyan & Zisserman , 2014b ) . Deep neural network requires large amount of data to be trained but how much data we should feed to a deep neural network to gain optimum performance is not well established . Deep neural networks behave “ double-descent ” curve while traditional machine learning models stuck to the “ bell-shaped ” curve as deep neural networks have larger model complexity ( Belkin et al. , 2019 ) . However , in deep neural network the data interpolation is reduced as the data are fed into the deeper layers of the network . This raises a critical question : Can we predict whether the deep neural network keeps learning or not based on the training data behavior ? Convolutional Neural Network ( CNN ) gains impressive performance on computer vision task ( He et al. , 2016a ) . Deeper layer based CNN tends to achieve higher accuracy on vision task ( e.g. , image classification ) ( Sinha et al. , 2020 ) . In terms of computational time saving , light-weighted CNN architectures are introduced which have a trade-off between speed and accuracy . However , when a CNN architecture reaches its ’ model capacity and stops significant learning from the training data remains unclear . In training phase , all training data are fed into the CNN as an epoch . Current practice is to use many epochs ( e.g. , 200∼500 ) to train a CNN model . The optimum number of epochs required to train a CNN model is not well researched . To infer whether the model keeps learning or not in each epoch , validation data are used alongside training data . Traditionally , training of the model is stopped when the validation error or generalization gap starts to increase ( Goodfellow et al. , 2017 ) . Generalization gap indicates model ’ s capability to predict unseen data . However , this current approach is based on trial-and-error . Our research objective is to replace this trial-and-error based approach by algorithmic approach . We hypothesize that a layer after convolution reaches its near optimal learning capacity if the produced data have significantly less variation . We use our hypothesis to identify the epoch where all the layers reach their near optimal learning capacity which also represents the model ’ s near optimal learning capacity . Thus , our hypothesis predicts the near optimal epoch number to train a CNN model without using any validation data set . The selection of optimal epoch number to train a deep neural network is not well established . Followings are some of the recent works that use different epoch number for their experiments : Zhang & He ( 2020 ) use 186 epochs , Piergiovanni & Ryoo ( 2020 ) use 256 epochs , Peng et al . ( 2020 ) use 360 epochs , Khalifa & Islam ( 2020 ) use 150 epochs . For CIFIR10 and CIFIR100 dataset Li et al . ( 2020 ) and Reddy et al . ( 2020 ) use 200 epochs . For ResNet and VGG architecture , Kim et al . ( 2020 ) use 200 epochs . Dong et al . ( 2020 ) and Liu et al . ( 2020 ) also use 200 epochs for their experiment . Huang et al . ( 2020 ) use 50∼500 epochs as a range . Curry et al . ( 2020 ) use 1000 epochs for their custom dataset . In short , most deep neural models adapt a safe epoch number . As illustrated by Figure 1 , our hypothesis can be deployed as a plug-and-play to any CNN architecture . Our hypothesis does not introduce any additional trainable parameter to the network . At the end of each epoch , our hypothesis verifies the models ’ learning capacity . The training is terminated when the model reaches its near optimal learning capacity . The main contributions of this paper can be summarized as : • We introduce a hypothesis regarding near optimal learning capacity of CNN architecture . • We examine the data variation across all the layers of a CNN architecture and correlate to its near optimal learning capacity . • The proposed hypothesis can predict the near optimal epoch number to train a CNN model without using any validation dataset . • The implementation of the proposed hypothesis can be deployed as plug-and-play to any CNN architecture and does not introduce any additional trainable parameter to the network . • To test our hypothesis , we conduct image classification experiments on six CNN architectures and three datasets . Adding the hypothesis to the existing CNN architectures save 32 % to 78 % of the computational time . • Finally , we provide detail analysis of our hypothesis on different phases of training and how we obtain the near optimal epoch number . 2 RELATED WORK Validation data are used along side training data to identify generalization gap ( Goodfellow et al. , 2017 ) . Generalization refers the model ’ s capability to predict unseen data . Increasing generalization gap indicates that the model is going to overfit . It is recommended to stop training the model at that point . However , this is a trial and error based approach that has been widely used in current training strategy . In order to use this strategy , validation dataset is required . In terms of model complexity , modern neural networks have more complexity compared to the classical machine learning methods . In terms of biasvarience trade off for generalization of neural networks , traditional machine learning methods behave the “ bell-shaped ” , and modern neural networks behave the “ double descent curve ” ( Belkin et al. , 2019 ) . To the best of our knowledge , there is no previous work that investigates at what optimal epoch a CNN model stops learning . However , there are CNN archi- tectures that aim at obtaining best possible accuracy under a limited computational budget based on different hardware and/or applications . This results a series of works towards light-weight CNN architectures and have speed-accuracy trade-off , including Xception ( Chollet , 2017 ) , MobileNet ( Howard et al. , 2017 ) , ShuffleNet ( Zhang et al. , 2018 ) , and CondenseNet ( Huang et al. , 2018 ) . They use FLOP as an indirect metric to compare computational complexity . ShuffleNetV2 ( Ma et al. , 2018 ) uses speed as a direct metric while considering memory access cost , and platform characteristics . However , considering epoch number as metric to analyze computation on CNN or at what optimal epoch CNN reaches optimal learning capacity is not well researched . For a specific dataset and CNN architecture , the usual practice is to adapt a safe epoch number . However , the epoch number selection is random and an arbitrary safe number is picked for most of the experiments . This inspires us to conduct our research work to find out when CNN almost stops learning and to predict near optimal epoch number to train any CNN architecture regardless of dataset . 3 TRAINING BEHAVIOR ON DEEP NEURAL NETWORK . 3.1 CONVOLUTIONAL NEURAL NETWORK ( CNN ) . In deep learning , a typical CNN is composed of stacked trainable convectional layers ( LeCun et al. , 1998 ) . In one epoch ( e ) , the entire training data is sent by multiple iteration ( t ) in batch size ( N ) . The entire training data sent in one epoch is expressed as e = Nt . The input tensor X is organised by batch size N , channel number c , height h , and width w as X ( N , c , h , w ) . A typical CNN convolution operation on n-th layer can be mathematically represented by Equation 1 , where θw are the learned weights of the convolutional kernel . Xn = ( θw ~ Xn−1 ) ( 1 ) 3.2 STABILITY VECTOR During training phase , we examine whether the deep leaning model is learning or not by measuring data variation after convolution operation . We introduce stability vector S to measure data variation . In every epoch , we construct stability vector for each layer of the deep neural network . The stability vector for e-th epoch and n-th layer is denoted by Sne . At each epoch and each layer , we construct stability vector Sne by computing stability value for all the iterations ( t ) of an epoch . At t-th iteration , after the convolution of n-th layer , we measure stability value ( element of a stability vector ) αnt by computing the standard deviation value of Xnt as αnt = σ ( Xnt ) . The pro- cess of constructing the elements of stability vector is shown in Figure 2 . After t iterations at n-th layer and e-th epoch , we have the stability vector Sne = [ αn1 , αn2 , . . . , αnt ] . The process of constructing stability vectors for all the layers ( i.e. , layers 1 to n ) after t number of iterations at epoch e is shown in Figure 3 . At every epoch , we have n number of stability vectors ( i.e. , based on number of layers ) where each vector has size t . 3.3 LAYER AND MODEL STABILITY . We compute the mean of stability vector , µen , at e-th epoch and n-th layer by the following equation : µen = 1 t t∑ i=1 αni ( 2 ) We define a function pr that rounds a number to decimal places r. Thus , if µen = 1.23456 , p 2 ( µen ) will return 1.23 . For each layer n , we compare the mean of stability vector with previous epoch by rounding to decimal places r by using the following equation : δn = p r ( µen ) − pr ( µe−1n ) ( 3 ) At n-th layer and e-th epoch , if δn equals zero , we consider that the n-th layer is stable on e-th epoch . If all layers shows the stability by ∑n i=1 δi = 0 , then it indicates the possibility that the CNN model becomes stable ( i.e. , reaches its near optimal learning capacity ) and it can not extract significant information from the training data . To make sure that the CNN model reaches its near optimal learning capacity , we verify the ∑n i=1 δi = 0 for two more epochs and if the result remains same , we reach to the conclusion that the model reaches near optimal learning capacity and we terminate the training phase . The trained model is now ready for testing environment . All the variables we use in our hypothesis are not trained via back-propagation , and do not introduce any trainable parameter to the network . 3.3.1 RESNET18 STABILITY ON CIFAR100 DATASET . On CIFAR100 dataset , the total number of training sample is 50000 . We consider 64 as the batch size for training ( i.e. , N = 64 ) . So , we need 5000064 = 782 iterations ( i.e. , t = 782 ) in an epoch ( e ) to use entire training data . At epoch e and layer n , the first iteration constructs the first element ( i.e. , αn1 ) of stable vector Sne . In ResNet18 architecture , at epoch e , there are 18 layers and for each layer we construct one stability vector , so we have in total 18 stability vectors ( i.e. , S1e , S2e , . . . , S18e ) . The length of each stability vector is 782 because each epoch consists of 782 iterations ( Figure 3 ) . Table 1 shows the p2 ( µen ) values for epoch 73 to 76 . As the δn is 0 for four consecutive epochs , our hypothesis terminates the ResNet18 training on CIFAR100 dataset at epoch 76 .

Summary: The paper describes a method to define the optimal number of epochs for training process without using validation set.  The work proposes to compute, for each epoch and for each layer, the stability vector defined as the vector containing the standard deviation of the activations for each CNN layer, for all iterations of a particular epoch.  At each epoch, authors compute the mean of the stability vector for each layer (average over the iterations). If the difference between the mean of the stability vectors of two successive epochs for a layer is small, we can assume that the layer is stable for the considered epochs. If this happens for all layers, we can stop the training process.   The proposed approach is of simple implementation and the authors tested it on 6 different CNN based architectures for classification on CIFAR10, CIFAR100 and SVHN datasets.