When do Convolutional Neural Networks Stop Learning?

Abstract: Convolutional Neural Networks (CNNs) is one of the most essential architectures that has shown impressive performance in computer vision tasks such as image classification, detection, and segmentation. In training phase of CNN, an arbitrary number of epochs is used to train the neural networks. In a single epoch, the entire training data---divided by batch size---are fed to the network. However, the optimal number of epochs required to train a neural network is not well established. In practice, validation data is used to identify the generalization gap. To avoid overfitting, it is recommended to stop training when the generalization gap increases. However, this is a trial and error based approach. This raises a critical question: Is it possible to estimate when neural networks stop learning based on only the training data? In this research work, we introduce the stability property of data in layers and based on this property, we predict the near optimal epoch number of a CNN. We do not use any validation data to predict the near optimal epoch number. We experiment our hypothesis on six different CNN models and on three different datasets (CIFIR 10, CIFIR 100, SVHN). We save on average 58.49\% computational time to train a CNN model. Our code is available at https://github.com/PaperUnderReviewDeepLearning/Optimization.

Reviews and author responses:

The paper proposes a data stability measure for early stopping in training various CNN architectures (RestNet18, VGG16), and a shallow CNN. Experiments demonstrate that comparing the means of layer weights across epochs to a certain number of decimal places allows for a significant (~30%-~70%) computational time savings with a relatively small drop in accuracy. Methodology is thoroughly quantitatively evaluation on classification networks, and a detailed ablation study in presented.  Strengths:
- A simple method for early stopping based on network layer weight averages
- Offers a significant computational time savings
- Technique can be used with any network architecture
- Code is provided
 
Weaknesses:
- Evaluation is performed only on classification networks. How well would this approach for detection, segmentation? How does the number of layers affect performance?
- It is surprising that precision of only two decimal places is needed, given that performance is averaged over such a large number of parameters. 
- As neural networks have a large number of parameters and are ever growing, what is the memory and computational cost of the proposed approach? How much does it slow down training
- In Table 3, using the proposed method mainly makes results worse for CIFAR10, but improves results for CIFAR100, SVHN in some cases. A discussion of these findings would be helpful. A
- Number of grammatical errors make the paper difficult to follow
- Limited discussion of previous works, especially in early stopping and regularization methods, and no comparison to previous works, e.g., the basic validation set strategy.

Very similar ideas are found in the following papers:
Mahsereci, M., Balles, L., Lassner, C., & Hennig, P. (2017). Early stopping without a validation set. arXiv preprint arXiv:1703.09580.

Bonet, D., Ortega, A., Ruiz-Hidalgo, J., & Shekkizhar, S. (2021). Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation. arXiv e-prints, APSIPA 2021

I would also suggest to include some references to uncertainty estimation in neural networks, e.g., 
Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., & Wilson, A. G. (2019). A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 13153-13164. The paper proposes a simple technique for early stopping in neural networks. However, limited discussion and comparisons to previous methods makes it challenging to understand how this technique can be used in practice. I would suggest the authors increase the discussion of related work, compare existing early stopping methods stated above, and explain the computational complexity and memory overhead required. I would also like to see an evaluation on non-classification tasks. 


The authors propose a method to perform early stopping in CNNs based on the training data alone, by measuring when the mean of the variance of the convolutional layers does not change much across several epochs. They show that for several architectures and datasets their method could be used to save computation time, in the cost of minor changes in the accuracy. The paper in its current state feels like a very rough draft. As a non-English speaker, I prefer not to judge a work by the level of its English, but in this case, it is extreme. The paper is full of grammatical errors, spelling errors, and even misuses the names of commonly used datasets (CIFIR instead of CIFAR, for example).

The introduction of the paper is long and repetitive and goes over some points several times. The related work section cites works which I do not understand why should be relevant to this analysis.

The suggested method is also problematic. The authors introduce a new hyper-parameter, "r", which decides when the variance of each convolutional layer did not change "much", so learning should seize. I would expect this hyper-parameter to drastically change when increasing the learning rate and is highly dependant on the chosen architecture and the training data. Tuning this hyper-parameter will require some validation dataset, hence undermining the biggest contribution the paper claims to have - that its proposed method can do not require any additional validation dataset.
 At the current state of this work, I do not think it meets the standards of a Tier 1 conference.
I highly recommend rejecting this paper at the current state.

The paper presents a method to measure the stability of vectors obtaining during training of a CNN as proxy measure for convergence of the model. This is used in a early-stopping fashion to allow interruption of the training procedure, instead of training for a fixed number of iterations/epochs. In comparison with training for a fixed number of epochs and using Curriculum by Smoothing (CBS), the proposed method is competitive in terms of final accuracy, while reducing the number of total epochs. Also, stability measured in 3 benchmark datasets behave similarly. Strengths
- The idea of computing stability of training per-layer and use it to stop learning has merit and it is worth pursuing.
- There is empirical evidence that support the hypothesis of stability as a proxy measure for the rate of learning/changes in the model which can be helpful not only by reducing number of epochs and saving computational time, but also potential to prevent overtraining
- I liked the analysis of initial-curved-stable phases, which may help evaluating convergence in practice

Weaknesses
- My main concern with the paper is the comparison and baselines used. Although CBS and vanilla-learning approaches are possible baselines, I missed the use of a regular early-stopping approach using the difference of the losses of two consecutive iterations as a stopping criterion, and a small validation set to avoid overtraining. Although the proposed method do not use/require a validation set, for the proposed benchmark datasets, it is feasible to separate at least 5% of the data for this purpose. Note the authors do not work on the premise of small data.
- Comparing 200 epochs vs a reduced number of epochs only by its absolute value results in a poor discussion. The authors could have reported for example the "estimated generalization" which is the difference between the training and validation/test loss.  The final loss value of all models should also be reported so that some conclusions could be drawn from this.
- The ablation study does not look like an "ablation", but just a combination of the proposed method with CBS. I recommend the authors to look for the definition of ablation.
- It is true that a fixed "safe" number of epochs is used, but this is because this approach allows inspecting for previous epoch and selecting the model that better fits some criterion (e.g. training vs validation loss). By stopping the training using the proposed method, maybe we could not know what could happen in future iterations
- Including a joint analysis with learning rate scheduling would also be important. For example, in cosine annealing, the premise of the phases that appear using a fixed learning rate can drastically change.

 The paper presents interesting ideas and show experiments with promising results. However, better baselines should be evaluated and more discussion around the effects of early stopping vs stability is important before this paper is ready for publication. I believe the claims are not well supported without comparing with basic stopping criteria (with or without a validation set) and discussing the results. In summary, I believe it is a good initial set of results, but not a work that is ready/mature enough for publication.

The paper describes a method to define the optimal number of epochs for training process without using validation set.

The work proposes to compute, for each epoch and for each layer, the stability vector defined as the vector containing the standard deviation of the activations for each CNN layer, for all iterations of a particular epoch.

At each epoch, authors compute the mean of the stability vector for each layer (average over the iterations). If the difference between the mean of the stability vectors of two successive epochs for a layer is small, we can assume that the layer is stable for the considered epochs. If this happens for all layers, we can stop the training process. 

The proposed approach is of simple implementation and the authors tested it on 6 different CNN based architectures for classification on CIFAR10, CIFAR100 and SVHN datasets. The approach is simple, explained quite clearly and its implementation is straightforward.

The main assumption of the work is that the average (over the iteration) of the standard deviation of the activations of the layers is representative of the status of the learning process.

I have some concerns about this hypothesis:
1. The standard deviation is a measure of dispersion around the mean but the histograms of the activations are not provided to understand if there is a measure that better takes in account the variability inside the activations (e.g. Inter-quartile range could be more precise to capture the effective variability)
2. The average (eq. 2) can mask some fluctuations that can occur in different iterations. In this way if there are similar changes in two iterations but with opposite sign, this information is lose and we can conclude that, for that epoch, there is a sort of stability but this is not the case. 

Moreover, in my opinions, other weakness are:
- This process depends also on the selected optimization algorithm and/or initial seed and/or some tricks that can be implemented (e.g. reduce learning rate on plateau) but these situations are not considered neither tested. For example the different initial seeds are not properly considered in the experiments.
- The limitations of the approach should be clearly described (e.g. is it applicable only to Conv Layers or also to Fully connected layers? Is it applicable in presence of Batch Normalization? etc.)
- Also Batch Normalization involves the estimation of the standard deviation of the activations. Even though its role is different, introduce regularity during the learning, it is neither described nor simply cited.
- The authors described the impact of the value of r on the selection of the "optimal" epoch and it emerges that increasing the value of r doesn't lead to a good stopping criterion. This high sensitivity to the number of significative digits seems to suggest that the proposed approach isn't enough robust. 
- The authors tested their approach comparing it with the fixed epochs defined in some works (i.e. 200). Since the best practice, as also authors describe in the introduction, is the early stopping, a comparison with this approach should be done and the Computational Time Saving should be tested respect to the early stopping situation. 
- The authors cite the Double Descent phenomenon but the epoch-wise double descent is not cited and, in this case, it is important.
- The "Related work" section should be increased since several works that try to assess the learning status are not cited (e.g. loss landscape, relation with the flatness/curvature of the loss, weight watchers etc.) 

Other observations:
- More details should be provided about the optimization algorithms used in tested architecture (e.g. type, learning rate, weights' initialization, etc.)
- From Figure 1 looks like the Stability Vector is computed only for Conv Layers (also the caption confirms that) but in the introduction the authors wrote that "we examine the data variation across all layers of a CNN architecture") 
- In the Figure 2, it is used $S_i$ instead of $S_{ie}$
- The $\alpha$ should contain also the information about the related epoch (e.g. equation 2 or Figure 3), for instance $\alpha_{nt}^e$ 
- The authors should clarify that the operation used in eq. 1 performed by CNNs is different by the mathematical convolution operation, it is a correlation.
- $X_{nt}$ is not properly defined
- In the paper often CIFIR is used instead of correct CIFAR
- Figure 5 would be clearer if the authors presented the complete plot for each layer differentiating the several regions by colors.
- The title, captions, legends and descriptions of **all** figures should be improved.
- Page 9: dataser --> dataset
- Page 3: convectional --> do you mean: conventional ? The work describes a simple criterion to decide when stop the learning procedure.
Even though there are some interesting aspects, this work looks like an initial investigation but more experimental and/or theoretical demonstrations are required.

Meta-review: This work presents a simple method for early stopping that is based on layer statistics. The reviewers have all commented on the work's relative lack of novelty, poor writing and insufficiently general empirical evidence for the method working. There are very few baselines being compared and little in terms of ablation studies. All the reviewers have provide extensive constructive comments about how this work can be improved and while there was no rebuttal or discussion, I feel that there is sufficient feedback in the process for the authors to improve this work further.

In conclusion: while the idea of using stability of layer statistics has merit, at this point this work is not ready to be published at ICLR.

Decision: Reject