Title: On The Difficulty of Warm-Starting Neural Network Training
Authors: Jordan T. Ash
Ryan P. Adams
Venue: ICLR 2020
Keywords: deep learning
neural networks

Abstract: In many real-world deployments of machine learning systems, data arrive piecemeal. These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily financial data) or active, where samples are selected according to a measure of their quality (e.g., experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data. We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point. Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate---to "warm start'' the optimization rather than initialize from scratch---and see reductions in wall-clock time. However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the final training losses are similar. While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start. Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems. In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs. Although the present investigation did not lead to a solution, we hope that a thorough articulation of the problem will spur new research that may lead to improved methods that consume fewer resources during training.

TL;DR: We empirically study the gap in generalization between warm-started and randomly-initialized neural networks.

Review discussion:
This paper examines the problem of warm-starting the training of neural networks.
In particular, a generalization gap arises when the network is trained on the full training set from the start versus being warm-started, where the network is initially (partially) trained on a subset of the training set, then switched to the full training set. This problem is practical, as it is often preferable to train online while data is collected to make up-to-date predictions for tasks (such as in online advertising or recommendation systems), but it has been found that retraining is necessary in order to obtain optimal performance. The paper also mentions active learning, domain shift, and transfer learning as two other relevant and important problems.
The paper attempts to investigate this phenomena from a few different avenues: (1) simple experiments segmenting the training set into two different subsets, then training to completion or partial training on the first subset before switching to training on the full set; (2) looking at the gradients of warm-started models; (3) adding regularization; (4) warm-starting all layers, then training only last layer; (5) perturbing the warm-started parameters.
Strengths:
I believe very strongly in the practical impact of the problems presented in this paper. These indeed are challenging problems that are relevant to industry that have not been given sufficient attention in the academic literature. I appreciate the initial experimentation on this subject, and the clear demonstration of the problem through simple experiments. The paper is also well-written.
Weaknesses:
Some questions I had include:
- Why is the Pearson correlation between parameters of the neural network a good way of measuring the correlation to their initialization?
- Why is it surprising that the magnitude of the gradients of the "new" data is higher than at a random initialization?
- Why does this phenomena occur even though the data is sampled from the same distribution?
- Does this work have any relationship with work on generalization such as:
[1] Recht, Benjamin, et al. "Do CIFAR-10 classifiers generalize to CIFAR-10?." arXiv preprint arXiv:1806.00451 (2018).
[2] Recht, Benjamin, et al. "Do ImageNet Classifiers Generalize to ImageNet?." arXiv preprint arXiv:1902.10811 (2019).
etc.
Although I like the topic of this paper, the investigation seems too preliminary at this point. There is no clear hypothesis towards answering the problems proposed in the paper. There is also no analysis, which places the burden on the numerical experiments to demonstrate something interesting, and the experiments seem sparse and small-scale. For these reasons, I am inclined to reject this paper at this time, but I strongly encourage further exploration into the topic.
Some potential questions or directions could include:
1. What if only a single epoch of training is used on 50% of the data? Does the gap appear in that setting? I ask because one would expect that a single epoch of training on 50% of the data, then training on new data would be equivalent to training on the full dataset from the start.
2. How does this gap change with respect to the (relative) amount of new data introduced into the problem?
For example, if one were to only add a single datapoint to the training set, would one still observe this behavior? Could one potentially add data more incrementally (rather than half of the training set) and potentially mitigate this drastic change in the problem?
3. There are optimization algorithms specifically designed for stochastic optimization (with a fixed distribution) versus for online optimization (online gradient, Adagrad). Is the online optimization framework perhaps more "realistic" than the stochastic optimization framework in these streaming/warm-starting settings?
Thank you for your review. Our investigation is intended to identify the warm-start phenomenon as something that the community should focus on. We allocate the ten pages of our article towards carefully defining the problem, zooming in on the effect, and doing thoughtful experiments that demonstrate that there does not seem to be a trivial solution.
While the investigation is not complete, we see this work as similar to “Intriguing Properties of Neural Networks” and “Understanding Deep Learning Requires Rethinking Generalization“, which both stimulated important research by studying unintuitive empirical properties of neural networks.
Despite the importance of these articles, they don’t posit solutions to the problems they propose.
Weaknesses:
1.  Why is the Pearson correlation between parameters of the neural network a good way of measuring the correlation to their initialization?
Other correlation measurements give nearly identical results. In Appendix Figure 11, we've made the same plot but for Spearman's Correlation, Cosine Distance, and Euclidean Distance as the metrics to measure difference between warm-start initialized parameters and final parameters.
2. Why is it surprising that the magnitude of the gradients of the "new" data is higher than at a random initialization?
We say in our article that it is surprising that these gradient norms are much larger than those in the randomly-initialized model. This is surprising because a large gradient magnitude suggests the data induce a large change in the model. This would be unexpected in a warm-started model because it has already learned a hypothesis on data emitted from the same distribution. We might instead expect a larger gradient norm in a randomly-initialized model because the initial hypothesis in this setting is further from the correct one. We have added some additional explanation to the article.
3. Why does this phenomena occur even though the data is sampled from the same distribution?
This is exactly the question we are trying to investigate in our paper.
Like we say, although we don’t present a solution here, we believe it is important to deliver careful empirics that outline the problem and its difficulty.
Potential directions:
1. What if only a single epoch of training is used on 50% of the data? Does the gap appear in that setting? I ask because one would expect that a single epoch of training on 50% of the data, then training on new data would be equivalent to training on the full dataset from the start.
A single epoch of 50% data will not cause this effect. This is because showing 50% of samples once then switching to 100% of samples is functionally identical to just training on 100% of samples.
Still, as Figure 7 shows, damage caused by warm-starting becomes significant after only a few epochs.
2. How does this gap change with respect to the (relative) amount of new data introduced into the problem?
For example, if one were to only add a single datapoint to the training set, would one still observe this behavior? Could one potentially add data more incrementally (rather than half of the training set) and potentially mitigate this drastic change in the problem?
Adding data to the training set in smaller batches is exactly the situation described in the “Passive Online Learning”. As we say in the article, we gradually add batches of 100 points and train until convergence. Figure 2 demonstrates that the magnitude of the generalization gap increases in proportion to the amount of batches added, eventually becoming quite large.
3. There are optimization algorithms specifically designed for stochastic optimization (with a fixed distribution) versus for online optimization (online gradient, Adagrad). Is the online optimization framework perhaps more "realistic" than the stochastic optimization framework in these streaming/warm-starting settings?
Adagrad would be worth trying for the online learning experiments, but we see this phenomenon even when the classifier observes only one large new batch of data (e.g. Table 1 and Figure 4).
This paper conducted an empirical study on why training with warm starting has worse generalization ability than learning from scratch. The paper is interesting, however, it has something unclear to me, as explained below.
1)	The scale and diversity of the study can be improved. Only three models and three datasets were examined, which might not be representative enough.
For example, the popular Transformer model, the large-scale datasets like ImageNet, the language understanding and machine translation tasks, etc. were not included in the study. This may make the study less relevant to many important tasks and domains.
2)	The interesting and highlight part of the paper is that it studies many different factors and aspects, including the influence of batch size, learning rate, regularization, moment, denoising, etc.
However, I kind of feel that the experimental setting has some fatal problems, which makes the experimental results not convincing. The authors partitioned the dataset into two halves, using the first half for pre-training, and then use the whole datasets for continued training.
Although the partitioning is random, given the limited size of the datasets, such a treatment will change the underlying distribution (frequency of the samples during training). The first half of the dataset plays a more important role in training: it was used during pre-training, and also used in training.
So somehow the first half was used twice, or at least used more than once,  depending on the numbers of epochs in pre-training and training. This distribution change may make the training a little biased, and at least it is not a fair comparison with learning from scratch (the latter will not have such bias in data).
So for a fair comparison, one needs to add some baselines to understand the influence of data frequency change. Without the understanding from this angle, the study may be mis-leading.
**I read the author rebuttal, however, I still think the experiments are not comprehensive and the use of data partitions in the experiments are problematic (nothing to do with realistic or not, just for fair comparison with learning from scratch).
So I would not change my assessment.
Thank you for your review.
1. The motivation of the paper is not very strong. Even the authors themselves acknowledged that this topic has not received much direct attention from the research community. There are only a very small number of related works, and most of the motivation is based on the authors’ own experiments. I am not criticizing this – but if the problem is general and important in practice, it is hard to believe that other people have not found it and have not conducted studies on it.
We strongly disagree that research in an understudied area implies work that is not well motivated. Being unable to effectively warm-start neural network applications has tremendous practical implications related to time and energy required to fit performant predictors.
2. The scale and diversity of the study can be improved. Only three models and three datasets were examined, which might not be representative enough.
For example, the popular Transformer model, the large-scale datasets like ImageNet, the language understanding and machine translation tasks, etc. were not included in the study. This may make the study less relevant to many important tasks and domains.
We use a breadth of tools to examine this behavior across three models and datasets, although we agree that these experiments have focused on image recognition problems. We further cite work that anecdotally describes the practical benefit of random-restarting, for several architectures, datasets, and learning scenarios. This is clearly a general problem with warm-starting neural networks not restricted to those discussed in this article.
Of course, more and larger experiments are always desirable, but even reporting the current experimental results required pushing three tables of results into supplemental material.
3. The interesting and highlight part of the paper is that it studies many different factors and aspects, including the influence of batch size, learning rate, regularization, moment, denoising, etc.
However, I kind of feel that the experimental setting has some fatal problems, which makes the experimental results not convincing. The authors partitioned the dataset into two halves, using the first half for pre-training, and then use the whole datasets for continued training.
Although the partitioning is random, given the limited size of the datasets, such a treatment will change the underlying distribution (frequency of the samples during training). The first half of the dataset plays a more important role in training: it was used during pre-training, and also used in training.
So somehow the first half was used twice, or at least used more than once,  depending on the numbers of epochs in pre-training and training. This distribution change may make the training a little biased, and at least it is not a fair comparison with learning from scratch (the latter will not have such bias in data).
So for a fair comparison, one needs to add some baselines to understand the influence of data frequency change. Without the understanding from this angle, the study may be mis-leading.
Indeed, samples in the 50% of data used for the initial round of training are used again for fitting the warm-started models. This is the most realistic scenario — in an online learning framework, one wouldn’t fit the model using only the newest data, especially when it’s drawn from the same distribution as the initial data. It would be unsurprising, for example, to find that a randomly-initialized model with access to the entire dataset outperforms that same model, regardless of its initialization, with access to only a restricted subset of those data. Here we show that all things but initialization held equal, two models given access to the same data can have significantly different generalization performance. This is interesting because the warm-started initialization parameters are the result of training using data emitted from the same distribution as those in the learning problem at hand; conventional intuition suggests that they should consequently work well.
The author proposes different approaches to the problem of  "warm-started" neural networks. Models trained from scratch on the whole dataset have better performance than "warm-started" models, which are trained with weights initialized from training using part of the available data.  The authors change hyperparameters like batch size and learning rate and demonstrate a tradeoff between generalization performance of the model and time, required for its training. We can also see that the choice of hyperparameters, necessary for the best performance, levels benefit in time from "warm starting."
The core idea of the paper is the investigation of various possible causes of difficulty of "warm start" to reduce training time without damaging a generalization performance. The authors investigate the dependence of this effect with gradient values, regularization, part of "warm started" layers, noising weights, catastrophic forgetting, and so on.
The authors describe different problems where this research can be useful and tries to shed light on the causes of this problem, but the solution is not found. This article can be helpful for future researchers as they continue research in this direction from a warm start.
However, it is hard to judge how valuable this contribution is.
Also, see a few minor comments:
1. Maybe it should be useful to include in Table 1 results for models trained using only 50% of data.
2. Typo: NVIDA -> NVIDIA (p. 6)
3. It seems that the problem lies in the area of the complex learning landscape for optimization of Neural Networks as we end up in the worse local optimum if we use a warm start. Maybe one should attack the problem with this direction, as there are several papers that investigate how the loss function landscape looks like e.g. [1]
4. It seems that the behavior becomes worse if we increase the complexity of the model. Investigation of the dependence of severity of warm start effect on e.g. number of layers in the network can be useful.
Also, it can be possible to gain some theoretical insights and provable results while dealing with simpler models. [1.] H. Li, Zh. Xu et. al. Visua
5. It might be useful to research the dependence of "warm start" effect from the portion of data on which the model was pre-trained. As I understand situations, where 80-90% of data are used in the first round of training, are more common.
6. The authors provide us with graphics of accuracy on training dataset for "warm started" and trained from the scratch models. We can see that both models reach 100% train accuracy, and the authors state that it is impossible to spot the "warm start" problem on the training dataset because the metrics are equal. Maybe it could be useful to show graphics of loss function.
Despite of similar 100% train accuracy, final losses might be different("warm start" loss > from scratch loss). It may mean that in case of warm start we reach a local minimum, but not the best one.
7. It might be a good idea to add to Table 2 results for not-regularized model in order to compare results with and without regularization and figure out what effect on "warm started" models regularization have.
Thank you for your review.
1. Maybe it should be useful to include in Table 1 results for models trained using only 50% of data.
We have included this information in Appendix Table 6 of the updated paper.
2. Typo: NVIDA -> NVIDIA (p. 6)
Typo fixed. Thanks for pointing this out.
3. It seems that the problem lies in the area of the complex learning landscape for optimization of Neural Networks as we end up in the worse local optimum if we use a warm start. Maybe one should attack the problem with this direction, as there are several papers that investigate how the loss function landscape looks like e.g. [1]
Overall, somehow examining the geometry of the loss surface seems like a useful direction. In our setting, the geometry of the loss surface will change between the first and second round of training, with the two geometries being more similar if more data are present in the first round of training, and less similar if less data are available in the first round of training.
Of course, one can’t know the degree of dissimilarity without having access to the new, larger set of data. This is a significant deviation from [1], but we’ve included the citation in our related work section.
4. It seems that the behavior becomes worse if we increase the complexity of the model. Investigation of the dependence of severity of warm start effect on e.g. number of layers in the network can be useful.
Also, it can be possible to gain some theoretical insights and provable results while dealing with simpler models.
We agree this is an interesting experiment. We’ve added results for a few different ResNet depths to the appendix of our paper (Tables 7 and 8).  Warm starting is always worse than random initializations, but the gap does not seem to grow with depth.
5. It might be useful to research the dependence of "warm start" effect from the portion of data on which the model was pre-trained. As I understand situations, where 80-90% of data are used in the first round of training, are more common.
The pretraining scenario is discussed in section 2.3 and Figure 3. We show that as more data are available for the second-round of training, pretraining becomes more harmful than helpful.
Along these lines, we’ve added Figure 10 to the appendix of our article, demonstrating how generalization performance varies as a function of the fraction of training data available in the first round of model fitting.
Unlike Figure 3, Figure 8 is for the scenario where both rounds of model training are on CIFAR-10 data. The severity of the warm-start generalization gap in inversely proportional to the proportion of data available during the first round of training.
6. The authors provide us with graphics of accuracy on training dataset for "warm started" and trained from the scratch models. We can see that both models reach 100% train accuracy, and the authors state that it is impossible to spot the "warm start" problem on the training dataset because the metrics are equal. Maybe it could be useful to show graphics of loss function.
Despite of similar 100% train accuracy, final losses might be different("warm start" loss > from scratch loss). It may mean that in case of warm start we reach a local minimum, but not the best one.
We added this information in Table 8 of the Appendix (as a companion to the table that varies ResNet depth). Interestingly, warm-started models seem to achieve lower training loss when compared to their randomly-initialized peers.
7. It might be a good idea to add to Table 2 results for not-regularized model in order to compare results with and without regularization and figure out what effect on "warm started" models regularization have.
All results presented here, including those in Table 2, are for non-regularized models unless otherwise stated. We experiment with regularization only in Section 3.4.

Meta-review:
role: PC
data: comment: The paper addresses the question of why warm starting could result in worse generalization ability than training from scratch.  The reviewers agree that increasing the circumstances in which warm starting could be applied is of interest, in particular to reduce training time and computational resources.  However, the reviewers were unanimous in their opinion that the paper is not suitable for publication at ICLR in its current form.  Concerns included that the analysis was not sufficiently focused and the experiments too small scale.  As the analysis component of the paper was considered to be limited, the experimental results were insufficient on the balance to push the paper to an acceptable state.

Decision: Reject