Title: Introspection:Accelerating Neural Network Training By Learning Weight Evolution
Authors: Abhishek Sinha
Aahitagni Mukherjee
Mausoom Sarkar
Balaji Krishnamurthy
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Optimization

Abstract: Neural Networks are function approximators that have achieved state-of-the-art accuracy in numerous machine learning tasks. In spite of their great success in terms of accuracy, their large training time makes it difficult to use them for various tasks. In this paper, we explore the idea of learning weight evolution pattern from a simple network for accelerating training of novel neural networks.

We use a neural network to learn the training pattern from MNIST classification and utilize it to accelerate training of neural networks used for CIFAR-10 and ImageNet classification. Our method has a low memory footprint and is computationally efficient. This method can also be used with other optimizers to give faster convergence. The results indicate a general trend in the weight evolution during training of neural networks.

TL;DR: Acceleration of training by performing weight updates, using knowledge obtained from training other neural networks.

Review discussion:
I have questions regarding the experiment setups:
How are the hyper-parameter of the SGD are chosen? Are the learning rate and momentum tuned for the best training time performance?
In sec. 4.2.1, it seems MNIST1 model is more powerful than MNIST2. Why is the MNIST1 model perform worse than MNIST2?
It is a paper on improving training neural networks, it will be nice to include the plots that shows the actual training objective function.
Does the Introspection network cause any overfitting problem? From the plots shown by the author, it seems non of those plots are extensive enough for model to overfit on the validation set.
It seems all the models that authors have experimented with are vanilla convolutional neural networks. Does the I network work well for fully connected architecture? Is the I network trained on normal nets able to predict weight evolution for the models using batch norm or layer norm?
Thanks for the insightful comments this helped us in carrying out additional experiments
1.We chose the default SGD parameters as present in the tensorflow MNIST,CIFAR,Imagenet examples.And we were always training for best validation error.
2. The architecture of MNIST1 and MNIST2 has now been described in the paper.
3. The training loss was also going down faster when Introspection was applied. We are going to include the training plot in the next revision.
4. We tried Introspection network on multi layerd fc network with sigmoid activation and it converged faster. It also works with Batch Norm and layer Norm. The cifar network was using batch normalization. We will mention these in our next revision of the paper.
5.
Also we need to do the experiments for long duration to check if there are any improvements in accuracy or overfitting
My questions are same as AnonReviewer1.
How were the SGD parameters chosen? For best training loss, or best validation error?
How does the training loss behave?
Having read Sec. 4.3, I still have the question: how did you select the jump points for the reported results? You mention that using I later in training helped more, which implies that other jump points were also tried. What were the results then?
We chose the default SGD parameters as present in the tensorflow MNIST,CIFAR,Imagenet examples. And we were always training for best validation error.
The training loss was also going down faster when Introspection was applied. We are going to put that plot in the paper going forward.
Did the authors perform experiments varying hyperparameter k and the number of samples from the training history of weights?
I'd be interested to know if the introspection method would provide improvement in accuracy and training speed if batch normalization is used.
EDIT: Updated score. See additional comment.
I quite like the main idea of the paper, which is based on the observation in Sec. 3.0 - that the authors find many predictable patterns in the independent evolution of weights during neural network training. It is very encouraging that a simple neural network can be used to speed up training by directly predicting weights.
However the technical quality of the current paper leaves much to be desired, and I encourage the authors to do more rigorous analysis of the approach. Here are some concrete suggestions:
- The findings in Section 3.0 which motivate the approach, should be clearly presented in the paper. Presently they are stated as anecdotes.
- A central issue with the paper is that the training of the Introspection network I is completely glossed over. How well did the training work, in terms of training, validation/test losses? How well does it need to work in order to be useful for speeding up training? These are important questions for anyone interested in this approach.
- An additional important issue is that of baselines. Would a simple linear/quadratic model also work instead of a neural network? What about a simple heuristic rule to increase/decrease weights? I think it's important to compare to such baselines to understand the complexity of the weight evolution learned by the neural network.
- I do not think that default tensorflow example hyperparameters should be used, as mentioned by authors on OpenReview. There is no scientific basis for using them.
Instead, first hyperparameters which produce good results in a reasonable time should be selected as the baseline, and then added the benefit of the introspection network to speed up training (and reaching a similar result) should be shown.
- The authors state in the discussion on OpenReview that they also tried RNNs as the introspection network but it didn't work with small state size. What does "didn't work" mean in this context? Did it underfit? I find it hard to imagine that a large state size would be required for this task.
Even if it is, that doesn't rule out evaluation due to memory issues because the RNN can be run on the weights in 'mini-batch' mode. In general, I think other baselines are more important than RNN.
- A question about jump points:
The I is trained on SGD trajectories.
While using I to speed up training at several jump points, if the input weights cross previous jump points, then I gets input data from a weight evolution which is not from SGD (it has been altered by I). This seems problematic but doesn't seem to affect your experiments. I feel that this again highlights the importance of the baselines. Perhaps I is doing something extremely simple that is not affected by this issue.
Since the main idea is very interesting, I will be happy to update my score if the above concerns are addressed.
Your comments have been very insightful and helped us in bringing out several interesting properties and also helped frame the paper better.
The Section 3.0 “Patterns in Weight evolution” now has more details to quantify the weight evolution patterns.
Details on training of introspection network have been added to the section 4.1. We trained the Introspection model until the validation loss stopped decreasing. We till now didn’t stop the training of the introspection earlier to check if it gives better speedups during inference. This adds an additional dimension in our experiments, which we are yet to explore.
Also thanks to your suggestion of comparisons to base line methods, we now know that Introspection should work better that the baselines.
We have created several baselines in Section 4.3 by comparing the introspection against linear fit, quadratic fit, random noise with various jump ratios (K of Kx) . We also used linear Introspection network i.e. with no nonlinearity to create additional baselines. In all our experiments only quadratic fit managed to outperform Introspection for once but it was very unstable and finicky on jump points and jump ratios. It has a tendency to take the network towards extremely high loss from where the training never recovers.
We have now mentioned our hyperparameters for each experiment from section 4.2
Our thought behind the use of RNN was to remove the burden of sampling from our shoulders. The idea was to keep a state for each weight and update it repeatedly using the RNN after a fixed number of steps and predict the future weight value from this state. Now the dimensions in the state increases the amount of memory we would need to hold on the GPU i.e. a state size of N means we need memory for N*#weights. Yes we could transfer it to the disk and then do prediction but that will eat up time hence we were trying to keep the N small so that it can be kept on the GPU. We choose N in the range of 4 to 32 but none of them converged to anywhere close to the loss we were getting with plain Introspection. We will explore this direction further later on.
Yes you are right about jump points. The introspection is fed data that has been altered by I itself, but it still seems to work fine even though it has not seen such self altered data during training.
A reason for such behavior could be that because we are using just 4 values for prediction, it doesn’t matter if there are some sudden changes in weight evolution as the Introspection Network might have seen such values occurring naturally during its training. May be this could also suggest that our training of Introspection didn’t overfit.
We have now updated our paper with some early results on Inception V1 running on Imagenet classification task. We have put these results in the Appendix section and we are going to update the results when we have more data from training. We are expecting the training to complete in a week or two. Looking at the initial trends we think that this trend will continue as per our earlier observations with other networks.
The authors have addressed all my comments, and worked hard to improve the paper. In my opinion, the new revision is very interesting and has high potential significance. The authors do not attempt to oversell the idea and present extensive experimental evaluation. I have updated my score to reflect my satisfaction with the updated version.
Your constructive comments helped a lot and thanks for updating the score.
The paper reads well and the idea is new.
Sadly, many details needed for replicating the results (such as layer sizes of the CNNs, learning rates) are missing.
The training of the introspection network could have been described in more detail.
Also, I think that a model, which is closer to the current state-of-the-art should have been used in the ImageNet experiments. That would have made the results more convincing.
Due to the novelty of the idea, I recommend the paper. I would increase the rating if an updated draft addresses the mentioned issues.
Thanks for the insightful comments they helped in framing the paper better.
We have now mentioned the layer sizes, learning rates and other hyperparameters for every network we have used starting from section 4.2.
Introspection training has been detailed out in section4.1
We used our method on Imagenet classification to show that this can be used on huge datasets and is also efficient enough in terms of compute and memory, to be used with large networks. The current Introspection network has been trained using data just from MNIST N0 weight evolution and yet it was able to accelerate training of novel networks with different architectures, activations and datasets.
We are confident that by adding more training data and using a more complex Introspection network would be able to accelerate state of art networks. This can be part of future investigations along with the other directions mentioned in the paper.
Even though there is no page limit, I find the paper too long and would recommend to move major part of the figures to appendix/supplementary material.
Otherwise, updates address most of the concerns.
As per your recommendation of trying a network closer to state of art, we have now updated our paper with some early results on Inception V1 running on Imagenet classification task. We have put these results in the Appendix section and are going to update the results when we have more data from training. We are expecting the training to complete in a week or two. Looking at the initial trends we think that this trend will continue as per our earlier observations with other networks.
We have also reduced the number of pages and will be working to reduce them further.
Thanks for updating the score.
In this paper, the authors use a separate introspection neural network to predict the future value of the weights directly from their past history. The introspection network is trained on the parameter progressions collected from training separate set of meta learning models using a typical optimizer, e.g. SGD.
Pros:
+ The organization is generally very clear
+ Novel meta-learning approach that is different than the previous learning to learn approach
Cons:
- The paper will benefit from more thorough experiments on other neural network architectures where the geometry of the parameter space are sufficiently different than CNNs such as fully connected and recurrent neural networks.
- Neither MNIST nor CIFAR experimental section explained the architectural details
- Mini-batch size for the experiments were not included in the paper
- Comparison with different baseline optimizer such as Adam would be a strong addition or at least explain how the hyper-parameters, such as learning rate and momentum, are chosen for the baseline SGD method.
Overall, due to the omission of the experimental details in the current revision, it is hard to draw any conclusive insight about the proposed method.
Thanks for your comments and suggestions. They were very helpful and we have tried to address the issues you have raised.
We have now reported several experiments, which gives more insight into our approach. We have reported experiments on fully connected networks and RNNs as well. We observed that the same Introspection network was able to accelerate the training of fully connected and RNN as well.
We have now mentioned the mini-batch sizes, layer sizes, learning rates and other hyperparameters for every network that we have used.
We have added several baseline in section 4.3 and have also compared SGD vs SGD + introspection vs Adam vs Adam + Introspection with plots in section on “Comparison with Adam optimizer”
The Introspection Network was able to accelerate when applied over SGD and Adam.
We have now updated our paper with some early results on Inception V1 running on Imagenet classification task. We have put these results in the Appendix section and are going to update the results when we have more data from training. We are expecting the training to complete in a week or two. Looking at the initial trends we think that this trend will continue as per our earlier observations with other networks.
The revised paper demonstrated the effectiveness of the introspection network under different optimizers and addressed most of my concerns. I will increase my score accordingly. It is interesting to see that Adam + introspection provides a substantial gain for the fully connected architecture but not as much for the convolutional architecture. I am wondering if the weight initialization and the performance of the introspection network are correlated.
Namely, if the weights of a network are initialized to a wrong scale, the introspection network can easily fix that in its first few jumps. Experiments along these lines could further improve the strength of this paper.
Thanks for updating the score.
We have revised the paper with updated results on experiments with Inception V1 network, which continues to show promise.

Meta-review:
role: PC
data: comment: Interesting paper and clear accept. Not recommended for an oral presentation because of weaknesses in the empirical contribution that make the significance of the results unclear.

Decision: Accept (Poster)