Title: INTROSPECTION:ACCELERATING NEURAL NETWORK TRAINING BY LEARNING WEIGHT EVOLUTION

Abstract: Neural Networks are function approximators that have achieved state-of-the-art accuracy in numerous machine learning tasks. In spite of their great success in terms of accuracy, their large training time makes it difficult to use them for various tasks. In this paper, we explore the idea of learning weight evolution pattern from a simple network for accelerating training of novel neural networks.
We use a neural network to learn the training pattern from MNIST classification and utilize it to accelerate training of neural networks used for CIFAR-10 and ImageNet classification. Our method has a low memory footprint and is computationally efficient. This method can also be used with other optimizers to give faster convergence. The results indicate a general trend in the weight evolution during training of neural networks.

1. INTRODUCTION:

Deep neural networks have been very successful in modeling high-level abstractions in data. However, training a deep neural network for any AI task is a time-consuming process. This is because a large number of parameters need to be learnt using training examples. Most of the deeper networks can take days to get trained even on GPU thus making it a major bottleneck in the large-scale application of deep networks. Reduction of training time through an efficient optimizer is essential for fast design and testing of deep neural nets.

In the context of neural networks, an optimization algorithm iteratively updates the parameters (weights) of a network based on a batch of training examples, to minimize an objective function. The most widely used optimization algorithm is Stochastic Gradient Descent. Even with the advent of newer and faster optimization algorithms like Adagrad, Adadelta, RMSProp and Adam there is still a need for achieving faster convergence.

In this work we apply neural network to predict weights of other in-training neural networks to accelerate their convergence. Our method has a very low memory footprint and is computationally efficient. Another aspect of this method is that we can update the weights of all the layers in parallel.

2. RELATED WORK:

Several extensions of Stochastic Gradient Descent have been proposed for faster training of neural networks. Some of them are Momentum (Rumelhart et al., 1986) , AdaGrad (Duchy et al., 2011) , AdaDelta (Zeiler, 2012) , RMSProp (Hinton et al., 2012) and Adam (Kingma & Ba, 2014) . All of them reduce the convergence time by suitably altering the learning rate during training. Our method can be used along with any of the above-mentioned methods to further improve convergence time.

In the above approaches, the weight update is always a product of the gradient and the modified/unmodified learning rate. More recent approaches (Andrychowicz et al., 2016) have tried to learn the function that takes as input the gradient and outputs the appropriate weight update. This exhibited a faster convergence compared to a simpler multiplication operation between the learning rate and gradient. Our approach is different from this, because our forecasting Network does not use the current gradient for weight update, but rather uses the weight history to predict its future value many time steps ahead where network would exhibit better convergence. Our approach generalizes better between different architectures and datasets without additional retraining. Further our approach has far lesser memory footprint as compared to (Andrychowicz et al., 2016) . Also our approach need not be involved at every weight update and hence can be invoked asynchronously which makes it computationally efficient.

Another recent approach, called Q-gradient descent (Fu et al., 2016) , uses a reinforcement learning framework to tune the hyperparameters of the optimization algorithm as the training progresses. The Deep-Q Network used for tuning the hyperparameters itself needs to be trained with data from any specific network N to be able to optimize the training of N . Our approach is different because we use a pre-trained forecasting Network that can optimize any network N without training itself by data from N .

Finally the recent approach by (Jaderberg et al., 2016) to predict synthetic gradients is similar to our work, in the sense that the weights are updates independently, but it still relies on an estimation of the gradient, while our update method does not.

Our method is distinct from all the above approaches because it uses information obtained from the training process of existing neural nets to accelerate the training of novel neural nets.

3. PATTERNS IN WEIGHT EVOLUTION:

The evolution of weights of neural networks being trained on different classification tasks such as on MNIST and CIFAR-10 datasets and over different network architectures (weights from different layers of fully connected as well as convolutional architectures) as well as different optimization rules were analyzed. It was observed that the evolution followed a general trend independent of the task the model was performing or the layer to which the parameters belonged to. A major proportion of the weights did not undergo any significant change. Two metrics were used to quantify weight changes:

• Difference between the final and initial values of a weight scalar: This is a measure of how much a weight scalar has deviated from its initial value after training.In figure 4 we show the frequency histogram plot of the weight changes in a convolutional network trained for MNIST image classification task, which indicates that most of the weight values do not undergo a significant change in magnitude. Similar plots for a fully connected network trained on MNIST dataset ( figure 6 ) and a convolutional network trained on CIFAR-10 dataset (figure 8 ) present similar observations.

• Square root of 2nd moment of the values a weight scalar takes during training: Through this measure we wish to quantify the oscillation of weight values. This moment has been taken about the initial value of the weight. In figure 5 , we show the frequency histogram plot of the second moment of weight changes in a convolutional network trained for the MNIST digit classification task, which indicates that most of the weight values do not undergo a significant oscillations in value during the training. Similar plots for a fully connected network trained on MNIST (figure 7 ) and a convolutional network trained on CIFAR-10 ( figure 9 ) dataset present similar observations.

A very small subset of the all the weights undergo massive changes compared to the rest.

The few that did change significantly were observed to be following a predictable trend, where they would keep on increasing or decreasing with the progress of training in a predictable fashion.

In figures 1, 2 and 3 we show the evolution history of a few weights randomly sampled from the weight change histogram bins of figures 4,6 and 8 respectively, which illustrates our observation. 

3.1. WEIGHT PREDICTION:

We collect the weight evolution trends of a network that is being trained and use the collected data to train a neural network I to forecast the future values of each weight based on its values in the previous time steps. The trained network I is then used to predict the weight values of an unseen network N during its training which move N to a state that enables a faster convergence. The time taken for the forecast is significantly smaller compared to the time a standard optimizer (e.g. SGD) would have taken to achieve the same accuracy. This leads to a reduction in the total training time. The predictor I that is used for forecasting weights is a comparatively smaller neural network, whose inference time is negligible compared to the training time of the network that needs to be trained(N). We call this predictor I Introspection network because it looks at the weight evolution during training.

The forecasting network I is a simple 1-layered feedforward neuralnet. The input layer consists of four neurons that take four samples from the training history of a weight. The hidden layer consists of 40 neurons, fully connected to the input layer, with ReLU activation. The output layer is a single neuron that outputs the predicted future value of the weight. In our experiments four was minimum numbers of samples for which the training of Introspection Network I converged.

The figure 10 below shows a comparison of the weight evolution for a single scalar weight value with and without using the introspection network I. The vertical green bars indicate the points at which the introspection network was used to predict the future values. Post prediction, the network continues to get trained normally by SGD, until the introspection network I is used once again to jump to a new weight value.

4. EXPERIMENTS:



4.1. TRAINING OF INTROSPECTION NETWORK:

The introspection network I is trained on the training history of the weights of a network N 0 which was trained on MNIST dataset.The network N 0 consisted of 3 convolutional layers and two fully connected layers, with ReLU activation and deploying Adam optimiser. Max pooling(2X2 pool size and a 2X2 stride) was applied after the conv layers along with dropout applied after the first fc layer. The shapes of the conv layer filters were [5, 5, 1, 8] , [5, 5, 8, 16] and [5, 5, 16, 32] respectively whereas of the fc layer weight were [512, 1024] and [1024, 10] Since a large proportion of weights remain nearly constant throughout the training, a preprocessing step is done before getting the training data for I. The large number of weight histories collected are sorted in decreasing order on the basis of their variations in values from time step 0 to time step t. We choose 50% of the training data from the top 50th percentile of the sorted weights, 25% from the next 25th percentile(between 50 to 75th percentile of the sorted weights) and the remaining 25% from the rest (75th to 100th percentile). Approximately 0.8 million examples of weight history are used to train I. As the weight values are very small fractions they are further multiplied by 1000 before being input to the network I. The expected output of I, which is used for training I using backpropagation, is a single scalar the value of the same weight at step 2t. This is an empirical choice. For example, any step kt with k > 1 can be chosen instead of 2t. In our experiments with varying the value of k, we found that the value of k = 2.2 reached a slightly better validation accuracy than k = 2.0 on MNIST dataset (see figure 15 ) but, on the whole the value of k = 2.0 was a lot more consistent in its out-performance at various points in its history. All the results reported here are with respect to the I trained to predict weight values at 2t. 

4.2. USING PRE-TRAINED INTROSPECTION NETWORK TO TRAIN UNSEEN NETWORKS:

The introspection network once trained can be then used to guide the training of other networks. We illustrate our method by using it to accelerate the training of several deep neural nets with varying architectures on 3 different datasets, namely MNIST, CIFAR-10 and ImageNet. We note that the same introspection network I, trained on the weight evolutions of the MNIST network N 0 was used in all these different cases.

All the networks have been trained using either Stochastic Gradient Descent, or ADAM and the network I is used at a few intermediate steps to propel the network to a state with higher accuracy.We refer to the time step at which the introspection network I is applied to update all the weights as a "jump point".

The selection of the steps at which I is to be used is dependent on the distribution of the training step t used for training I. We show the effect of varying the timing of the initial jump and the time interval between jump points in section 4.2.2. It has been observed that I gives a better increase in accuracy when it is used in later training steps rather than in the earlier ones.

All the networks trained using I required comparatively less time to reach the same accuracy as normal SGD training. Also, when the same network was trained for the same time with and without updates by I, the former is observed to have better accuracy. These results show that there is a remarkable similarity in the weight evolution trajectories across network architectures,tasks and datasets.

4.2.1. MNIST:

Four different neural networks were trained using I on MNIST dataset: 1. A convolutional neural network M N IST 1 with 2 convolutional layer and 2 fully connected layers(dropout layer after 1st fc layer is also present)with ReLU acitvations for classification task on MNIST image dataset.Max pooling(2X2 pool size and a 2X2 stride) was applied after every conv layer. The CNN layer weights were of shape [5, 5, 1, 8] and [5, 5, 32, 64] respectively and the fc layer were of sizes [3136, 1024] and [1024, 10] .The weights were initialised from a truncated normal distribution with a mean of 0 and std of 0.01. The network was trained using SGD with a learning rate of 1e-2 and batch size of 50.

It takes approximately 20,000 steps for convergence via SGD optimiser. For M N IST 1 , I was used to update all weights at training step 3000, 4000, and 5000. 2. A convolutional network M N IST 2 with 2 convolutional layer and 2 fully connected layers with ReLU acitvations. Max pooling(2X2 pool size and a 2X2 stride) was applied after every conv layer. The two fc layer were of sizes [800, 500] and [500, 10] whereas the two conv layers were of shape [5, 5, 1, 20] and [5, 5, 20, 50] respectively. The weight initialisations were done via xavier intialisation. The initial learning rate was 0.01 which was decayed via the inv policy with gamma and power being 1e -4 and 0.75 respectively. Batch size of 64 was used for the training.It takes approximately 10,000 steps for convergence . The network I was used to update weights at training step 2500 and 3000. 3. A fully connected network M N IST 3 with 2 hidden layers each consisting of 256 hidden units and having ReLU acitvations. The network was trained using SGD with a learning rate of 5e -3 and a batch size of 100. The initial weights were drawn out from a normal distribution having mean 0 and std as 1.0. For this network the weight updations were carried out at steps 6000, 8000 and 10000. For the MNIST2 network, the figure 12 shows that to reach an accuracy of 99.11%, the number of iterations required by normal SGD was 6000, whereas with the application of the introspection network I, the number of iterations needed was only 3500, which represents a significant savings in time and computational effort. The initial drop in accuracy seen after a jump in M N IST 2 figure 12 can be attributed to the fact that each weight scalar is predicted independently, and the interrelationship between the weight scalars in a layer or across different layers is not taken into consideration. This interrelationship is soon reestablished after few SGD steps. This phenomenon is noticed in the CIFAR and ImageNet cases too. We observed that for the CIF AR 1 network that in order to reach a validation accuracy of 85.7%, we need 40,000 iterations with normal SGD without any intervention with the introspection network I. In all the four sets where the introspection network was used, the target accuracy of 85.7% was reached in approximately 28,000 steps. This shows that the introspection network is able to successfully generalize to a new dataset and new architecture and show significant gains in training time.

On CIF AR 1 , the time taken by I for prediction is negligible compared to the time required for SGD. So the training times in the above cases on CIF AR 1 can be assumed to be proportional to the number of SGD steps required.

A comparison of the validation accuracy with and without updates by I at the four different sets of jump points are shown in figures 16, 17, 18 and 19. The results show that the while choice of jump points have some effect on the final result, the effects are not very huge. In general, we notice that better accuracy is reached when the jumps take place in later training steps. et al., 2015) dataset. It has 5 conv layers and 3 fully connected layers . Max pooling and local response normalization have been used after the two starting conv layers and the pooling layer is there after the fifth conv layer as well. We use SGD with momentum of 0.9 to train this network, starting from a learning rate of 0.01. The learning rate was decreased by one tenth every 100, 000 iterations. The mini-batch size was 128. It takes approximately 300,000 steps for convergence. The weight updates were carried out at training steps 120, 000 , 130, 000 , 144, 000 and 160, 000 .

We find that in order to achieve a top-5 accuracy of 72%, the number of iterations required in the normal case was 196,000. When the introspection network was used, number of iterations required to reach the same accuracy was 179,000. The results on Alexnet 1 show that our approach has a small memory footprint and computationally efficient to be able to scale to training practical large scale networks.

4.3. COMPARISON WITH BASELINE TECHNIQUES:

In this section we provide a comparison with other optimizers and simple heuristics which can be used to update the weights at different training steps instead of updations by introspection network.

4.4. COMPARISON WITH ADAM OPTIMIZER:

We applied the introspection network on M N IST 1 and M N IST 3 networks being trained with Adam optimizer with learning rates of 1e -4 and 1e -3. The results in figure 22 and figure 23 show that while Adam outperforms normal SGD and SGD with introspection, we were able to successfully apply the introspection network on Adam optimizer and accelerate it.

For M N IST 1 the max accuracy achieved by Adam with introspection was 99. The max accuracy achieved with introspection applied was 99.21% whereas quadratic fit it was 99.19%. We note that even though the best performing quadratic fit eventually almost reaches the same max accuracy than that achieved with introspection network, it required considerable experimentation to find the right jump ratio.A unique observation for the quadratic fit baseline was that it would take the accuracy down dramatically, upto 9.8%, from which the training often never recovers. Sometimes,the optimizers (SGD or Adam) would recover the accuracy, as seen in figure 24 . Moreover, the quadratic fit baseline was not able to generalize to other datasets and tasks. The best performing jump ratio of 1.25 was not able to outperform Introspection on the CIFAR-10 dataset, as seen in figure 25 .

In the CIFAR-10 case, The maximum accuracy achieved via updations by introspection was 85.6 which was achieved after 25500 steps, whereas with updations by quadratic fit, the max accuracy of 85.45 was achieved after 27200 steps.

For the normal training via SGD without any updations after 30000 steps of training, the max accuracy of 85.29 was achieved after 26500 steps, whereas the same accuracy was achieved by introspection after only 21200 steps and after 27000 steps via updation by quadratic.

4.4.2. FITTING LINEAR CURVE:

Instead of fitting a quadratic curve to each of the weights we tried fitting a linear curve. Experiments were performed on M N IST 1 for jump ratios of 1.1 and 1.075 as the higher ratios would cause the model to diverge after 2 or 3 jumps.The result has been shown below in figure 26 . As no significant improvement in performance was observed the experiment was not repeated over cifar.

4.5. LINEAR INTROSPECTION NETWORK:

We removed the ReLU nonlinearity from the introspection network and used the same training procedure of the normal introspection network to predict the future values at 2t. We then used this linear network on the M N IST 1 network. We found that it gave some advantage over normal SGD, but was not as good as the introspection network as shown in figure 27 . Hence we did not explore this baseline for other datasets and networks.

4.5.1. ADDING NOISE:

The weight values were updated by adding small gaussian random zero mean noise values . The experiment was performed over M N IST 1 for two different std. value, the results of which have been shown below in figure 28 . Since no significant improvement was observed for the weight updations via noise for MNIST, the experiment was not performed over cifar-10.

5. LIMITATIONS AND OPEN QUESTIONS:

Some of the open questions to be investigated relate to determination of the optimal jump points and investigations regarding the generalization capacity of the introspection network to speed up training in RNNs and non-image tasks. Also, we noticed that applying the jumps in very early training steps while training AlexN et 1 tended to degrade the final outcomes. This may be due to the fact that our introspection network is extremely simple and has been trained only on weight evolution data from MNIST. A combination of a more powerful network and training data derived from a diverse set may ameliorate this problem.

6. CONCLUSION:

We introduced a method to accelerate neural network training. For this purpose, we used a neural network I that learns a general trend in weight evolution of all neural networks. After learning the trend from one neural network training, I is used to update weights of many deep neural nets on 3 different tasks -MNIST, CIFAR-10, and ImageNet, with varying network architectures, activations, optimizers, and normalizing strategies(batch norm,lrn). Using the introspection network I led to faster convergence compared to existing methods in all the cases. Our method has a small memory footprint, is computationally efficient and is usable in practical settings. Our method is different from other existing methods in the aspect that it utilizes the knowledge obtained from weights of one neural network training to accelerate the training of several unseen networks on new tasks. The results reported here indicates the existence of a general underlying pattern in the weight evolution of any neural network.

A APPENDIX:

In this section, we report some initial results of applying the introspection network I (trained on the weight evolution of MNIST network N 0) to accelerate the training of inception v1 network (Szegedy et al., 2014) . We trained the inception v1 network on imagenet dataset with a mini-batchsize of 128 and a RMS optimizer(decay 0.9, momentum 0.9, epsilon 1.0) starting from a learning rate of 0.01 with a decay of 0.94 after every 2 epochs. The network training is still in progress, and we will eventually report on the final outcome. However we thought it would be valuable to share the preliminary results all the same. We found that applying introspection network seems to be reducing the training time quite significantly. In Figures 29 and 30 , we see that applying the introspection network leads to a gain of at least 730,000 steps.After training for around 1.5 million steps, the maximum accuracy achieved by normal training was 68.40%, whereas with introspection applied after every 300k steps the max accuracy achieved was 69.06%.The network achieved the max accuracy of 68.40% after only 852k steps. With introspection applied at steps 200k, 400k and 600k the max accuracy achieved was 68.69% and it reached the max accuracy achieved by the normal training of model after only 944k steps.

However, we also observed that choosing the jump points early in the training does not lead to eventual gains, even though a significant jump in accuracy is observed initially. Figure 31 shows the flattening of the test accuracy after a set of early jumps. It remains to be seen if further interventions later in the training can help maintain the initial accelerated convergence. 

:



Figures and Tables:

Figure 1: Deviation of weight values from initialized values as a convolutional network gets trained on MNIST dataset using SGD optimizer.

Figure 3: Deviation of weight values from initialized values as CNN gets trained on CIFAR-10 dataset using SGD optimizer.

Figure 4: log-Frequency distribution of difference between weight values before and after training for a network N 0 trained on MNIST dataset using SGD optimizer.

Figure 6: log-Frequency distribution of difference between weight values before and after training for a fully-connected network trained on MNIST dataset using Adam optimizer.

Figure 10: Example of weight update using Introspection Network.

4. A RNN M N IST 4 used to classify MNIST having a LSTM cell of hidden size of 128 followed by a fc layer of shape[128, 10]  for classification. The RNN was trained on Adam optimizer with a learning rate of 5e -4 and a batch size of 128. The weight updations for this network were done at steps 2000,3000 and 4000. Since the LSTM cell uses sigmoid and tanh activations, the RNN M N IST 4 allows us to explore if the introspection network, trained on ReLU can generalize to networks using different activation functions.A comparison of the validation accuracy with and without updates by I is shown in figures 11, 12 ,13 and 14. The green lines indicate the steps at which the introspection network I is used. For the M N IST 1 network with the application of the introspection network I at three points, we found that it took 251 seconds and 20000 SGD steps to reach a validation accuracy of 98.22%. In the same number of SGD steps, normal training was able to reach a validation accuracy of only 97.22%. In the same amount of time (251 seconds), normal training only reached 97.92%. Hence the gain in accuracy with the application of introspection network translates to real gains in training times.

Figure 11: Validation accuracy plot for M N IST 1

Figure 12: Validation accuracy plot for M N IST 2

Again the time taken by I for prediction is negligible compared to the time required for SGD. A comparison of the validation accuracy with and without updates by I is shown in figure 20. The green lines indicate the steps at which the introspection network I is used. The corresponding plot of loss function against training steps has been shown in figure 21.

Figure 21: Plot of loss function vs training steps for AlexN et 1 on ImageNet

Figure 23: Test accuracy comparison for M N IST 3 for SGD and Adam optimiser in the presence and absence of introspection.

Figure 24: Comparison of test accuracy for M N IST 1 with weight updations by Introspection and quadratic fit.

1

Figure 25: Comparison of test accuracy for CIFAR-10 with weight updations by Introspection and quadratic fit.

Figure 27: Validation accuracy plot for M N IST 1 using an introspection network without nonlinearity

Figure 28: Test accuracy for M N IST 1 with weight updations via gaussian noise.

Figure 29: Test accuracy plot for Inception V1 network with weight updates via introspection network at steps 2 × 10 5 , 4 × 10 5 and 6 × 10 5 (pink curve) and at steps 3 × 10 5 , 6 × 10 5 and 9 × 10 5 (blue curve)

Which is an RNNFor M N IST 3 after 15000 steps of training,the max accuracy achieved by normal training of network via Adam optimizer was 95.71% whereas with introspection network applied the max accuracy was 96.89%. To reach the max accuracy reached by normal training , the modified network(weights updated by I) took only 8300 steps.For M N IST 4 after 7000 steps of training, the max accuracy achieved by normal training of network was 98.65% achieved after 6500 steps whereas after modification by I it was 98.85% achieved after 5300 steps. The modified network(weights updated by I) reached the max accuracy achieved by normal network after only 4200 steps. It is notable that the introspection network I trained on weight evolutions with ReLU activations was able to help accelerate the convergence of an RNN network which uses sigmoid and tanh activations.CIF AR 1 were done to investigate two issues. The first was to investigate if the introspection network trained on MNIST weight evolutions is able to generalize to a different network and different dataset. The second was to investigate the effect of varying the timing of the initial jump, the interval between successive jumps and the number of jumps. To investigate these issues, four separate training instances were performed with 4 different set of jump points:1. Set 1 : Weight updates were carried out at training steps 12000 and 17000. 2. Set 2 : Weight updates at steps 15000 and 18000 . 3. Set 3 : Weight updates at steps 12000 , 15000 and 19000 . 4. Set 4 : Weight updates at steps 14000 , 17000 and 20000 .

Table content: <table><tr><td/><td>0.98</td><td/><td/><td/><td/><td/><td/><td/><td/><td>0.99</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.96</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.985</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.94</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.98</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Test loss</td><td>0.92</td><td/><td/><td/><td/><td/><td/><td/><td/><td>Test accuracy</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.975</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.9</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.97</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.88</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>With introspection network</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">With introspection network</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Without introspection network</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Without introspection network</td></tr><tr><td/><td>0.86</td><td>2000</td><td/><td>4000</td><td>6000</td><td>8000</td><td>10000</td><td>12000</td><td>14000</td><td>1500 0.965</td><td>2000</td><td>2500</td><td>3000</td><td>3500</td><td>4000</td><td>4500</td><td>5000</td><td>5500</td><td>6000</td><td>6500</td><td>7000</td></tr><tr><td/><td/><td/><td/><td/><td/><td>training steps</td><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">training steps</td><td/><td/><td/></tr><tr><td colspan="10">Figure 13: Validation accuracy plot for</td><td colspan="11">Figure 14: Validation accuracy plot for</td></tr><tr><td colspan="3">M N IST 3</td><td>Test accuracy</td><td>0.978 0.98 0.982 0.984 0.986 0.988 0.99 0.992 0.994</td><td/><td colspan="6">0.6 M N IST 4 0.4 0.8 1 1.2 1.4</td><td/><td/><td>1.6</td><td/><td>1.8</td><td colspan="2">Jump(*2.2) Jump(*2) Jump(*1.5) Jump(*1.3)</td><td>2</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>training steps</td><td/><td/><td/><td/><td/><td/><td/><td colspan="2">x 10 4</td></tr><tr><td colspan="21">Figure 15: Comparison of introspection networks trained with different jump ratios on M N IST 1</td></tr><tr><td colspan="21">network with Adam optimizer.Jump of 2.0 has a more consistent out performance compared to a</td></tr><tr><td colspan="14">jump value of 2.2 even though it reaches a slightly higher accuracy</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="5">4.2.2 CIFAR-10</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="21">We applied our introspection network I on a CNN CIF AR 1 for classifying images in the CIFAR10</td></tr><tr><td colspan="21">(Krizhevsky, 2009) dataset. It has 2 convolutional layers, 2 fully connected layer and a final soft-</td></tr><tr><td colspan="21">max layer with ReLU activation function. Max pooling (3X3 pool size and a 2X2 stride) and batch</td></tr><tr><td colspan="21">normalization has been applied after each convolutional layer. The two conv layer filter weights</td></tr></table>

34%, by normal Adam was 99.3%, by SGD with introspection was 99.21% and by normal SGD was 99.08% . With introspection applied on Adam the model reaches the max accuracy as achieved by normal Adam after only 7200 steps whereas the normal training required 10000 steps.For M N IST 3 the max accuracy achieved by Adam with introspection was 96.9%, by normal Adam was 95.7%, by SGD with introspection was 94.47% and by normal SGD was 93.39% . With introspection applied on Adam the model reaches the max accuracy as achieved by normal Adam after only 8800 steps whereas the normal training required 15000 steps.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.97</td><td/><td/><td/><td/><td/></tr><tr><td/><td>0.99</td><td/><td/><td/><td/><td/><td/><td/><td/><td>0.96</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.95</td><td/><td/><td/><td/><td/></tr><tr><td/><td>0.985</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Test accuracy</td><td>0.98</td><td/><td/><td/><td/><td/><td/><td/><td>Test accuracy</td><td>0.93 0.94</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.92</td><td/><td/><td/><td/><td/></tr><tr><td/><td>0.975</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Introspection on adam</td><td>0.91</td><td/><td/><td/><td/><td/><td>Introspection on adam</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Introspection on Sgd</td><td/><td/><td/><td/><td/><td/><td/><td>Introspection on Sgd</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Normal Adam</td><td/><td/><td/><td/><td/><td/><td/><td>Normal Adam</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Normal Sgd</td><td/><td/><td/><td/><td/><td/><td/><td>Normal Sgd</td></tr><tr><td/><td>0.97</td><td>3000</td><td>4000</td><td>5000</td><td>6000</td><td>7000</td><td>8000</td><td>9000</td><td>10000</td><td>2000 0.9</td><td>4000</td><td>6000</td><td>8000</td><td>10000</td><td>12000</td><td>14000</td></tr><tr><td/><td/><td/><td/><td/><td>training steps</td><td/><td/><td/><td/><td/><td/><td/><td>training steps</td><td/><td/></tr><tr><td colspan="10">Figure 22: Test accuracy comparison for</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="10">M N IST 1 for SGD and Adam optimiser in</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="10">the presence and absence of introspection.</td><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2017):
EDIT: Updated score. See additional comment. I quite like the main idea of the paper, which is based on the observation in Sec. 3.0 - that the authors find many predictable patterns in the independent evolution of weights during neural network training. It is very encouraging that a simple neural network can be used to speed up training by directly predicting weights. However the technical quality of the current paper leaves much to be desired, and I encourage the authors to do more rigorous analysis of the approach. Here are some concrete suggestions: - The findings in Section 3.0 which motivate the approach, should be clearly presented in the paper. Presently they are stated as anecdotes. - A central issue with the paper is that the training of the Introspection network I is completely glossed over. How well did the training work, in terms of training, validation/test losses? How well does it need to work in order to be useful for speeding up training? These are important questions for anyone interested in this approach. - An additional important issue is that of baselines. Would a simple linear/quadratic model also work instead of a neural network? What about a simple heuristic rule to increase/decrease weights? I think it*s important to compare to such baselines to understand the complexity of the weight evolution learned by the neural network. - I do not think that default tensorflow example hyperparameters should be used, as mentioned by authors on OpenReview. There is no scientific basis for using them. Instead, first hyperparameters which produce good results in a reasonable time should be selected as the baseline, and then added the benefit of the introspection network to speed up training (and reaching a similar result) should be shown. - The authors state in the discussion on OpenReview that they also tried RNNs as the introspection network but it didn*t work with small state size. What does *didn*t work* mean in this context? Did it underfit? I find it hard to imagine that a large state size would be required for this task. Even if it is, that doesn*t rule out evaluation due to memory issues because the RNN can be run on the weights in *mini-batch* mode. In general, I think other baselines are more important than RNN. - A question about jump points: The I is trained on SGD trajectories. While using I to speed up training at several jump points, if the input weights cross previous jump points, then I gets input data from a weight evolution which is not from SGD (it has been altered by I). This seems problematic but doesn*t seem to affect your experiments. I feel that this again highlights the importance of the baselines. Perhaps I is doing something extremely simple that is not affected by this issue. Since the main idea is very interesting, I will be happy to update my score if the above concerns are addressed.