HOW IMPORTANT ARE NETWORK WEIGHTS? TO WHAT EXTENT DO THEY NEED AN UPDATE?
Authors: Fawaz Sammani
Mahmoud Elsayed
Abdelsalam Hamdi
Venue:  (2020)
Keywords: weights update
weights importance
weight freezing

Abstract: In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss. Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better). We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19,
ResNet-110 and DenseNet-121). On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully trained model. Our source code can be found in the appendix.

Under review as a conference paper at ICLR 2020
How IMPORTANT ARE NETWORK WEIGHTS? To WHAT EXTENT DO THEY NEED AN UPDATE?
Anonymous authors Paper under double-blind review
ABSTRACT
In the context of optimization, a gradient of a neural network indicates the amount specific weight should change with respect the loss. Therefore, small gradients indicate good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and sometimes better). We experiment on the MNIST, CIFARIO 10 and Flickr8k datasets using several architectures (VGG19, ResNet-110 and DenseNet- 121). On CIFARIO 10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet- 11 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet- 121 parameters results in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting a better BLEU-4 score than the fully-trained model. Our source code can be found in the appendix.
INTRODUCTION
The immense success of deep neural networks we witnessing since the deep learning revolution occurred is surprising. A large variety of vision and language applications ranging from image classification, object detection, image synthesis, image super-resolution, image captioning, language modeling....etc. has proved that neural networks possess powerful capability of learning very complex data. However, training these networks to perform as expected is very time-consuming and requires powerful graphical processing units (GPUs). A recently published open-source project by NVIDIA claimed that training a generative adversarial network (GAN) took more than days on Tesla V100 GPUs.
claimed
However, we argue that a lot of parameters involved during training are important for update only for the first few epochs (in our experiments, the first two epochs only), and can be frozen for the rest of the training epochs. The backpropagation algorithm is the base algorithm used to optimize deep neural networks. For each weight, a gradient is computed with respect to the loss which indicates the amount a weight should change. Large gradients correspond to a large change that will occur in the weight, while small ones (near zero) indicate that the weight is nearly optimized and does not need much change. In particular, if gradient for a particular weight is zero or close to zero, this means that it has either reached its optimal solution, or it is stuck at a saddle point. The former means that the weight has a good value and is less likely to change throughout the training and can be kept frozen. In this paper, we wish to show the redundancy of weights influence and can be kept frozen during training. In particular, we demonstrate that fully training model with all its weights required for the first two epochs only. To justify this, we propose an experimental technique named Partial Backpropagation, which freezes weights that have gradients very near to zero and are less likely to change, with the rest of the weights trained normally. This induces very slight drop in accuracy (and no harm in accuracy for lesser freezing). An overview of
neural network that have no
https://gthuu..co//// http:///gihuuu.co///Waaeeeeeee Woliaaaooo/iiiiisleseeeeng

---

Under review as a conference paper at ICLR 2020
dBly)
4B19)
dbng
dBia)
dBlg)
dBly)
dus
diss
W2
W+
E(y)
E
Wg
Wg
Ws
W44
We
dBly)
dhia
dBly)
d
shes
duy
dota
(a) Full Backpropagation
(b) Experimental Partial Backpropagation
Figure The difference between (a) full backpropagation and (b) our proposed experimental technique. In (b), weights which have nearly zero gradients are less likely change throughout the training and are therefore kept frozen without being updated. These frozen weights are shown in red.
60000
125000
250000
300000
10000
20000
50000
100000
50000
5
10
0.0
5.0
7.5
30.0
6
(a) Layer
(b) Layer 7
(c) Layer 10
(d) Layer 13
Figure 2: The histogram plot of gradients for (a) layer 3, (b) layer 7, (c) layer 10 and (d) layer 13 of a VGG19 convolutional network. large number of gradients have values very close to indicating that a lot of weights in these layers have reached to a good value and are less likely to change throughout the training.
experimental technque is shown in Figure 1. Note that in Figure 1(b), the red weights frozen and not removed or zeroed out.
We can further visualize the histogram of gradients across the network layers have a better understanding of their distributions. In Figure 2, we visualize the distribution of gradients from several layers in VGG19 convolutional network (Simonyan & Zisserman, 2015). particular, visualize the gradients of layers 3, 7, 10 and 13 after training for 2 epochs. We can see large number of gradients with values very near to zero, suggesting that been optimized and are less likely to change throughout the training.
weights in these layers have already
RELATED WORK
We discuss two topics closely (but not exactly) related to our work, namely weight pruning and progressive freezing of hidden layers.
2.1 WEIGHT PRUNING
The reduction of the heavy inference cost of deep networks in low-loooooo oowrresuurce settings is the primary purpose of weight pruning. Neural Networks usually contain large number of parameters that
2

---

Under review as a conference paper at ICLR 2020
make them hard to bring into effective action on embedded devices such as mobile phones. The hardship here happens due to both the network size and evaluation time. Significant redundant parameters could be reduced resulting in compressed network with the unnecessary computation being alleviated while maintaining the same accuracy. In short, the main objective of weight pruning techniques is the reduction of the energy and storage needed to process inference on deep neural networks. Hence, they can be deployed on embedded devices. (Han et al.) 2015) learn only the significant connections to reduce storage. A neural network is first trained to learn the significant connections. Subsequently, the insignificant connections are pruned out and the remaining weights are then fine-tuned. In particular, after the initial training phase, all weights lower than a certain threshold are eliminated. This results in sparse network which is then re-trained so the remaining weights can compensate for the removed ones. The phases of pruning and retraining is an iterative process. After many iterations, the minimum number of weights could be found. In (Babaeizadeh) ct al., 2016). two neurons with highly correlated activations are merged into one instead of removing neuron, which keeps the signals inside the network as close possible to the original one. If the highest correlated neurons are not fully correlated, merging them into one neuron could possibly alter the accuracy factor. (Han et al. 2017) proposed leene--parse-denss training flow. Other works also focus on pruning convolutional filters for efficient inference. In (Molchanov et al. (0177), a new criterion based on Taylor expansion which approximates the change in the cost function induced by pruming network parameters is proposed. Similarly, (Li et al. 2017) proposed an acceleration method for convolutional networks where different filters which have a small influence on the output accuracy are pruned. By pruning whole filters in the network with their corresponding feature maps, the computation cost is greatly minimized. Moreover, (Liu et al., 2017) enforces channel-level sparsity in the network to reduce the model size of a CNN, which decreases the run-time memory and lower the number of computing operations with no harm on the accuracy factor. (Zhang et al. 2018)) presented systematic weight pruning framework of DNNs using the alternating direction method of multipliers
(ADMM). A recenetly proposed netowrk, namely slimmable neural network (Yu et al. 2018) presents simple and general method to train a single neural network which is capable of being exucted at different widths. Rather than training an individual network with different width configurations, a shared network with switchable batch normalization could be trained. During runtime, the network can change its width on the fly according to the resource limitation. Other works on compressing deep neural networks include (Luo et al. 2017 and (Ullrich et al. 2017).
However, an important aspect of weight pruning models is that deep neural networks are mainly trained for the objective of reducing computation at testing time for efficient implementation on hardware systems, which is an iterative training process and requires a lot of procedures and algorithms to be involved during training, which in turn increases the training time. In contrast to our work, we wish to demonstrate the number of redundant parameters, which if frozen and not zeroed out, can have a very slight (or no) influence on the overall aceuracy. Since the two objectives are different, we cannot directly compare weight pruning methods with our experimental technique.
2.2 PROGRESSIVE FREEZING OF HIDDEN LAYERS
In (Brock et al. 2017). the authors proposed training each layer for set portion of the training schedule, progressively "freezing out" layers and excluding them from the backward pass. This technique is motivated by the fact that in many deep networks, initial layers consume most of the cost, but have the fewest parameters and tend to converge to reasonable simple structures, implying that they do not require much update the later layers where most of the parameters This concept aims to reduce training time, however works on freezing complete layers which may contain informative gradients, and corresponding weights avoid getting updated. The study reports training time improvements with loss/no loss in accuracy, however does not report the number of parameters frozen in the overall network. Moreover, the study "progressively" freezes some layers in the network, implying that important weights at specific times in the training schedule experience an update. Therefore, it is also hard directly compare Freezout with our experimental technique.
populated.
3 EXPERIMENTAL TECHNIQUE
As mentioned in Section 1, weights with gradients very near to zero imply that the particular weight is nearly optimized and is less likely to change throughout the training and can therefore be kept
3

---

consists of all Under review as a conference paper at ICLR 2020
frozen. Our experimental technique freezes weights that have gradients very near to from the third epoch onwards, while training rest of the weights normally. All weights with gradients below a particular threshold are frozen. In this section, we first describe how calculate the threshold of gradients and then elaborate on the proposed experimental technique.
3.1 THRESHOLDING TECHNIQUE
We start by extracting the non-zero gradients of the neural network. After training, a slight amount of weights may posses zero gradients, which are not considered when performing thresh- olding. Given the gradients of each layer I, where i indicates the layer number, we have:
(dE(v) 4 dd
G
dE(g)
dE(y) Note that each term dE(y)
the gradients in the layer i. For convenience, we divide each gradient by the corresponding learning rate obtain the original gradient (non-scaled by the learning rate), and take its absolute value dBly)
der This is because we are interested in the magnitude of the gradient, and not its direction. We then plot the kernel density estimation (KDE) G. given by the equation:
1
x 2(1)
f(x)
K
where K is the kernel function and h is the kernel bandwidth. From f(x)), we find the region which has high likelihood of the gradients observation, obtaining the new range of gradients G,, The new range G, is then divided into b sub-ranges and the mean of each sub-range b is obtained to form G. = μr pb.............. ................................ -..
where each gradient in G, assigned to its corresponding mean value. We set to 300. We then apply Otsu thresholding technique to find the optimal threshold value. We re-write the mathematical formulation of otsu's 's method operating on the mean values in Ga. The within-class variance is calculated as:
( (t) +
(2)
where,
(t) = Σ -1
P(i) & (x(t) = E P(i)
(t) E
PP())
& (2(() Ei=1+1 9z(()
:P(()
(3)
(
o1 - m (0)
P( & oB(1) El l (2)
P( (1(
where is the mean value and represents the threshold at each of the 300 steps, (i.e. = " n² ................................ p.)). After calculating the within-class variance of all values in Ga, we select the mean value from GJ with the least within-class variance as our final threshold value.
3.2 EXPERIMENTAL ALGORITHM
We first start by training the network with full backpropagation for epochs. is necessary that the network first understands the data it is dealing with and performs weights update. After the second epoch, the network is considered to have known what weights are optimized and what weights further need optimization. We then perform full backpropagation on the first batch of the third epoch, however before updating the weights, we perform the gradient mask generation. Given the Jacobian matrix of the gradients with respect to the loss F at the first batch of the third epoch:
0fi
afi
Dw
8w.
J
aF.. Owy
&
Dwn.
we divide each gradient by the corresponding learning rate to obtain the original non-scaled gradient, and take its absolute value aF(y)
since interested in the magnitude of the gradient and not its direction. Ω is the number of epochs to train for with full backpropagation at the beginning
de
before starting partial backpropagation. As discussed earlier, we set Ω = 2. For each layer i n,
4

---

Under review as a conference paper at ICLR 2020
create mask m, that has the same shape of the gradient matrix J. We then set the corresponding mask value for each gradient in the layer i to if the gradient is smaller than the threshold t, and 1 otherwise. Thus, we have a list of masks for each layer: = (m, m........ number of layers. We then perform weight update only on the weights corresponding to an entry of 1 in the corresponding mask m,. For the rest of the batches in the third epoch and all the remaining epochs, we use the same mask m. Gradients and weight updates are only calculated for entries that correspond to I in the mask m,. If the weight corresponds to an entry of O the mask, its gradient is set to O and thus no weight update will happen to it (frozen weight). Therefore, the mask is a one-time calculation that depends on one batch the third epoch, which makes critical to have a reasonable batch size that contains all possible non-linear data combinations. In our experiments, we set the batch size to 100. Note that generating a mask requires one full iteration (one batch) of backpropagation. We have also experimented with changing the mask every several epochs (changing the mask each 35 epochs), and found out that it leads to the same performance as using the same mask for all epochs. Moreover, we experimented with performing full backpropagation at the last epoch in order to fine-tune the network so that the frozen weights could possibly change if needed. However, found that the accuracy maintains as it is, implying that the frozen weights with least gradients are already optimized from the first two epochs, which validates our methodology.
where is the
4
EXPERIMENTS
4.1 RESULTS WITH CNNS
We start our experiments on the CIFAR-I0 dataset using several CNN architectures. The CIFAR-10 dataset consists of 60000 32x32 colour images with 10 classes and 6000 images per class. There are 50,000 training images and 10,000 test images. Figure 3 shows the training and validation plots for both full backpropagation (FB) and experimental technique (PB). We use four different architectures: VGG19 (Simonyan & Zisserman, 2015), ResNet- 10 (He et al. 2016), DenseNet-121 (Huang et al., 2017) and LeNet-5 (LeCu) 1998). The construction details are as follows: For VGG19, we construct x 64, 'M', 2 x 128, 'M', 4 x 256, 'M',4 512, 'M', 4 x 512, 'M'] with 3x3 convolutions, where 'M' stands for max-pooling. For ResNet-110, we follow the proposed CIFAR- 10 architecture (He al. 2016) (i.e. 54 blocks with [16, 32, 64] channels). For DenseNet-121, we use [6, [2.12,24.66] blocks with a growth rate of 12. Finally, for LeNet-5, the number of channels are [8.32]. Table demonstrates the results obtained, while Table demonstrates the average performance over 3 runs.
the number of channels
The lowest freezing percentage is witnessed by residual networks. This is expected since these type of networks only learn the residual information and thus a lot of redundancy is already removed. Moreover, we find that VGG19 experiences 9.4% parameters with zero gradients by default (without applying our experimental technique), while LeNet-5 experiences 31.8% by default. Moreover, to demonstrate best performance of our methodology (i.c. highest freezing percentage with lowest accuracy drop), we slightly tweak the original threshold value obtained as discussed in section 3.1, and report the tweaked threshold values in Table 1.
It is worth noting that when training the networks, we avoid using any regularization techniques. In particular, we don't ' use data augmentation, neither weight decay or dropout. We focus on delivering our experiments under normal settings. Models are all trained for 110 epochs. We start with learning rate of 0.001 and decay the learning rate by a factor of 0. at epoch 40 and 80. We train all networks with AMSGrad (Reddi et al. 2018) which takes the maximum between the current and the previous exponentially weighted average of gradients.
4.2 RESULTS WITH RNNS (USING LSTMS)
In order to experiment on complex real-life applications, we trained an image captioning model with visual attention mechanism as proposed in (Xu et al., 2015) on the Flickr8k dataset (Hodosh et al., 2013) using our experimental technique (PB). Flickr8k is dataset that contains 8,000 images with up to captions for each image. We divide the dataset into 6,000 training images and 1,000 validation and testing images, each. In our setup, and randomly sample from any present caption (5-present captions) times if the 5 captions are not present, resulting in 30,000 training captions and 5,000 validation and testing captions, each. We
the 5 captions for each image (if all 5 are present)
5

---

Under review as a conference paper at ICLR 2020
12
90
Full Backprop
85
Partial Backprop
80
Validaion Accuracy
75
Train Loss
0.8
60
55
0.4
50
Rull Backprop
Partial Backprop
Epochs
Epochs
(a)
(b)
1.6
Full Backprop
88
1.4
Partial Backprop
83
1.2
Validation Accuracy
733
Train
0.6
0.4
0.2
58
Full Backprop
Partial Backprop
20
100
Epochs
Epochs
(d)
1.4
Full Backprop
Backprop
1
Validation Accurscy
75
0.6
70
0.2
full Backprop
Partial Backprop
40
60
100
40
60
100
Epochs
Epochs
(e)
(f)
0.18
1000
0.16
Full Backprop
99
0.14
Portial Backprop
98
0.1
Validation
0.06
0.04
92
Full Backprop
0.02
91
Partial
90
Epochs
Epochs
Figure 3: The training loss (left) and validation accuracy (right) plots for (a.b) VGG19 trained on CIFAR10, (c.d) ResNet-110 10 trained on CIFAR10, (e,f) DenseNet- 121 trained on CIFAR10 10 and (g.h) LeNet-5 trained on MNIST

---

Under review as a conference paper at ICLR 2020
Network
Dataset
Threshold
V. Accuracy (FB)
Accuracy (PB)
Frozen
VGG19 ResNet-110 CIFARIO
CIFARIO
0.04
88.12 %
87.88%
79.68%
0.19
83.72%
82.81 %
49.85%
DenseNet-121
CIFARIO
0,45
88.20%
87.63%
69.07%
LeNet-5
MNIST
0.50
99.25%
99.22%
93.70%
Table 1: Performance CIFARIO and MNIST datasets for different types of CNNs with the threshold, validation accuracy for full backpropagation (FB), validation accuracy for experimental technique (PB) (PB) and the freezing ratio in parameters which is calculated as: (total gradients non-zero gradients / total gradients) 100
Network
Dataset
Threshold
V. Accuracy (FB)
Accuracy (PB)
Frozen
VGGI9
CIFARIO
0.04
88.12 %
87.70%
80.45%
ResNet-110
CIFARIO
0.19
83.72%
82.83%
51.53%
DenseNet-12
CIFARIO
0.45
88.20%
87.52%
69.33%
LeNet-5
MNIST
0.50
99.25%
99.22%
92.36%
Table Average Performance over
CIFARIO and MNIST datasets for 4 different types of CNNs with the threshold, validation accuracy for full backpropagation (FB), validation accuracy for our experimental technique (PB) and the freezing ratio in parameters which is calculated as: (total gradients non-zero gradients / total gradients) x 100
resize all images to 256 x 256 pixels. We use single layer LSTM with a hidden size of 512. The batch size is 60. We use the Adam optimizer (Kingma Ba 2015) with an initial learning rate of 5e-4 and anneal the learning rate by a factor of 0.8 once the BLEU- score shows no improvement for 3 consecutive epochs. The word embedding and attention size is set to 512. We train for a maximum of 15 epochs with early stopping if the validation BLEU-4 score has not improved for 10 consecutive epochs. When sampling, we use a beam size of 3. We use BLEU (Papineni et al. 2001) with up to 4 grams (BLEU-1, BLEU-2, BLEU-3, BLEU-4) soft attention variant Xu et al. (2015). The best BLEU-4 score obtained when training with full backpropagation is 0.099. When applying experimental technique (PB), using threshold of 0.05, we obtain 61.81% % redundant parameters which are frozen from the third epoch onwards. Under this setting, obtain higher BLEU-4 score of 0.101 than the fully trained model. The freezing ratio is calculated as: (total gradients non-zero gradients / total gradients) x 100. Figure shows some generated captions. For evaluation results on all BLEU scores, see Figure
evaluation metric. We experiment on the
Figure 4: Results on training image captioning model with visual attention mechanism (Xu et al.) 2015) on the Flickr8k dataset using experimental technique (PB). The predicted word and attention weights for each timestep are shown for 4 generated captions of different lengths
7

---

Under review as a conference paper at ICLR 2020
0.35
0.3
BLEU-n Score
0.25
0.2
0.15
0.1
0.05
10
12
14
16
Epochs
BLEU-4 (FB)
BLEU-4 (PB)
BLEU-3 (FB)
BLEU-3 (P8)
BLEU-2
BLEU-2
Figure 5: BLEU-n scores (where is the number of grams) reported on the the Flickr8k dataset for image captioning with attention mechanism using both full backpropagation (FB) and our experimental technique (PB).
CONCLUSION
We provided an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. Through our experiments, we emphasized the number of redundant parameters that earry no informative gradient, which if frozen from the third epoch onwards, slightly effect (and in sometimes do not) the overall accuracy of the model. To prove our concern, ran experiments on the MNIST and CIFAR datasets using several CNN architectures (VGG19, ResNet- 10 and DenseNet- 121), well as the Flick8k dataset using image captioning architecture composed of LSTM networks with attention mechanism. Our experiments successfully prove the concern of this paper.
REFERENCES
Mohammad Babaeizadeh, Paris Smaragdis, and Roy H. Camphell. Noiseout: simple way prune neural networks. CoRR, 1b0/211 2016.
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. CoRR, abs/1706.0098s. 2017.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In NIPS, 2015.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro, and William J. Dally. Dsd: Dense-sparse-dens. training for deep neural networks. In ICLR. 2017.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as ranking task: Data, models and evaluation metries. In J. Artif. Intell. Res., 2013.
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412,698s 2015.
8

---

Under review as a conference paper at ICLR 2020
Yann LeCun. Gradient- based learning applied to document recognition. 1998.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. CoRR, abs/1608.08710 10, 2017.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2755-2763, 2017.
Hao-HHoo Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5068-5076, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2001.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. CoRR, abs/1904.09237, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/ 1459, 2015.
Karen Ullrich, Edward Meeds, and Max Welling, Soft weight-sharing for neural network compression. CORR, Abs//702.04400 2017.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan R. Salakhutdinov, Richard S, Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In JCML, 2015.
Jincheng Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas S. Huang. Slimmable neural networks. CoRR, abs/1812.08992 2018.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yuankai Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In ECCV, 2018

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
We provide our base code in this appendix. We use PyTorch to implement experiments. We modified publicly available codes for the architectures we used.
import torch
import torch
import torch.nn. nn. functional as F
import torchvission transforms transforms
import torchvision. datasets as datasets plt
import seghhotiib.pppleee
import numpy as np
import math
from collections import Counter from tensorboard logger import confiqure, log_value import torch, backends cudnn as cudnn get_ipython () run_ wwwrrrunnemaaeeeeeeeeeeee ("matllotii"" "inline")
batch_size 100
epochs = 110
# Set normal_traii to True to disable partial backpropagation
normal_train False
mnist = False
cifar10 True
tensorboard True
best_precl
print_freq = 100
calculated False
name = "densenet"
save True load checkpoint False start_epoch, = 0
Epoch after which to start partial backprop (epoch starts at 0)
start_pb_epoctt cudrn.benchmark True
def zeroGrad (optimizer, net, masks)
if masks is None:
masks list ()
for param in net parameters
gradients param. gorram.gradddeteee detach
gradients = gradients / wtw.iiieeaeaaagrroppep.... [0] ["1r"]
all pol_pvostiie aba
thresh 0.45
mask = torch. ones_nikeeeeeeee (all (all_psstiilll
param. gradd-aaraaaaak
baaks.appent (mask) return masks, net
else:
for i,param in enumerate mat..paraeeeers )
param. grad* [i]
return masks, net
def count2eroGrads (net):
zeros
for param in net patameaaesssss () grad param. grad. detach detac ()
10

---

Under review as a conference paper at ICLR 2020
zeros torch.sume (grad 0) int ()) ) ettm ()
return 2eros
def countNonZeroGrads (net):
non_zeros 0
for param net (etapaammeters()
grad param. detacd
non_zeros += torch. sum (grad != 0) .int ()) iitem ()
return non_zeros
def count_parameters (model):
return sum (p. numel () for in model parameters () if p.requires_grad
def set_ set_learning_rate (optimizer, new_lr): 1r): for param_group in optimizer.param.grooo. paraanggrooss: paran_group(" (' rew_lr
transformation transforms. Compose ([transforms. ToTensor
Compooe
transforms Normalize (0. 4914, 0.4822, 0.4465),
(0.2023, 0.1994, 0.2010)
train_dataset datasets. CIFAR10 (root="./cifarlddas
train=True,
download=True,
transformetrrassoomatiiin
test_dataset Catasets.CIFAN /root=",,cifaarldddtt
train=False,
download-Tzue,
tensorboard: configure ("runs/ss" (name)
trade_loader torch. utilh. D (dataset = train_dataset, batch_size batch_size, shuffle True)
torch. utils data
wtw.:llss.aaaa.a.aaaaaaaaaa.eeee DataLoader (dataset test test_dataset, batch_size = batch_size, shuffle False)
**ク88044 LeNet and VGG19 дес**эннндне6
$ from models import model CNN ()
# model = VGG (VGG1 9)
#AS RESNETCIFAR #USAS:***** from resnetcifar import resnet
model = resnet. resset.resnet1000
### DENSENET н*еееАеннесс
from models import
model = densenet_cifan CUDA terch. is_availllaa
CUDA:
11

---

Under review as a conference paper at ICLR 2020
model model.
criterion nn nn. mm..crosstttrooyosss
optimizer = torch. opt.m...mmmmmmmmm.m Adam (model.parameee parameters (),
amsgrad=True)
if coad_checkpoint:
checkpoint torch. load "best_model_pooh__.))
start_epoch = checkpoint ['epoch']
best_precl checkpoint | ('best_preckl
model_loaa_sttte__diit (checkpoint ('annttssssteeee state_dicct
class AverageMeter (object):
def
_init_ _init_(self]] relf.resetlo ()
def
reset (self):
self.va 0
self.avg 0
self. sum
0
def
update (self, val, n=1);
self.val val self.sum == val
self.count += n
self.avg self. self.count
def accuracy (output, target, top)=
"""Computes the precisionek for the specified values of k"n maxk max (topk)
batch_size target size (0)
pred output. topk (maxk, 1, True, True)
pred = pred.t
correct pred. eq (target. view (1, -1) expand_as (pred)
res []
for k in topk:
correct_k correct [:k] www.wwwwll float () sum (0)
res rep.ad (correct_k.o mul (100.0 batch_size)
return re
def train (taiiinteind,,,,,r model, criterion, optimizer, epoch):
global calculated, masks #global learning rate losses AverageMeter ()
top1 = AverageMeter ()
switch to train mode
model.
for i, (input, target) in enumerate (train_loadez) target target cuda (async=True)
input input. cuda () input_var torch. www.grada.rrrdd..oomoo periabbe(innuul
target_var = torch. autograd Variable (target)
# compute output
output model (input_val varr
loss criterion (output, target_var)
12

---

Under review as a conference paper at ICLR 2020
# measure accuracy and record loss
precl accuracy (output data, target, topk= )1,)) [0]
losses update (loss. [0)a input.size (0))
uppl.e (precl [0], input.size(0) # compute gradient and do SGD step model model.zero_grad
loss loss.backward()
if not normal_train:
if calculated False and epodt>start_pébeeodh calculated True
masks, model zeroGrad (optimizer, model, masks None)
else:
if bpoch>star__pb_poo
masks, model aeroGrad
model, masks)
if print_freq== 0:
# Make sure gradients are zeroed out print ("Number Zero Gradients: ", count2eroGrads (model)) #Make sure optimizer parameters and model parameters are the same pm set (p..date_ptr)) for p model.parameters set (p.data_ptr() for in optimizer pawteee.prragggrrrueeee [0] [params] print ("Pointer Status:", pm po)
optimizer.step!
if print_freq 0:
nonzeros_param countNon7eerGGrad (model)
original_param count_parametters (model)
reduction ((original_paran nonzeros_param [riginal_param) 100
print ("Epoch: [10)]((1)((2}]}}**
"Loss
((loss. avg: .4g:
"Prec81 (topl.val:.3f. ([topl.avg) avg:.3f)) .t". format (epoch,
len (train_loader
loss=losses, topl=top1))
print ("Non-Zero Gradients: ()" formatt mhonzeros_paraaee print ("Total Gradients: ()".formal format (original_ param) print ("Reduction: (:.3f) format (reduction)
log to TensorBoard if tensorboard:
log_value ("train_loss", losses. avg, epoch) val_v ("train_ace"" topl.avg, epoch)
def validate (val_loades model, criterion, epoch)
losses = AverageMeter topl AverageMeter
switch to evaluate mode evae)
for (input, target) in enumerate Ival_loader? target target.. cuda (async=True) input input.cudal
input_var torch. aut wettttorrd..aariall... Variable (input, volatile=True) target_var torch. torch. autograd, Variable (target, volatile-True) # compute output
output model sinput_var)
13

---

Under review as a conference paper at ICLR 2020
loss criterion (output, target_ var) # measure accuracy and record loss
precl accuracy (output data, target, topk= (1,)) [0]
losses update (loss. data input.size (0))
topl update (prec1 [0], input.size(0))
if print_freq 0:
("Test: [10)/[1)]{*
"Loss (loss.val:.45) (Looss.ag::... 4f)) \t"
"Prec01 (top1.val:.35) (vtopl.avg:... " format (i,
ln (val_loaden) loss=losses,
topl=top1))
print ("* Precê1 (topl.avg:.3ff ". format (top1=top1)) log to TensorBoard
if tensorboard:
log_value ("val_loss", losses.ave avg, epoch)
Log_value ("val_acc", topl.avg, epoch) return topl.avg
for epoch in range range (start_epoch, epochs)
if epoch == 40:
ceg_laariigg_rateepppiiiii
new_lr 0.0001)
if epoch == 80: pet_llamminng_atteepptiiiii new 1r 0.00001)
(aantttain_ooadeee model, criterion, optimizer, epoch) precl validate (val_loader, model, criterion, epoch)
is_best = precl > best_prec1
best_precl max (precl, best_precl)
if save and is_best:
torch. save (""epoc";; epoch,
"state_dict" stae__state_dict (),
"best_prec1": best_precl),
pth" . format (epoch, prec1) )
print ("Best accuracy: best_predd it)
14


Reviews:
[{"review_id": "SkgTDt7pYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the importance of a neural networks weights and to which extend do they need to be updated. Particularly, the authors show that freezing weights which have small gradient in the very beginning of the training only results in a very slight drop in the final accuracy.\n\nThis paper should be rejected because (1) the paper only provides some empirical results on freezing network network weights, I don't think there are much insights and useful information; (2) To my knowledge, the phenomenon that only a few parameters are important has been observed before by many papers.\n\nGiven that, I vote for a rejection.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HylH3mj2Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents the empirical observation that one can freeze (stop updating) a significant fraction of neural network parameters after only training for a short amount of time, without hurting final performance too much. The technical contribution made by this paper is an algorithm for determining which weights to freeze, called partial backpropagation, and an empirical validation of the algorithm on various models for image recognition.\n\nThe observation that weights can be frozen is somewhat interesting, although similar findings have been reported before. \nIt's not clear the proposed algorithm is useful. The authors mention that fully parameterized models are expensive to run, but they don't demonstrate any speed-ups using their approach. Such speed-up would also not be expected since the forward pass of the algorithm cannot get faster by freezing weights, and the impact on the backward pass is limited. I'd be willing to raise my rating if the authors can convince me of the usefulness of their algorithm.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxqGH95Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, the authors performed an empirical study on the importance of neural network weights and to which extent they need to be updated. Some observations are obtained such as from the third epoch on, a large proportion of weights do not need to be updated and the performance of the network is not significantly affected.\n\nOverall speaking, the qualitative result in the paper has already been discovered in many previous work, although the quantitative results seem to be new. However, there is large room to improve regarding the experimental design and the comprehensiveness of the experiments. Just name a few as follows:\n\n1)\tFor different models and different tasks, the quantitative results are different. There is no deep discussion on the intrinsic reason for this, and what is the most important factor that influences the redundancy of weight updates. The authors came to the conclusion that from the third epoch on, no need to update most of the weights. “3” seems to be a magic number to me. Why is it? No solid experiments were done regarding this, and no convincing analysis was made.\n\n2)\tThe datasets used in the experiments are not diverse enough and are not of large scale. For example, the CIFA-10 and MNIST datasets are relatively of small scale. What if the datasets are much larger like ImageNet. In such more complicated case, will the weight updates still be unnecessary? Will the ratio and the epoch number change? What is the underlying factor determining these? For another example, there are many NLP datasets for language understanding and machine translation, which are of large scale. Why choosing an image captioning dataset (which I do not agree to be real-life experiments when compared with language understanding and machine translation)? Can the observations generalizable to more complicated tasks and datasets?\n\n3)\tThe models studied in the paper are also a little simple, especially for the text task. Why just using a single-layer LSTM? Why not popularly used Transformer? \n\nAs a summary, for an empirical study to be convincing, the tasks, datasets, scales, model structures, detailed settings, and discussions are the critical aspects. However, as explained above, this paper has not done a good job on these aspects. Significantly more work needs to be done in order to make it an impactful work. \n\n*I read the author rebuttal, but would like to keep my rating unchanged.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors demonstrate that starting from the 3rd epoch, freezing a large fraction of the weights (based on gradient information), but not entire layers, results in slight drops in performance.

Given existing literature, the reviewers did not find this surprising, even though freezing only some of a layers weights has not been explicitly analyzed before. Although this is an interesting observation, the authors did not explain why this finding is important and it is unclear what the impact of such a finding will be. The authors are encouraged to expand on the implications of their finding and theoretical basis for it. Furthermore, reviewers raised concerns about the extensiveness of the empirical evaluation.

This paper falls below the bar for ICLR, so I recommend rejection.
Decision: Reject