Title: DSD: Dense-Sparse-Dense Training for Deep Neural Networks
Authors: Song Han
Jeff Pool
Sharan Narang
Huizi Mao
Enhao Gong
Shijian Tang
Erich Elsen
Peter Vajda
Manohar Paluri
John Tran
Bryan Catanzaro
William J. Dally
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1 accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50 by 1.1%, respectively. On the WSJ’93 dataset, DSD improved DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S step. At testing time, DSD doesn’t change the network architecture or incur any inference overhead. The consistent and significant performance gain of DSD experiments shows the inadequacy of the current training methods for finding the best local optimum, while DSD effectively achieves superior optimization performance for finding a better solution. DSD models are available to download at https://songhan.github.io/DSD.

TL;DR: DSD effectively achieves superior optimization performance on a wide range of deep neural networks.

Review discussion:
When applying DSD, do the training time increase? by how much?
After training with DSD, how much memory storage is saved?
Does DSD also improve inference time compared with models trained without DSD?
Thanks for your questions, I will try to clarify them here and make sure it's reflected in the final paper.
(1) When applying DSD, the training time increases if using pretrained model.
For example, as shown in Table 2, it takes 60 epochs to train the BVLC GoogleNet with conventional method, while DSD training adds 33 extra epochs. The training time per epoch is the same. We believe such increase is worthwhile because conventional training can't get the same accuracy as DSD training even trained for 93 epochs.
However, we found training time can be shortened by *avoiding using pretrained model*, i.e. pruning the model before it converges.  In section 4.5, the DeepSpeech model is pruned early, one iteration of DSD training took 50+50+50=150 epochs, and the conventional training also took 150 epochs, but DSD training achieves better performance. This shows that we can reduce the DSD training time by pruning early. A similar finding is observed in DeepSpeech-2 in section 4.6.
(2,3) DSD training produces the same model architecture; it doesn't save the storage, nor does it increase the storage. It also has the same inference time. The advantage of DSD training is that it produces better accuracy.
This brings an interesting question of optimizing accuracy, speed, and storage at the same time.
Instead of performing DSD, we can do DSDS, which still produces better accuracy model (see Table7 for DeepSpeech and Table9 for DeepSpeech2), but ends up with a sparse model, which has the potential of reduced storage by applying Deep Compression, and faster inference by using EIE accelerator. Thanks for pointing this out and we'll add the discussion in the next version.
1) What’s the difference between W and \tilde{W} in Algorithm 1? (I'm assuming none?)
2) Algorithm 1 says that in the sparse training phase the matrix W stays the same, the gradients are all computed as before, but then only the “Mask” weights are updated.
But this is in contrast to Figure 2 (and I think the text) which implies that what happens in practice is that connections corresponding to low-norm weights are actually removed/set to zero, and then training continues as if the network was much sparser. Is this in fact a contradiction, or is there something I’m missing here?
2a) If the weights are indeed removed, is there a temporary increase in the loss when this happens? (i.e., between Fig 2a and 2b?)
1) What’s the difference between W and \tilde{W} in Algorithm 1? (I'm assuming none?)
That's right; This is a typo and the tilde should be omitted. We'll correct this in the next version.
2) how to update and apply mask:
This is not a contradiction. Low-norm weights are removed by setting them to zero. This is relected in both the algorithm and the figure. During the sparse phase, the gradients are computed and applied to all the weights, but then the weights are multiplied (element-wise) by the sparsity mask: W(t) = W(t) * Mask.  This mask is '1' for large weights, but '0' for weights under the pruning threshold.
So, any weights that are pruned are forced to remain ‘0’, and the weight matrix becomes a sparse matrix.
2a) If the weights are indeed removed, is there a temporary increase in the loss when this happens? (i.e., between Fig 2a and 2b?)
Exactly, there is a temporary increase in loss at the initial pruning.  The sparse training phase recovers this accuracy loss, as in prior work (Han et al. NIPS'15).
Thanks for pointing out these confusion - we will clarify the algorithm in the paper.
Which learning rates are used for the baseline methods?
For example, in Tables 2 and 3, LR is 1e-2 for the baseline.  The baselines achieve results that seem to be worse than what is reported in the corresponding papers (GoogLeNet 10.96% top-5 vs. 10.07% in CVPR 2015, VGG-16 11.32% vs. 7.32%).
As I understand, the DSD method continues learning using a smaller learning rate (1e-4) and seems to run for more epochs (60+11+22 for GoogLeNet comparison?).
Is there a comparison with standard training, dropping the learning rate upon "convergence" and continuing to learn?
We did not train and report the accuracy of the baseline models ourselves.
Instead, our baseline models and their accuracies are obtained from the widely used Caffe model zoo (https://github.com/BVLC/caffe/wiki/Model-Zoo). The reported accuracy is without any data augmentation. We use single-scale, single-crop, no jittering, which matches well with the accuracy report from Caffe model zoo.
On the contrary, the accuracy reported in the papers used data augementation. Data augmentation is orthogonal to learning algorithm, neither the baseline nor our DSD model use data augmentation for evaluation, so we believe we have a fair comparison to show the effectiveness of DSD training.
- For GoogLeNet, 1e-2 is the initial learning rate, the full learning rate schedule (such as stepsize) could be found here: https://github.com/BVLC/caffe/blob/master/models/bvlc_googlenet/solver.prototxt. We used this pretrained BVLC GoogleNet from Caffe model zoo (https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet), which reports 11.1% top5 error on a center crop.  We saw 10.96% top5 error on the same baseline model, which is slightly better.
- For VGG16, 1e-2 is also the initial learning rate. The baseline model is obtained from: http://www.robots.ox.ac.uk/~vgg/software/very_deep/caffe/VGG_ILSVRC_16_layers.caffemodel. Again the performance reported (7.32% top5 error) is for a multi-scale evaluation; for a single-crop, single-scale from images with their smallest side resized to 224, our baseline of 11.32% top5 error is slightly better than that reported by other users of the same model (https://gist.github.com/ksimonyan/211839e770f7b538e2d8#gistcomment-1403727).
Yes, DSD method continues learning using a smaller learning rate and run for more epochs. We have compared using standard training, dropping the learning rate upon "convergence" and continuing to learn for more epochs. For VGG16, this converged to 29.3%/10.0% error, as compared to 27.19%/8.67% error produced by DSD which is significantly lower.
So even worked on a model that is not fully converged on imagenet, DSD can win the *fully converged* model by a large margin (27.2% compared with 29.3% top1 error).
To further verify the advantage of DSD training is statistically significant, we did T-test by repeating the experiment 16 times on cifar10 using ResNet-20. The results demonstrate the DSD training achieves significant improvements from both the baseline model (T-test result with p<0.001) and conventional fine tuning (T-test result with p<0.001). The above T-test results are in the appendix.
This paper presents a training strategy for deep networks.
First, the network is trained in a standard fashion.
Second, small magnitude weights are clamped to 0; the rest of the weights continue to be trained.
Finally, all the weights are again jointly trained.  Experiments on a variety of image, text, and speech datasets demonstrate the approach can obtain high-quality results.
The proposed idea is novel and interesting.  In a sense it is close to Dropout, though as noted in the paper the deterministic weight clamping method is different.
The main advantage of the proposed method is its simplicity.  Three hyper-parameters are needed: the number of weights to clamp to 0, and the numbers of epochs of training used in the first dense phase and the sparse phase.  Given these, it can be plugged in to training a range of networks, as shown in the experiments.
The concern I have is regarding the current empirical evaluation.  As noted in the question phase, it seems the baseline methods are not trained for as many epochs as the proposed method.  Standard tricks, such as dropping the learning rate upon "convergence" and continuing to learn, can be employed.  The response seems to indicate that these approaches can be effective.  I think a more thorough empirical analysis of performance over epochs, learning rates, etc. would strengthen the paper.  An exploration regarding the sparsity hyper-parameter would also be interesting.
We thank the reviewer for the comments. The reviewer's concern is unnecessary.
We have compared using DSD method and baseline method for the same number of epochs in Deep Speech (Table 7) and Deep Speech2 (table 9). Baseline training fully converged to WER of 28.03%/33.55% with 150 epochs, while DSD training achieved a better 27.90%/32.20% WER with the same total 50+50+50=150 epochs.
Similar for Deep Speech2 (Table 9), baseline training fully converged to WER of 9.55%/14.52% with 60 epochs, while DSD training achieved a better 9.11%/13.96% WER with the same total 20+20+20=60 epochs. For both Deep Speech and Deep Speech2 we trained the baseline models by ourselves so we make sure it's fully converged.
Thus it is a fair comparison that DSD is advantageous.
The overall observation is: training for same #epochs, DSD can perform better; training for another DSD iteration (DSDSD), DSD can perform even better.
Even worked on a model that is not fully converged on imagenet, DSD can win the *fully converged* model by a large margin (27.2% compared with 29.3% top1 error).
To further verify the advantage of DSD training is statistically significant, we did T-test by repeating the experiment 16 times on cifar10 using ResNet-20 to compare the error rate of DSD training and conventional fine-tuning (by dropping the learning rate upon "convergence" and continuing to learn) under the same number of epochs. The DSD training on average achieved Top-1 testing error of 7.89%, which is 0.37% absolute improvements (4.5% relative improvements) from the baseline model and relatively 1.1% better than what the conventional fine-tuning achieves. The results demonstrate the DSD training achieves significant improvements from both the baseline model (T-test result with p<0.001) and conventional fine tuning (T-test result with p<0.001). The above T-test results are in the appendix.
The T-test experiment also shows that DSD could reduce the variance of learning. In the 16 repeated experiments, DSD training has lower standard deviation of errors  (sparse phase std=0.05%, dense phase std=0.03%) compared with their counterparts using conventional fine-tuning (first phase std=0.08%, second phase std=0.04%). Each phase corresponds to the same number of epochs.
Hope these empirical result helps clarify the question. We already incorporated related results at Table 7/9 and at the end of page 6 and also in appendix-A, and we'd be happy to add more discussion to the paper.
Summary:
The paper proposes a model training strategy to achieve higher accuracy. The issue is train a too large model and you going to over-fit and your model will capture noise. Prune models or make it too small then it will miss important connections and under-fit.
Thus, the proposed method involves various training steps: first they train a dense network, then prune it making it sparse then train a sparse network and finally they add connections back and train the model as dense again (DSD). The DSD method is generic method that can be used in CNN/RNN/LSTM. The reasons why models have better accuracy after DSD are: escape of saddle point, sparsity makes model more robust to noise and symmetry break allowing richer representations.
Pro:
The main point that this paper wants to show is that a model has the capacity to achieve higher accuracy, because it was shown that it is possible to compress a model without losing accuracy. And lossless compression means that there’s significant redundancy in the models that were trained using current training methods. This is an important observation that large models can get better accuracies as better training schemes are used.
Cons & Questions:
The issue is that the accuracy is slightly increased (2 or 3%) for most models. And the question is what is the price paid for this improvement? Resource and performance concerns arises because training a large model is computationally expensive (hours or even days using high performance GPUs).
Second question, can I keep adding Dense, Sparse and Dense training iterations to get higher and higher accuracy improvement? Are there limitations to this DSDSD… approach?
We thank the reviewer for the kind words in the review.
1.a) The 2% or 3% increase of accuracy for a well-designed model is non-trivial. The baseline ResNet-50 top5 error is already below 7%, improving on such good model is not an easy task.  I have chatted with a notable web company in China, they pay 1Million CNY to buy more dataset just to increase the relative accuracy by only 5%, while the performance improvement of DSD training is 3.6%-13.7%.
1.b) Fine-tuning a model with DSD is indeed extra time, but the longer training time is dwarfed by the time it took to do design space exploration for new topology/architecture and tune hyperparameters.  For the tens to hundreds of models that were trained and then discarded, only one is fine-tuned with DSD.
1.c) We also explored DSD training with shorter epochs, this is possible by pruning early, i.e. prune before the dense model fully converges. In the DeepSpeech experiments, weights are pruned early after training the initial model prematurely for only 50 epochs. Shown in Table 10, the baseline training converged to WER of 28.03%/33.55% with 150 epochs, while DSD training converged to 27.90%/32.20% WER with 50+50+50=150 epochs.
So, same epochs, better accuracy. With another SD iteration it can do even better.
2) Yes, we can do DSDSD. We have already listed the DSDSD result in Table 10 and Table 12, where we have five rows of result for each DSDSD step. The limitation to this approach is the diminishing returns as we add more iterations.
Training highly non-convex deep neural networks is a very important practical problem, and this paper provides a great exploration of an interesting new idea for more effective training.  The empirical evaluation both in the paper itself and in the authors’ comments during discussion convincingly demonstrates that the method achieves consistent improvements in accuracy across multiple architectures, tasks and datasets. The algorithm is very simple (alternating between training the full dense network and a sparse version of it), which is actually a positive since that means it may get adapted in practice by the research community.
The paper should be revised to incorporate the additional experiments and comments from the discussion, particularly the accuracy comparisons with the same number of epochs.
We thank the reviewer for the comments. The same #epochs experimental results are addressed in the reply to reviewer1, to facilitate reading let me paste here:
We have compared DSD v.s. conventional training for the same number of epochs in Deep Speech (Table 7) and Deep Speech2 (Table 9). Baseline training fully converged to WER of 28.03%/33.55% with 150 epochs, while DSD training achieved a better 27.90%/32.20% WER with the same total 50+50+50=150 epochs.
Similar for Deep Speech2 (Table 9), baseline training fully converged to WER of 9.55%/14.52% with 60 epochs, while DSD training achieved a better 9.11%/13.96% WER with the same total 20+20+20=60 epochs. For both Deep Speech and Deep Speech2, we trained the baseline models by ourselves so we make sure it's fully converged.
Thus it is a fair comparison that DSD is advantageous.
Even worked on a model that is not fully converged on imagenet, DSD can win the *fully converged* model by a large margin (27.2% compared with 29.3% top1 error).
To further verify the advantage of DSD training is statistically significant, we did T-test by repeating the experiment 16 times on cifar10 using ResNet-20 to compare the error rate of DSD training and conventional fine-tuning (by dropping the learning rate upon "convergence" and continuing to learn) under the same number of epochs. The DSD training on average achieved Top-1 testing error of 7.89%, which is 0.37% absolute improvements (4.5% relative improvements) from the baseline model and relatively 1.1% better than what the conventional fine-tuning achieves. The results demonstrate the DSD training achieves significant improvements from both the baseline model (T-test result with p<0.001) and conventional fine tuning (T-test result with p<0.001). The above T-test results are in the appendix.
The T-test experiment also shows that DSD could reduce the variance of learning. In the 16 repeated experiments, DSD training has lower standard deviation of errors  (sparse phase std=0.05%, dense phase std=0.03%) compared with their counterparts using conventional fine-tuning (first phase std=0.08%, second phase std=0.04%). Each phase corresponds to the same number of epochs.
Hope these empirical result helps clarify the question. We already incorporated related results at Table 7/9 and at the end of page 6 and also in appendix-A, and we'd be happy to add more discussion to the paper.

Meta-review:
role: PC
data: comment: Important problem, simple (in a positive way) idea, broad experimental evaluation; all reviewers recommend accepting the paper, and the AC agrees. Please incorporate any remaining reviewer feedback.

Decision: Accept (Poster)