Title: DSD: DENSE-SPARSE-DENSE TRAINING FOR DEEP NEURAL NETWORKS

Abstract: Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1 accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50 by 1.1%, respectively. On the WSJ'93 dataset, DSD improved DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S step. At testing time, DSD doesn't change the network architecture or incur any inference overhead. The consistent and significant performance gain of DSD experiments shows the inadequacy of the current training methods for finding the best local optimum, while DSD effectively achieves superior optimization performance for finding a better solution. DSD models are available to download at https://songhan.github.io/DSD.

1. INTRODUCTION:

Deep neural networks (DNNs) have shown significant improvements in many application domains, ranging from computer vision (He et al. (2015) ) to natural language processing (Luong et al. (2015) ) and speech recognition (Amodei et al. (2015) ). The abundance of powerful hardware makes it easier to train complicated DNN models with large capacities. The upside of complicated models is that they are very expressive and can capture the highly non-linear relationship between features and output. The downside of such large models is that they are prone to capturing the noise, rather than the intended pattern, in the training dataset. This noise does not generalize to new datasets, leading to over-fitting and a high variance. Initialization: W (0) with W (0) ∼ N (0, Σ) Output :W (t) .

----------------Initial Dense Phase ---------------while not converged do W (t) = W (t-1) -η (t) ∇f (W (t-1) ; x (t-1) ); t = t + 1; end -----------------Sparse Phase ------------------/ / initialize the mask by sorting and keeping the Top-k weights. S = sort(|W (t-1) |); λ = S ki ; M ask = 1(|W (t-1) | > λ); while not converged do W (t) = W (t-1) -η (t) ∇f (W (t-1) ; x (t-1) ); W (t) = W (t) • M ask; t = t + 1; end -----------------Final Dense Phase ----------------while not converged do W (t) = W (t-1) -η (t) ∇f (W (t-1) ; x (t-1) ); t = t + 1; end goto Sparse Phase for iterative DSD;

In contrast, simply reducing the model capacity would lead to the other extreme, causing a machine learning system to miss the relevant relationships between features and target outputs, leading to under-fitting and a high bias. Bias and variance are hard to optimize at the same time.

To solve this problem, we propose a dense-sparse-dense training flow (DSD), a novel training strategy that starts from a dense model from conventional training, then regularizes the model with sparsityconstrained optimization, and finally increases the model capacity by restoring and retraining the pruned weights. At testing time, the final model produced by DSD still has the same architecture and dimension as the original dense model, and DSD training doesn't incur any inference overhead. We experimented DSD training on 7 mainstream CNN / RNN / LSTMs and found consistent performance gains over its comparable counterpart for image classification, image captioning and speech recognition.

2. DSD TRAINING FLOW:

Our DSD training employs a three-step process: dense, sparse, re-dense. Each step is illustrated in Figure 1 and Algorithm 1. The progression of weight distribution is plotted in Figure 2 .

Initial Dense Training::

The first D step learns the connection weights and importance via normal network training on the dense network. Unlike conventional training, however, the goal of this D step is not only to learn the values of the weights; we are also learning which connections are important. We use a simple heuristic to quantify the importance of the weights using their absolute value. Sparse Training: The S step prunes the low-weight connections and trains a sparse network. We applied the same sparsity to all the layers, thus there's a single hyper parameter: the sparsity, the percentage of weights that are pruned to 0. For each layer W with N parameters, we sorted the parameters, picked the k-th largest one λ = S k as the threshold where k = N * (1 -sparsity), and generated a binary mask to remove all the weights smaller than λ. Details are shown in Algorithm 1 .

We remove small weights because of the Taylor expansion. The loss function and its Taylor expansion are shown in Equation (1)(2). We want to minimize the increase in Loss when conducting a hard thresholding on the weights, so we need to minimize the first and second terms in Equation 2. Since we are zeroing out parameters, ∆W i is actually W i -0 = W i . At the local minimum where ∂Loss/∂W i ≈ 0 and ∂ 2 Loss ∂W 2 i > 0, only the second order term matters. Since second order gradient ∂ 2 Loss/∂W 2 i is expensive to calculate and W i has a power of 2, we use |W i | as the metric of pruning. Smaller |W i | means a smaller increase to the loss function.

EQUATION

EQUATION

Retraining while enforcing the binary mask in each iteration, we converted a dense network into a sparse network that has a known sparsity support and can fully recover or even increase the original accuracy of initial dense model under the sparsity constraint. The sparsity is the same for all the layers and can be tuned using validation. We find a sparsity value between 25% and 50% generally works well in our experiments.

Final Dense Training::

The final D step recovers the pruned connections, making the network dense again. These previously-pruned connections are initialized to zero and the entire network is retrained with 1/10 the original learning rate (since the sparse network is already at a good local minima).

Hyper parameters like dropout ratios and weight decay remained unchanged. By restoring the pruned connections, the final D step increases the model capacity of the network and makes it possible to arrive at a better local minima compared with the sparse model from the S step.

To visualize the DSD training flow, we plotted the progression of the weight distribution in Figure 2 . The figure is plotted using GoogLeNet's inception_5b3x3 layer, and we found this progression of weight distribution very representative for VGGNet and ResNet as well. The original distribution of weight is centered on zero with tails dropping off quickly. Pruning is based on absolute value so after pruning the large center region is truncated away. The un-pruned network parameters adjust themselves during the retraining phase, so in (c), the boundary becomes soft and forms a bimodal distribution. In (d), at the beginning of the re-dense training step, all the pruned weights come back again and are reinitialized to zero. Finally, in (e), the pruned weights are retrained together with the un-pruned weights. In this step, we kept the same learning hyper-parameters (weight decay, learning rate, etc.) for pruned weights and un-pruned weights. Comparing Figure (d) and (e), the un-pruned weights' distribution almost remained the same, while the pruned weights became distributed further around zero. The overall mean absolute value of the weight distribution is much smaller. This is a good phenomenon: choosing the smallest vector that solves the learning problem suppresses irrelevant components of the weight vector ( Moody et al. (1995) ). Distillation: Model distillation (Hinton et al. (2015) ) is a method that can transfer the learned knowledge from a large model to a small model, which is more efficient for deployment. This is another method that allows for performance improvements in neural networks without architectural changes.

Model Compression: Both model compression (Han et al. (2016; 2015) ) and DSD training use network pruning (LeCun et al. (1990) ; Hassibi et al. (1993) ). The difference is that the focus of DSD training goes beyond maintaining the accuracy. DSD is able to further improve the accuracy by considerable margins. Another difference is that DSD training doesn't require aggressive pruning. A modestly pruned network (50%-60% sparse) can work well. However, model compression requires aggressively pruning the network to achieve high compression rates.

Sparsity Regularization and Hard Thresholding: the truncation-based sparse network has been theoretically analyzed for learning a broad range of statistical models in high dimensions (Langford et al. (2009) ; Yuan & Zhang (2013) ; Wang et al. (2014) ). A similar training strategy with iterative hard thresholding and connection restoration is proposed by Jin et al. (2016) during the same time period as, but independently from, DSD. Sparsity regularized optimization is heavily applied in Compressed Sensing (Candes & Romberg (2007) ) to find optimal solutions to the inverse problems in highly under-determined systems based on the sparsity assumption.

4. EXPERIMENTS:

We applied DSD training to different kinds of neural networks in different domains. We found that DSD training improved the accuracy for all these networks compared to the baseline networks that were not trained with DSD. The neural networks are chosen from CNN, RNN and LSTMs; the datasets covered image classification, speech recognition, and caption generation. For networks trained for ImageNet, we focus on GoogLeNet, VGG and ResNet, which are widely used in research and production. An overview of the networks, dataset and accuracy results are shown in Table 1 . For the convolutional networks, we do not prune the first layer during the sparse phase, since it has only 3 channels and is very sensitive to pruning. The sparsity is the same for all the other layers, including convolutional and fully-connected layers. We do not change any other training hyper-parameters, and the initial learning rate at each stage is decayed the same as conventional training. The epochs are decided by when the loss converges. When the loss no longer decreases, we stop the training.

4.1. GOOGLENET:

We experimented with the BVLC GoogLeNet (Szegedy et al. (2015) ) model obtained from the Caffe Model Zoo (Jia (2013) ). It has 13 million parameters and 57 convolutional layers. We pruned each layer (except the first) to 30% sparsity. Retraining the sparse network gave some improvement in accuracy due to regularization, as shown in Table 2 . After the final dense training step, GoogLeNet's error rates were reduced by 1.12% (Top-1) and 0.62% (Top-5) over the baseline.

We compared DSD v.s. conventional training for the same number of epochs by dropping the learning rate upon "convergence" and continuing to learn. The result is shown as LLR (lower the learning rate). The training epochs for LLR is equal to that of Sparse+re-Dense as a fair comparison. LLR can not achieve the same accuracy as DSD. Sparse: a man is sitting on a bench with his hands in the air.

DSD: a man is sitting on a bench with his arms folded.

Baseline: two dogs are playing together in a field.

Sparse: two dogs are playing in a field.

DSD: two dogs are p l a y i n g i n t h e grass.

Baseline: a boy in a red shirt is climbing a rock wall. Sparse: a young girl is jumping off a tree.

DSD: a young girl in a pink shirt is s w i n g i n g o n a swing.

Baseline::

a basketball player in a red uniform is playing with a ball. Sparse: a basketball player in a blue uniform is jumping over the goal. DSD: a basketball player in a white uniform is trying to make a shot. In the pruning step, we pruned all layers except W s , the word embedding lookup table, to 80% sparse. We used a higher sparsity than CNN's experiments based on the validation set of flickr8k. We retrained the remaining sparse network using the same weight decay and batch size as the original paper. The learning rate is tuned based on the validation set, shown in Table 5 . Retraining the sparse network improved the BLUE score by [1.2, 1.1, 0.9, 0.7]. After getting rid of the sparsity constraint and retraining the dense network, the final results of DSD further improved the BLEU score by [2.0, 2.1, 2.0, 1.7] over baseline.

The BLEU score is not the sole criteria measuring auto-caption system. We visualized the captions generated by DSD training in Figure 3 . In the first image, the baseline model mistakes the girl with a boy and the girl's hair with a rock wall; the sparse model can tell that it's a girl; and the DSD model can further identify the swing. In the the second image, DSD training can more accurately tell the player is in a white uniform and trying to make a shot, rather than the baseline just saying he's in a red uniform and playing with a ball. The performance of DSD training generalizes beyond these examples; more image caption results generated by DSD training are provided in the Appendix.

4.5. DEEPSPEECH:

We explore DSD training on speech recognition tasks using both Deep Speech 1 (DS1) and Deep Speech 2 (DS2) networks (Hannun et al. (2014) ; Amodei et al. (2015) ).

The DS1 model is a 5 layer network with 1 Bidirectional Recurrent layer, as described in Table 6 .

The training dataset used for this model is the Wall Street Journal (WSJ), which contains 81 hours of 2015) due to two factors. First, in DeepSpeech2, the models were trained using much larger data sets containing approximately 12,000 hours of multi-speaker speech data. Secondly, WER was evaluated with beam search and a language model in DeepSpeech2; here the network output is obtained using only max decoding to show improvement in the neural network accuracy, and filtering out the other parts.

The first dense phase was trained for 50 epochs. In the sparse phase, weights are pruned in the Fully Connected layers and the Bidirectional Recurrent layer only (they are the majority of the weights). Each layer is pruned to achieve the same 50% sparsity and trained for 50 epochs. In the final dense phase, the pruned weights are initialized to zero and trained for another 50 epochs. For a fair comparison of baseline, we used Nesterov SGD to train, reduce the learning rate with each re-training, and keep all other hyper parameters unchanged. The learning rate is picked using our validation set.

We first wanted to compare the DSD results with a baseline model trained for the same number of epochs. The first 3 rows of Table 7 shows the WER when the DSD model is trained for 50+50+50=150 epochs, and the 6th line shows the baseline model trained by 150 epochs (the Same #Epochs as DSD). DSD training improves WER by 0.13 (WSJ '92) and 1.35 (WSJ '93) given the same number of epochs as the conventional training.

Given a second DSD iteration, accuracy can be further improved. In the second DSD iteration, each layer is pruned away 25% of the weights. Similar to the first iteration, the sparse model and subsequent dense model are further retrained for 50 epochs. The learning rate is scaled down for each re-training step. The results are shown in Table 7 . Compared with the fully trained and converged baseline, the second DSD iteration improves WER by 0.58 (WSJ '92) and 1.96 (WSJ '93), a relative improvement of 2.07% (WSJ '92) and 5.84% (WSJ '93). So, we can do more DSD iterations (DSDSD) to further improve the performance. Adding more DSD iterations has a diminishing return.

4.6. DEEPSPEECH 2:

To show how DSD works on deeper networks, we evaluated DSD on the Deep Speech 2 (DS2) network, described in Here we show again that DSD can be applied multiple times or iteratively for further performance gain. A second iteration of DSD training achieves better accuracy as shown in Table 9 . For the second sparse iteration, 25% of parameters in the Fully Connected layer and Bidirectional Recurrent layers are pruned. Overall DSD training achieves relative improvement of 5.55% (WSJ '92) and 7.44% (WSJ '93) on the DS2 architecture. These results are in line with DSD experiments on the smaller DS1 network. We can conclude that DSD re-training continues to show improvement in accuracy with larger layers and deeper networks.

5. DISCUSSION:

Dense-Sparse-Dense training changes the optimization process and improves the optimization performance with significant margins by nudging the network with pruning and re-densifying. We conjecture that the following aspects contribute to the efficacy of DSD training.

Escape Saddle Point: Based on previous studies, one of the most profound difficulties of optimizing deep networks is the proliferation of saddle points (Dauphin et al. ( 2014)). Advanced optimization methods have been proposed to overcome saddle points. For a similar purpose but with a different approach, the proposed DSD method overcomes the saddle points by pruning and re-densifying framework. Pruning the converged model perturbs the learning dynamics and allows the network to jump away from saddle points, which gives the network a chance to converge at a better local or global minimum. This idea is also similar to Simulated Annealing ( Hwang (1988) ). While Simulated Annealing randomly jumps with decreasing probability on the search graph, DSD deterministically deviates from the converged solution achieved in the first dense training phase by removing the small weights and enforcing a sparsity support. Similar to Simulated Annealing, which can escape sub-optimal solutions multiple times in the entire optimization process, DSD can also be applied iteratively to achieve further performance gains, as shown in the Deep Speech results.

Significantly Better Minima: After escaping saddle point, DSD achieved better minima. We measured both the training loss and validation loss, DSD training decreased the loss and error on both the training and the validation sets on ImageNet. We have also validated the significance of the improvements compared with conventional fine-tuning by t-test, shown in the appendix.

Regularized and Sparse Training::

The sparsity regularization in the sparse training step moves the optimization to a lower-dimensional space where the loss surface is smoother and tend to be more robust to noise. More numerical experiments verified that both sparse training and the final DSD reduce the variance and lead to lower error (shown in the appendix).

Robust re-initialization: Weight initialization plays a big role in deep learning (Mishkin & Matas (2015) ). Conventional training has only one chance of initialization. DSD gives the optimization a second (or more) chance during the training process to re-initialize from a more robust sparse training solution. We re-densify the network from the sparse solution which can be seen as a zero initialization for pruned weights. Other initialization methods are also worth trying.

Break Symmetry: The permutation symmetry of the hidden units makes the weights symmetrical, thus prone to co-adaptation in training. In DSD, pruning the weights breaks the symmetry of the hidden units associated with the weights, and the weights are asymmetrical in the final dense phase.

6. CONCLUSION:

We introduce DSD, a dense-sparse-dense training framework that regularizes neural networks by pruning and then restoring connections. Our method learns which connections are important during the initial dense solution. Then it regularizes the network by pruning the unimportant connections and retraining to a sparser and more robust solution with same or better accuracy. Finally, the pruned connections are restored and the entire network is retrained again. This increases the dimensionality of parameters, and thus model capacity, from the sparser model. Baseline: a group of people are standing in front of a building. Sparse: a group of people are standing in front of a building. DSD: a group of people are walking in a park.

Baseline: a dog runs through the grass.

Sparse: a dog runs through the grass. DSD: a white and brown dog is running through the grass.

Baseline: a group of football players in red uniforms. Sparse: a group of football players in a field. DSD: a group of football players in red and white uniforms.

Baseline: a group of people sit on a bench in front of a building. Sparse: a group of people are standing in front of a building. DSD: a group of people are standing in a fountain.

Baseline: a man in a black jacket and a black jacket is smiling. Sparse: a man and a woman are standing in front of a mountain. DSD: a man in a black jacket is standing next to a man in a black shirt.

Baseline:a young girl in a red dress is holding a camera. Sparse: a little girl in a pink dress is standing in front of a tree. DSD: a little girl in a red dress is holding a red and white flowers.

Baseline: a man in a red jacket is standing in front of a white building. Sparse: a man in a black jacket is standing in front of a brick wall. DSD: a man in a black jacket is standing in front of a white building.

Baseline: a man in a red shirt is standing on a rock. Sparse: a man in a red jacket is standing on a mountaintop. DSD: a man is standing on a rock overlooking the mountains.

Baseline: a group of people are sitting in a subway station. Sparse: a man and a woman are sitting on a couch. DSD: a group of people are sitting at a table in a room.

Baseline: a soccer player in a red and white uniform is running on the field. Sparse: a soccer player in a red uniform is tackling another player in a white uniform. DSD: a soccer player in a red uniform kicks a soccer ball.

Baseline: a young girl in a swimming pool.

Sparse: a young boy in a swimming pool. DSD: a girl in a pink bathing suit jumps into a pool.

Baseline: a soccer player in a red and white uniform is playing with a soccer ball. Sparse: two boys playing soccer. DSD: two boys playing soccer.

Baseline: a girl in a white dress is standing on a sidewalk. Sparse: a girl in a pink shirt is standing in front of a white building. DSD: a girl in a pink dress is walking on a sidewalk.

Baseline: a boy is swimming in a pool. Sparse: a small black dog is jumping into a pool. DSD: a black and white dog is swimming in a pool.

Baseline: a snowboarder flies through the air.

Sparse: a person is snowboarding down a snowy hill. DSD: a person on a snowboard is jumping over a snowy hill.

Baseline: two young girls are posing for a picture. Sparse: a young girl with a blue shirt is blowing bubbles. DSD: a young boy and a woman smile for the camera.

Baseline: a man in a red shirt is sitting in a subway station. Sparse: a woman in a blue shirt is standing in front of a store. DSD: a man in a black shirt is standing in front of a restaurant.

Baseline: a surfer is riding a wave. Sparse: a man in a black wetsuit is surfing on a wave. DSD: a man in a black wetsuit is surfing a wave.

Baseline: a man in a red shirt is standing on top of a rock. Sparse: a man in a red shirt is standing on a cliff overlooking the mountains. DSD: a man is standing on a rock overlooking the mountains.

Baseline: a group of people sit on a bench. Sparse: a group of people are sitting on a bench. DSD: a group of children are sitting on a bench.

Baseline: a little boy is playing with a toy. Sparse: a little boy in a blue shirt is playing with bubbles. DSD: a baby in a blue shirt is playing with a toy.

Baseline: a brown dog is running through the grassy.

Sparse: a brown dog is playing with a ball. DSD: a brown dog is playing with a ball.

Baseline: a boy in a red shirt is jumping on a trampoline. Sparse: a boy in a red shirt is jumping in the air. DSD: a boy in a red shirt is jumping off a swing.

Baseline: a man is standing on the edge of a cliff. Sparse: a man is standing on the shore of a lake. DSD: a man is standing on the shore of the ocean.

Baseline: two people are riding a boat on the beach. Sparse: two people are riding a wave on a beach. DSD: a man in a yellow kayak is riding a wave.

Baseline: a black and white dog is running on the beach. Sparse: a black and white dog running on the beach. DSD: a black dog is running on the beach.

Baseline: a man and a dog are playing with a ball. Sparse: a man and a woman are playing tug of war. DSD: a man and a woman are playing with a dog.

Baseline: a group of people are standing in a room. Sparse: a group of people gather together. DSD: a group of people are posing for a picture.

Baseline: a man in a red jacket is riding a bike through the woods.

Sparse: a man in a red jacket is doing a jump on a snowboard. DSD: a person on a dirt bike jumps over a hill.

Baseline: a man in a red jacket and a helmet is standing in the snow. Sparse: a man in a red jacket and a helmet is standing in the snow. DSD: a man in a red jacket is standing in front of a snowy mountain.

2

:



Figures and Tables:

Figure 1: Dense-Sparse-Dense Training Flow. The sparse training regularizes the model, and the final dense training restores the pruned weights (red), increasing the model capacity without overfitting.

Figure 2: Weight distribution of a layer of GoogLeNet at different points in DSD training: the original GoogLeNet (a), pruned (b), after retraining with the sparsity constraint (c), ignoring the sparisty constraint and recovering the zero weights (d), and after retraining the dense network (e).

Figure 3: Visualization of DSD training improving the performance of image captioning.

DSD training achieves superior optimization performance. We highlight our experiments using GoogLeNet, VGGNet, and ResNet on ImageNet; NeuralTalk on Flickr-8K; and DeepSpeech-1&2 on the WSJ dataset. This shows that the accuracy of CNNs, RNNs, and LSTMs can be significantly improved with DSD training. Our numerical results and empirical tests show the inadequacy of current training methods for which we have provided an effective solution.B. APPENDIX: MORE EXAMPLES OF DSD TRAINING IMPROVES THE CAPTIONS GENERATED BY NEURALTALK (IMAGES FROM FLICKR-8K TEST SET)Baseline: a man in a red shirt and jeans is riding a bicycle down a street. Sparse: a man in a red shirt and a woman in a wheelchair. DSD: a man and a woman are riding on a street.Baseline: two girls in bathing suits are playing in the water. Sparse: two children are playing in the sand. DSD: two children are playing in the sand.

Overview of the neural networks, data sets and performance improvements from DSD.

Table content: <table><tr><td colspan="2">Neural Network Domain</td><td>Dataset</td><td colspan="2">Type Baseline</td><td>DSD</td><td colspan="2">Abs. Imp. Rel. Imp.</td></tr><tr><td>GoogLeNet</td><td>Vision</td><td colspan="2">ImageNet CNN</td><td colspan="2">31.1% 1 30.0%</td><td>1.1%</td><td>3.6%</td></tr><tr><td>VGG-16</td><td>Vision</td><td colspan="2">ImageNet CNN</td><td colspan="2">31.5% 1 27.2%</td><td>4.3%</td><td>13.7%</td></tr><tr><td>ResNet-18</td><td>Vision</td><td colspan="2">ImageNet CNN</td><td colspan="2">30.4% 1 29.2%</td><td>1.2%</td><td>4.1%</td></tr><tr><td>ResNet-50</td><td>Vision</td><td colspan="2">ImageNet CNN</td><td colspan="2">24.0% 1 22.9%</td><td>1.1%</td><td>4.6%</td></tr><tr><td>NeuralTalk</td><td colspan="3">Caption Flickr-8K LSTM</td><td>16.8 2</td><td>18.5</td><td>1.7</td><td>10.1%</td></tr><tr><td>DeepSpeech</td><td>Speech</td><td>WSJ'93</td><td>RNN</td><td colspan="2">33.6% 3 31.6%</td><td>2.0%</td><td>5.8%</td></tr><tr><td>DeepSpeech-2</td><td>Speech</td><td>WSJ'93</td><td>RNN</td><td colspan="2">14.5% 3 13.4%</td><td>1.1%</td><td>7.4%</td></tr><tr><td colspan="8">1 Top-1 error. VGG/GoogLeNet baselines from the Caffe Model Zoo, ResNet from Facebook.</td></tr><tr><td colspan="6">2 BLEU score baseline from Neural Talk model zoo, the higher the better.</td><td/></tr><tr><td colspan="8">3 Word error rate: DeepSpeech2 is trained with a portion of Baidu internal dataset with only max</td></tr><tr><td colspan="4">decoding to show the effect of DNN improvement.</td><td/><td/><td/></tr><tr><td colspan="2">3 RELATED WORK</td><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td/><td colspan="3">: DSD results on GoogLeNet</td><td/><td/></tr><tr><td>GoogLeNet</td><td colspan="5">Top-1 Err Top-5 Err Sparsity Epochs LR</td></tr><tr><td>Baseline</td><td>31.14%</td><td>10.96%</td><td>0%</td><td>250</td><td>1e-2</td></tr><tr><td>Sparse</td><td>30.58%</td><td>10.58%</td><td>30%</td><td>11</td><td>1e-3</td></tr><tr><td>DSD</td><td>30.02%</td><td>10.34%</td><td>0%</td><td>22</td><td>1e-4</td></tr><tr><td>LLR</td><td>30.20%</td><td>10.41%</td><td>0%</td><td>33</td><td>1e-5</td></tr><tr><td>Improve (abs)</td><td>1.12%</td><td>0.62%</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Improve (rel)</td><td>3.6%</td><td>5.7%</td><td>-</td><td>-</td><td>-</td></tr><tr><td>4.2 VGGNET</td><td/><td/><td/><td/><td/></tr></table>

He et al. (2015)) were the top performer in the 2015 ImageNet challenge. The baseline ResNet-18 and ResNet-50 models are provided byFacebook (2016). We prune to 30% sparsity uniformly, and a single DSD pass for these networks reduced top-1 error by 1.26% (ResNet-18) and 1.12% (ResNet-50), shown in Table4. A second DSD iteration can further improve the accuracy. As a fair comparison, we continue train the original model by lowering the learning rate by another decade, but can't reach the same accuracy as DSD, as shown in the LLR row.

Table content: <table><tr><td/><td/><td colspan="3">: DSD results on VGG-16</td><td/><td/><td/></tr><tr><td/><td>VGG-16</td><td colspan="5">Top-1 Err Top-5 Err Sparsity Epochs LR</td><td/></tr><tr><td/><td>Baseline</td><td>31.50%</td><td>11.32%</td><td>0%</td><td>74</td><td>1e-2</td><td/></tr><tr><td/><td>Sparse</td><td>28.19%</td><td>9.23%</td><td>30%</td><td>1.25</td><td>1e-4</td><td/></tr><tr><td/><td>DSD</td><td>27.19%</td><td>8.67%</td><td>0%</td><td>18</td><td>1e-5</td><td/></tr><tr><td/><td>LLR</td><td>29.33%</td><td>10.00%</td><td>0%</td><td>20</td><td>1e-7</td><td/></tr><tr><td colspan="2">Improve (abs)</td><td>4.31%</td><td>2.65%</td><td>-</td><td>-</td><td>-</td><td/></tr><tr><td colspan="2">Improve (rel)</td><td>13.7%</td><td>23.4%</td><td>-</td><td>-</td><td>-</td><td/></tr><tr><td>4.3 RESNET</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="6">Deep Residual Networks (ResNets, Table 4: DSD results on ResNet-18 and ResNet-50</td><td/><td/></tr><tr><td/><td colspan="2">ResNet-18</td><td colspan="2">ResNet-50</td><td/><td/><td/></tr><tr><td/><td colspan="7">Top-1 Err Top-5 Err Top-1 Err Top-5 Err Sparsity Epochs LR</td></tr><tr><td>Baseline</td><td>30.43%</td><td>10.76%</td><td>24.01%</td><td>7.02%</td><td>0%</td><td>90</td><td>1e-1</td></tr><tr><td>Sparse</td><td>30.15%</td><td>10.56%</td><td>23.55%</td><td>6.88%</td><td>30%</td><td>45</td><td>1e-2</td></tr><tr><td>DSD</td><td>29.17%</td><td>10.13%</td><td>22.89%</td><td>6.47%</td><td>0%</td><td>45</td><td>1e-3</td></tr><tr><td>LLR</td><td>30.04%</td><td>10.49%</td><td>23.58%</td><td>6.84%</td><td>0%</td><td>90</td><td>1e-5</td></tr><tr><td>Improve (abs)</td><td>1.26%</td><td>0.63%</td><td>1.12%</td><td>0.55%</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Improve (rel)</td><td>4.14%</td><td>5.86%</td><td>4.66%</td><td>7.83%</td><td>-</td><td>-</td><td>-</td></tr></table>

DSD results on NeuralTalk

Table content: <table><tr><td>NeuralTalk</td><td colspan="7">BLEU-1 BLEU-2 BLEU-3 BLEU-4 Sparsity Epochs LR</td></tr><tr><td>Baseline</td><td>57.2</td><td>38.6</td><td>25.4</td><td>16.8</td><td>0</td><td>19</td><td>1e-2</td></tr><tr><td>Sparse</td><td>58.4</td><td>39.7</td><td>26.3</td><td>17.5</td><td>80%</td><td>10</td><td>1e-3</td></tr><tr><td>DSD</td><td>59.2</td><td>40.7</td><td>27.4</td><td>18.5</td><td>0</td><td>6</td><td>1e-4</td></tr><tr><td>Improve(abs)</td><td>2.0</td><td>2.1</td><td>2.0</td><td>1.7</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Improve(rel)</td><td>3.5%</td><td>5.4%</td><td>7.9%</td><td>10.1%</td><td>-</td><td>-</td><td>-</td></tr><tr><td>4.4 NEURALTALK</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Deep Speech 1 Architecture The validation set consists of 1 hour of speech. The test sets are from WSJ'92 and WSJ'93 and contain 1 hour of speech combined. The Word Error Rate (WER) reported on the test sets for the baseline models is different from Amodei et al. (

Table content: <table><tr><td>Layer ID</td><td>0</td><td>1</td><td>2</td><td/><td>3</td><td/><td>4</td><td>5</td></tr><tr><td>Type</td><td>Conv</td><td>FC</td><td>FC</td><td colspan="3">Bidirectional Recurrent</td><td>FC</td><td>CTCCost</td></tr><tr><td colspan="5">#Params 1814528 1049600 1049600</td><td>3146752</td><td/><td>1049600</td><td>29725</td></tr><tr><td/><td colspan="7">Table 7: DSD results on Deep Speech 1: Word Error Rate (WER)</td></tr><tr><td/><td colspan="7">DeepSpeech 1 WSJ '92 WSJ '93 Sparsity Epochs LR</td></tr><tr><td/><td colspan="2">Dense Iter 0</td><td>29.82</td><td>34.57</td><td>0%</td><td>50</td><td>8e-4</td></tr><tr><td/><td colspan="2">Sparse Iter 1</td><td>27.90</td><td>32.99</td><td>50%</td><td>50</td><td>5e-4</td></tr><tr><td/><td colspan="2">Dense Iter 1</td><td>27.90</td><td>32.20</td><td>0%</td><td>50</td><td>3e-4</td></tr><tr><td/><td colspan="2">Sparse Iter 2</td><td>27.45</td><td>32.99</td><td>25%</td><td>50</td><td>1e-4</td></tr><tr><td/><td colspan="2">Dense Iter 2</td><td>27.45</td><td>31.59</td><td>0%</td><td>50</td><td>3e-5</td></tr><tr><td/><td>Baseline</td><td/><td>28.03</td><td>33.55</td><td>0%</td><td>150</td><td>8e-4</td></tr><tr><td/><td colspan="2">Improve(abs)</td><td>0.58</td><td>1.96</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td colspan="2">Improve(rel)</td><td>2.07%</td><td>5.84%</td><td>-</td><td>-</td><td>-</td></tr><tr><td>speech.</td><td/><td/><td/><td/><td/><td/></tr></table>

This network has 7 Bidirectional Recurrent layers with approximately 67 million parameters, around 8 times larger than the DS1 model.

Table content: <table/>



Table content: <table/>

Deep Speech 2 Architecture Layers are pruned. The Baseline model is trained for 60 epochs to provide a fair comparison with DSD training. The baseline model shows no improvement after 40 epochs. With one iteration of DSD training, WER improves by 0.44 (WSJ '92) and 0.56 (WSJ '93) compared to the fully trained baseline.

Table content: <table><tr><td>Layer ID</td><td>0</td><td>1</td><td>2</td><td>3 -8</td><td>9</td><td>10</td></tr><tr><td>Type</td><td colspan="2">2DConv 2DConv</td><td>BR</td><td>BR</td><td>FC</td><td>CTCCost</td></tr><tr><td>#Params</td><td>19616</td><td colspan="4">239168 8507840 9296320 3101120</td><td>95054</td></tr><tr><td/><td colspan="5">Table 9: DSD results on Deep Speech 2 (WER)</td><td/></tr><tr><td colspan="7">DeepSpeech 2 WSJ '92 WSJ '93 Sparsity Epochs LR</td></tr><tr><td colspan="2">Dense Iter 0</td><td>11.83</td><td>17.42</td><td>0%</td><td>20</td><td>3e-4</td></tr><tr><td colspan="2">Sparse Iter 1</td><td>10.65</td><td>14.84</td><td>50%</td><td>20</td><td>3e-4</td></tr><tr><td colspan="2">Dense Iter 1</td><td>9.11</td><td>13.96</td><td>0%</td><td>20</td><td>3e-5</td></tr><tr><td colspan="2">Sparse Iter 2</td><td>8.94</td><td>14.02</td><td>25%</td><td>20</td><td>3e-5</td></tr><tr><td colspan="2">Dense Iter 2</td><td>9.02</td><td>13.44</td><td>0%</td><td>20</td><td>6e-6</td></tr><tr><td/><td>Baseline</td><td>9.55</td><td>14.52</td><td>0%</td><td>60</td><td>3e-4</td></tr><tr><td colspan="2">Improve(abs)</td><td>0.53</td><td>1.08</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="2">Improve(rel)</td><td>5.55%</td><td>7.44%</td><td>-</td><td>-</td><td>-</td></tr></table>

Peer review (iclr2017):
Training highly non-convex deep neural networks is a very important practical problem, and this paper provides a great exploration of an interesting new idea for more effective training. The empirical evaluation both in the paper itself and in the authors’ comments during discussion convincingly demonstrates that the method achieves consistent improvements in accuracy across multiple architectures, tasks and datasets. The algorithm is very simple (alternating between training the full dense network and a sparse version of it), which is actually a positive since that means it may get adapted in practice by the research community. The paper should be revised to incorporate the additional experiments and comments from the discussion, particularly the accuracy comparisons with the same number of epochs.