Title: Faster CNNs with Direct Sparse Convolutions and Guided Pruning
Authors: Jongsoo Park
Sheng Li
Wei Wen
Ping Tak Peter Tang
Hai Li
Yiran Chen
Pradeep Dubey
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: Phenomenally successful in practical inference problems, convolutional neural networks (CNN) are widely deployed in mobile devices, data centers, and even supercomputers.
The number of parameters needed in CNNs, however, are often large and undesirable. Consequently, various methods have been developed to prune a CNN once it is trained. 
Nevertheless, the resulting CNNs offer limited benefits. While pruning the fully connected layers reduces a CNN's size considerably, it does not improve inference speed noticeably as the compute heavy parts lie in convolutions. Pruning CNNs in a way that increase inference speed often imposes specific sparsity structures, thus limiting the achievable sparsity levels.

We present a method to realize simultaneously size economy and speed improvement while pruning CNNs. Paramount to our success is an efficient general sparse-with-dense matrix
multiplication implementation that is applicable to convolution of feature maps with kernels of arbitrary sparsity patterns. Complementing this, we developed a performance model that predicts sweet spots of sparsity levels for different layers and on different computer architectures. Together, these two allow us to demonstrate 3.1-7.3x convolution speedups over dense convolution in AlexNet, on Intel Atom, Xeon, and Xeon Phi processors, spanning the spectrum from mobile devices to supercomputers.


TL;DR: Highly-performance sparse convolution outperforms dense with only 70% sparsity. Performance model that guides training to find useful sparsity range, applied to AlexNet and GoogLeNet

Review discussion:
This paper tackles the problem of compressing trained convnets with the goal of reducing memory overhead and speeding up the forward pass. As I understand it, the main contribution of this work is to develop fast convolution routines for sparse conv weights int he case of general sparsity (as compared with structured sparsity). They evaluate their method on both AlexNet and GoogLeNet as well as on various platforms. The authors make code available online. The paper is well written and does a good job of putting this work in the context of past model reduction techniques.
My main request of the authors would be to provide a concise summary of the speedup/memory gains achievable with this new work compared with previously published work. The authors do show the various sparsity level obtained with various methods of pruning but it is unclear to me how to translate the information given in the paper into an understanding of gains relative to other methods.
Please note that, in addition to our fast convolution algorithms, another important contribution is the performance model guiding the pruning process that allows pursuing desired speedup and model size reduction without falling into combinatorial number of choices offered by multiple layers. Our performance model can also guide other methods discussed in related work as shown by our application to dynamic network surgery (GDNS in Figure 4(a)).
Thanks for the suggestion on summarizing inference speedups and model size reductions of related work. A quick summary is shown below, which we will also consider including in our paper. It is important to note that our work achieves highest speedup without accuracy loss among all the techniques below. The speedups shown that are not our own measurements should be taken with a grain of salt because 1) many papers only provide relative speedups to a baseline whose efficiency is suboptimal (e. g. in some cases, the baseline is Caffe running on CPU, which is known to be suboptimal as it is tuned for GPU), and 2) what some papers report as "speedup" is actually FLOP reduction, not actual timing measurements.
As we did in our paper, for more scientific comparison among different CNN speedup techniques, we recommend using dense matrix multiplication (GEMM) FLOP/s of the evaluated platform as the baseline, because many platforms readily have vendor-provided extensively-optimized GEMM implementations which can be a proxy of highly-optimized dense CNN implementation. This also aligns with a long-accepted common practice in the high-performance computing (HPC) community. We omit Denton et al. 2014, Jaderberg et al. 2014, and Lebedev et al. 2015 in the summary because they report improvements in a subset of conv and fc layers.
AlexNet
GESL (ours, 0% top-1 accuracy drop):                8.5x smaller model,                      2.5x speedup,                                      4.2x FLOP reduction
DNS (0.5% top-1 accuracy drop):                    17.7x smaller model,                      1.0x speedup (not enough sparsity in conv layers), 2.8x FLOP reduction
SSL (0% top-1 accuracy drop):                       1.01x smaller model (no sparsity in fc), 1.5x speedup,                                      1.3x FLOP reduction
Lebedev and Lempitsky (1.3% top-1 accuracy drop):   2.9x smaller model,                      3.0x speedup? (not sure if this is a real speedup or FLOP reduction)
Liu et al. (1% top-1 accuracy drop):                1.04x smaller model (no sparsity in fc), 4.4x speedup (not sure if lowering overhead included)
Kim et al. (1.7% top-5 accuracy drop):              5.5x smaller model,                      1.8x speedup,                                      2.7x FLOP reduction
Tai et al. (0.4% top-5 accuracy drop):              5.0x smaller model,                      1.8x speedup,                                      5.3x FLOP reduction
GoogLeNet
GESL (0.2% top-1 accuracy drop):                    3.3x smaller model,                      2.0x speedups in conv and fc layers,               3.0x FLOP reduction
DNS (our own evaluation, 2.5% top-1 accuracy drop): 1.5x smaller model,                      2.0x speedups in conv and fc layers,               2.6x FLOP reduction
SSL (our own evaluation, 2% top-1 accuracy drop):   2.1x smaller model,                      speedup N/A yet,                                   2.3x FLOP reduction
Kim et al. (0.2% top-5 accuracy drop):              1.3x smaller model,                      1.2x speedup,                                      1.3x FLOP reduction
Ioannou et al. (0.4% top-1 accuracy drop):          1.7x smaller model,                      speedup N/A,                                       1.4x FLOP reduction
Tai et al. (0.4% top-5 accuracy drop):              2.8x smaller model,                      1.2x speedup,                                      2.9x FLOP reduction
The authors provide a well engineered solution to exploiting sparsity in convolutional layers of a deep network by recasting it as sparse matrix-vector multiplication. This leads to very nice speedups and the analysis of when this is possible is also useful for practitioners. My main concern with this paper is that the "research" aspect of it seems rather minimal, and it's mostly about performance engineering and comparisons. It is upto the area chairs to decide how well such a paper fits in at ICLR.
We respectfully ask the reviewer to reconsider the assessment that there is a lack of research contribution in the present paper. It is well accepted that deep learning is enabled by three similarly crucial components – algorithm, data, and performance. Each component benefited from a succession of original research efforts which are still active at present. These include the invention of back propagation (past) to new optimization algorithms for training (present), the creation of the ImageNet dataset (past) to the Re’s ‘’data programming” approach (present), the lowering method that transforms convolution to matrix products (past) to computer architecture for special-purpose hardware accelerator (present).
The present paper belongs to the performance research area of sparsification.
While sparsification has been effective in memory footprint reduction, it has hitherto limited success in inference throughput enhancement. The greatly enhanced throughput reported here cannot be achieved solely by our direct sparse convolution technique (which is a new fast algorithm by itself). Preserving inference accuracy is an implicit requirement for all sparsification endeavor. Sparsification for performance enhancement therefore cannot be successful without understanding where to “use the sparsification budget” for most effective performance gain. And a simplistic “engineering” trial-and-error approach is infeasible for the combinatorial number of choices offered by tens of layers and hundreds of channels. We successfully identified sparsification targets by combining a high-resolution performance model and a sparsity allocation methodology, both of which contain original research that are applicable to performance research in general.
The paper details an implementation of sparse-full convolutions and a model to work out the potential speed-up of various sparsity levels for CNNs.
The first contribution is more about engineering, but the authors make the source code available which is greatly appreciated.
The second contribution is perhaps more interesting, as so far pruning methods have focused on saving memory, with very modest speed gains. Imbuing knowledge of running speed into a pruning algorithm seems like the proper way to tackle this problem. The authors are very methodical in how they build the model and evaluate it very thoroughly.
It seems that the same idea could be used not just for pruning existing models, but also when building new architectures: selecting layers and their parameters as to achieve an optimal throughput rate. This could make for a nice direction for future work.
One point that is missing is some discussion of how transferable the performance model is to GPUs. This would make the technique easier to adopt broadly.
Other areas for improvement: The points in Figure 4 are hard to distinguish (e.g. small red circle vs. small red square), and overall the figure could be made bigger; specifying whether the "base learning rate" in Section 3 is the start or end rate of the annealing schedule; typos: "punning" (p.4), "spares" (p.5).
We thank the reviewer’s appreciation of our sparse convolution algorithms as well as guided pruning techniques and the comment that the performance-model-guided techniques can be used in future work.
Indeed, there could be very interesting directions for future work on building new neural network architectures with our techniques. As shown in GoogLeNet designs, building new network architectures with reduced computing demand is important.
However, as shown in our paper, the actual performance improvement is not a simple linear function of reduced FLOP count;
instead, the convolution kernel size, input and output channel dimensions, and characteristics of a target platform among others, determine the actual speed of a neural network  on the target platform. Our performance-model-guided approach can accurately project actual speedup of a neural network on specific target platforms (CPU/GPU/FPGA/ASIC).
Consequently, design decisions on new network architecture can be custom made for a specific hardware platform in question. We will add an elaboration on this to our current paper.
Our model is very transferable to other platforms including GPUs.
For example, with a typical 90% sparsity, both Pascal Titian-X and P100 GPUs are projected to achieve similar ~3x speedups over their dense baselines.
Moreover, our model also reveals that sparse convolution becomes memory bandwidth bound and thus provide diminishing speedups earlier on Titan-X than on P100. This is because Titan-X has a much higher flop to byte ratio than P100 equipped with a new high bandwidth memory technology, HBM2. These insights shed light on designing sparse convolution on GPUs.

Meta-review:
role: PC
data: comment: While the core ideas explored in this paper are quite limited in algorithmic novelty (e.g., the direct sparse convolutions), the reviewers largely feel that the paper is well written, experiments are carefully done on multiple architectures and system issues are discussed in-depth. Given the interest in the ICLR community around performance characterization and acceleration of CNNs in particular, this paper offers an interesting perspective.

Decision: Accept (Poster)