Title: The Power of Sparsity in Convolutional Neural Networks
Authors: Soravit Changpinyo
Mark Sandler
Andrey Zhmoginov
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning

Abstract: Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training time, and prediction accuracy? A surprisingly effective approach to trade accuracy for size and speed is to simply reduce the number of channels in each convolutional layer by a fixed fraction and retrain the network. In many cases this leads to significantly smaller networks with only minimal changes to accuracy. In this paper, we take a step further by empirically examining a strategy for deactivating connections between filters in convolutional layers in a way that allows us to harvest savings both in run-time and memory for many network architectures. More specifically, we generalize 2D convolution to use a channel-wise sparse connection structure and show that this leads to significantly better results than the baseline approach for large networks including VGG and Inception V3.

TL;DR: Sparse random connections that allow savings to be harvested and that are very effective at compressing CNNs.

Review discussion:
In the conclusion section, the paper says: "For example, this simple method achieves AlexNet-level accuracy with fewer than 400K parameters and VGG level with roughly 1M parameters." I couldn't see these results in any of the graphs. By looking at the graphs, my understanding is there was no experiment conducted with AlexNet and the number of parameters needed to achieve VGG level accuracy is the same as the dense connected convolutional networks. Can you please clarify this statement from the conclusion?
Right, we did not apply our approach to AlexNet, and we simply take VGG accuracy from Fig. 5. We have modified the paper to make it more clear. Sorry for the confusion. We simply want to say that we can achieve high accuracy with a small number of parameters.
Computational cost is reported in terms of multiply-adds.  On what real architectures (current or possible future) would this result in wall-time speedups?  On many arch implementations, elements stored in contiguous memory regions are faster to load (amortized per element) compared to spread-out random reads.  Is it possible to measure wall-time in addition to multiply-adds?
Thank you for your comment. You are absolutely right that any architecture from last 10 years makes it prohibitively expensive to do random memory access.
However we don't use sparse matrices, we only introduce sparsity from channel to channel.
Thus, we can use all the benefits of caching becuase the basic operation (convolve image plane with k \times k filter) is unchanged.
In other words, if naive implementation of dense convolution looks like this:
for i in input_channel
for o in output_channel
# this oepration is the expensive one since it has to go over entire image
result[o] += convolve(input[i]  w[i, o, ...])
similar sparse convolution will look like this:
for i in input_channel
for index, o, in enumerate(output_channels_connected_to_i)
# this oepration is expensive  since it has to go over entire image
result[o] += convolve(input[i]  w[i, index, ...])
in this implementation setup, the speed up will be mostly in terms of madds, since the basic operation (convoving *entire* image plane) is unchanged. We will include this in the updated version of the paper
To answer your question we do not have optimized version of sparse convolutions yet, and Cudnn implementation of dense convolutions is very fast (yet not publicly available) (e.g. X00% faster than reference GPU implementation)  so we can not report any speed up compared to that. Our naive implementation gives comparable speed up to naive dense gpu implementation.
We believe this what makes this result interesting.
In particular it might guide design for hardware manufacturing and/or algorithmic research to optimize these use cases.
The paper experiments with channel to channel sparse neural networks.
The paper is well written and the analysis is useful. The sparse connection is not new but has not been experimented on large-scale problems like ImageNet. One of the reasons for that is the unavailability of fast implementations of randomly connected convolutional layers.
The results displayed in figures 2, 3, 4, and, 5 show that sparse connections need the same number of parameters as the dense networks to reach to the best performance on the given tasks, but can provide better performance when there is a limited budget for the #parameters and #multiplyAdds.
This paper is definitely informative but it does not reach to the conference acceptance level, simply because the idea is not new, the sparse connection implementation is poor, and the results are not very surprising.
Thank you for the detailed review.
Regarding the sparse implementation issue, we wholeheartedly agree that gains on modern GPUs still need to be realized.
However, given the amount of engineering invested in implementation of CUDA convolution operators, matching it will require a similar effort.
Nevertheless, we believe that results like ours help guide the future hardware and software optimization for neural networks. We also note that the story is different in embedded applications where CUDA does not exist yet, but this is beyond the scope of the paper.
In the related work section, we provide a comprehensive overview regarding where our novelty lies in comparison to relevant ideas.
Besides being the first to provide evidence and analysis on the usefulness of sparsity in large convolutional networks, our approach is very simple but at the same time preserving “regularity” in structure and therefore efficiency. We believe this is an important knowledge. We also introduce an incremental training technique to help speeding up the training further.
This paper aims to improve efficiency of convolutional networks by using a sparse connection structure in the convolution filters at each layer.  Experiments are performed using MNIST, CIFAR-10 and ImageNet, comparing the sparse connection kernels against dense convolution kernels with about the same number of connections, showing the sparse structure (with more feature maps) generally performs better for similar numbers of parameters.
Unfortunately, any theoretical efficiencies are not realized, since the implementation enforces the sparse structure using a zeroing mask on the weights.
In addition, although the paper mentions that this method can be implemented efficiently and take advantage of contiguous memory reads/writes of current architectures, I still find it unclear whether this would be the case:  The number of activation units is no smaller than when using a dense convolution of same dimension, and these activations (inputs and outputs) must be loaded/stored.  The fact that the convolution is sparse saves only on the multiply/addition operation cost, not memory access for the activations, which can often be the larger amount of time spent.
The section on incremental training is interesting, but feels short and preliminary, and any gains here also have yet to be realized.  The precision is no better than for the original network, and as mentioned above, the implementation of the sparse structure is no faster than the original.
Overall, the method and evaluations show that the basic approach has promise.
However, it is unclear how real gains (in either speed or accuracy) might actually be found with it.  Without this last step, it still seems incomplete to me for a conference paper.
Thank you for your thoughtful and detailed review.
We want to make two points regarding the memory cost.
First, for a convolutional layer, there are two memory components: the memory needed to store activations (e.g. num_channels * width * height) vs. the memory needed to store convolutional weights (e.g. num_input_channels * num_output_channels * kernel_size). It is important to stress that, for many layers (especially for later “thicker” layers), the cost is often dominated by the latter component rather than the former.
Thus, storing sparse matrices reduces memory requirement during inference time.
Furthermore, it can also significantly reduce memory needed to store trained models.
Second, while memory write access can be expensive, in matrix multiplication, vast majority of the time is spent accumulating the result rather than saving it to memory. That’s why in general matrix multiplication is much slower than addition. In this case, each output value requires kernel_size * num_input_channels computations plus memory/input access for all the weights, and sparse convolutions require a comparable number of memory access to their corresponding reduced-depth dense convolutions (which could be much smaller than what the full dense convolutional operator would require).
Regarding the gains, we wholeheartedly agree that gains on modern GPUs still need to be realized.
However, given the amount of engineering effort invested in implementation of highly efficient CUDA convolution operators, matching it will require a similar effort.
Nevertheless, we believe that results like ours help guide the future hardware and software decisions about neural networks. We also note that the story is different in embedded applications where CUDA does not exist yet, but this is beyond the scope of the paper.
While incremental training gives no better accuracy than training the full network from the beginning, it is very likely that incremental training will be more efficient. Not only is the network during early stages of incremental training is sparser, but also it has far fewer parameters than the full model.
The paper is about channel sparsity in Convolution layer.
The paper is well written and it elaborately discussed and investigated different approaches for applying sparsity.  The paper contains detailed literature review.
In result section, it showed the approach gives good results using 60% sparsity with reducing number of parameters, which can be useful in some embedded application with limited resource i.e. mobile devices.
The main point is that the paper needs more detailed investigation on different dropout schedule.
As mentioned implementation details section, they deactivate the connections by applying masks to parameter tensors, which is not helpful in speeding up the training and computation in convolution layer. They can optimize implementation to reduce computation time.
Thanks for your thoughtful review. We agree that coming up with a different/better dropout schedule is an interesting and important research direction. We will add this to the future work section.

Meta-review:
role: PC
data: comment: The reviewers agreed that the main contribution is the first empirical analysis on large-scale convolutional networks concerning layer-to-layer sparsity. The main concerns were that of novelty (connection-wise sparsity being explored previously but not in large-scale domains) and the importance given the current state of fast implementations of sparsely connected CNNs. The authors argued that the point of their paper was to drive software/hardware co-evolution and guide the next generation of, e.g. CUDA tools development. The confident scores were borderline while the less confident reviewer was pleased with the paper so I engaged the reviewers in a discussion post author-feedback. The consensus was that the paper presented promising but not fully developed nor convincing research.

Decision: Reject