Title: Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
Authors: Arash Ardakani
Carlo Condo
Warren J. Gross
Venue: ICLR 2017
Keywords: Deep learning
Applications
Optimization

Abstract: Recently deep neural networks have received considerable attention due to their ability to extract and represent high-level abstractions in data sets. Deep neural networks such as fully-connected and convolutional neural networks have shown excellent performance on a wide range of recognition and classification tasks. However, their hardware implementations currently suffer from large silicon area and high power consumption due to the their high degree of complexity. The power/energy consumption of neural networks is dominated by memory accesses, the majority of which occur in fully-connected networks. In fact, they contain most of the deep neural network parameters. In this paper, we propose sparsely-connected networks, by showing that the number of connections in fully-connected networks can be reduced by up to 90% while improving the accuracy performance on three popular datasets (MNIST, CIFAR10 and SVHN). We then propose an efficient hardware architecture based on linear-feedback shift registers to reduce the memory requirements of the proposed sparsely-connected networks. The proposed architecture can save up to 90% of memory compared to the conventional implementations of fully-connected neural networks. Moreover, implementation results show up to 84% reduction in the energy consumption of a single neuron of the proposed sparsely-connected networks compared to a single neuron of fully-connected neural networks.

TL;DR: We show that the number of connections in fully-connected networks can be reduced by up to 90% while improving the accuracy performance.

Review discussion:
Does the Asic implementation of the sparsely-connected network also provide a speed-up over fully connected layers? Can you add the speed numbers to Table 6?
Do you have any plans to experiment with Imagenet dataset?
We thank the reviewer for the comments.
The number of inputs of each neuron determines the latency (or speed) of the architecture.
Regardless of the output value of SNG, inputs are fed into each neuron sequentially in each clock cycle in the proposed architecture.
Therefore, the latency of the proposed sparsely-connected architecture is the same as the conventional fully-connected architecture. The detailed discussion is provided in Section 5, paragraph 4. The latency value has also been added to Table 6 based on your comment.
We have not considered ImageNet dataset for this conference paper due to the time constraints we have. Currently, we are extending the proposed architecture to an accelerator for sparsely-connected networks which will be tested using ImageNet dataset.
The results looks good but the baselines proposed are quite bad.
For instance in the table 2 "Misclassification rate for a 784-1024-1024-1024-10 " the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see "significant" compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers...
In CIFAR-10 experiments, i do not understand  why "Sparsely-Connected 90% + Single-Precision Floating-Point" is worse than "Sparsely-Connected 90% + BinaryConnect".
So it is better to use binary than float.
Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations).
Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision.
In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines.
We sincerely thank the reviewer for the comments.
Reviewer comment: For instance in the table 2 "Misclassification rate for a 784-1024-1024-1024-10 " the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see "significant" compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers...
Response: We fully agree with the reviewer that misclassification rate (MCR) of 0.8% can be obtained for a purely fully-connected network with the configuration of 784-1024-1024-1024-10 if data augmentation methods, unsupervised pre-training or other fine-tuning techniques are used.
Despite the good MCR that the floating point networks with data augmentation methods, unsupervised pre-training or other fine-tuning techniques provide, they require a high number of bits, huge amount of memories and costly multipliers for hardware implementation. In order to address the aforementioned issues, different pruning and binarization/ternarization techniques were introduced in literature to make the hardware implementation of neural networks feasible. In this paper, we also mainly focused on reducing the connections of binarized/ternarized networks while improving their MCR.
Of course, the proposed method is not restricted to only binarized/ternarized networks and can be used also in floating point networks.
Reviewer comment: In CIFAR-10 experiments, i do not understand why "Sparsely-Connected 90% + Single-Precision Floating-Point" is worse than "Sparsely-Connected 90% + BinaryConnect".
So it is better to use binary than float.
Response: In the original paper on binary precision [*], it was shown that the binary precision can yield a better performance than the floating point precision. This is because the binarization technique works as a regularization method to prevent over-fitting. In our work, we observed the sparsification of the network connections can work as a regularization method too.
[*] Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David, “BinaryConnect: Training Deep Neural Networks with binary weights during propagations”, NIPS 2015.
Reviewer comment: Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations).
Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision.
Response: The main goal of this paper is to propose a method to significantly reduce the number of parameters of fully-connected networks for hardware implementations. Literature mainly focuses on floating-point precision: however, floating point is very costly to implement in hardware.
On the other hand,  binarized/ternarized networks are well-suited for hardware implementations, and thus we focused on these kind of networks.
As mentioned earlier, no data augmentation or regularization was used in this paper. We have not investigated the effect of using the augmentation techniques yet.
However, while we mostly focus on and compare to binarized/ternarized networks due to their more efficient hardware implementation, the proposed technique is not restricted to binarized/ternarized networks and can be applied to the floating point precision networks on top of any other optimization technique.
Reviewer Comment: In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines.
Response: We agree that with the convolutional network we used in the paper, floating point precision can get a MCR below 9% on CIFAR10 while using data augmentation or regularization methods. In this paper, we showed that we can still get a MCR below 9% on CIFAR10 while using the proposed pruning technique and binarized/ternarized weights without any data augmentation or regularization method.
Although the floating point networks yield state-of-the-art results on different datasets while using data-augmentation techniques, they require a high number of bits and costly multipliers for hardware implementation.
However, binarized networks require only a single bit for representation of weights and no multiplication is required for hardware implementation of these networks.
Therefore, binarized/ternarized networks are mainly used for hardware implementation of the inference engine, and we showed that we can optimize these networks by pruning the weights in an efficient way and achieving a better MCR.
From my original comments:
The results looks good but the baselines proposed are quite bad.
For instance in the table 2 "Misclassification rate for a 784-1024-1024-1024-10 " the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see "significant" compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers...
In CIFAR-10 experiments, i do not understand  why "Sparsely-Connected 90% + Single-Precision Floating-Point" is worse than "Sparsely-Connected 90% + BinaryConnect".
So it is better to use binary than float.
Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations).
Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision.
In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines.
The authors reply still does not convince me.
I still think that the same technique should be applied on more challenging scenarios.
We sincerely thank the reviewer for the comments. We are currently running simulations on more challenging scenarios, and we will do our best to upload the results before the submission deadline.
We have added some results on a new baseline using data augmentation (affine transformation method) for the MNIST dataset in Table 2 of the revised manuscript. Our simulation results show that when data augmentation is used, our method can drop up to 50% and 70% of connections from BinaryConnect and TernaryConnect networks, respectively, without any compromise in performance.
However, using data augmentation results in a better misclassification rate when it is used on networks trained with single-precision floating-point weights, as shown in Table 2. In this case, our method can drop up to 90% of connections without any performance degradation.
At the moment, we are are running simulations on the CIFAR 10 dataset with data augmentation. We hope to be able to include the results before the deadline.
We have added some results on a new baseline using data augmentation (affine transformation method) for the CIFAR10 dataset in Table 3 of the revised manuscript. Our simulation results show that binarized/ternarized networks acquired with our pruning technique still result in a better MCR while using data augmentation. Based on our experimental results, binarized/ternarized networks yield a better MCR than Single-Precision Floating-Point networks since these networks prevent the model from overfitting as it was also shown in the original binarized/ternarized paper. In this paper, we experimentally showed that our pruning technique can also yield a further improvement in accuracy while having a significantly lower connections.
Of course, the reason behind this improvement requires more investigations and we will include this in our future works.
The paper proposes a sparsely connected network and an efficient hardware architecture that can save up to 90% of memory compared to the conventional implementations of fully connected neural networks.
The paper removes some of the connections in the fully connected layers and shows performance and computational efficiency increase in networks on three different datasets. It is also a good addition that the authors combine their method with binary and ternary connect studies and show further improvements.
The paper was hard for me to understand because of this misleading statement: In this paper, we propose sparsely-connected networks by reducing the number of connections of fully-connected networks using linear-feedback shift registers (LFSRs). It led me to think that LFSRs reduced the connections by keeping some of the information in the registers.
However, LFSR is only used as a random binary generator. Any random generator could be used but LFSR is chosen for the convenience in VLSI implementation.
This explanation would be clearer to me: In this paper, we propose sparsely-connected networks by randomly removing some of the connections in fully-connected networks. Random connection masks are generated by LFSR, which is also used in the VLSI implementation to disable the connections.
Algorithm 1 is basically training a network with back-propogation where each layer has a binary mask that disables some of the connections. This explanation can be added to the text.
Using random connections is not a new idea in CNNs. It was used between CNN layers in a 1998 paper by Yann LeCun and others: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf It was not used in fully connected layer before. The sparsity in fully connected layer decreases the computational burden but it is difficult to speed up.
Also the author's VLSI implementation does not speed up the network inference.
How are the results of this work compared to Network in Network (NiN)? https://arxiv.org/abs/1312.4400 In NiN, the authors removed the fully connected layers completely and used a cheap pooling operation and also got improved performance. Are the results presented here better? It would be more convincing to see this method tested on ImageNet, which actually uses a big fully connected layer.
Increased my rating from 5-6 after rebuttal.
Reviewer comment: The paper proposes a sparsely connected network and an efficient hardware architecture that can save up to 90% of memory compared to the conventional implementations of fully connected neural networks. The paper removes some of the connections in the fully connected layers and shows performance and computational efficiency increase in networks on three different datasets. It is also a good addition that the authors combine their method with binary and ternary connect studies and show further improvements.
Response: We sincerely thank the reviewer for the comments.
Reviewer comment: The paper was hard for me to understand because of this misleading statement: In this paper, we propose sparsely-connected networks by reducing the number of connections of fully-connected networks using linear-feedback shift registers (LFSRs). It led me to think that LFSRs reduced the connections by keeping some of the information in the registers.
However, LFSR is only used as a random binary generator. Any random generator could be used but LFSR is chosen for the convenience in VLSI implementation.
This explanation would be clearer to me: In this paper, we propose sparsely-connected networks by randomly removing some of the connections in fully-connected networks. Random connection masks are generated by LFSR, which is also used in the VLSI implementation to disable the connections.
Response: We thank the reviewer for pointing out this unclear statement. We have modified the explanation according to your comment.
Reviewer comment: Algorithm 1 is basically training a network with back-propagation where each layer has a binary mask that disables some of the connections. This explanation can be added to the text.
Response: We thank the reviewer for the comment. We agree with the reviewer that Alg. 1 is the back-propagation algorithm with a binary mask. We believe that summarizing the proposed method as an algorithm would be clearer to the public reviewer. As suggested, we added the explanation to the text.
Reviewer comment: Using random connections is not a new idea in CNNs. It was used between CNN layers in a 1998 paper by Yann LeCun and others: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf It was not used in fully connected layer before.
Response: We thank the reviewer for pointing this out.
Regarding the 1998 paper, we agree with the reviewer that a pruning technique was used.
However, we believe that the presented pruning technique follows a pattern provided in Table I of the 1998 paper: it is not declared that the pattern was generated randomly, and no explanation is provided on this matter.
Moreover, this pruning technique was only used for one convolutional layer (from S2 to C3) with 37% less connections.
Besides, the method presented in the 1998 paper only applies a high level pruning technique in a “binary” way: it either eliminates or maintains each complete 5*5 filter, and it does not drop connections within the filter.
Also, in the aforementioned paper no explanation is provided on the effect of the presented pruning technique in terms of both misclassification rate (MCR) and hardware implementation.
Reviewer comment: The sparsity in fully connected layer decreases the computational burden but it is difficult to speed up.
Also the author's VLSI implementation does not speed up the network inference.
Response: We thank the reviewer for the comment. The current VLSI implementations of fully-connected layers mainly suffer from copious power/energy consumption due to the energy/power-hungry memory accesses [*], [**].
That is why many works recently have focused on pruning techniques to reduce the memory accesses and consequently power/energy consumption. We agree with the reviewer that VLSI implementation of our proposed pruning method does not speed up the process compared to conventional VLSI implementations.
However, the proposed VLSI implementation provides other advantages.
First, we show that the proposed pruning technique can be simply performed using the conventional back-propagation algorithm where each layer has a binary mask, as opposed to other pruning techniques that usually require an additional training stage.
Secondly, we show that the proposed technique can drop up to 90% of connections while improving MCR. More importantly, the proposed technique can be easily implemented using LFSR units as shown in Fig. 3 while highly reducing power/energy consumption and required memory size, and thus making fully connected network implementable in practice. VLSI implementations of fully-connected layers generally speed up the computations compared to GPUs/CPUs as shown in [*].
[*] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally. EIE: Efficient inference engine on compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243–254, June 2016. doi: 10.1109/ISCA.2016.30.
[**] Matthieu Courbariaux and Yoshua Bengio. BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016.
Reviewer comment: How are the results of this work compared to Network in Network (NiN)? https://arxiv.org/abs/1312.4400 In NiN, the authors removed the fully connected layers completely and used a cheap pooling operation and also got improved performance. Are the results presented here better? It would be more convincing to see this method tested on ImageNet, which actually uses a big fully connected layer.
Response: We thank the reviewer for the comment. The MCRs of the NiN paper are summarized in Table 5 of our paper. The simulation results show that the proposed technique has a better MCR compared the NiN paper on CIFAR10 and SVHN datasets.
Regarding the ImageNet dataset, we are currently working on the extension on this paper: it will include tests results obtained on ImageNet.
Increased my grade to 6.
Experimental results look reasonable, validated on 3 tasks.
References could be improved, for example I would rather see
Rumelhart's paper cited for back-propagation than the Deep Learning book.
We sincerely thank the reviewer for pointing this out. The mentioned paper is now added to the manuscript. We will also double check the references to make sure that the original papers are cited.

Meta-review:
role: PC
data: comment: After discussion, the reviewers unanimously recommend accepting the paper.

Decision: Accept (Poster)