Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a sparsely connected network and an efficient hardware architecture that can save up to 90% of memory compared to the conventional implementations of fully connected neural networks. 
The paper removes some of the connections in the fully connected layers and shows performance and computational efficiency increase in networks on three different datasets. It is also a good addition that the authors combine their method with binary and ternary connect studies and show further improvements.
The paper was hard for me to understand because of this misleading statement: In this paper, we propose sparsely-connected networks by reducing the number of connections of fully-connected networks using linear-feedback shift registers (LFSRs). It led me to think that LFSRs reduced the connections by keeping some of the information in the registers. However, LFSR is only used as a random binary generator. Any random generator could be used but LFSR is chosen for the convenience in VLSI implementation. 
This explanation would be clearer to me: In this paper, we propose sparsely-connected networks by randomly removing some of the connections in fully-connected networks. Random connection masks are generated by LFSR, which is also used in the VLSI implementation to disable the connections.
Algorithm 1 is basically training a network with back-propogation where each layer has a binary mask that disables some of the connections. This explanation can be added to the text.
Using random connections is not a new idea in CNNs. It was used between CNN layers in a 1998 paper by Yann LeCun and others: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf It was not used in fully connected layer before. The sparsity in fully connected layer decreases the computational burden but it is difficult to speed up. Also the author's VLSI implementation does not speed up the network inference.
How are the results of this work compared to Network in Network (NiN)? https://arxiv.org/abs/1312.4400 In NiN, the authors removed the fully connected layers completely and used a cheap pooling operation and also got improved performance. Are the results presented here better? It would be more convincing to see this method tested on ImageNet, which actually uses a big fully connected layer. 

Increased my rating from 5-6 after rebuttal.

review_title: null

final_score: rating: null
review_content: The results looks good but the baselines proposed are quite bad.

For instance in the table 2 "Misclassification rate for a 784-1024-1024-1024-10 " the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see "significant" compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers...

In CIFAR-10 experiments, i do not understand  why "Sparsely-Connected 90% + Single-Precision Floating-Point" is worse than "Sparsely-Connected 90% + BinaryConnect". So it is better to use binary than float. 

Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations). Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision. 

In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines.



review_title: Results better but not good baselines

final_score: rating: 6: Marginally above acceptance threshold
review_content: From my original comments:

The results looks good but the baselines proposed are quite bad.

For instance in the table 2 "Misclassification rate for a 784-1024-1024-1024-10 " the result for the FC with floating point is 1.33%. Well far from what we can obtain from this topology, near to 0.8%. I would like to see "significant" compression levels on state of the art results or good baselines. I can get 0,6% with two FC hidden layers...

In CIFAR-10 experiments, i do not understand  why "Sparsely-Connected 90% + Single-Precision Floating-Point" is worse than "Sparsely-Connected 90% + BinaryConnect". So it is better to use binary than float. 

Again i think that in the experiments the authors are not using all the techniques that can be easily applied to float but not to binary (gaussian noise or other regularizations). Therefore under my point of view the comparison between float and binary is not fair. This is a critic also for the original papers about binary and ternary precision. 

In fact with this convolutional network, floating (standard) precision we can get lower that 9% of error rate. Again bad baselines.

----

The authors reply still does not convince me.

I still think that the same technique should be applied on more challenging scenarios.


review_title: Baseline results...

final_score: rating: 7: Good paper, accept
review_content: Experimental results look reasonable, validated on 3 tasks. 
References could be improved, for example I would rather see
Rumelhart's paper cited for back-propagation than the Deep Learning book.

review_title: Neural Nets for embedded devices

final_score: rating: null
review_content: Does the Asic implementation of the sparsely-connected network also provide a speed-up over fully connected layers? Can you add the speed numbers to Table 6?
Do you have any plans to experiment with Imagenet dataset?
review_title: Request for more numbers

Meta-review: After discussion, the reviewers unanimously recommend accepting the paper.
Decision: Accept (Poster)