Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper shows how SNNs can integrate backpropagation with a second accumulation module that discretizes errors into spikes. In other words, the authors show how to translate event-based information propagation (used by SNNs) into a backpropagation method, which is the main contribution. The description of establishing the equivalence between an ANN and SNN in Section 3 is mostly well done. They perform empirical studies on MNIST and CIFAR-10 to demonstrate the effectiveness of SpikeGrad. 


= Main Concerns =

1. It seems not clear to me why it is a good idea to introduce a second compartment with a threshold in each neuron as described in Eqn. 6. 
2. I very much like the idea of "translating" an SNN into an ANN. I'm a bit confused about the computational complexity estimation of the SNN. In particular, it is not clear to me what is the practical implication of {n-n_min}/n_min < 0.035. Furthermore, in https://openreview.net/forum?id=rkg6PhNKDr, for ANNs on CIFAR-10, freezing 80% of the VGG19 parameters from the third epoch onwards only results in 0.24% drop in accuracy. I wonder if the advantage of SNN over ANN is still huge in this case. 
3. I do not think experiments on MNIST are very useful, as the task is a toy task. I would suggest running at least one more experiment on CIFAR-100 or TinyImagenet. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: EDIT After Rebuttal: My understanding of the contributions of this paper has improved. I now increase my score to a weak accept.

This paper proposes a new backpropagation algorithm learning algorithm "SpikeGrad" for Spike-based neural network paradigm. Simulating this algorithm on a classical hardware would require a lot of time-steps. To circumvent this, they show how to construct a corresponding artificial neural net (that can be trained using the traditional gradient based algorithms) which is equivalent to the spiking neural net. Using this equivalence they simulate a large scale SNN on many real-world dataset (first paper to do so). In particular, they use MNIST and CIFAR-10 for this purpose. They show that training a fixed architecture using their method is comparable to other prior work which uses high-precision gradients to train them. They also show how to exploit sparsity of the gradient in the back propagation for SNN.

This paper is hard-to-follow for someone not familiar with the background material. In particular, without looking at prior literature it was hard to understand that "integrate and fire neuron model" is essentially the feedforward mechanism for the SNN. I would suggest the authors make this a bit more explicit. Moreover, it would serve the structuring of the paper to have a formal "Preliminaries" section, where all known stuff goes. It was hard to discern what is new in this paper, and what is from prior work and these are mixed in section 2. For instance, section 2 states "SpikeGrad" algorithm; but the main contribution (ie., the back propagation algorithm) only appears in the middle of this section. Likewise, I think section 3 can be arranged better. In particular, the equivalence is a "formal" statement and thus, could be stated as a theorem followed by a proof. It will also make it explicit as to what does it mean by an "equivalent" network. In fact, it is still not clear to me at this point what that statement means. Could you please elaborate this in the rebuttal? 

Regarding the conceptual contribution of this paper, if I understood things correctly, the main claim is that they give a new way to train SNN whose performance on MNIST and CIFAR-10 is comparable to other works. The second contribution is that they give the equivalence between ANN and SNN (point above). It is also unclear to me what the point regarding the sparse gradient in the backpropagation in the experimental section is trying to make? Could you please clarify this in the rebuttal as well?

At this point, the writing of this paper leaves me with many unanswered questions that needs to be addressed before I can make a more informed decision. Please provide those in the rebuttal and based on those will update my final score. But with my current understanding of this paper, I think this does not meet the bar. The contributions in this paper do not seem too significant.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a first framework of large-scale spiking neural network that exploits the the sparsity of the gradient during backpropagation, to save training energy. 
Later, it provides detailed analysis to show the equivalence of accumulated response and the corresponding integer activation ANN.

The paper is clearly written. The forward and backward process with the spike activation and error activation function respectively to save energy is clearly demonstrated. The response equivalence of the proposed architecture and integer ANNs provides theoretical gurantee for the good performance in training accuracy.  

My only concern is the lack of empirical support for the energy saving of the proposal. In order to show the effectiveness of the proposal, the authors should also provide time consumptions of the SNN and normal ANN. A mere comparison on sparsity doesn't really show the advantage of the proposal, since there is some computational overhead. For a system-level improvement, it's not sufficient to show the epoch-operation relation.
If the authors could provide wall clock time comparisons, I will consider raising my score. 

==========
I find the response of the authors reasonable and address some of my concerns. Therefore I'm willing to raise my score to 6. 
review_title: Official Blind Review #3

Meta-review: This paper proposes a learning framework for spiking neural networks that exploits the sparsity of the gradient during backpropagation to reduce the computational cost of training. The method is evaluated against prior works that use full precision gradients and shown comparable performance. Overall, the contribution of the paper is solid, and after a constructive rebuttal cycle, all reviewers reached a consensus of weak accept. Therefore, I recommend accepting this submission.
Decision: Accept (Poster)