Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes spike-element-wise ResNet, in an effort to achieve identity mapping. The original Spiking ResNet put the element-wise operation before the activation function. However, with the LIF activation function, the activation will be binarized to spikes and thus the identity mapping is not fulfilled. Thus, the authors put the element-wise operation after the LIF activation. 


main_review: **Post Rebuttal**

I have seen the author's response. Overall, I still think the author's response does not fully address my first question (why not use QNN if the neuromorphic HW can support up to 8-bit spikes?). Their responses referring that 1) SNN uses binary activation, 2) multiplication-free computation and 3) higher adversarial robustness all are the property of conventional SNN rather than the multi-bit SNN. However, I also acknowledge that the authors made adequate efforts for rebuttal, which I indeed appreciate. I will increase my score to 6 and I hope authors can rigorously discuss the point of this multi-bit spike in the next version. 

PS: According to your baseline paper, the tdBN-Spiking-ResNet-50 has 65% accuracy while your result is 58%?

-----


Overall, the authors propose a reasonable architecture to address the residual mapping issue in the spiking networks. However, the overall algorithm seems to be a minor solution, as the only difference is the swap of two operations. This work main experiment with  ImageNet experiments, which I appreciate.  


**Pros**:

+ The topic of this paper is interesting, the residual path is an important technique in ANN, and I am happy to see a paper study this architecture in SNN. 

+ The experiments in ImageNet show improvements. 

**Cons**:

- My biggest concern of this work is the non-binary spikes in the network. Although the authors propose two variant element-wise ops: namely AND and IAND, they finally adopt the ADD operation, which means the spikes after the operation is not binarized. And the bit-width that is used to represent this spike will continue to increase. This seems to be an undesired characteristic in SNN since the neuromorphic hardware operates with binary activation. Thus the energy consumption may be significantly higher than the spiking resnet.

- In my view, the spiking resnet is more biology-plausible. Is there any biological evidence that two spikes can be fused together? 
limitations_and_societal_impact: My concerns are listed above. In my opinion, it would be natural that multi-bit spikes outperform binary spikes, therefore I would reject this submission for now. If the authors can clarify this, I will change my scores.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors proposed the spike-element-wise (SEW) ResNet to train deep SNNs with ResNet modules.  They achieved very excellent results compared to the SOTA. Although simple, the proposed methods turned out to be very effective in the practice. 
main_review: 1. The methods and results were written clearly except for some minor grammar issues. Please check more carefully in the later version. 

2. As the authors also state in lines 166-176, the SEW block is an extension of the RBA block. Thus, it suffers the same shortcoming that it leads to a non-negative output from the transform $\mathcal{F}$, which would impact the representational ability. 

3. For the comparison with other methods, the simulation for SEW is with T=4. Is there any reason for not using T=6, the same as Spiking ResNet-34? Does the proposed method require more memory for training? Another issue here is that the references to the ANN2SNN methods are not accurate here. For example, Deng & Gu, 2021 ICLR can get 67.54% with T=128 and Li, 2021 ICML can get 71.12\% with T = 64. 

4. The demonstration of gradient amplitude is a bit misleading. I suggest that the authors subtract the gradient calculated by $S_{j}^l[t]=1$ and $\sigma'(1-V_{th})=1$ out so that $0$ indicates vanishment. 

5. The problem of SpikeNet may be relieved by choosing surrogate gradients for the identity map. This setup with the same training strategy as the SEW-Net can serve as a benchmark to compare as well. 

6. It would be more informative to see further results on DVS-CIFAR-10. 
limitations_and_societal_impact: Not included. The authors may add a paragraph to discuss the limitation and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper wants to apply the residual block into deep spiking neural networks to achieve better classification accuracy. They proposed a spike element-wise (SEW) to realize residual learning. This top and idea are very important in training a robust deep spiking neural network and the residual block is approved useful in the deep learning field. The authors evaluated the SEW on several datasets such as ImageNet and DVS gesture datasets, the experimental results show the proposed method can achieve better performance. 

main_review: A spiking neural network (SNN) is considered a promising approach to build an intelligent machine as efficient as the human brain. This paper investigated the learning of deep SNN by converting residual network which allowed a network to go extremely deep and has achieved great success in pattern classification tasks. This paper analyzed the drawbacks of some current spiking resnet methods, aiming at this, the paper introduced SEW block as formulated as equation 9. Through the identity mapping with ADD, AND, and IAND, the binary spiking signals could be calculated and communicated between neural layers. 

Different from ANN2SNN methods, this paper applied a spiking bp-based method to training the proposed network model. Table 3 shows the experimental results on ImageNet, the authors reported impressive results that the classification accuracy is positive with fewer time steps. 

limitations_and_societal_impact: This paper still existed some problems that I hope the authors could illustrate in a clearer way. 

1.	The authors argued that they were the first time to directly training deep SNNs with more than 100 layers. I don’t think this is the core contribution in this paper, because of the residual block, the spiking could be deeper. In my opinion, SEW structure is the most important point in this paper, and directly training a 50-layer and 100-layer snn is not a huge breakthrough. Otherwise, if they could give a more detailed analysis about why other methods can’t train a 100-layer snn except section 3.2, it may be more reasonable. 
2.	Why the RBA block can be seen as a special case of the SEW block? I mean SEW is another kind of RBA with binary input and output. 
3.	Equ. 11 is wonderful, how about other bit operations?
4.	Fig. 5 a seems strange, please give more explanations. 
5.	When the input is aer format, how did you deal with DVS input? 
6.	If you can analyze the energy consumption as reference[15] did, this paper would be more solid.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The present paper is concerned about a residual architecture for spiking neural networks. First, the authors point out two drawbacks of the existing Spiking ResNet. The first drawback is that Spiking ResNet cannot easily represent identity mapping for some neuronal dynamics. The second drawback is that even if a block of Spiking ResNet represents the identity mapping, the gradient of its output with respect to the input is not equal to 1, and therefore, a sequence of Spiking ResNet blocks suffers from vanishing or exploding gradients. These two drawbacks hinder us from training a deep Spiking ResNet.
The authors present the spike-element-wise (SEW) residual block to address the drawbacks mentioned above. Since the drawbacks are caused by the architecture where the skip connection is fed into a spiking neuron, the idea is to skip the spiking neuron and directly add/multiply/IAND the residual mapping and the input spike sequence. The authors discussed that the SEW block can address the drawbacks.
The authors present two sets of experiments, one uses ImageNet and the other uses DVS Gesture data set. The experimental results suggest that the proposed SEW ResNet successfully addresses the drawbacks and the deeper the network is the better the performance is.

main_review: I cannot judge whether the proposed architecture is novel or not, because I am working on a learning algorithm of SNN rather than its architecture. However, provided that the proposed architecture is novel, I think this paper is good because it clearly states the drawbacks of the existing method and the proposed method successfully addresses it.

One minor concern is that there are many typos;
- l.271 "indicating that indicating that" -> "indicating that"
- l.275 "Fig. 7 shows" -> "Fig. 8 shows" ?
- l.276 "Fig. 8" -> "Fig. 6" ?
to name a few. I would recommend the authors to go through the paper to improve the quality of the paper.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: By allowing multi-bit spiking, a novel residual block structure is proposed for neuromorphic deep learning. Thorough experiments were run to compare performance and power consumption. Reviewers generally agreed that this contribution is interesting for the spiking neural network community. They also agreed that the writing can be improved. Please make sure to include the results included in the rebuttal in the main paper as well.
Decision: Accept (Poster)