Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a Spiking Neural Network (SNN) training method that jointly optimizes input spike encoding parameters, spiking neuron parameters (membrane leak and voltage threshold), and weights in an end-to-end fashion using gradient descent. Compared to SNNs with only weight optimization, the SNN trained by the proposed method significantly decreases the inference latency and results in high activation sparsity with minimal accuracy decrease.

Highlights:

1. Energy-efficiency is currently the main advantage of SNN. Two primary factors that determine the energy-efficiency of an SNN are inference latency and activation sparsity. The method proposed in the paper directly targets both factors, and the intuitions are very clear.

2. The results show the joint training approach directly contributes to the improvement of SNN's inference latency and activation sparsity (as shown in Fig. 2 and Fig. 3 in the paper). It's also interesting to see the method can learn low membrane leak and high voltage threshold at the same time for later layers of VGG16, which significantly improve activation sparsity.

3. The paper proposes a new spike generation function for LIF neurons in equation (2) of section 3. The function places the voltage threshold in the computational graph of backpropagation, making training voltage threshold possible in an elegant way.

Concerns:

1. The reviewer’s main concern with the paper is its limited novelty. The joint end-to-end training of neuron parameters has already been proposed in a recent paper ([Bojian Yin et al. 2020]). Moreover, in addition to gradient-based training, there are also local learning methods like intrinsic plasticity that tune the neuron parameters based on the activity of the SNN (for example, [Wenrui Zhang et al. 2019], [Anguo Zhang et al. 2019]). The paper needs to better position the proposed method with respect to these existing related works.

2. Table 3 shows the computation overheads as a result of the direct encoding layer is around 1%. However, the large number of encoded spiking activities can slow down the inference speed when the SNN is deployed on currently available neuromorphic chips (for example, Intel's Loihi or IBM's TrueNorth) since these chips lack an efficient solution for injecting external spikes. It will be great if the paper can show how the DIET-SNN performs with different numbers of convolution channels in the encoding layer and examine if using fewer input neurons for encoding will hurt the accuracy.

3. The proposed method uses backpropagation through time (BPTT) to train the SNN. Many recent papers that train SNN using spike-train level learning methods also achieve low latency and sparse synaptic activities (for example, [Jibin Wu et al., 2019], [Yingyezhe Jin et al., 2018]). The paper lacks experiments for comparing the performance, inference latency, and activation sparsity with these existing methods.

4. The ablation study in section 6 uses IF neurons for the first 3 experiments and LIF neurons for the last experiment which seems unfair. The ablation study should show that DIET-SNN can do better than an SNN using LIF neurons with fixed (preset) membrane leak and voltage threshold. The current ablation study cannot rule out the possibility that an SNN with fixed (preset) low membrane leak and high voltage threshold performs as well as or even better than DIET-SNN in terms of accuracy and activation sparsity.

5. The ablation study in section 6 shows the performance of only one inference timesteps value for each SNN. To show a complete picture of how the performance changes with the latency, the paper needs to train each SNN with the same group of latency values (for example, all 4 SNN trained with 150, 35, 25, and 20 timesteps).

6. In section 6, the paper claims that optimizing the threshold has similar consequences as optimizing the weights. The co-optimization of them can lead to faster convergence and lower latency. However, there are no experiments to support this argument. The reviewer suggests conducting additional experiments for this.


I want to thank the author for addressing my concerns. Many of my concerns are resolved. I have updated my rating after the discussion. I agree with the authors that the novelty of the submission lies in scaling up the existing methods. However, I'm not sure if this is enough for the paper's scientific significance required by this conference. 


Bojian Yin et al. 2020, Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks

Wenrui Zhang et al. 2019, Information-theoretic intrinsic plasticity for online unsupervised learning in spiking neural networks

Auguo Zhang et al. 2019, Fast and robust learning in Spiking Feed-forward Neural Networks based on Intrinsic Plasticity mechanism

 Jibin Wu et al., 2019, A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks

Yingyezhe Jin et al., 2018, Hybrid macro/micro level backpropagation for training deep spiking neural networks
review_title: Interesting results for direct-trained low-latency SNN but with limited novelty and incomplete ablation studies

final_score: rating: 3: Clear rejection
review_content: Strength:
I appreciate the experimental results demonstrated on challenging datasets like CIFAR100 and ImageNet.

Weakness
(1) There are two existing papers emphasizing direct training SNN with extremely low latency [1][2]. Therefore, the authors should comment more on how the proposed method is different or is better compared to these two papers (they use a smaller number of time steps.). In addition, I hope the authors to show the performance comparison with these two references. In the experimental results, the paper compares performance with [1]. However, the network size is significantly different. I think a comparison of the same network size can help to demonstrate the effectiveness of the proposed method.

(2) To my understanding, the method proposed in this paper has nothing different compared to the existing BPTT method with the surrogate gradient. I hope the authors can claim clearly the novelty of this paper. For example, how is the proposed method different or better from [3] and [4]. As far as I know, the only difference is that the threshold and leaky parameters are also trained in this paper. However, this can also be easily done in other existing methods like [2][3][4]. There's no doubt their performance can also be improved since it introduces more tunable parameters.

(3) Tuning threshold and leaky parameters may be unbiologically plausible.


(4) What is the weight optimization method? As shown in Table 1, it seems the weight optimization contributes more to the performance than the proposed method.


[1] Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., & Shi, L. (2019, July). Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 1311-1318).
[2] Zhang, W., & Li, P. (2020). Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks. arXiv preprint arXiv:2002.10085.
[3] Wu, Y., Deng, L., Li, G., Zhu, J., & Shi, L. (2018). Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12, 331.
[4] Shrestha, S. B., & Orchard, G. (2018). Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems (pp. 1412-1421).

review_title: This paper proposed a hybrid of ANN-SNN conversion and direct SNN training. It can train the network with small number of time steps.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Tile: training SNNs for visual categorization with few timesteps

PROS
* high accuracy with only 20-25 timesteps

CONS
* nothing really new

The authors train convolutional SNNs for image classification, using surrogate gradient learning (SGL). They combine 4 mechanisms:

1) Hybrid SNN Training. First train an ANN with the same architecture, and use the resulting weights as initial values for SGL, to accelerate convergence. This has already been proposed in (Rathi et al., 2020).
2) Direct Input Encoding. Instead of converting the image RGB values into spike trains using Poisson rate coding, the authors feed the analog RGB values in the first convolutional layer, which treats them as input current, and emits spikes using the LIF neuron model. This allows using fewer time steps, since Poisson rate coding imposes long time windows to estimate rates and average out noise. But again, this is not new. It's been done for example in (Rueckauer et al., 2017; Lu & Sengupta, 2020), which they cite.
3) Leak timescale training. The time scale of the leak is an important parameter in SNNs. It can be trained by SGL, just like the weights. Again this is not new. See for example (which should be cited):
Fang W (2020) Leaky Integrate-and-Fire Spiking Neuron with Learnable Membrane Time Parameter.
Yin B, Corradi F, Bohté SM (2020) Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks. arXiv.
Zimmer R, Pellegrini T, Singh Fateh S, Masquelier T (2019) Technical report: supervised training of convolutional spiking neural networks with PyTorch. arXiv.
4) Threshold training. Again this is not new. For example Zimmer et al 2019 (ref above) did it already. Also, training both the weights and the thresholds seems redundant, since only the ratio between weights and threshold matters. About that the author say:
"Intuitively, the effect of optimizing either weights or thresholds in SNNs with
IF neurons should have similar consequences. But the threshold affects the activity of each neuron, whereas the weights are shared among multiple neurons."
I don't understand this sentence. Weights are shared only in the convolutional layers. Plus earlier the authors said that the threshold is the same for all neurons of a given layer.

So in short, the authors successfully combine known approaches, but do not propose anything new at a conceptual or theoretical level. In my opinion this paper is mostly an engineering effort. That being said, it seems that no one can claim a better accuracy on CIFAR and ImageNet using this nb of timesteps (or fewer).

MINOR POINTS:

* How much longer is convergence if the authors skip the ANN training, and start the SGL training from random weights?
* p2: "The length of timesteps" -> "The number of timesteps"
* "The neurons in the convolutional and linear layers are defined by the LIF model" I guess the authors meant "dense" or "fully connected" instead of "linear", since the LIF is not linear
* I liked a lot the "iso-accuracy" analysis of section 6. But I suggest the authors do the same for their energy analysis of section 5. The SNNs consume less than the ANNs (Table 3) but are also less accurate (Table 1).
 

review_title: Training SNNs for visual categorization with few timesteps

Meta-review: The manuscript presents a training method for Spiking Neural Networks (SNN). The method jointly optimizes input spike encoding parameters, spiking neuron parameters (membrane leak and voltage threshold), and weights in an end-to-end fashion using gradient descent. SNNs are very interesting for energy-efficient implementations of neural networks. Their energy efficiency strongly depends on inference latency (SNNs compute in time, unlike feed-forward ANNs) and activation sparsity. 

All reviewers acknowledged that the approach directly improves inference latency and activation sparsity on large convolutional models at very good performance levels.

The main concern of all reviewers was the limited conceptual novelty. The paper combines some known techniques (hybrid SNN training, direct input encoding, training of neuron parameters like leak time constant and threshold) and scale the setup up to large networks and datasets (e.g. ImageNet).

In summary, the paper presents impressive results, but the conceptual innovation is missing. 



Decision: Reject