Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the problem of improving spiking neural net training with better initialization. They observe that error backpropagation efficiency in SNNs depends on availability of responding neurons and show that traditional surrogate backprop methods are inefficient as they take many iterations to move weights to the regime supporting neuronal responses. 
The authors approximate the SNN neuron model (LIF, IF etc) I/O function using a piecewise-linear  iterative expression and show that clipping the output to (0,1) and normalizing the variance of the random weight initialization to insure that neurons produce substantial responses even with initial random weights results improves SNN training substantially. The improvement is to a large extent due to early onset of convergence. The authors demonstrate the superiority of their approach on MNIST, N-MNIST, DVS-MNIST and CIFAT datasets and present extensive experiments using different optimization algorithms and response functions. 

main_review: The paper addresses an important problem of inefficiencies in trainign SNN models and presents a compelling argument for the inefficient SNN initialization and demonstrate that improving it so that neurons are responding from training onset can increase both convergence and test accuracy.

While the paper is technically sound and offers a substantial result, the presentation is quite  convoluted and could be substantially streamlined and would benefit dramatically from a read by someone fluent in English. Just to mention a few issues, the introduction section is overly verbose and does not seem to be always to the point. Many statements are non-informative, like "neurons respond in a suitable region..." or "producing proper amount of spike" where "suitable region" and "proper amount" is never defined. Incorrect use of terms abound (non-differential => non-differentiable etc) that can seriously obscure the authors' intentions.

Minor issues:
Figure 2: Dashed lines referenced but not displayed; what is the difference; why the actual neuron responses (circles) are not shown for the full range of inputs?



summary_of_the_review: While the paper presents an important and technically sound contribution, the quality of presentation dramatically reduces the value of the manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper enhances the efficient BPTT training of SNNs by proposing an initialization method to match the response of spiking neurons in initial training. Their method bridges the spiking neuron response to the wisdom of traditional deep learning training, which may have an influence on future research like ANN-to-SNN conversions with LIF neurons or other SNN training methods. The authors conduct experiments on CIFAR10, MNIST, and neuromorphic datasets to show the efficacy of their technqiue. One of the main contribution is their theorteical analysis of iterative systems to model first-order integrate-and-fire neurons and investigate the response curve.

main_review: The authors have presented interesting results. The discrepancy between forward spike activation function and backward surrogate gradient function during backprop restricts training capability in SNNs. Modern deep learning relied on tricks like ReLU, initialization to balance forward/backward variance of activations (Glorot/He), skip connections etc. to overcome the vanishing gradient problem. The weight initialization approach in this work aims to do similar things. 

While the authors have compared with recent works, I would like to bring the attention of authors to recent works from a group that authors have cited in their work, that use batch norm and threshold initialization as a way to mitigate the gradient training issue. I feel these two works are closely related to the authors work and both yield better performance results with very interesting SNN dynamics advantages like low latency etc. Since the authors have missed out comparing their work to these to works which are more in line with what authors are trying to do, I am a little concerned about the sanity and the novelty of this work.
 
[R1] Kim, Y., & Panda, P. (2020). Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. arXiv preprint arXiv:2010.01729.
[R2] Kim, Y., & Panda, P. (2021). Optimizing deeper Spiking Neural Networks for Dynamic Vision Sensing. Neural Networks.

In [R1], Kim et al. present a temporal batchnorm technique to improve the gradient vanishing/explosion. I feel this work is very related to [R1]. So, it will be prudent for the authors to comment and show a comparsion. Further, in [R1] , the authors are able to train large-scale datasets from scratch like CIFAR100, TinyImagenet with a very low latency on rate-coded inputs. I am wondering if the author's method is scalable to non-trivial datasets beyond CIFAR10, MNIST (which are easy to get good accuracy with).

Based on this, my next question is, can authors comment how weight initialization impacts the overall spike activity, latency of processing of the network.

In [R2], Kim et al. propose a threshold initialization technique to improve the trainign of SNNs for DVS datasets and they looked at interesting datasets like N-Caltech (more complex than N-MNIST) as well as DVS-CIFAR10, DHP etc. Again, since the authors have looked into small scale datasets, it makes me question the scalability of their work. I think the authors should comment on this and also highlight how their work is better or orthogonal or complementary to these two works.



summary_of_the_review: The results are interesting. I feel the authors should make a more thorough investigation of recent work that highlight similar ideas and yield better results on more complex and large-scale datasets. I am giving a rating of 5, but I am willing to chnage my score if the authors can convince me of the scalability and the novelty of their approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the effects of weight initialization for deep spiking neural networks (SNNs) on training speed and final accuracy. The main finding is that popular initialization methods for conventional ANNs and RNNs do not match the specific dynamics of SNNs, in particular they ignore the need to have sufficient firing early on to generate good gradients. The paper derives a theoretical first-order approximation on the response curve, and from it a weight initialization scheme that initializes the weights in a region where neurons fire from the first epoch on, but also avoid explosion or reduction of activities throughout the network. In various experiments on standard and spiking variants of MNIST and CIFAR, the initialization scheme shows good performance and fast convergence for different neuron types (e.g. non-leaking vs. fast leaking), and different encoding schemes. For CIFAR10 the results are significantly better than for other initialization schemes.
main_review: Whereas previous work on Deep SNN has often borrowed techniques from standard ANNs, this paper goes beyond that and develops SNN-specific methods that can accelerate Deep SNN training and lead to better accuracies. This is achieved from theoretical analysis of SNN neuron models and network dynamics, and the results seem to indicate that the method is generally superior to non-SNN-specific methods under a variety of parameters and datasets. If this method generalizes to larger tasks and networks, this could become a new standard for SNN training, and therefore a significant contribution to the SNN literature.

Strenghts:
+ derivation of the weight initialization method from theory
+ convincing results of experimental validation on a variety of datasets
+ comparison to a number of standard initialization methods from the ANN world,which do not have SNN-specific adaptations
+ possibly very broad application field, could become a new standard

Weaknesses:
- occasional unclear writing and grammar errors which should be fixed
- tested only on MNIST and CIFAR variants, it is unclear whether the method generalizes to larger networks

Recommendation:
- I would suggest including at least one larger experiment that shows that the weight initialization scales to larger networks, and is not an effect of the relatively easy datasets that were considered
summary_of_the_review: The presented method could become important for further studies on Deep SNNs, as a replacement of poorly fitting weight initialization schemes borrowed from ANNs. It remains to be shown that the method scales to more difficult tasks and larger networks. If the method still works there, this could be a paper with very high impact.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  This paper focuses on the parameter initialization problem of training SNNs. The authors derive the theoretical response of spiking neurons, and propose an initialization method based on slant asymptote, which can overcome the gradient vanishing problem. The results show that the proposed method can effectively improve training speed and accuracy.
main_review: Strengths:
This paper proposes a weight initialization method to enhance the efficient BPTT training of SNNs. The derivation and validation of the weight initialization method appears to be sound. It is interesting to see the authors relate the spiking neuron response to the traditional deep learning training. This work may also inspire research like ANN-to-SNN conversions with LIF neurons.
Weaknesses:
(1) The related work section should be introduced in more detail. Is there any other study about weight initialization of SNN? 
(2) Why is the weight initialization more effective on the CIFAR10 than the MNIST?
(3) I suggest to demonstrate the effect of the initialization method on more complex dataset..

summary_of_the_review: Overall, it is a nice paper with solid theoretical basis. It has the potential to convince researchers to adopt SNNs and exploit learning of deep SNN. I consider to raise my score if the authors can resolve the concerns above.
review_title: null

Meta-review: The paper derives a new parameter initialization for deep spiking neural networks to overcome the vanishing gradient problem.

During the review, concerns were expressed about how well the method would scale to larger neural networks. It was also questioned how this parameter initialization technique compares with a recently proposed batch normalization technique, especially when training larger neural network on more challenging datasets. There were also concerns raised about the readability of the paper.

I commend the authors for improving the readability of their paper in their revision. I also commend them for taking the time to implement the comparisons requested by the reviewers. These new comparisons revealed that batch normalization and its recently proposed variant were superior to the initialization method on its own, and that the initialization proposed in the paper did not significantly improve performance when paired with batch norm [[1](https://openreview.net/forum?id=T8BnDXDTcFZ&noteId=yIAPcSbUAQ0)]. The authors also acknowledged based on the new results, that their proposed parameter initialization scheme appears to fail to scale to more complex datasets and networks, especially relative to competing methods, which invalidates a key claim that their approach can "accelerate training and get better accuracy compared with existing methods" [[2](https://openreview.net/forum?id=T8BnDXDTcFZ&noteId=j12fwayWEb)].

The recommendation is to reject the paper in its current form.
Decision: Reject