Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work finds it unclear to determine which surrogate gradient to apply in different tasks or networks in SNN. Seeking to solve this problem, the authors have done some experiments with different surrogate gradients across tasks and networks, and come to the conclusion that the derivative of fast sigmoid outperforms other chosen surrogate gradients. Besides that, the authors have also done research on how different characteristics of surrogate gradients and initialization affect surrogate gradient learning and found that low dampening, high sharpness, low tail-fatness, orthogonal initialization, and high initial firing rate could help improve the network’s performance. Lastly, the authors provide a theoretical solution based on bounding representations to find a surrogate gradient and initialization that could improve accuracy.
strength_and_weaknesses: Strength: 1. Training details are explicit, including network architecture and hyper-parameters. 2. Conclusion from the experiments is helpful in choosing surrogate gradient.

Weaknesses: It doesn’t seem reasonable to make the neuron as sensitive to the network history as to new input in section 3.5.2, since the network history is an accumulation of knowlegde learnt from past so it contains more knowledge than new input.

Minor typos:

Sec2, line 3 "y_L the network output" -> "y_L is the network output" or "y_L means the network output"
clarity,_quality,_novelty_and_reproducibility: The authors of this work do not promise any reproducibility in text.


summary_of_the_review: The discussion on how to choose SG in SNN is meaningful and missed before.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This article aims to experimentally and theoretically define the best SG across different stress tests to reduce the future need for grid search and proposes techniques to improve the performance of SNN training from multiple perspectives, such as the dampening of the SG and the normalization method.
strength_and_weaknesses: Strengths:
1.	The authors provide empirical results on how to select SGs and initial methods to enhance the final performance.
2.	The authors designed a new kind of SG that is suitable for all the datasets they used.
Weaknesses:
1.	Lack of comparison with other existing work.
2.	This paper only provides experiments on very simple datasets and network structures. More complex cases need to be included.
3.	The logic of this paper is confusing, e.g. many important figures are placed in the appendix, making it difficult for the readers to read.
4.	Their main findings are obtained through some experiments with grid search while lacking logic proof. So the article's novelty and contribution are modest.

Questions:
1. In Sec 3.1.2, what is the complete model structure for testing different SGs?
2. 1.The datasets used in the experiments seem to be relatively small datasets. How does the experiment perform on larger datasets? I think the baseline of this work should be compared with the existing work.
3. 1.The SG used doesn't guarantee $\int_{-\infty }^{+\infty }f(x) = 1$. Maybe that's why the other SGs don't perform well?
4. 1.In sec 3.4.1.Is there a missing equation before "where p is the firing rate"?
5. 1.Fig 5. If the baseline is replaced with other initialization methods, will the results change?

clarity,_quality,_novelty_and_reproducibility: The paper seem to be a rush re-wrap from its long preprint. Thus the writing and readability are not sufficiently good. 
summary_of_the_review: Overall, it adds an interesting paragraph to the training of SNN with LIF but the performance is not convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers different spiking neuron models and explores hyperparameters on multiple tasks in order to facilitate efficient training via the surrogate gradient method. The study focuses primarily on the shape of the surrogate gradient function. They find that low dampening, high sharpness and low tail-flatness generally lead to better performance. More precisely, the fast-sigmoid derivative appears as the best choice of surrogate gradient function. Secondly, they perform a study of different weight initialisations, which reveals that “He” gives the best outliers, “orthogonal normal” the best mean accuracy and “bigamma” the lowest variance. Thirdly, they demonstrate that using high initial firing rates combined with a sparsity encouraging loss term induces better generalisation. Finally, they investigate various conditions to ensure stable gradients and show that these can also lead to better performances.
strength_and_weaknesses: Strengths

The paper aims to facilitate the training of spiking neural networks via surrogate gradients, which remains more challenging and less documented compared to more popular, non-spiking artificial networks.
SNNs are applied to rather diversified tasks involving image recognition, spoken digit recognition and language modelling.
Some findings can lead to helpful, practical solutions for SNN training.

Weaknesses

The overall narrative of the paper needs substantial improvements. A fair number of (sub)sections do not lead smoothly into the next ones. The number of components tested in the study is rather ambitious, which can be okay, but the different findings often feel disconnected, which makes the general direction hard to follow. 
For instance, the authors define a spiking version of the LSTM, but the results with it are only shown to be inferior to those obtained with simpler, more commonly used and more biologically plausible LIF and ALIF models. Even if interesting, the inclusion of this spiking LSTM does not seem justified and feels out of the way in terms of the general objectives of the paper.
The use of the different models and data sets does not feel coherent. Some hypotheses are tested on all tasks and models, whereas others for instance only use the LIF on a single task.
The paper feels more like a report than an actual article.

Remarks

The SHD data set only seems to have training and validation splits in its default version. It would be worth explaining what test split you are using.
The spiking LSTM, which is said to be defined in Appendix G, is missing (maybe appendixes are simply not included for this review process, in which case, this remark can be discarded).
clarity,_quality,_novelty_and_reproducibility: The paper is not very clearly written.  It is probably reproducible, but the novelty is not so high.
summary_of_the_review: The authors test a variety of techniques, albeit somewhat inconsistently.  As a review the paper could be useful, but overall it comes across as rather rushed and incomplete.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: As claimed by this work, the authors present a framework for designing the best surrogate gradients of SNNs with LIF neurons. 

Unfortunately, this paper is hard to follow due to the unclear organization and non-standard presentation. The claimed contributions can be roughly divided into two terms, i.e., summarizing the observed results from experiments and presenting the theoretical understanding of SG design. I cannot demonstrate the correctness of the observed results since there lacks apposite definitions or measures of many terminologies, such as low dampening, high sharpness, and low tail-fatness.

More importantly, this paper provides a theoretical method for SG choice to improve the experimental performance. What are the assumptions and conclusions of this theoretical investigation?

Besides, I would appreciate to have a clear clarification on the experimental details before showing the experimental results, such as shown in Subsection 3.5.5. 
strength_and_weaknesses: Weaknesses:
The weaknesses are detailed above, including:
1. This paper is hard to follow due to the unclear organization and non-standard presentation.
2. Low quality. There lacks apposite definitions or measures of many terminologies, such as low dampening, high sharpness, and low tail-fatness.
3. The assumptions and conclusions of the theoretical investigation is far from clear.
clarity,_quality,_novelty_and_reproducibility: nothing to mentioned
summary_of_the_review: Overall, I believe that the presence of this paper is poor, and thus, it is difficult to verify the quality of its work, which would move the paper away from the accepted standard.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores the use of surrogate gradients in spiking neural networks. It examines how different shapes of surrogate gradient affect performance and explores a theoretical construction of a good initialization for surrogate gradient parameters that reduce the burden of hyperparameter searches.

The strength of this paper is that it is exploring an important question and could potentially be very useful for researchers working on spiking neural networks.

The weakness of this paper, agreed by all reviewers, is a significant lack of clarity, and in turn, reproducibility.

The authors responded to the reviewers concerns, but it was not enough to change the low scores (average score of 3.5). Based on this, a decision of reject was reached.
justification_for_why_not_higher_score: Based on the reviewer comments, this paper is not sufficiently well-written for acceptance at ICLR.
Decision: Reject