Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors outline a principled approach--finite difference gradients (FDG)--for choosing a surrogate gradient for spiking neural networks. FDG is demonstrated to approximate gradient descent in classic rate-based ANNs very well, approximate classic rectangular surrogate gradient function descent in SNNs for medium neighborhoods and thus theoretically justify the method, and be useful in finding an optimal neighborhood in which to approximate a gradient for SNNs.  This last step is then used to optimize a continuously varying surrogate spike gradient estimator (Dspike), modeled as a tanh approximation with the gain as a free parameter. Using the chosen Dspike function (which depends on the time step), the authors then demonstrate the appreciable accuracy and energy efficiency improvements of the resultant SNN.
main_review: The authors' approach is original (to the reviewer's knowledge), intuitive, and of significant interest. The approach and results are well motivated, well written, and understandable, and appear to represent an appreciable advance in the field of SNNs.  However, some of the comparisons that have been made leave some room for questions as to the specific improvements that Dspike provides. Indeed, some of the improvements may be greater than suggested. Addressing this would be helpful for readers to better assess the impact of the authors' work.

In particular, Table 2 shows that Dspike achieves greater accuracy than all other surveyed methods.  However, the comparisons vary in important categories, in particular, architecture. Dspike may well do better than suggested compared to STBT-tdBN for both CIFAR 10 and CIFAR 10-DVS, since the former method utilized ResNet-18 and the latter ResNet-19.  Meanwhile, the comparisons to BinarySNN and Diet-SNN are less clear since they use VGG-15 and VGG-16, respectively. It would be helpful if, in these comparisons, the comparisons involved only a change of method and not architecture as well. Note that Table 4 does indeed allow such comparisons, and the resultant improvements are potentially somewhat more modest.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies the differentiable training problem of spiking neural network (SNN), which is a promising alternative to the widely used ANN. The authors first identify the problem of the current surrogate gradient of SNN and propose a new surrogate gradient estimator. I really like the author's research method of thinking about problems from experiments, but the result of the experiment is just a tuning experiment of some parameters, and I did not see the author's insight into the existing SNN training problems. Furthermore, the Spike method proposed by the author lacks clear motivation. The proposed Dspike method is just a differentiable piecewise function, which does not have much methodological contribution. The author exaggerated the experimental settings and did not present the experimental results of neuromorphic images. In summary, I think this article does not meet the standards of the NeurIPS conference.

The author’s reply addressed most of my concerns. I agree that AdderNet is another line of work with different purpose. I can see that the authors are very thoughtful in the model design, however it is not well-described in the main paper. I have raised my rating by 1.
main_review: Main Comments:
1.	AdderNet [a] is also a multiplication-free neural network architecture. What is the advantage and disadvantages of Spiking Neural Network?
2.	How did Time-dependent Batch Normalization (TdBN) relevant to this paper? What is the motivation for using time-dependent batch normalization? The authors argued that TdBN is designed to normalize spatial-temporal data (Line 103-104), Are there any experimental results that argument?
3.	What is the motivation of the Time Inheritance Training (TIT) proposed in Section 4.4? 
4.	The authors argue that they conduct experiments on both static and dynamic datasets (Line 51), however, the authors only reported results on static datasets (CIFAR and ImageNet Sec 5.1 and Sec 5.2), and I didn’t find any results related to dynamic dataset (including Supplementary materials ). Where are the results on the dynamic dataset? In Line 235, the authors mentioned a “neuromorphic image dataset”, is that the dynamic dataset? Where are the experimental results on that dataset?

limitations_and_societal_impact: Yes, the authors addressed the limitations and potential improvements.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new method called Differentiable Spike (Dspike) to compute the surrogate gradient for backpropagation of SNNs. In addition, an online optimization method is proposed to optimize the hyperparameter of Dspike. 
main_review: Strength: I appreciate that this paper demonstrates results on more complicated datasets like CIFAR 100 and ImageNet.

Weakness: The proposed method is still in the frame of surrogate gradient.

Please see the comments below for details.
limitations_and_societal_impact: 1. The main issue of this paper is novelty. Overall, this main idea of this paper is a surrogate function with the dynamic hyperparameter tuning method to enable wide range of smoothness. However, it is still in the framework of surrogate gradient method and I cannot view it as a significant improvement over previous works. This work reports a good results on several challenging datasets like CIFAR10-DVS, CIFAR100 and ImageNet. I appreciate the authors' efforts in the experiments and I think the performances reported in this paper are better than state-of-the-art results. However, in my opinion, this is more about engineering works for running experiments and hyperparameters tuning.

2. For the FDG computation of the experiments, the authors set epsilon to 0.1 which is empirically coming from the toy experiments on MNIST. However, I think this parameter should depend on datasets and cannot find a universal optimal choice. Can the authors comment on why the epsilon is fixed in the experiments?

3. In table 2, I hope the authors can compare the proposed method with existing works using the same network size or similar number of tunable parameters. Thus, we can understand the improvement brought by the proposed method.

4. For section 5.3, I hope the author can analyze the complexity of the proposed method compared to the existing methods. For example, compared with the STBP method, how much improvement the proposed method can achieve and how much additional cost do the dynamical temperature tuning and time inheritance training method require?
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Summary.

This paper attempts to facilitate the surrogate gradient descent in training SNNs. The proposed FDG quantitatively analyzes the gradients in the training procedure. And then, the authors propose a Dspike function, which is claimed as a proper searching function for the optimal surrogate gradients.  Numerical experiments demonstrate the superiority of the proposed models to several contenders in CIFAR and ImageNet data sets.

Contributions.

I have to admit that it's hard to understand and follow the objective and soundness of this paper. At first glance at the Tables in Section 5, the proposed methods achieve a commendable result compared to other SNN models. However, I can't entirely agree with the author's claimed contributions, especially the first and second ones.

As claimed that "through which (FDG) we are able to not only explain why it (surrogate gradient) works well but also indicate how we can improve its (gradient) adaptivity to data through training", so what are the answers of why and how? In detail,

[1] The derivation of FGD can be traced back to the directional derivative and difference in mathematics. It is reasonable to use Eq. (10) to approximate the instantaneous derivative at a certain point in ANNs, which have continuous and differentiable activations. The high-similarity curves in experiment $\mathrm{I}$ verify my conjectures. It's a good point.

[2] However, the finite difference technique is not suitable for estimating the derivative of holder-continuous and non-differentiable functions. There may be a large gap between the instantaneous and average derivatives. Thus, the similarity between FDG and surrogate gradients in SNNs cannot support the "soundness of surrogate gradients" in Line 175 due to the imprecision of FDG.

[3] Further, how to conclude that "This observation concludes that surrogate gradient can predict the change rate of loss in a relatively larger neighborhood" between Lines 175-176? Doesn't dissimilarity mean invalidity? I hope the authors can clarify this point.

[4] My conjecture in [2] can be verified by Figure 2. At the initial state that epoch = 0 or 1, parameters $\mathbf{W}$ may be far away from the optimal solution. In this case, FDG is not similar to the surrogate gradient if employing small $\epsilon$. As the setting in the experiment that "weight decay (L2 regularization) is set to 0.0001", the parameters may converge to the optimal local minima as weights decay to 0.0001. Thus, FDG still works, although $\epsilon$ is small. So it may be a practical trick but not holds as the authors theoretically claimed, right?

About the claim that "optimal surrogate gradient can be found by choosing temperature wisely" in Lines 210-211. In other words, the Dspike function is effective since optimal surrogate gradient can be found by choosing temperature wisely. How to evaluate this point? I would have appreciated more clarity on the experimental results not as omitted by "As did in former toy experiments (Sec. 4.2), we verify the cosine similarity between our Dspike gradients and the FDG at the beginning of each training epoch and update the  temperature to better match the FDG."

In summary, I find the interests of this work and the proposed model may be effective in practice. However, this conclusion should be testified either theoretically or experimentally. Of course, this paper still needs improvement in these two aspects. So the current version is improper for acceptance.
main_review: Clarity and Quality.

[1] The formalization and theoretical derivation of this paper are clear.

[2] The conclusion and proof of Lemma 4.1 are valid and correct. Unfortunately, this lemma only describes the properties of Dspike function. Can the function set of Dspike functions with learnable parameters contain the surrogate gradients obtained by the common-used approaches, such as Eqs. (7) or (8)?

[3] What's the computational complexity of FDG? It is better to show the number of paras, computational overhead, and storage costs.

[4] As shown in Table 2, adding TIT will reduce the performance of Dspike. TIT is not the contribution of this work; why do this here?

[5] The experimental results in Table 2 are not promising. Taking the results in CIFAR-100 as an example, both architecture and Dspike functions are different. So how is the improved accuracy attributed to the power of the Dspike function with FDG? In my mind, ablation experiments are desired. Besides, the comparative models are so scant.

limitations_and_societal_impact: As mentioned above.
review_title: null

Meta-review: The authors present a method to train spiking neural networks (SNNs) using an adaptive surrogate gradient method.
Training of SNNs is difficult due to the well-known problem of non-existing derivative of the spiking neuron output. This is usually tackled using a surrogate gradient method. However, the functional form of the surrogate gradient is quite arbitrary.

The authors first analyze the training behavior of SNNs. They propose to use a parameterized function for the surrogate gradient (sg) and estimate its best actual shape based on a comparison with a finite-difference gradient. Then they use this sg for parameter updates. They test their method on challenging data sets such as CIFAR100 and ImageNet and show excellent performance.

The paper is very well written and clear. The adaptive surrogate gradient is a novel contribution that has potential to impact future SNN training methods. The experimental results are compelling.

Weaknesses and potential improvements: 
The functional form of the adaptive surrogate gradient lacks motivation.
Reviewers have identified some unclear or over-stated statements. The authors should refine these statements accordingly. 


Decision: Accept (Poster)