Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an effective way to train the spike thresholds of spiking neural networks, while the naive surrogate gradient method fails to do so. The proposed method achieves SOTA performance on several image classification tasks and an object detection task.
strength_and_weaknesses: Strength:

1. The proposed method is simple, and can be directly plugged into other existing methods.

2. There are experiments on non-classification tasks. It is encouraged to explore the performance of SNNs on various tasks.

3. It is interesting to see the potential of quantized small-scale SNNs.

Weaknesses:

1. In table 2, does "GPW for all" mean that all neurons share the same learnable Vth? And does "SGP" mean that each layer has its own Vth? For the experiments shown in Fig. 2, does each layer have its own trainable Vth?
For each experiment, the authors need to make clear statements on whether all the neurons share the same Vth, or each layer has its own Vth, or each neuron has its own Vth. I also suggest the authors to combine eqns. 6 and 11 in one place, and clarify the range of the summation in eqn. 11.


2. I think the catastrophic performance drop in fig. 2a is not hard to understand. Let me first guess that each layer has its own trainable Vth. For one layer, you need to sum up dL/dVth_i for each neuron i in that layer to get dL/dVth. Since dL/dVth_i, dL/du_i, and dL/dWij have the same order, |dL/dVth| will be hundreds of or thousands of times large than |dL/dWij|, depending on the model width. Therefore, a learning rate suitable for updating Wij may be too large for updating Vth. So we can easily overcome the issue by using a small learning rate for Vth (e.g., set init lr=0.1 and 0.001 for W and Vth, respectively). Your proposed method basically shrinks dL/dVth_i with a sigmoid function, so it has a similar effect as using smaller lr.  Can you compare the two methods experimentally? Can you show the benefit of your proposed voltage-dependent shrinkage scheme?

3. I am concerned about the direct performance gain of the proposed method. The authors do not show the performance when disabling threshold training and keeping other settings the same. So it is not clear whether the sota results are achieved due to the proposed method or, say, network structures. Table 2 seems to show the direct effect of the proposed method, but the baseline uses a different network structure (the title of table 2 is misleading!). I would appreciate it if the authors could conduct ablation studies.

Minor: 
1. Fig.1: DSR-VGG11 adopts learnable thresholds, as repeatedly pointed out in the manuscript.
2. In eqns. 6 and 9, a minus sign is missed. ds/du and ds/dVth have opposite signs.
clarity,_quality,_novelty_and_reproducibility: Clarity: clear, but should state the training setting more clearly in sec 5. Refer to weakness 1.

Quality: This work makes sense, but does not provide juicy information.

Novelty: The idea of training thresholds is not novel. But the authors make the idea work for the surrogate training framework using a simple trick.

Reproducibility: I think the method is easy to implement.
summary_of_the_review: I give a "reject" because:

1. I challenge sec 4 and the motivation of the proposed method totally. You do not need to create a hypothesis and then give some observations. The issue is easy to handle: shrink dl/dVth or shrink the corresponding lr. See weakness 2.

2. The experiments are not carefully designed. I give some examples as following. In fig. 2(a), you do not fine-tune the lr for training vth (see bullet 1). In tables 1 and 2, DSR and TET use vgg-11, but you use vgg-9 (btw, I am confused about the term "true binary spike", DSR just uses binary spike). No ablation studies, which are super important in this work, see weakness 3.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors address the thorny problem of training spiking deep neural networks that are marred with problems of non-differentiable nonlinearity, sparse responses and limited expressive power. The authors propose an adaptive threshold mechanism so that the threshold is trained using a different differentiation path from that used for the weight updates.
In extensive experiments the authors demonstrate advantages of their approach.
strength_and_weaknesses: 
Strength 
The paper offers a new view on learning adaptive thresholds in SNNs. Their SNNs acheve competitive performance on standard tasks even with by an order of magnitude fewer parameters. The paper is well organized and clearly written.

Weaknesses
While the authors bring up the question of biological relevance of SNNs and their learning mechanisms, they don't elaborate on this topic and don't provide evidence that their approach is more biologically realistic as compared to other SNN training schemes.
The authors do not comment if their code will be publicly available raising questions about replicability of their research.


clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and quite convincingly. 
summary_of_the_review: This is an impressive piece of work substantially improving on SOTA SNN training. However, questions remain about replicability of this work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a layer-wise learnable threshold approach for SNNs. The proposed method design two separate surrogate gradient paths for the gradient of membrane potential and learnable threshold. Different classification and object detection tasks reflect the effectivenesses of the proposed LTSNN training scheme.
strength_and_weaknesses: 
Weaknesses and questions:

-1. The experiments recorded in Table 1 are not clear to me. 

       a. Why do "such constraints in the training process largely limits the learnability of SNN"? What's the definition of "learnability"?
       b. What is the definition of "true binary spikes"? Biologically, an action potential is triggered when the membrane potential is higher than a threshold. The output potential should NOT be a binary value in any case. The spike (action potential) is binary. 
       c. How hardware deals with layer-wise varied spikes is out of the scope of the paper. I assume the "hardware" means neuromorphic hardware. Biologically, each neuron has a different dynamic threshold. The current neuromorphic hardware can barely mimic the simplest biological neuron. Thus, algorithm and theoretical study should not be limited or favored to existing neuromorphic hardware. 
       d. DSR and LTSNN used different base models, VGG-11 for DSR and VGG-9 for LTSNN. Is it fair?

-2. The falsified hypothesis 1 is based only on VGG-9 and DVS-CIFA10 is not convincing. In other words, how general of observation 1? In addition, I am not convinced by the experiments associated with Figure 2. If we stop training STSG before 80 epoch, STSG also has stabilized training, right? In other words, why the authors stopped training LTSNN at epoch 100? What would happen if the authors train LTSNN 200 epoch?

 -3. Super unclear of section 5.3! The L in Eq. 12 is the same as the one in Eq. 13? What's L_{CE}, MSE, O(t), and T? More importantly, why the MSE regularizer is necessary? 

-4. For the experiments associated with Table 4, why did the authors not use the same architecture of other competing approaches? What's the logic behind the experimental design?
clarity,_quality,_novelty_and_reproducibility: The motivation for the work is very unclear to me at the moment. In addition, I did not see much novelty either. Writing needs significant improve!
summary_of_the_review: Too many things are unclear to me at the moment, as discussed in the section on "strengths and weaknesses." In addition, many works have developed different dynamic threshold schemes for SNNs. The authors did not mention them at all. For example, in "Biologically Inspired Dynamic Thresholds for Spiking Neural Networks," the authors define dynamic thresholds without learning. More importantly, I do not think the proposed approach is able to imitate "the self-adaptiveness of the biological nervous system." 

In summary, I do not recommend the submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper propose LT-SNN, a novel SNN training algorithm with self-adaptive learnable potential threshold to improve SNN performance. To stabilize the SNN training even further, the author propose separate surrogate gradient path (SGP), a simple-yet-effective method that enables the smooth learning process of SNN training. The proposed method is validate on multiple event-based datasets, including both image classification and object detection tasks.
strength_and_weaknesses: Strength

1. The proposed method is shown to outperform SOTA on multiple event-based datasets.
2. The paper is well written and easy to follow.


Weakness

1. I don't see a clarification for the problem importance. Why learnable threshold is important in SNN? If the authors want to make SNN more biologically plausible, should the threshold lie in the specific range? I don't see any discussion or insight in the paper.
2. The proposed method is not well motivated. For example, the authors propose a separate gradient path in sec. 5.1, but why we should use a separate path instead of single one? Is there any insight or theoretical guarantee? Also, for Eq.9, the authors select the sigmoid function as the activation. Why would the authors select this particular activation function? Is there any biologically insight?
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is clearly written.

Quality:
The experiments show good results. But the problem importance and  the motivation of the proposed method is missing.

Novelty:
The novelty is trivial.  

Reproducibility:
The authors fail to provide code.
summary_of_the_review: Although the paper gets better results with more parameters, the problem importance and the motivation of the proposed method is unclear. Therefore, I give a weak reject for the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel training method for Spiking Neural Networks. Typically, the non-differentiability of the spiking neuron was by-passed with a differentiable approximation (e.g. the triangle function). However, each spike depends on two variables: the previous neuron's voltage, and a voltage threshold above which the spike is created. In prior works, the network optimized over the voltage during training, but kept the voltage threshold as a constant, due to training instability issues. This work resolves these instability issues by introducing a separate gradient approximation for the voltage threshold, consisting of the triangle function modulated by a sigmoid function. During backpropagation, the voltage and voltage threshold will receive separate gradients based on this method. Empirical experiments are performed to demonstrate the stability of this method, and the authors demonstrate significant improvements in experiments for image classification and object detection over prior works, with much smaller model sizes.
strength_and_weaknesses: Overall, the motivation and method is very intuitive, and the proposed solution is simple and makes sense. The authors provide a good amount of background, and justify well the need for the proposed contribution. While the method is quite simple, the improvements in the experiments are quite impressive, especially considering the significant reduction in model size. 

One open question is how much the regularization in Section 5.3 contributes to the work. It seems to only be briefly mentioned, but is not ablated, and it's unclear how important this is to the improvements seen in the experiments.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper itself is well written, and the simplicity of the method lends itself to ease of replicability. Overall, the work seems original, and the results are impressive.
summary_of_the_review: Overall, the proposed method seems like a simple modification to existing SNN networks which provides a significant improvement. However, I may be missing some context as to the novelty of this work.

EDIT: After reviewing the discussions from the other reviewers, it seems like there are legitimate concerns about the novelty and experiments of this work. Taking this in mind, I am modifying my review to a 5 (although more of a 5.5).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper originally got two 8s (accept, good paper), one 5 (marginally below threshold) and two 3s (reject). The major challenges inclide unconvincing (and even unfiar) experimental comparisons, unconvincing motivation of the proposed method, unsatisfactory paper writing, limited novelty, etc. There are many discussion between the authors and reviewers. After rebuttal, one reviewer agreed on the novelty and experiment issues and downgraded from 8 to 5, while the other reviewer raised from 3 to 5. Since only one reviewer was positive towards the paper, the AC deemed that the paper is not good enough for ICLR, thus recommended rejection, yet thanked the authors for taking great efforts on the rebuttal.
justification_for_why_not_higher_score: The paper should be rejected for sure.
Decision: Reject