Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors proposed a learnable (authors called dynamic) threshold scheme and a moderate dropout method for SNNs. The effectiveness
of the proposed approach is reflected by classification tasks and DenseNet. Based on the experimental results, the proposed method requires fewer time steps for achieving state-of-the-art classification performance. 
strengths_and_weaknesses: Strengths:

1. Looking into authentic bio-plausible dynamic threshold schemes is a promising research direction for SNNs.
2. Even though the authors did not develop an actual dynamic threshold scheme, assigning different thresholds to different SNN neurons is interesting. 

Weaknesses:

-1. If I understand correctly, the proposed threshold scheme is not dynamically changed during inference. But, it is just learnable parameters during a training process. If so, I do not think it is a "dynamic" threshold scheme, which is just a learnable threshold scheme. 

-2. The validation mainly relies on DesneNet. But the authors used 8-layer FC SNN for MNIST. How does the DesneNet work with the MNIST dataset? Does the proposed method work with other architectures well, e.g., ResNet, VGG?  

-3. lines 282: "much fewer time steps needed" does not mean "suitable for processing real-time data." It depends on how much time spend on each time step. The logic does not hold here. The authors should report the processing speed for each time step.

-4. There are so many "-" in table 2. The authors did not explain why which I think is very necessary. 

-5. The authors claimed "less memory space" but without providing corresponding proofs. 

In summary, I think the validation is not convincing. If the proposed learnable threshold scheme only works with DenseNet, it is just an application. To make it more compelling, the authors should show the generalization of the proposed method. In addition, the authors should also show the experimental results of other tasks besides classification tasks. However, based on the current manuscript, I cannot see any generality of the proposed approach.  

questions: Please see weaknesses. 
limitations: 1. The proposed approach is not a dynamic threshold scheme. I suggest the authors call it a learnable threshold scheme. 
2. The authors should compare other existing dynamic threshold schemes to the proposed learnable threshold, e.g., r1 and r2.
3. The validation only based on DesneNet is very limited. The authors should provide more experimental results to show the effectiveness of the proposed approach. 


r1: Yunzhe Hao, Xuhui Huang, Meng Dong, and Bo Xu. A biologically plausible supervised learning method for spiking neural networks using the symmetric stdp rule. Neural Networks, 121:387–395, 2020.
r2: TaeyoonKim,SumanHu,JaewookKim,JoonYoungKwak,JongkilPark,SuyounLee,InhoKim,Jong- Keuk Park, and YeonJoo Jeong. Spiking neural network (snn) with memristor synapses having non-linear weight update. Frontiers in computational neuroscience, 15:22, 2021.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors use dynamic thresholding to incorporate heterogeneity in the neuronal parameters of spiking neurons. Dynamic thresholding utilizes learnable threshold values and enables flexible neuronal mechanisms across layers with proper information flow within the network, and fast network convergence. The authors also propose a moderate dropout method to serve as an enhancement technique to minimize inconsistencies between independent dropout runs. The authors show that these methods increase the performance and robustness of the model for image classification. 
strengths_and_weaknesses: Strengths:
The paper addresses a key problem that plagues current SNN research - i.e., homogeneity in the neuronal parameters. Though some recent papers have shown increased performance and robustness when heterogeneity is introduced in the neuronal parameters, the process of hyperparameter optimization is complex especially when the number of neurons is large. This paper uses a backpropagation-based method to find the optimal hyperparameters for dynamic thresholding which is great.

Though the performance of the model is promising, there are some key weaknesses that caught my attention.
Weakness:
The concept of introducing heterogeneity is not novel as several recent papers have introduced the concept and shown great results. 
The paper shows results on small datasets like MNIST, CIFAR10, NMNIST - similar results have already been shown in recent related works.
The authors introduce two very different concepts - dynamic thresholding and moderate dropout which seems to be two independent modifications both of which are supposed to greatly increase the computational load

questions: Since the authors introduce a dynamic thresholding technique, it would be great if the authors could show the increase in computational load due to this. 

Also, a comparative study on the training times for the DTMD with the static thresholding case would be also helpful. A table showing the percentage change in performance on complex datasets with the increase in training time/computational load (measured using MACs/FLOPs)

The illustration showing the KL divergence of different runs of SNN sub-models is a bit unclear and not properly explained in the main text. It would be great if the authors could furnish some more details. Especially, the authors are claiming the output of the spiking neural network as a probability distribution, on which they are taking the KL divergence which seems unclear.

The authors use the “KL divergence between the different output distributions from sub-network runs is incorporated into the loss function according to Eq. 12, with the overall objective being to minimize the loss, leading to increased consistencies among different runs of the network.” However, the authors add the two KL divergences to get the overall loss. However, this is still not a metric as it still does not satisfy the triangle inequality i.e., it does not measure the distance between the two distributions. Thus, I believe effectively trying to minimize the symmetrized KL divergence counteracts and nullifies each other. It would be great if the authors could show an ablation study with and without using the symmetrized KL divergence and using each of the asymmetric KL divergences also.

limitations: The authors did not discuss the limitations of their work or the potential negative social impact of their work in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors inspired by the neuroscience observation that different neuronal mechanisms exist in disparate brain regions to propose a new spiking neuronal mechanism, named dynamic thresholding. It is clear that the goal of this study is to mimic the brain's ability to build different types of neurons that facilitate the transmission of information in a deep spiking model. Overall, this work is interesting.

---------------------------------------------------------------
The author has addressed all my concerns and I would like to upgrade my rating.
strengths_and_weaknesses: Strengths:
1. The new dynamic thresholding mechanism.
2. Moderate random deactivation technique based on DenseNet structure is used to enhance SNN modeling capability.

Weakness:
1. Comparison and analysis with other dynamic thresholding techniques is lacking.
2. The authors claim that output decoding can enhance the robustness of the system, but there are no experiments related to robustness in the manuscript. The reviewer suggests that if possible the authors need to simply supplement the SNR based input noise experiments to verify the robustness of the system. Unless the robustness claimed by the authors is not in the traditional sense.
3. The moderate dropout uses KL fused into the loss function, but the cross-entropy loss of the classification task already includes this component. The reviewer suggests that  the authors elaborate further.
questions: 1. One of my main concerns is that the author gives a github link in the abstract that provides the code, but the anonymous link to github should have this generated (i.e., https://anonymous.4open.science/). I am not sure if this issue violates the double-blind policy. If it does not violate it, I think this article should be WEAK ACCEPT.
2. The authors should explain the problem that the proposed model on the N-MNIST dataset in Table 2 has a slightly improvement in both performance and timesteps.
limitations: The authors have clarified.
review_title: null

Meta-review: metareview: This paper introduces an trainable threshold and a dropout variant to improve training of spiking neural networks. There were serious issues raised by the reviewers primarily about 1) the idea of having trainable spiking thresholds being not new, and 2) the paper providing insufficient computational validation. The authors addressed the second point. About the first point, the authors provided the following: "The aim of our work is not to show that neuronal heterogeneity is a critical property for spiking neurons and can benefit SNN performance, some other works [R7, R8] have already introduced that. In this work, we want to show a practical threshold optimization methodology to enhance neuronal heterogeneity to those who are already interested in this special property of SNNs and expecting an easy implementation." The AC agrees that the paper achieves this goal, and, after much thought, agreed that this is a sufficient contribution for acceptance to NeurIPS.

In addition, as a minor point, the AC agrees with the opinion of Reviewer BGk9 that the term "dynamic thresholding" is misleading. 
confidence: Less certain
Decision: Accept