Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a NAS optimization algorithm for SNN search.
strengths_and_weaknesses: +The authors present interesting results with the differentiable NAS search.
-There are two major works related to NAS for SNNs that has been recently out [5], [11]. The authors have not cited these works. It makes me wonder what is the author's contributiona s compared to these works. [5]  talks about the fact that training SNN using standard NAS methods might be too complex because SNNs need large training time, so they come up with a NAS without tarining technique. [11] talks about a differentiable NAS technique. Both works show good results on a avraiety of datasets, and talk about the intricacies of architecture search.
-The authors have also compared their technique to select works in table 1.  There is a lot of work from Priya Panda's group at Yale, Emre Neftci's group, and many others with regard to SNN training that show SOTA results on DVS and static datasets. The authors have failed to acknowledge most recent works.

Below is a list of publications (not exhaustive) that the author should check:
[1] Towards spike-based machine intelligence with neuromorphic computing K Roy, A Jaiswal, P Panda Nature 575 (7784), 607-617

[2] Enabling spike-based backpropagation for training deep neural network architectures C Lee, SS Sarwar, P Panda, G Srinivasan, K Roy Frontiers in neuroscience, 119

[3] Rate Coding Or Direct Coding: Which One Is Better For Accurate, Robust, And Energy-Efficient Spiking Neural Networks? Y Kim, H Park, A Moitra, A Bhattacharjee, Y Venkatesha, P Panda ICASSP 2022-2022

[4] Neuromorphic Data Augmentation for Training Spiking Neural Networks Y Li, Y Kim, H Park, T Geller, P Panda arXiv preprint arXiv:2203.06145

[5] Neural architecture search for spiking neural networks Y Kim, Y Li, H Park, Y Venkatesha, P Panda arXiv preprint arXiv:2201.10355

[6] Optimizing deeper spiking neural networks for dynamic vision sensing Y Kim, P Panda Neural Networks 144, 686-698

[7] Federated Learning with Spiking Neural Networks Y Venkatesha, Y Kim, L Tassiulas, P Pand IEEE Transactions on Signal Processing 2021

[8] Beyond classification: directly training spiking neural networks for semantic segmentation Y Kim, J Chough, P Panda arXiv preprint arXiv:2110.07742

[9] Revisiting batch normalization for training low-latency deep spiking neural networks from scratch Y Kim, P Panda Frontiers in neuroscience, 1638

[10]Na, Byunggook, et al. "AutoSNN: Towards Energy-Efficient Spiking Neural Networks." arXiv preprint arXiv:2201.12738 (2022).
questions: See my above comments.
limitations: See weakness section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work is aimed to search for both the optimal SNN architecture and hyperparameters of surrogate gradient (SG) functions. In the architecture search phase, they use DARTS and refine the search to different granularities (layer-level and cell-level). The search for SG function (DGS) focuses on optimizing the temperature of the Dspike SG function. The results show that searched architecture achieve SOTA  performance on image classification and event-based stereo matching task.
strengths_and_weaknesses: **Pros**
1. The search for the architecture alone significantly increases the performance of image classification tasks, which reveals the potential to be applied to various more complicated tasks.
2. The idea of searching hyperparameter of SG function is novel, simple but effective.

**Cons**
1. The idea of applying NAS on SNNs is not novel till the deadline of NeurIPS submission. SNASNet[1] and AutoSNN[2] have proposed that NAS methods can be used for searching the structure of SNNs. The latter has been accepted at ICML2022.
2. The articulation of the training pipeline is not highlighted and is somewhat unclear to me. See the **Questions** below.
3. The trials of search on SG functions are confined to the Dspike function.

[1] Youngeun Kim, et al. "Neural architecture search for spiking neural networks." *arXiv preprint arXiv:2201.10355* (2022).

[2] Byunggook Na, et al. "AutoSNN: Towards Energy-Efficient Spiking Neural Networks." *arXiv preprint arXiv:2201.12738* (2022).

questions: 1. The comparison of previous NAS work on SNNs should be added.
2. The training pipeline is unclear to me. Does the training pipeline include **both** ANN-to-SNN conversion and direct training using SG? I notice that L158 suggests that ANN layers are converted to SNN layers. If so, the comparison to some previous work using only directly trained SNNs is not fair.
3. Why Dspike is the only tested SG function? As mentioned in ref. 64, the choice of SG function affects the viable range of hyperparameters. The range determines the difficulty level of search. As there has been no claimed best SG function till now and ref. 64 points out that some different SG functions actually have the same maximum performance, the author should try some popular SG functions like SuperSpike[1], triangle[2], ArcTan[3], etc., to show the versatility of this method.

[1] Friedemann Zenke, and Surya Ganguli. "Superspike: Supervised learning in multilayer spiking neural networks." *Neural computation* 30.6 (2018): 1514-1541.

[2] Guillaume Bellec, et al. "Long short-term memory and learning-to-learn in networks of spiking neurons." *NeurIPS* (2018).

[3] Wei Fang, et al. "Incorporating learnable membrane time constant to enhance learning of spiking neural networks." *ICCV*. 2021.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors propose a differentiable hierarchical search framework for spiking neurons, where spike-based computation is realized on both the cell and the layer level search space. Meanwhile, the authors find effective SNN architectures under limited computation cost. In order to avoid the standard SG approach that leads the network into suboptimal solutions, the authors propose a differentiable surrogate gradient search method where the SG function can be efficiently optimized locally in parallel. Finally, this work shows some interesting results on the image classification tasks.
strengths_and_weaknesses: Strengths:
1. A hierarchical differentiable surrogate gradient search framework is proposed to obtain better performance of the spiking model.
2. Significant improvements in energy savings on deep stereo.

Weakness：
1. In terms of writing, some methods that were not proposed in the work were placed in the methods section. There are also some typos in terminology.
2. The results of the ablation experiments and the analysis of some elements do not match.
3. The font of the figure seems to be a small and not clear enough, which leads to a very careful reading to find valuable information.
4. The percentage improvement of the proposed method varies greatly on the two image classification datasets. Even the improvement on CIFAR-10 is only 0.18.
questions: 1. My core concern lies in the choice of the surrogate gradient (SG) function in the methods section. First, it looks like the SG function is from existing work [31]. If this is true, the reviewer suggests that the authors put this subsection in the background section. Second, what is the motivation for choosing such a SG function? Because, the motivation for the design of the SG function has been given in the existing studies [1-2]. It is clear that SG methods are a key part of the direct training of spiking neurons, and the choice of such methods also affects the search results, and the reviewers suggest that the authors explain the motivation clearly.

[1]"Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks," Nat. Mach. Intell. 3(10): 905-913 (2021)

[2]"A Hybrid Spiking Neurons Embedded LSTM Network for Multivariate Time Series Learning under Concept-drift Environment," in IEEE Transactions on Knowledge and Data Engineering, doi: 10.1109/TKDE.2022.3178176.

2. The parallel optimization involved in the search process needs further explanation by the authors.

3. There is a significant difference in performance improvement on the CIFAR-10 and CIFAR-100 datasets, and the authors need to explain the reason for this phenomenon.

4. The ablation experiments concluded the important value of temporal dynamics in the spiking neuron, but why the MDE metric and Median depth error metric appear opposite on split3. i.e., DARTS-BN^D better than DARTS-SNN^D in terms of Median depth error metric-split3.

5. In Table III, the experimental results show that the power consumption of the proposed method is significantly lower. However, the reviewers note that the SNN model is tested on a chip and whether it is the device that leads to a more recent reduction in power consumption.

limitations: The authors illustrate the limitations of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this submission draft, the authors device a differentiable hierarchical search framework tailored for SNNs. In the meantime, this framework is able to search the surrogate gradient in a differentiable manner. Their methods are validated on the CIFAR dataset and an event-based deep stereo dataset. 
strengths_and_weaknesses: Overall this is an interesting work. The authors come up with an end-to-end differentiable framework that solves two critical problems in SNN: the architecture and the surrogate gradient. 


1. Developing SNN-oriented architectures are novel and necessary. Even though this work is not the first trial in the community. 
2. Searching the SG is interesting and I am glad to see a learning-based method to address the issue. 
2. The results on the CIFAR10/100 dataset are promising.
questions: 1. Can authors elaborate on the search space in this work, especially when compared to the search space in DARTS and NAS-Bench-101?

2. Can the authors provide more details on the difference in searching gradients with Dspike in Section 3.4? Currently, the only saying is *" [31] demonstrates that by optimizing the width (or temperature) of the SG function the performance of SNN can be improved"*, but there is no drawbacks analysis and to which degree that this DGS method can improve. 
limitations: 1. Need to include two prior SNN NAS papers in the discussion or experiments. See references below.


2. A critical problem is that there is no comparison between the searched architecture and the ResNets used in other works. What if the searched architecture has a higher capacity than ResNets?

3. An ablation study on the DGS is recommended, the authors should compare static temperature gradient, DGS, and [31] on the same neural architecture and under the same training receipt.

4. Better to have an ImageNet result. 



------

**References**

Na B, Mok J, Park S, et al. AutoSNN: Towards Energy-Efficient Spiking Neural Networks[J]. arXiv preprint arXiv:2201.12738, 2022.

Kim Y, Li Y, Park H, et al. Neural architecture search for spiking neural networks[J]. arXiv preprint arXiv:2201.10355, 2022.
review_title: null

Meta-review: metareview: This paper proposes a new architecture search algorithm for spiking neural networks (SNNs). The key insight is to optimize both the cell and the architecture level of the SNN. Convincing numerical results are provided on image classification tasks (CIFAR10, CIFAR100, and an event-based stereo task). 

One concern raised by the reviewers regards the comparison to existing work (some of which appears to be very recent). This point is raised by all the four reviewers (although it has led to a rather large variance in their initial assessments). After an in-depth discussion between authors and reviewers and a discussion between AC and reviewers as well, it appears that this concern has been addressed in a satisfactory way. Other concerns (e.g., training pipeline and versatility by reviewer cjsQ) have been also resolved, and the remaining ones (measuring energy accurately as mentioned by reviewer LhUf, and computational overhead on neuromorphic hardware as mentioned by reviewer hUzC) have been regarded as out of scope.

In summary, the reviewers have found the authors’ response convincing and have reached a consensus towards accepting the paper. After my own reading of the manuscript, I agree with this assessment and I am happy to recommend acceptance. As a final note, I would like to encourage the authors to include in the camera ready the discussions related to the feedback from the reviewers. 

confidence: Certain
Decision: Accept