Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors propose a conversion algorithm for deploying SNNs.
strength_and_weaknesses: -Not novel and results are not good. Today, many works are focused on direct training precisely to reduce the overall timestep count in SNNs that can lead to very expensive timestep computations during inference[1]. The authors work still gives best conversion accuracy at T>100. This is not acceptable anymore. I suggest the authors to go over recent conversion works that have strived to created conversion SNNs at T<10 timesteps [2].

[1] Yin, Ruokai, et al. "SATA: Sparsity-Aware Training Accelerator for Spiking Neural Networks." arXiv preprint arXiv:2204.05422 (2022).

[2] Zheng, Hanle, et al. "Going deeper with directly-trained larger spiking neural networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.
clarity,_quality,_novelty_and_reproducibility: Not novel at all!
summary_of_the_review: In summary, this paper's attempt is worthy to be credited but it really lacks good motivation and novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper improves upon the conversion error of two step ANN to SNN conversion based technique to yield sota accuracy.  In particular, it presents an efficient ANN-SNN conversion mechanism based on the SlipReLU and shifted SlipReLU activation function replacing the traditional ReLU to improve the accuracy of the converted SNN. The results look promising and competitive compared to the related works. The theoretical analysis of error strengthens the paper.
strength_and_weaknesses: ### Strengths
=============

Well written.

Well motivated.

Result is good and inspiring.

Analysis of the various conversion error is present in details. Some are taken from earlier research though.

### Weakness:
===============

The experimental results section is weak. The authors should provide results on larger datasets to show the efficacy.

The author should do an ablation study with SlipReLU and shifted SlipReLU.

It would be interesting to see how such ReLU performs under various noisy inputs as discussed in [1-2], particularly, whether they can also maintain some inherent robustness with such non-linear functions, under two-step conversion based framework.

Please put some light on to the selection of input type (direct, rate etc.) on the performance with  SlipReLU.

lack of reproducibility and missing details of experimental setup.

[1] Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations, ECCV 2020.

[2] HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training With Crafted Input Noise, ICCV 2021.
clarity,_quality,_novelty_and_reproducibility: clarity: 9/10
Quality: 8/10
Novelty: 8/10
Reproducibility: 8/10
summary_of_the_review: Overall, this is a well written well motivated paper that requires further experiments to clarify the impact of the proposed non linearity in SNN paradigm.

Post rebuttal: the contributions are new with detailed additional results, major concerns are addressed with results on further complex datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors devise a two-step conversion framework for ANN-SNN conversion. Unlike previous methods which consider the ANN-SNN conversion error, this work considers the error between source ANN and altered ANN as well as altered ANN with SNN. The idea is generally sound and simple to understand. Yet in my view, the two-step conversion naturally has some deficiencies with than one-step conversion (see weakness section for details). 



strength_and_weaknesses: Strength:

+ The structure and presentation of this paper are neat and easy to understand.

+ The perspective of the trade-off between the distance to ANN and the distance to SNN is novel and an interesting activation function is proposed (SlipReLU). 

Weakness:

- The major concern in this work is the first term in Eq. 5, which hypothesize that the output between ReLU-based ANN and customized ANN should be considered, is not fully verified in experiments. As far as I interpret Eq. 5, the SlipReLU should have higher ANN accuracy but also higher conversion error than QCFS, because it has less distance to ANN while more distance to SNN compared with QCFS. 
And ideal result in SlipReLU is that it shows higher ANN accuracy and despite the higher conversion error, SlipReLU can still achieve higher SNN accuracy than QCFS. However, the results in paper does not verify the tradeoff or did not explicitly discuss the tradeoff in experiments. Ablation studies on the weight of combination, and more analysis on ANN, SNN accuracy tradeoff should be provided. 

- Two-step conversion certainly has its pros, but it also has cons. Two step method must train their own ANN models, while one-step models can utilize directly-trained checkpoints in open-sourced projects. Some large-scale datasets and models, are require a lot resource to train the ANN, for example, ImageNet. Thus, ImageNet results are also anticipated. 




clarity,_quality,_novelty_and_reproducibility: + Clarity: This paper is sufficiently clear. 

+ Quality: This paper's quality is good. 

+ Novelty: An extension from previous work, but it needs more exresults to support its correctness. 

+ Reproducibility: Not reproducible at this time. 
summary_of_the_review: My reasons for the weak rejection are insufficient analysis and the lack of large-scale experiments. Were they addressed by the authors, I can increase my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an efficient ANN-SNN conversion mechanism based on the SlipReLU activation function. It replaces the traditional ReLU to improve the accuracy of the converted SNN. The results look promising and competitive compared to the related works.
strength_and_weaknesses: Strengths:
1. The tackled problem is relevant to the scope of ICLR.
2. The contributions look solid.
3. The results look promising.

Weaknesses:
1. It would be interesting to have more details about more intuitions about the design decision made when developing the proposed SlipReLU activation function.
2. The description of the proposed method in Section 4 looks vague and hard to follow. It is recommended to use schemes, and examples to ease the discussion.
3. Please describe the experimental setup and tool flow used to generate the results shown in Section 6.
4. It would be useful to provide the source code for reviewers' inspection during the rebuttal.

clarity,_quality,_novelty_and_reproducibility: Clarity: 6/10

Quality: 8/10

Novelty: 8/10

Reproducibility: 5/10
summary_of_the_review: Good paper where few concerns should be clarified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new activation function SlipReLU for ANN-SNN conversion to jointly reduce the conversion error and ANN performance loss. Reviewers have quite diverse opinions on the paper. One reviewer recommends strong reject due to the lack of novelty and not good results, as well as missing comparisons with direct training methods of SNNs. Two reviewers recommend accept for good results and one reviewer recommend marginally above the acceptance threshold after the rebuttal. After reading the paper, the AC found that some concerns should be noticed.

First, there exists many inconsistencies between the results in the paper and those in previous works. For example, the reported results and trends for different network architectures of QCFS (ICLR 2022) are inconsistent with those in the original paper. The authors explained that they adjusted the hyperparameters for each method, according to the suggestions of reviewers, to get better results than the original ones, but for the VGG-16 and ResNet-20 networks, results of QCFS in this paper are much lower than those in the original paper, and for VGG-16, the original QCFS results are even better than those of the proposed method in Table 1 (when T is small). This was not properly explained. Besides, for direct training methods, some of the reported results in the rebuttal and revision are different from those in the original papers. For example, the authors reported 88.23% accuracy for TSSL-BP but in the original paper the result is 91.41%. And for STBP, GDDP, IIR-SNN, LTH, and NAS, the reported results are all inconsistent with the original paper. Since there are so many inconsistencies, some being better and some being worse, the authors' explanation on QCFS and TET did not fully resolve the concern of the AC. 

Second, the large-scale ImageNet results are missing. For the conversion and training of SNNs, CIFAR results may not easily generalize to large-scale evaluation, especially with small time steps. These results are important and all previous ANN-SNN works reported results on ImageNet. The AC regreted that the authors could only provide results on Tiny-ImageNet due to computational constraints (but somehow the authors promised to provide the results on ImageNet if the paper is accepted).

Third, for theoretical analysis, a discussion on the assumptions is missing, e.g., the range of $v^{l}(T)$. This assumption is important to only consider quantization error in the analysis. However, in practice the assumption may not hold and there exists “unevenness error” (as stated in QCFS (ICLR 2022)), and recent works reveal that such kind of error is much larger than quantization error for deep networks (Meng et al., Training much deeper spiking neural networks with a small number of time-steps, Neural Networks, 2022). There should be proper discussions for the error analysis and important assumptions. This paper hardly goes beyond existing analysis results and does not theoretically show the advantage of SlipReLU. Moreover, the AC did not see much flavor of "unified optimization" as many of the steps were not deduced from optimization.

Given the above concerns and the overall scores, the AC deemed that the paper does not reach the acceptance threshold in its current form. So the AC recommended rejection, yet thanked the authors for taking great efforts in the rebuttal.

justification_for_why_not_higher_score: The paper is not good enough. The meta-review clearly stated the reasons. But the AC wouldn't mind it be bumped up as three reviewers were positive towards the paper and only one gave a harsh score: strong reject.
Decision: Reject