Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an interesting training algorithm for training SNNs from scratch with only one timestep.
strength_and_weaknesses: Very comprehensive results

Simple yet effective idea

Since the authors use a regularization technique, I am wondering if the authors can shed light on how their method differs from previous temporal BN methods proposed by prior works that have shown accuracy improvement while decreasing the total timesteps [1, 2]. 

[1]Kim, Y., & Panda, P. (2020). Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, 1638. 

[2] Zheng, Hanle, et al. "Going deeper with directly-trained larger spiking neural networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.

There is a plethora of works today on SNN algorithmic training- precisely talking about how we can get improved accuracy with less timesteps. But, I am more concerned by the fact that in such large-scale settings, are SNNs going to be actually advantageous? The authors show some simplistic energy estimation results which is grossly approximate. For true energy estimation, they have to consider memory access and data access energy which turns out to expend a lot of computations in SNNs given their repeated time-wise computation. In a recent work [3], true energy estimation on a systolic accelerator precisely shows that SNNs are not very advantageous over ANNs because repeated timestep computations will lead to redundant access of weights and membrane potentials is going to further add to energy unless we really improve the sparsity rate. Can the authors kindly comment on this - and it may be worthwhile for authors to include a discussion n the relevance of using more mainstream tools for energy estimation rather than just doing analytical modeling of FLOPS count?

 [3] Yin, Ruokai, et al. "SATA: Sparsity-Aware Training Accelerator for Spiking Neural Networks." arXiv preprint arXiv:2204.05422 (2022).

Coming to my next point, there is a recent work [4] that explores sparse SNNs using lottery ticket hypothesis or NAS [R5]  to truly take advantage of SNNs energy efficiency over ANNs. Can the authors comment on how their model in terms of parameter count compares to these sparse SNN models which in fact show SOTA accuracy on CIFAR10,100 with very low timestep count? 

[4] Kim, Youngeun, et al. "Lottery Ticket Hypothesis for Spiking Neural Networks." arXiv preprint arXiv:2207.01382 (2022).

[R5] Kim, Youngeun, et al. "Neural architecture search for spiking neural networks." arXiv preprint arXiv:2201.10355 (2022).

Finally, I think it is well known that SNNs will be more suited to DVS or event based tasks as compared to standard digital camera recognition models. Recent works have shown superiority of SNNs over ANNs on these neuromorphic datasets [5, 6, 7]. Can the authors run their model on one of these datasets and compare to [5,6,7]? 

[5] Li, Yuhang, et al. "Neuromorphic Data Augmentation for Training Spiking Neural Networks." arXiv preprint arXiv:2203.06145 (2022).

[6] Kim, Youngeun, and Priyadarshini Panda. "Optimizing deeper spiking neural networks for dynamic vision sensing." Neural Networks 144 (2021): 686-698. 

[7] Kim, Y., Chough, J., & Panda, P. (2022). Beyond classification: directly training spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineering.
clarity,_quality,_novelty_and_reproducibility: he contributions are clear and the results are good. I am not sure about novelty since it is a mix of methods that have existed in SNN/ANN literature and putting together seem to make the model better. -I am concerned about whether this is truly advanategous SNN framework as I have raised questions around SNN implementation and the sparsity in weakness.
summary_of_the_review: I am not very convinced by the novelty and true efficiency advantage of this method since many comparisons and experiments are limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to train one-time-step SNNs with a Hoyer regularizer and Hoyer spike layer. The proposed Hoyer spike layer uses adaptive threshold based on the Hoyer extremum of membrane potentials, and a Hoyer regularization for membrane potentials is added in the loss function. Experiments on static image classification and object detection tasks demonstrate competitive performance and energy efficiency.
strength_and_weaknesses: Strengths:

1. This paper conducts extensive experiments on static image tasks, including classification and detection, with different network structures, and considers many aspects such as accuracy, energy efficiency, and training and inference time. The effect of quantization of weights is also considered.

2. This paper considers comparisons with AddNNs and BNNs and reports promising results.

Weakness:

1. The “one-time-step SNN” in the paper is actually an ANN with binary activations. It does not consider temporal dynamics of neuron models, so it should not belong to “spatio-temporal computing paradigm” as mentioned in the first sentence of the abstract. It cannot handle temporal inputs such as speech or DVS data, which may possibly be more suitable for SNN with temporal processing ability. So this paper is more close to binary or quantized ANNs compared with general SNNs. The introduction, related work, and comparisons should have more focus on those ANNs than SNNs, and some introduction such as the dynamics of the LIF neuron model in Section 2.1 almost has no relation with models in this paper.

2. The motivation and the effect to introduce Hoyer regularizer and Hoyer spike layer are not clear enough. As introduced in Section 2.2, Hoyer regularizer is first introduced to induce sparsity. Then adding this regularizer for membrane potentials will encourage membrane potentials towards 0, implying that spikes are encouraged to be sparser. However, in Section 3.1, it is said that “it is crucial to reduce the value of the threshold to generate enough spikes for better network convergence”, which is *contradictory* to the regularizer that encourages fewer spikes. Why to combine these two opposite components and why it can work? Besides, there is no explanation for why “Hoyer extremum” is introduced to adjust the threshold and why it can work to balance the threshold and include proper data distribution as shown in Figure 1(a), why Eq. (4) considers clip function, why combining this “Hoyer extremum” with the trainable threshold can be better than the single trainable threshold? There lack significant explanations for the methods.

3. Some descriptions are imprecise. For example, in Section 3.1, “This is because if a neuron does not emit a spike, the pre-synaptic weight connected to it cannot be updated”. What does “pre-synaptic weight” mean? I suppose the authors mean that the synaptic weight will not be updated if the pre-synaptic neuron does not emit a spike when gradients of $w_{ij}$ from neuron $i$ to $j$ are calculated by $g_{u_j}*o_i$, where $g_{u_j}$ is the gradient for $u_j$. 
And in Section 3.3, “Note that unlike existing multi-time-step SNN architectures that avoid BN due to difficulty in convergence”. This is not up-to-date since most recent SNN works use BN to achieve high performance [1,2,3,4].

4. The energy results in Table 6 are strange. As introduced in the “Effect on Quantization” paragraph, transforming the full-precision ACs to 2-6 bit ACs will lead to 4.8-15.2 reduction in compute energy. Why in Table 6, there are more than 400-1000 times reduction? Not to mention that only convolutional layers are quantized as said in the paragraph.

5. The comparison with BNN and AddNN in Table 7 is not clear enough. The reported accuracy of the method is based on full-precision (also note that the full-precision results in Table 6 is inconsistent with Table 4 and Table 7) but the reported energy is based on 2-bits quantization, which is inconsistent. And how is the energy of BNN and AddNN calculated? Is AddNN also quantized? As for the proposed methods, even with quantization, it is unclear to me why the energy can be much lower than BNN with only pop-count operations. I think pop-count operations are much more efficient than ACs. These details should be reported and explained.

6. The reference style in the paper is with poor readability. The authors had better distinguish parenthetical and narrative types, i.e. \citep and \citet in the latex.

[1] Zheng, H., Wu, Y., Deng, L., Hu, Y., & Li, G. (2021). Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 12, pp. 11062-11070).

[2] Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., & Gu, S. (2021). Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 23426-23439.

[3] Deng, S., Li, Y., Zhang, S., & Gu, S. (2021). Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In International Conference on Learning Representations.

[4] Meng, Q., Xiao, M., Yan, S., Wang, Y., Lin, Z., & Luo, Z. Q. (2022). Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12444-12453).
clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: The introduction of the method details is clear, but there lacks a reasonable explanation for the motivation and effects of the method. Also, there are some questions regarding the experimental results and comparisons. (see above weakness)

Novelty: The Hoyer regularizer is introduced by previous works. The Hoyer spiking layer is new.

Reproducibility: Good.

summary_of_the_review: In summary, this paper lacks some significant explanations for the methods despite the good performance, and there are some questions about the experiments that should be tackled. Besides, the paper should have more focus on the more similar binary or quantized neural networks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a training framework (from scratch) for one- time-step SNNs that uses a novel variant of the recently proposed Hoyer regularizer. By estimating the threshold of each SNN layer as the Hoyer extremum of a clipped version of its activation map, the approach not only downscales the value of the trainable threshold, but also shifts the membrane potential values away from the threshold. The approach outperforms existing spiking, binary, and adder neural networks in terms of accuracy-FLOPs trade-off for complex image recognition tasks.


%%%%%%%%%%%%%

Update:
The authors have addressed part of my initial concern about bio-plausibility and limitation of BNN. Yet, I don't think the proposed method is that novel. So I change my score to 5. 
strength_and_weaknesses: Strength

1. The paper is well-written and easy to follow.
2. The experiments show the method with low-latency.

Weaknesses
1. The problem importance is unclear. One-time-step SNNs is simply binary network. The lack of dynamics make it less biologically plausible.
2. Limited novelty. The proposed method is simply training with HOYER regularizer, while similar method has been proposed in [1]


[1]Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. In International Conference on Learning Representa- tions, 2020.
clarity,_quality,_novelty_and_reproducibility: The paper is clear. The quality is fair. The novelty is limited. There is no code provided, so it is not reproducible.
summary_of_the_review: The paper fail to clarify the problem importance, and the proposed method is limited novel. So I vote for a reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a new method to train spiking neural networks (SNNs) with only one timestep. The method is based on a new regularization term ("Hoyer"), and a new threshold adjustment method ("Hoyer spike", Eq 4).


strength_and_weaknesses: STRENGTHS:

The accuracy they reach with only one timestep is impressive, better than the previous state-of-the-art.

WEAKNESSES:

The theory is unclear:

* "we estimate the value of the Hoyer extremum as He(ul) ="
I think what follows is not the value of H but the value of the element of u_l at the extremum.

* a SNN with one timestep is actually a somewhat degenerated case because there is no temporal integration. It boils down to a vanilla feed-forward artificial neural network, with Heaviside as the activation function. This also corresponds to the first neuron model proposed by McCulloch and Pitts and 1943, also known as threshold gates. This should be discussed.

* according to Table 5 what matters is the Hoyer spike, not the Hoyer regularization. This raises a question: could Hoyer spike be used alone, without the Hoyer regularization?

* does the approach extend to multistep SNNs?


clarity,_quality,_novelty_and_reproducibility: * Will the authors share their code?

* Eq 1 is misleading, there is only one type step here.

* |u_l| is described in the text before it's actually used in the equations

* what are the blue areas on Fig 1?

* what is the dataset in Table 1?

* https://arxiv.org/abs/2102.04159 and https://arxiv.org/abs/2007.05785 should be included in Table 4


summary_of_the_review: A potentially promising method, but some aspects are unclear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an SNN model with one timestep only. The key to the model is the use of the Hoyer extremum as a spiking threshold. The Hoyer regularization is a technique previously used for DNN regularization. The proposed SNN corresponds to a DNN with binary activation functions. The backprop pipeline includes the gradient of the threshold function (nondifferentiable), which is approximated to a simple boxcar function with a sufficient width (0-2) to cover the range of inputs. The performance of the proposed model was evaluated in terms of classification accuracy (on CIFAR 10/ImageNet and VOC2007) and computational efficiency. Given the use of binary activation functions, accuracy degradation was inevitable compared with the DNN counterpart. Instead, the authors place emphasis on computational efficiency that may be enhanced given the use of binary activations that may avoid FP multiplications, this is very arguable though. 
strength_and_weaknesses: Strength:\
The proposed model was verified on various datasets (CIFAR10, ImageNet, VOC2007) and compared with previous models including DNNs, BNNs, SNNs in a systematic manner.

Weaknesses:

1. SNN comes into its own when encoding time-varying data, so that it should be able to extract spatiotemporal patterns of event stream through time. Therefore, SNN should be defined in a time domain. The authors proposed time-independent SNNs given the use of a single timestep, which are not SNNs as a matter of fact given the key to SNNs, i.e., dynamic models. Therefore, the advantages of the time-independent SNN over DNNs are not sure. In fact, the model used in this work is a DNN with binary activations rather than SNN. 

2. The CE comparison between the proposed model and DNN in Eq. 14 is not fair. The proposed model includes additional operations that are not addressed in the CE calculation. They include the division in Eq. 3 and comparison with the threshold in FP in Eq. 4. Note that the comparison with the FP number is much more expansive than the comparison in ReLU, which can simply be done by seeing the sign bit of an input. Moreover, I wonder if FP multiplications are avoided by using binary activations (0/1 or 0.0000…/1.0000…) because this depends on hardware used. I recommend the authors to specify the hardware considered for the proposed model.

3. Where is the multiplier $\lambda_H$ in Eq.6?
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well organized and clearly written.

Reproducibility: it cannot be evaluated since no code is provided. Also, no hyperparameters are given in the manuscript.

Quality and Novelty: As for the weaknesses mentioned in the above section, I found that the significance and novelty are marginal. 
summary_of_the_review: The proposed model (SNN with one timestep) corresponds to a DNN with binary activations. Therefore, the model cannot leverage the rich dynamics inherently given to SNNs, and thus the proposed model is not able to process time-varying event stream. Given this reason, I found marginal significance in this work. As well, Novelty is limited given that the Hoyer regularization concept was proposed previously. Regarding technical aspects of the proposed model, the loss in accuracy due to the use of binary activations is obvious, and the computational efficiency estimation is also very arguable.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to use a Hoyer regularizer and Hoyer spike layer to improve the training of one-time-step SNNs. The paper receives a borderline score and goes through intensive discussions between the reviewers and the authors. It has also been discussed in a panel via a virtual meeting. The reviewers’ original concerns are summarized as follows:
- The position of the considered one-time-step SNN in the areas of SNN and BNN, as well as its significance and biological plausibility
- The limited novelty of applying Hoyer regularizer
- The clarity of the motivation and explanation for the Hoyer regularizer and Hoyer spike layer
- Some details of experiment results, missing comparisons and experiments

During the discussion period, the authors made revisions to explain the position of this work as a continuum between sparsity-induced BNNs and SNNs for static vision tasks, emphasize the novelty of Hoyer spike layer, and supplement additional experiment results to better explain the method and demonstrate extensions. The reviewers acknowledged that some concerns are addressed, but still had concerns about the novelty, the explanation for the method, the bio-plausibility and the network capacity, as well as the clarity of experimental settings and hardware consideration. 

During the panel discussion, the reviewers acknowledged the good empirical results, while raising concerns in two main aspects. First, about hardware consideration and energy efficiency. One-time-step SNN is more like ANN with binary activation rather than SNN. If training one-time-step SNN does not target for neuromorphic chips with bio-plausibility, it should compare with BNN more on common hardware such as GPU, which may hardly utilize the sparsity. Current results do not show advantages over BNN in a consistent evaluation. Second, about explanation for the method. The inconsistency of the Hoyer regularizer and Hoyer spike layer on unnormalized and normalized membrane potentials with trainable threshold is not explained well. Applying the Hoyer regularizer is not theoretically novel, so the reasonable explanation for the Hoyer spike layer is important.

Given the comments and discussions, the current version of the paper needs much additional revision to fully address concerns. 

So based on the majority of negative rating from reviewers, the AC recommended rejection and all the reviewers (including qbCE) agreed on this decision during the virtual meeting. The AC appreciated the authors’ great efforts and hoped that the authors could use the constructive feedback to improve the paper. BTW, the AC wanted to point out that Eqn. (1) and (10) are incorrect.
justification_for_why_not_higher_score: The paper is not good enough for ICLR.
Decision: Reject