HOYER REGULARIZER IS ALL YOU NEED FOR EXTREMELY SPARSE SPIKING NEURAL NETWORKS
Authors: Gourav Datta
Zeyu Liu
Peter Anthony Beerel
Venue: Submitted to ICLR 2023 (2023)

Abstract: Spiking Neural networks (SNN) have emerged as an attractive spatio-temporal
computing paradigm for a wide range of low-power vision tasks. However, state-
of-the-art (SOTA) SNN models either incur multiple time steps which hinder their
deployment in real-time use cases or increase the training complexity significantly.
To mitigate this concern, we present a training framework (from scratch) for one-
time-step SNNs that uses a novel variant of the recently proposed Hoyer regularizer. We estimate the threshold of each SNN layer as the Hoyer extremum of a
clipped version of its activation map, where the clipping threshold is trained using
gradient descent with our Hoyer regularizer. This approach not only downscales
the value of the trainable threshold, thereby emitting a large number of spikes for
weight update with limited number of iterations (due to only one time step) but
also shifts the pre-activation values away from the threshold, thereby mitigating
the effect of noise that can degrade the SNN accuracy. Our approach outperforms
existing spiking, binary, and adder neural networks in terms of accuracy-FLOPs
trade-off for complex image recognition tasks. Downstream experiments on object detection also demonstrate the efficacy of our approach.

Under review as a conference paper at ICLR 2023
HOYER REGULARIZER IS ALL YOU NEED FOR ULTRA LOW-LATENCY SPIKING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Spiking Neural networks (SNN) have emerged as an attractive spatio-temporal computing paradigm for a wide range of low-power vision tasks. However, state- of-the-art (SOTA) SNN models either incur multiple time steps which hinder their deployment in real-time use cases or increase the training complexity significantly. To mitigate this concern, we present a training framework (from scratch) for one- time-step SNNs that uses novel variant of the recently proposed Hoyer regu- larizer. We estimate the threshold of each SNN layer as the Hoyer extremum of a clipped version of its activation map, where the clipping threshold is trained using gradient descent with our Hoyer regularizer. This approach not only down- scales the value of the trainable threshold, thereby emitting large number of spikes for weight update with a limited number of iterations (due only one time step) but also shifts the membrane potential values away from the threshold, thereby mitigating the effect of noise that can degrade the SNN accuracy. Our ap- proach outperforms existing spiking, binary, and adder neural networks in terms of the accuracy- -LOPss trade-off for complex image recognition tasks. Downstream experiments on object detection also demonstrate the efficacy of our approach. Codes will be made publicly available.
INTRODUCTION & RELATED WORKS
Due to its high activation sparsity and use of cheaper accumulates (AC) instead of expenyee haltiply--nd-acccumue
(MAC), SNNs have emerged a promising low-power alternative to compute- and eemmory-xppessiee deep neural networks (DNN) (Indiveri et al., 20 Pfeiffer et al., 2018; Cao et al., 2015). Because SNNs receive and transmit information via spikes, analog inputs have to be encoded with a sequence of spikes using techniques such as rate coding (Dichl et al., 2016), temporal coding (Comsa et al., 2020), direct encoding (Rathi et al., 2020a) and rank-order coding (Kheradpisheh et al., 2018). In addition to accommodating various forms of spike encod- ing, supervised training algorithms for SNNs have overcome various roadblocks associated with the discontinuous spike activation function (Lee et al., 2016; Kim et al., 2020). Moreover, previous SNN efforts propose batch normalization (BN) techniques (Kim et al., 2020; Zheng et al., 2021) that leverage the temporal dynamics with rate/direct encoding. However, most of these efforts require multiple time steps which increases training and inference costs compared to non-spiking counter- parts for static vision tasks. The training effort is high because backpropagation must integrate the gradients over an SNN that is unrolled once for each time step (Panda et al., 2020). Moreover, the multiple forward passes result in an increased number of spikes, which degrades the SNN's energy efficiency, both during training and inference, and possibly offsets the compute advantage of the ACs. The multiple time steps also increase the inference complexity because of the need for input encoding logic and the increased latency associated with requiring one forward pass per time step. To mitigate these concerns, we propose one-time-step SNNs that do not require any non-spiking DNN pre-training and are more compute-efficient than existing multi-time-step SNNs. Without any temporal overhead, these SNNs are similar vanilla feed-forward DNNs, with Heaviside activation functions (McCulloch & Pitts, 1943). These SNNs are also similar to sparsity-induced or uni-polar binary neural networks (BNNs) (Wang et al., 2020b) that have 0 and as two states. However, these BNNs do not yield SOTA accuracy like the bi-polar BNNs (Diffenderfer & Kailkhura, 2021) that has and -1 one time-step, however, it required CNN pre-training, followed by iterative SNN training from 5 to 1 steps, significantly increasing the training complexity, particularly for ImageNet-level tasks. Note
states. A recent SNN work (Chowdhury al., 2021) also proposed the use of

---

Under review as a conference paper at ICLR 2023
that there have been significant efforts in the SNN community to reduce the number of time steps via optimal DNN-to-SNN conversion (Bu et al., 2022b; Deng et al., 2021), lottery ticket hypothesis (Kim et al., 2022c), and neural architecture search (Kim et al., 2022b). However, none of these techniques have been shown to train one-time-step SNNs without significant accuracy loss.
Our Contributions. Our training framework is based on novel application of the Hoyer regular- izer and a novel Hoyer spike layer. More specifically, our spike layer threshold training-input dependent and is set to the Hoyer extremum of a clipped version of the membrane potential tensor, where the clipping threshold (existing SNNs use this as the threshold) is trained using gradient descent with our Hoyer regularizer. In this way, compared to SOTA one-time-ster non non-iteratively trained SNNs, our threshold increases the rate of weight updates and our Hoyer regularizer shifts the membrane potential distribution away from this threshold, improving convergence.
We consistently surpass the accuracies obtained by SOTA one-time-step SNNs (Chowdhury al., 2021) on diverse image recognition datasets with different convolutional architectures, while reduc- ing the average training time by ~9xx. Compared binary neural networks (BNN) and adder neural network (AddNN) models, our SNN models yield similar test accuracy with a ~5.5 x re- duction in the floating point operations (FLOPs) count, thanks the extreme sparsity enabled by our training framework. Downstream tasks on object detection also demonstrate that our approach surpasses the test mAP of existing BNNs and SNNs.
2 PRELIMINARIES ON HOYER REGULARIZERS
Based on the interplay between LI and L2 norms, new measure of sparsity was first introduced in (Hoyer, 2004), based on which reference (Yang al., 2020) proposed a new regularizer, termed the Hoyer regularizer for the trainable weights that was incorporated into the loss term to train DNNs. We adopt the same form Hoyer regularizer for the membrane potential train our SNN models
as (uu) = the superscript for the time step is omitted for simplicity. Compared the L1 and L.2 regularizers, the Hoyer regularizer has scale-invariance (similar to the LO regularizer). It is also differentiable almost everywhere, as shown in equation 1, where |uy] represents the element-wise absolute of the tensor uy.
(Kurtz et al., 2020). Here, || represents the Li norm of the tensor uy and
(H)) 2sign(u:) Ous
|uill
(1)24
122
|| |I (x))
(1)
|m 1
Letting the gradient
(H(w) Du.
0, we estimate the value of the Hoyer extremum Ext(w)
w
This extremum is actually the minimum. because the second derivative greater than zero for any value of the output element. Training with the Hoyer regularizer can effectively help push the activation values that are larger than the extremum (u,>Ex((u))) even larger and those that are smaller than the extremum (u <
(u:<Er((un) Ert(w:)) even smaller.
PROPOSED TRAINING FRAMEWORK
Our approach is inspired by the fact that Hoyer regularizers can shift the pre-activation distributions away from the Hoyer extremum in non-spiking DNN ( (Yang et al., 2020). Our principal insight is that setting the SNN threshold this extremum shifts the distribution of the membrane potentials away from the threshold value, reducing noise and thereby improving convergence. To achieve this goal for one-tine me-time-step SNNs we present a novel Hoyer spike layer that sets the threshold based upon a Hoyer regularized training process, as described below.
3.1 HOYER SPIKE LAYER
In this work, adopt time-independent variant of the popular Leaky Integrate and Fire (LIF) representation, as illustrated in Eq. 2, to model the spiking neuron with one time-step.
1, if 2 1;
U 1 21
o
0.
otherwise
where z denotes the normalized membrane potential. Such a neuron model with a unit step acti- vation function is difficult to optimize even with the recently proposed surrogate gradient descent
2

---

Under review as a conference paper at ICLR 2023
techniques for multi-time- SNNs (Panda et al., 2020; Panda & Roy, 2016), which either approx- imates the spiking neuron functionality with a continuous differentiable model or uses surrogate gradients to approximate the real gradients. This is because the average number of spikes with only one time step too low to adjust the weights sufficiently using gradient descent with only one iteration available per input. This is because if pre-synaptic neuron does not emit a spike, the synaptic weight connected to it cannot be updated because its gradient from neuron to j is calculated as Gay X0g, where Iu, is the gradient of the membrane potential új and o; is the out- put of the neuron Therefore, spikes for better network convergence. Note that sufficiently low value of threshold can gen- erate spike for every neuron, but that would yield random outputs in the final classifier layer. Previous works (Datta et al., 2021; Rathi et al., 2020a) show that the number of SNN time steps can be re-
crucial reduce the value of the threshold to generate enough
1.0
duced by training the threshold term
SOTA spike activetion Hoyer spikk
scale
Sown scaled gradient
oi using gradient descent. However, our experiments indicate that, for one-time-step SNNs, this approach still yields thresholds that produce significant drops in accuracy. In contrast, we propose to dynamically down-scale the threshold (see Fig. I(a)) based on the membrane poten- tial tensor using our proposed form of the Hoyer regularizer. In particu- lar, we clip the membrane potential tensor corresponding to each convolutional layer to the trainable threshold 1/th obtained from the gradient descent with our Hoyer loss, as detailed later in Eq. 11. Unlike existing approaches (Datta & Becrel, 2022; Rathi et al., 2020a) that require with to be initialized from pre-trained non-spiking model, our approach can be used to train SNNs from scratch with a Kaiming uniform initialization (He et al., 2015) for both the weights and thresholds. In particular, the down-scaled threshold value for each layer is computed as the Hoyer extremum of the clipped membrane potential tensor, as shown in Fig. I(a) and mathematically defined as follows.
clip
(e)
[b)
activation
dz.
H, H.(clip
My
1.0
Mr
1.0 2.0
Figure 1: (a) Comparison of our Hoyer spike activation function with existing activation functions where the blue distribution denotes the shifting of the membrane potential away from the threshold using Hoyer regularized training, (b) Proposed derivative of our Hoyer activation function.
1,
2 > 1
[1, if 2 Exl(z Or = hz())))= = {0. otherwise
21
clip
21,
if < 2 1
0,
if
z <
Note that our threshold (xt(z, ( is indeed less than the trainable threshold with used in earlier works (Datta & Beerel, 2022; Rathi et al., 2020a) for any output, and the proof is shown in Appendix A.I. Moreover, we observe that the Hoyer extremum in each layer changes only slightly during the later stages of training, which indicates that most likely an inherent attribute of the dataset and model architecture. Hence, to estimate the threshold during inference, we calculate the exponential average of the Hoyer extremums during training (similar to BN), and use the same during inference.
3.2
HOYER REGULARIZED TRAINING
The loss function (Lrotal) of our proposed approach is shown below in Eq. 4.
T-1
Leotal = LOE LH LOF + XH Σ H(n)
(4)
1=1
where LoE denotes the cross-entropy loss calculated on the softmax output of the last layer L,
and LH represents the Hoyer regularizer calculated on the output of each convolutional and fully- connected layer, except the last layer. The weight update for the last layer is computed
aLCE
aLH
OLCE duL +\B Dur dwL
dLH duL -(s 1)OL-1+A
HH (H(u))
AW
OwL
+AH Owl
dur dwr
du OL-1
(5)
OLCE alCE OuL = (8 - 7)
(6)
do Our do1-1
3

---

Under review as a conference paper at ICLR 2023
eah where a denotes the output softmax tensor, i.c., Di of where ui and un, denote the tth and tt elements of the membrane potential of the last layer L, and N denotes the number of classes. Note that y denotes the one-hot encoded tensor of the true label, and @H(u)) is computed using Eq. dug The last layer does not have any threshold and hence does not emit any spike.
For a hidden layer 1, the weight update is computed as
AW OLCE Ow;
A OLh ölor dor dz du ow, do: Oz Tu ow
+AH ØLn Ow Tr ow,
ŌLCE dor 01- ÔLH
+AH Su
01-1
do: dz of
where aLn can be computed as Oui
ÖLH 0LH 0ui+1 do: dzi (H
aLH
do: 1 (H())
w++1 az with Ou
dui+1 do: dz du:
ou:
dutt1
where 0Ln is the gradient backpropagated from the (1 1)th layer, that iteratively computed Surii from the last layer L (see Eqs. 6 and 9). Note that for any hidden layer I, there are two gradients that contribute the Hoyer loss with respect the potential un one is from the subsequent layer (I+1) and the other is directly from its Hoyer regularizer. Similarly, #LCE is computed iteratively, starting from the penultimate layer (L-1) defined in Eq. 6, as follows.
θLCE aLCE do,-1 0z411 0ui+1 do
aLCE 0o1+1 1 doy 0z otth
W1+1
do141 0z1,1 0ur+1 do:
All the derivatives in Eq. 8-11 can be computed by Pytorch autograd, except the spike derivative des whose gradient is zero almost everywhere and undefined at o-=0. We extend the existing idea 0z," of surrogate gradient (Neftci et al., 2019) to compute this derivative for one-time-step SNNs with Hoyer spike layers, as illustrated Fig. ((b) and mathematically defined as follows.
0o:
scalex1 i < z <
0zi 0
=
otherwise
(10)
where scale denotes a hyperparameter that controls the dampening of the gradient. Finally, the threshold update for the hidden layer computed as
Aoth th aLCE Outh
OLh aLCE do: dzi Outh
OLH ÖLCE do: -u Orth
OLH 0us41 0ut+1 Outh
+Au
+AH
do: Oz Ōuth
do: dz (uthi2
(11)
@ul+1 Dut1
do Outh
do u (2
Wi+1 Ozi
(12)
Outh do
Note that we use this v/a, which is updated in each iteration, to estimate the threshold value in our spiking model using Eq. 3.
3.3 NETWORK STRUCTURE
We propose a series of network architectural modifications of existing SNNs (Datta & Beerel, 2022; Chowdhury et al., 2021; Rathi et al., 2020a) for our one-time-step models. As shown in Fig. 2(a), for the VGG variant, we use the max pooling layer immediately after the convolutional layer that is common in many BNN architectures (Rastegari et al., 2016), and introduce the BN layer after max pooling. Similar to recently developed multi-time-step SNN models (Zheng et al., 2021: Li al., 2021b; Deng et al., 2022: Meng et al., 2022), we observe that BN helps increase the test accuracy with one time step. In contrast, for the ResNet variants, inspired by (Liu al., 2018), we observe models with shortcuts that bypass every block can also further improve the performance of the SNN. We also observe that the sequence of BN layer, Hoyer spike layer, and convolution layer outperforms the original bottleneck in ResNet. More details are shown in Fig. 2(b).
3.4 POSSIBLE TRAINING STRATEGIES
Based on existing SNN literature, we hypothesize a couple of training strategies that can used to train one-time-step SNNs, other than our proposed approach.
4

---

Under review as a conference paper at ICLR 2023
Proposed block
Max Pool
Conv
Max Pool
BN
Hoyer Spike
Dropout
Conv
Block
Downsample
Spike
Existing block
Hoyer
Conv
LIF Spike
Avg
Pool
Dropout
Conv
Conv
Spike
(a)
(b)
Figure 2: Spiking network architectures corresponding to (a) VGG and (b) ResNet based models.
Pre-trained DNN, followed by SNN fine-tuning, Similar to the hybrid training proposed in (Rathi et al., 2020b), we pre-train a non-spiking DNN model, and copy its weights to the SNN model. Initialized with these weights, we train a one-time-step SNN with normal cross-entropy loss.
Iteratively convert ReLU neurons to spiking neurons. First, we train DNN model which uses the ReLU function with threshold as the activation function, then we iteratively reduce the number of the ReLU neurons whose output activation values are multi-bit. Specifically, we first force the neurons with values the top N percentile spike (set the output be 1), and those with bottom N percentile percent to die (set the output be 0), and gradually increase the N until there is significant drop of accuracy or all neuron outputs are either or O. Proposed training from scratch. With our proposed Hoyer spike layer and Hoyer loss. we train a SNN model from scratch. Our results with these training strategies are shown in Table 1, which indicates that it difficult for training strategies that involve pre-training and fine-tuning to approach the accuracy of non-spiking models with one time step. One possible reason for this might be the difference in the distribution of the pre-activation values between the DNN and SNN models (Datta & Beerel, 2022). is also intuitive obtain one-time-step SNN model by iteratively reducing the proportion of the ReLU neurons from pretrained full- precision DNN model. However, our results indicate that this method also fails to generate enough spikes at time step required to yield close to SOTA accuracy. Finally, with our network structure modifications to existing SNN works, our Hoyer spike layer and our Hoyer regularizer, accuracy from scratch.
Table 1: Accuracies from different strategies to train one-step SNNs on CIFAR10
Pretrained DNN(%)
Spiking activity (%)
Training Strategies
Acc. (%)
93.15
91.39
23.56
Iterative training (N=10)
93.25
92.68
10.22
Iterative Training (N=20)
92.68
92.24
9.54
Proposed Training
93.13
22.57
train one-time-step SNN model with SOTA
EXPERIMENTAL RESULTS
Datasets & Models: Similar to existing SNN works (Rathi et al., 2020b:a), we perform object recognition experiments on CIFARIO (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) dataset using VGG16 (Simonyan Zisserman, 2014) and several variants of ResNet (He et al., 2016) architectures. For object detection, we use the MMDetection framework (Chen et al., 2019) with PASCAL VOC2007 and VOC2014 (Everingham et al., 2010) as training dataset, and bench- mark our SNN models and the baselines on the VOC2007 test dataset. We use the Faster R-CNN (Ren al., 2015) and RetinaNet (Lin et al., 2017) framework, and substitute the original backbone with our SNN models that are pretrained on ImageNetIK.
the Adam (Kingma & Ba, 2014) optimizer for VGG16, and use SGD optimizer for ResNet models. As shown in Table 2, we obtain the SOTA accuracy of 93.44% the accuracy of our ResNet- based SNN models on ImageNet also surpasses the existing works. On ImageNet, obtain 68.00% top-1 accuracy with VGG16 which is only ~2% lower compared to the non-spiking counterpart. All our SNN models yield a spiking activity of ~25% or lower on both CIFAR 10 and ImageNet, which is significantly lower compared to the existing multi-time-stey SNN models as shown Fig. 3.
Object Recognition Results:
For training the recognition models,
CIFARIO 10 with VGG16 with only one time step;
5

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of the test accuracy of our one-time-step SNN models with the non-spiking
DNN models for object recognition. Model" indicates that we remove the first max pooling layer.
Network
dataset
DNN Top 1(%)
SNN Top (%)
SNN Top 5 (%)
Spiking activity (%)
VGGI6
CIFARIO
94.10
93.44
97.88
21.87
ResNet18
CIFARIO
93.34
91.48
97.34
25.83
ResNet18*
CIFARIO
94.28
93.67
97.98
16.12
ResNet20
CIFARIO 10
93.18
92.38
97.63
23.69
ResNet34*
CIFARIO I0
94.68
93,47
97.86
16.04
ResNet50*
CIFARIO 10
94,90
93.00
97.86
17.79
VGG16
ImageNet
70.08
68.00
78.75
24.48
ResNet50
ImageNet
73.12
66.32
77.06
23.89
T=5 (5-0 43)
T=2 15=0 28)
T-5 15=0.311
T-2 (5=0 181
Tmx (6m0 35)
Tml (5mb 17)
1.2
[Smb 27)
(Smb
1.-
spking actloyy
spking activiy
1.0
CIFAR10 (+)
0.8
ImageNet
(b)
S10
0.6
25
0.4
1.5
0.2
0.2
1.0
Layer
10 11 12 14
Layer
10 14
0.0
CIFAR10
ImageNet
Figure 3: Layerwise spiking activities for VGG16 across time steps ranging from 5 to (average spiking activity denoted as S in parenthesis) representing existing low-latency SNNs including our work on (a) CIFARIO, (b) ImageNet, (c) Comparison of the total energy consumption between SNNs with different time steps and non-spiking DNNs.
Object Detection Results: For
Table 3: Comparison of our one-time-step SNN models with non-spiking DNN, BNN, and multi-step SNN counterparts VOC2007 test dataset.
object detection on VOC2007, compare the performance ob- tained by our spiking models with non-spiking DNNs and BNNs in Table 3. For two-stage architec- tures, such as Faster R-CNN, the mAP of our one-time-stey SNN models surpass the existing BNNs by >0.6% For one-stage archi- tectures, such as RetinaNet (cho- sen because of its SOTA perfor- mance), our one-time-step SNN models with a ResNet50 backbone yields mAP of 70.5% (highest among existing BNN, SNN, AddNNs), Note that spiking VGG and ResNet-based backbones lead to a significant drop in mAP with the YOLO framework that is more compatible with the DarkNet backbone (even existing DarkNet-based SNNs lead to very low mAp with YOLO as shown in Table 3). However, our models suffer 5.8-6.8% drop in mAP com- pared to the non-spiking DNNs which may be due to the significant sparsity and loss in precision.
Framework Faster R-CNN
Backbone
mAP(%)
Original ResNet50
79.5
Faster R-CNN
Bi-Real (Liu et al., 2018)
65.7
Faster R-CNN
ReActNet (Liu et al., 2020)
73.1
Faster R-CNN
Our spiking ResNet50
73.7
Retinanet
Original ResNet50
77.3
Retinanet
SNN ResNet50 (ours)
70.5
YOLO
SNN DarkNet (Kim et al., 2019)
53.01
SSD
BNN VGG16 (Wang et al., 2020c)
66.0
Accuracy Comparison: We compare our results with various SOTA ultra low-latency SNNs for image recognition tasks in Table 4. Our one-time-step SNNs yield comparable or better test accuracy compared to all the existing works for both VGG and ResNet architectures, with significantly lower inference latency. The only exception for the latency reduction is the one-time-step SNN proposed in (Chowdhury et al., 2021), however, increases the training time significantly illustrated later in Fig. 3. Other works that have training complexity similar or worse than ours, such as (Datta & Beerel, 2022) yields 1.78% lower accuracy with 2x more number of time steps. Across both CIFARIO 10 and ImageNet, our proposed training framework demonstrates 2-32x improvement in inference latency with similar or worse training complexity compared to other works while yielding better test accuracy. Table 4 also demonstrates that the DNN-SNN conversion approaches require more time steps compared our approach worse test accuracies.
We were unable find existing SNN works for two-stage object detection architectures.

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison of our one-time-step SNN models to existing low-latency counterparts. SGD and hybrid denote surrogate gradient descent and pre-trained DNN followed by SNN fine-tuning respectively. (qC, dL) denotes architecture with q convolutional and linear layers.
Reference
Training
Architecture
Acc. (%)
Time steps
Dataset: CIFARTO
(Deng et al., 2021)
DNN SNN conversion
VGGI6
92.29
16
(Wu et al., 2019)
SGD
2L
90.53
12
(Kundu al., 2021)
Hybrid
VGGT6
92.74
TO
(Wu et al., 202T)
Tandem Learning
2L,
90.98
(Bu ct al., 2022a)
DNN-SNN coonversion
VGG16
90.96
(Zhang & [.i, 2020)
SGD
5C,
91.41
(Rathi et al., 2020a)
Hybrid
VGGI6
92.70
(Zheng et al., 2021)
STBP-tdBN
ResNet19
93.16
(Datta & Beerel, 2022)
Hybrid
VGGT6
91.79
(Bu al., 2022b)
DNN-SNI conversion
VGGT6
91.18
(Fang et al., 2020)
SGD
SC
93.50
(Chowdhury et al., 2021)
Hybrid
VGGT6
93.05
(Chowdhury et al., 2021)
Hybrid
ResNet20
91.10
This work
Adam+Hoyer Reg.
VGG16
93.44
Dataset: ImageNet
(Li et al., 2021c)
DNN-SNN conversion
VGGT6
63.64
32
(Bu et al., 2022b)
DNN-SNY conversion
ResNet34
59.35
16
(Wu et al., 2021)
Tandem Learning
AlexNet
50.22
12
(Rathi al., 2020a)
Hybrid
VGGT6
69.00
(Fang et al., 2021)
SGD
ResNet34
67.04
(Fang et al., 2021)
SGD
ResNet152
69.26
(Rathi et al., 2020a)
Hybrid
VGGI6
69.00
(Zheng et al., 2021)
STBP-idNN
ResNet34
67.05
(Chowdhury et al., 2021)
Hybrid
VGGT6
67.71
This work
Adam+Hoyer Reg.
VGG16 68.00
Inference Efficiency: We compare the energy-efficiecy of our one-time-step SNNs with non- spiking DNNs and existing multi-time-step SNNs in Fig. 3. The compute-efficiency of SNNs stems from two factors:- 1) sparsity, that reduces the number of floating point operations in convolutional and linear layers compared to non-spiking DNNs according to SNN
S X DNN (Chowdhury et al., 2021). where St denotes the average number of spikes per neuron per infer- ence over all timesteps layer I. 2) Use of only AC (0.9pJ) operations that consume 5.1x lower compared to each MAC (4.6pJ) operation in 45mm CMOS technology (Horowitz, 2014) for floating- point (FP) representation. Note that the binary activations can replace the FP multiplications with logical operations, i.c., conditional assignment to with bank of AND gates. These replacements can be realized using existing hardware (eg. standard GPUs) depending on the compiler and the details of their data paths. Building custom accelerator that can efficiently implement these re- duced operations is also possible (Wang et al., 2020a; Frenkel et al., 2019; Lee Li, 2020). In fact, in neuromorphic accelerators such as Loihi (Davies et al., 2018), FP multiplications are typically avoided using message passing between processors that model multiple neurons.
The total compute energy (CE) of multi-time-stee time-step SNN (SNNcE) can estimated as
I.
SN =
DDNNEEDNN(Y DNN,N flpss +46++DN * 0.4+ Σ S * DNN, +.9 +..+DNN' DN
0.7
1=2
(13)
because the direct encoded SNN receives analog input in the first layer (1=1) without any sparsity (Chowdhury al., 2021; Datta & Beerel, 2022; Rathi at., 2020a). Note that DN DN Nçom denotes the total number of comparison operations in the layer 1 with each operation consuming 0.4pJ energy.
The CE of the non-spiking DNN (DNNce) is estimated DN Σie DNN- flops * 4.6, where we ignore the energy consumed by the ReLU operation since that includes only checking the sign bit of the input.
We compare the layer-wise spiking activities S for time steps ranging from 5 to in Fig. 3(a-b) that represent existing low-latency SNN works, including our work. Note, the spike rates decrease significantly with time step reduction from 5 to 1, leading to considerably lower FLOPSs in our

---

Under review as a conference paper at ICLR 2023
time-step SNNs. These lower FLOPs, coupled with the 5.1x reduction for AC operations leads to a 22.9x and 32.1x reduction in energy on CIFARIO and ImageNet respectively with VGG16. Though we focus on compute energies for our comparison, multi-time-step SNNs also incur large number of memory accesses as the membrane potentials and weights need to fetched from and read to the on-/off-chip memory for each time step. Our one-time-step models can avoid these repetitive read/write operations as does involve any state and lead to a ~Tx reduction in the number of memory accesses compared to a 7-time-step SNN model. Considering this memory cost and the overhead of sparsity (Yin al., 2022), as shown Fig. 3(c), our onetiimesttep SNNs lead to 1.08--4.774 and 22.5-3 33x.4x reduction of the total energy compared to multi-time-step SNNs and non-spiking DNNs respectively on a systolic array accelerator.
Training & Inference Time Requirements: Because SOTA SNNs require iteration over multiple time steps and storage of the mem- brane potentials for each neuron, their training and inference time can be substantially higher than their DNN counterparts. However, reduc- ing their latency to 1 time step can bridge this gap significantly, as shown in Figure 4. On average, our low-latency, one-time-step SNNs represent 2.38x and 2 reduction in train- ing and inference time per epoch respectively, compared to the multi-time-step training ap- proaches (Datta & Beerel, 2022; Rathi et al., 2020a) with iso-batch and hardware conditions. Compared the existing one-time-step SNNs (Chowdhury et al., 2021), we yield a 19x and x.25x reduction in training and inference time. Such significant savings in training time, which translates to power savings in big data centers, can poten- tially reduce Al's environmental impact.
22
Teres
Vileecc
Figure 4: Normalized training and inference time per epoch with iso-batch (256) and hardware (RTX 3090 with 24 GB memory) conditions for (a) CIFARIO and (b) ImageNet with VGG16.
Ablation Studies: We conduct ablation studies analyze the contribution of each technique in our proposed approach. For fairness, we train all the ablated models on CIFARIO 10 dataset for 400 epochs. and use Adam as the optimizer, with 0.0001 as the initial learning rate. Our results are shown in Table 5, where the model without Hoyer spike layer indicates that we set the threshold as ofth similar to existing works (Datta & Becrel, 2022; Rathi et al., 2020a) rather than our proposed Hoyer ex- tremum. With VGG16, our op- timal network modifications lead 1.9% increase in accuracy. Furthermore, adding only the Hoyer regularizer leads to negligible accuracy and spiking activity improvements. This might be because the regularizer alone may not be able sufficiently down-scale the threshold for optimal convergence with one time step. However, with our Hoyer spike layer, the accuracy improves by 2.68% to 93. 13% while also yielding 2.09% increase in spiking activity. We observe similar trend for our network modifications and Hoyer spike layer with ResNet18. 8. However, Hoyer regularizer substantially reduces the spiking activity from 27.62% to 20.50%. while also negligibly reducing the accuracy. Note that the Hoyer regularizer alone contributes to 0.20% increase in test accuracy on average. In summary, while our network modifications significantly increase the test accuracy compared to the SOTA SNN training with one time step, the combination of our Hoyer regularizer and Hoyer spike layer yield the SOTA SNN performance.
Table 5: Ablation study of the different methods in our proposed training framework on CIFAR10.
Network Hoyer
Hoyer Spike
Spiking Activity (%)
Arch.
Acc. (%)
Structure Reg.
VGG16
88.42
15.62
VGG16
X
X
90.33
20.43
VGG16
L
X
90.45
20.48
VGG16
X
V
92.90
21.70
VGG16
A
V
93.13
22.57
ResNet18
X
X
87.41
22.78
ResNet18
V
X
x
91.08
27.62
ResNet18
X
90.95
20.50
ResNet18
91.17
25.87
ResNet18
91.48
25.83
order to further improve the compute-efficiency our one-time- step SNNs, we perform quantization-awan training of the weights in our models to 2-6 bits.
Effect on Quantization:
8

---

Under review as a conference paper at ICLR 2023
This transforms the full-precision ACs to 2-6 6 bit ACs, thereby leading to a 4.8-13.8 reduction in compute energy as obtained from FPGA simula- tions on the Kintex7 platform using custom RTL. specifications. Note that we only quantize the con- volutional layers, as quantizing the linear layers lead to noticeable drop in accuracy. From the results shown in Table 6, when quantized to bits, our one-time-step VGG-based SNN incur negli- gible accuracy drop of only 0.02%. Even with 2- bit quantization, our model can yield an accuracy of .4% with any special modification, while still yielding a spiking activity of ~22%%
Table 6: Accuracies of weight quantized one- time-step SNN models based on VGGI6 on CIFARIO where FP is 32-bit floating point.
Spiking Activity (%)
Bits
Acc. (%)
CE (mJ)
FP
93.13
22.57
297.42
93.11
22.46
61.9
4
92.84
21.39
39.4
92.34
22.68
21.6
Comparison with AddNNs & BNNs: We compare the accu- racy and CE of our one-time- step SNN models with recently proposed AddNN models (Chen et al., 2020) that also removes mul- tiplications for increased energy- efficiency in Table 7. With the VGG16 architecture, on CIFAR 10, obtain 0.6% lower accuracy, while on ImageNet, we obtain 1.0% higher accuracy. Moreover, un- like SNNs, AddNNs do not involve any sparsity, and hence, consume ~5.55 more energy compared to our SNN models on average across both CIFARIO and ImageNet (see Table 7). We also compare our SNN models with SOTA BNNs in Ta- ble 7 that replaces the costly MAC operations with cheaper pop-count counterparts, thanks to the binary weights and activations. Both our full-precision and 2-bit quantized one-time-step SNN models yield accuracies higher than BNNs at iso-architectures on both CIFARIO and ImageNet. Additionally, our 2-bit quantized SNN models also consume 3.4x lower energy compared to the bi-polar networks (see (Diffenderfer & Kailkhura, 2021) in Table 7) due to the improved trade-off between the low spiking activity (~22% as shown in Table 7) provided by our one-time-step SNN models, and less energy due to XOR operations compared quantized ACs. On the other hand, our one-time-step SNNs consume similar energy compared to unipolar BNNs (see (Sakr et al., 2018; Wang et al., 2020b) in Table 7) while yielding 3.2% higher accuracy on CIFARIO at iso-architecture. The energy consumption is similar because the ~20 advantage of the pop-count operations is mitigated by the 22% higher spiking activity of the unipolar BNNs compared to our one-time-step SNNs.
Table 7: Comparison of our one-time-step SNN models to AddNNs and BNNs that also incur AC-only operations for improved energy-efficiency, where CE is compute energy
Reference
Dataset
Acc.(%)
CE (J)
BNNs
Sakr et al. (2018)
CIFAR10
89.6
0.022
Wang et al. (2020b)
CIFARTO
90.2
0.019
Wang al. (2020b)
ImageNet
59.7
3.6
Diffenderfer & Kailkhura (2021)
CIFARTO
91.9
0.073
AddNNs
Chen et al. (2020) (FP weights)
CIFARTO
93.72
1.62
Chen al. (2020) (2-bit weights)
CIFARIO
92.08
0.12
Chen et al. (2020) (FP weights)
ImageNet
67.0
77.8
Li et al. (2021a) (FP weights)
CIFARIO
91.56
1.62
Our SNNs
This work (FP weights)
CIFAR1O
93.44
0.297
This work (2-bit weights)
CIFARTO
92.34
0.021
This work (FP weights)
ImageNet
68.00
14.28
5 DISCUSSIONS & FUTURE IMPACT
Existing SNN training works choose ANN-SNN conversion methods to yield high accuracy or SNN fine-tuning yield low lateney or a hybrid of both for a balanced accuracy-latened trade-off. How- ever, none of the existing works can discard the temporal dimension completely, which can enable the deployment of SNN models in multiple real-time applications, without significantly increasing the training cost. This paper presents a SNN training framework from scratch involving novel combination of Hoyer regularizer and Hoyer spike layer for one time step. Our SNN models incur similar training time as non-spiking DNN models and achieve SOTA accuracy, outperforming the existing SNN, BNN, and AddNN models. However, our work can also enable cheap and real-time computer vision systems that might be susceptible to adversarial attacks. Preventing the application of this technology from abusive usage is important and interesting area of future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Optimized potential initialization for low- latency spiking neural networks. Proceedings the AAAI Conference Artificial Intelligence, 36(1):11-20, Jun. 2022a. doi: 10.1609/aaii.v33
http://aai.v661....... w3.... 19974. URL https:
http::/ //oss///j///... aaai. .rr//
hwt.:/ppppAAAaaatiiccccc wwwerticceeeveee/e/e 1194444. index.php/AAe
Tong Bu, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang. Optimal ANN- SNN conversion for high-accuracy and ultra-low-Iatene spiking neural networks. In Interna- tional Conference on Learning Representations, 2022b. URL https: httplp::///peeeevviieeee net//
Yongqiang Cao et al. Spiking deep convolutional neural networks for energy energy-efficient object recog- nition. International Journal of Computer Vision, 113:54-66 66, 05 2015.
Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprint arXiv: 01199000015... 2019.
Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. One timestep is all you need: Training spiking neural networks with ultra low latency. arXiv preprint 10.0599,,, 2021.
I. M. Comsa al. Temporal coding in spiking neural networks with alpha synaptic function. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), volume pp. 8889-5522555 2020.
Gourav Datta and Peter A. Beerel. Can deep neural networks be converted to ultra low-latency spiking neural networks? In 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE), volume 1, pp. 718-723, 2022. doi: 10.23919/DATE5411 1..2111..22222.7774
Gourav Datta et al. Training energy-efficient deep spiking neural networks with single-spike hybrid input encoding. In 2021 International Joint Conference on Neural Networks (IJCNN), volume 1, pp. 1-8, 2021. doi: 10.1 109//1CNN52387 220 195111953
M. Davies, N. Srinivasa. T. H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, Y. Liao, C. K. Lin, A. Lines, R. Liu, D. Mathaikutty, S. McCoy, A. Paul, Tse, G. Venkataramanan, Y. H. Weng. A. Wild, Y. Yang, and H. Wang. Loihi: A neu- romorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018. doi: 10. 1T009///MM.200811122103333
J. Deng. W. Dong, R. Socher, L.-J. Li. Li, and L. Fei-Fei. ImageNet: A Large-Scar Scale Hierarchical Image Database. In CVPRO9, 2009.
Shikuang Deng. Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representa- tions, 2022. URL https: http::::///pppeeeeee http///pppeeeeeeeeeeeeeeeeeeeeee net fortt//frmm?ii wiwdddd===ii323__1
Shikuang Deng et al. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations, 2021.
Peter U Diehl et al. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In 2016 IEFF International Conference Rebooting Computing (ICRC), pp. 1-8. IEEE, 2016.
James Diffenderfer and Bhavya Kailkhura. Multi-prize lottery ticket hypothesis: Finding accu-
rate binary neural networks by pruning randomly weighted network. In International Confer- ence on Learning Representations, 2021. URL https: http:://ppppevvieeeeeee ew. net/ forum? d=
U_ U_matt0b matt0biii........
10

---

Under review as a conference paper at ICLR 2023
M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zisserman. The paseal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303 June 2010.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. arXiv preprint ar arXiv:2007.0578: 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21056-21069, Curran Associates, Inc., 2021. URL httpss://proceedingg neurips cc pttppppppp/////illeeeee pdf.
Charlotte Frenkel, Martin Lefebvre, Jean-Didier Legat, and David Bol. A 0.086-mm² 12.7-pj/sop 64k-synapse 256-neuron online-learning digital spiking neuromorphic processor in 28-nm cmos. IEEE Transactions on Biomedical Circuits and Systems, 13(1):145-158, 2019, doi: 10.1109/
TBCAS.2018. 281..8
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv: 1502:01822.2 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Mark Horowitz.
Computing's energy problem (and what we can do about it). In 2014 IEEE Inter- national Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, 2014.
Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, 5(9), 2004.
Giacomo Indiveri et al. Frontiers in neuromorphic engineering. Frontiers in Neuroscience, 5, 2011.
Saeed Reza Kheradpisheh, Mohammad Ganjtabesh, Simon J. Thorpe, and Timothée Masquelier. STDP-based spiking deep convolutional neural networks for object recognition. Neural Networks, 99:56-67, Mar 2018. ISSN 0893-6080. doi: (0.10166..euunet2200
URL http://tp//c /dx
doi.org/10. 6 neunet. 2017. 12. 005.
Seijoon Kim, Scongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-effieient object detection, 2019.
Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 144:686-698, 2021. ISSN 0893-6080. doi: https://doi.or org// 10.10166j.euunet.22211 .990022. URL https: //ww://wwww//.///////////www sci encedi rect. com/ /ss//cciennceeee article/pii httpa//ceepp///00000000020200000
Youngeun Kim, Joshua Chough, and Priyadarshini Panda. Beyond classification: Directly training spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineer- ing, 2022a. URL http://oopssceiiccc iop. wwwg/arttgle////eoooooooo...eeee 11..1..18822232 4386/ ac9b86.
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. arXiv preprint arXiv:2201. 10355, 2022b.
Youngeun Kim, Yuhang Li. Hyoungseob Park, Yeshwanth Venkatesha, Ruokai Yin, and Priyadarshini Panda. Exploring lottery ticket hypothesis in spiking neural networks. arXiv preprint arXiv:2207.012 220.00188. 2022c.
Youngeun Kim al. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. arXiv preprint arXiv:2010.01729, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6412999 2014.
11

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Souvik Kundu et al. Towards low-latency energy-efficien deep SNNs via attention-guided com- pression. arXiv preprint arXiv:2107 12445, 2021.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting acti- vation sparsity for fast inference on deep neural networks. Hal Daumé III and Aarti Singh Machine Learning, volume 119 of
(eds.), Proceedings of the 37th International Conference Proceedings of Machine Learning Research, pp. 5533-5543, PMLR, 13-18 Jul 2020. URL http://ppoccediiieeee
w...m...rreee S.l...pressvvl11 /v119/ htwl/////kk/tta kurtz20a html.
Jeong-Jun Lee and Peng Li. Reconfigurable dataflow optimization for spatiotemporal spiking neu- ral computation on systolic array accelerators. In 2020 IEEE 38th International Conference on Computer Design (ICCD), volume pp. 57-64, 2020. doi: 10.1 (0.110//CCCDS0037..2000 ROD.DD55377.2200000000 2020.00027.
Jun Haeng Lee et al. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10, 2016.
Wenshuo Li, Hanting Chen, Minggiang Huang, Xinghao Chen, Chunjing Xu, and Yunhe Wang. Winograd algorithm for addemnt. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6307-6315, PMLR, 18-24 Jul 2021a. URL https: htt.tttpa:/pprreeeii..... /ttp///eooccee......... mlr. wal.ppres/vvll wttpp/eeeweee/.....tteett
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Dif- ferentiable spike: Rethinking gradient-descent for training spiking neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 23426-23439. Curran Asso- ciates, Inc., 2021b. URL https:t/ppocceediggng meweeeeddddiinee.eneneeeeeeeeeee neurips...... cc/pappt//22///ill/ pdf.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromor- phic data augmentation for training spiking neural networks. arXiv preprins arXiv:2002. 2022.
Yuhang Li et al. A free lunch from ANN: Towards efficient, accurate spiking neural networks calibration. arXiv preprint Stti::210.00660 2021c
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, arXiv preprint arXiv: (0ti:1708.0220 020000 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Twang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 722-737, 2018
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Tine Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In European conference on computer vision, pp. 143-159. Springer, 2020.
Warren McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115-133, 1943.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Train- ing high-performance low-latency spiking neural networks by differentiation on spike representa- tion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12444-12453, June 2022.
E. O. Neftei, H. Mostafa, and F. Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Pro- cessing Magazine, 36(6):51-63. 2019.
12

---

Under review as a conference paper at ICLR 2023
Priyadarshini Panda and Kaushik Roy. Unsupervised regenerative learning of hierarchical features in spiking deep networks for object recognition. arXiv preprint arXivv1602.01510, 2016.
Priyadarshini Panda et al. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14, 2020.
Michael Pfeiffer et al. Deep learning with spiking neurons: Opportunities and challenges. Frontiers in Neuroscience, 12:774, 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Nitin Rathi al. DIET-SNN: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008. 03658, 2020a.
Nitin Rathi et al. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.0/ 2020b.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Charbel Sakr, Jungwook Choi, Zhuo Wang, Kailash Gopalakrishnan, and Naresh Shanbhag. True gradient- based training of deep binary activated neural networks via continuous binarization. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), vol- ume 1, pp. 2346-2350, 2018. doi: 10.1.00/CSSS...00.... 244444444421888
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv::109.111 1556, 2014.
Dewei Wang, Pavan Kumar Chundi, Sung Justin Kim, Minhao Yang. Joao Pedro Cerqueira, Joon- sung Kang, Seungchul Jung, Sangjoon Kim, and Mingoo Scok. Always-on, sub-300-nw. event- driven spiking neural network based on spike-driven gating for ultra-low-powee lower intelligent device. In 2020 JEEE Asian Solid-Start Circuits Confer- ence (A-SSCC), volume 1, pp. 1-4, 2020a. doi: N0.1109/A-SSCC486 SSCCCCCC333330000933366660
and clock- and power-
Peisong Wang, Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. rized neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 34 (07):12122-12199, Apr. 2020b. URL https:/ http:///ss.aaa................a. aaai. org/ wwrg/indee iidex.inxx. phpp /AAA// article/view/5000
inducing bina-
Ziwei Wang, Ziyi Wu, Jiwen Lu, and Jie Zhou. Bidet: An efficient binarized object detector. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020c.
Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for effective training and rapid inference deep spiking neural networks. IEEE Transac- tions on Neural Networks and Learning Systems, 1(1):1-15, 2021. doi: 10.1109/TNLSS2021 3095724.
Yujie Wu et al. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 1311-1318, 2019.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differen- tiable scale-invariant sparsity measures. In International Conference on Learning Representa- fions, 2020.
Ruokai Yin, Abhishek Moitra, Abhiroop Bhattacharjee, Youngeun Kim, and Priyadarshini Panda. Sata: Sparsity-aware training accelerator for spiking neural networks. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 1(1):111, 2022. doi: 10. 1091/D./C.. 2022.3213211.
13

---

Under review as a conference paper at ICLR 2023
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spik- ing neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 12022-12033. Curran As- sociates, Inc., 2020. URL http::ttt///ttcm htR.hhttss////ooeeeeeeee wwwppeeeednns....o........ neurips ps. cc wtcp/paper/222//11111 pdf..
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (12):1062--1177,, May 2021. doi: 1.009/aaai.33511....a URL https: /ojs. aaai.
A APPENDIX
A.1 PROOF OF THRESHOLD DOWNSACLING WITH HOYER EXTREMUM
In order to prove that our Hoyer extremum is always less than or equal to wth we first prove the Hoyer extremum of 21 clip is less than equal to 1. Let use c; represent 2/ clip so Vj. cj s
|all2 Eg(r² I((d-mal(et) max(c))) S max(c)) $
Ext(ci) = |cill1 = Σd I
(14)
So the Hoyer extremum of zchip is always less than or equal one, and our Hoyer extremum of every layer / which is the product of ttth and Ext(z) is always less than equal ofth
A.2 EXPERIMENTAL SETUP
For training VGGI6 models, we using Adam optimizer with initial learning rate of 0.0001, weight decay of 0,0001, dropout of 0.1 and batch size of 128 in CIFAR10 for 600 epochs, and Adam optimizer with weight decay of 5e-6 and with batch size 64 in ImageNet for 180 epochs. For training ResNet models, we using SGD optimizer with initial learning rate of 0.1, weight decay of 0.0001 and batch size of 128 in CIFAR1O for 400 epochs, and Adam optimizer with weight decay of 5e and with batch size 64 in ImageNet for 120 epochs. We divide the learning rate by 5 at 60%, 80%, and 90% of the total number of epochs.
When calculating the Hoyer extremum implement two versions, that calculates the Hoyer extremum for the whole batch, while another that calculates it channel-wise. Our experiments show that using the channel-wise version can bring 0.1-0.37. increase in accuracy. All the experimental results reported in this paper use this channel-wise version.
For Faster R R-CNN, we use SGD optimizer with initial learning rate of 0.01 for 50 epochs, and divide the learning rate by 10 after 25 and 40 epochs each. For Retinanet, we use SGD optimizer with initial learning rate of 0.001 with the same learning rate scheduler as Faster R-CNN.
A.3 EXTENSION TO MULTIPLE TIME-STEPS
We extend our proposed approach to pilti--tmm-steep SNN models. As show in Table 8, as time step increases from 1 to 4, the accuracy of the model also increases from 93.44% to 94. 14%, which validates the effectiveness of our method. However, this accuracy increase comes at the cost of a significant increase in spiking activity (see Table 8), thereby increasing the compute energy and the temporal "overhead" increases, thereby increasing the memory cost due to the repetitive access of the membrane potential and weights across the different time steps.
A.4 EXTENSION TO DYNAMIC VISION SENSOR (DVS) DATASETS
The inherent temporal dynamics in SNNs may be better leveraged in DVS or event-based tasks (Deng al., 2022: Li al., 2022; Kim & Panda, 2021; Kim et al., 2022a) compared to standard static vision tasks that are studied in this work. Hence, we have evaluated our framework on the DVS-CIFARI 10 dataset, which provides each label with only 0.9k training samples, and is considered
14

---

Under review as a conference paper at ICLR 2023
Table 8: Test accuracy obtained by our approach with multiple time steps on CIFARIO.
Architecture
Time steps
Acc. (%)
Spiking activity (%)
VGG16
93.44
21.87
VGG16
93.71
44.06
VGG16
4
94.14
74.88
VGG16
94.04
101.22
ResNet18
1
91.48
25.83
RseNet18
91.93
33.24
Table 9: Comparison our one- and multi-time-step SNN models to existing SNN models on
DVS-CIFARI dataset.
Reference
Training
Architecture
Acc. (%)
Time steps
Deng al. (2022)
TET
VGGSNN
83.17
10
Deng et al. (2022)
TET
VGGSNN
75.20
4
Deng al. (2022)
TET
VGGSNN
68.12
Li et al. (2022)
tdBN+NDA
VGGIT
81.7
10
Kim Panda (2021)
SALT+Switchec BN
VGGI6
67.1
20
This work
Hoyer reg-
VGGSNN
83.68
10
This work
Hoyer reg. Hoyer reg.
VGGSNN
76.17
4
This work
VGGSNN
69.80
1
the most challenging event-based dataset (Deng et al., 2022). As illustrated in Table 9, we surpass the test accuracy of existing works (Li al., 2022; Kim & Panda, 2021) by 1.30% on average at iso-time-step and architecture. Note that the architecture VGGSNN employed in our work and (Deng al., 2022) based VGG11 with fully connected layers removed (Deng al., 2022) found that additional fully connected layers were unnecessary for neuromorphic datasets. In fact, our accuracy gain is more significant low time steps, thereby implying the portability of our approach to DVS tasks. Note that similar to static datasets, a large number time steps increase the temporal overhead in SNNs, resulting in large memory footprint and spiking activity.
A.5 FURTHER INSIGHTS HOYER REGULARIZED TRAINING
Since existing works (Panda et al., 2020) use surrogate gradients (and not real gradients) to update the thresholds with appropriate initializations. is difficult estimate the optimal value of the IF thresholds. On the other hand, our Hoyer extremums dynamically change with the activation maps particularly during the early stages of training (coupled with the distribution shift enabled by Hoyer regularized training), which enables our Hoyer extremum-basee scaled thresholds to be closer to optimal. In fact, shown from ablation studies in Table 5, our Hoyer extremum-based spike layer is more effective than the Hoyer regularizer which further justifies the importance of the combination of the Hoyer extremum with the trainable threshold. Additionally, function the membrane potential before computing the Hoyer extremum. This is done to get rid of few outlier values in the activation map that may otherwise unnecessarily increase the value of the Hoyer extremum, i.e., threshold value, thereby reducing the accuracy, without any noticeable increase in energy efficiency. In fact, the test accuracy with VGG16 on CIFAR10 drops by more than 1.4% (from 93.13% obtained by our training framework) to 91.7% without the clip function.
use the clip
A.6 TUNING SPIKING ACTIVITY WITH HOYER REGULARIZER Ay
We conduct experiments with different coefficients of Hoyer regularizer AØ to demostrate its impact on the trade-off between accuracy and spikinf activity. As shown in Table 10, we can clearly see that larger Hoyer regularizer alone can decrease the spike activity rate, while smaller Hoyer regularizer will increase the same. In fact, the spiking activity can be precisely tuned using A to yield range of accuracies. Interestingly. Hoyer-regularized training ResNet18 yields wider range of spiking activities and a narrower range of accuracies compared to VGG16. This might be because each architecture have different optimization headroom.
15

---

Under review as a conference paper at ICLR 2023
Table 10: Test accuracy obtained with different coefficients of Hoyer regularizer on CIFARIO.
Architecture
Acc. (%)
Spiking activity (%)
VGGI6
le-7
89.73
19.62
VGG16
le-8
90.33
20.43
VGG16 (with Hoyer spike layer)
le-7
92.93
21.61
VGG16 (with Hoyer spike layer)
lc-8
93.13
22.57
VGGI6 (with Hoyer spike layer)
lc-9
92.95
22.15
ResNet18
le-7
90.84
13.05
ResNet18
le-8
90.95
20.50
ResNet18
le-9
91.05
23.54
ResNet18 (with Hoyer spike layer)
le-8
91.48
25.83
ResNet18 (with Hoyer spike layer)
0
91.17
25.87
The Hoyer spike layer, when used with the Hoyer regularizer (with the optimal value of the efficient that yields the best test accuracy). increase the spiking activity for both VGG16 and ResNet18. Please check the 2.08% (from 20.48% to 22.57%) increase in spiking activity for VGGI6 and 5.33% (20.50% to 25.83%) for ResNet18. This is because the Hoyer spike layer downscales threshold value, enabling more neurons to spike.
Note that the Hoyer spike layer, when used without the Hoyer regularizer, may be unable tune the trade-off between spiking activity and accuracy. This is because we do not have any explicit regularizer co-efficient, and the Hoyer extremum may always lower the threshold value because it is computed based on the SGL-based trainable threshold which, without Hoyer regularizer, may be updated randomly (i.c., not systematic manner that may encourage sparsity). This is the reason we believe we do not observe any definitive trend for the trade-off between accuracy and spiking activity in this case. Note that all the results in Table reported as the mean from five runs with distinct seeds.
16

---

Under review as a conference paper at ICLR 2023
A.7 TRAINING ALGORITHM
Our proposed training framework that can yield accurate and sparse one-time-step SNN models is illustrated below.
Algorithm 1: Detailed Algorithm for training our one-time-step SNN model.
Input: runEpochs, numBatches, numlayers, initial weights W, initial thresholds et Training data {(a()) y ))) Hoyer regularizer coefficient Ag- Data: Hoyer extremum Ext Layer index =
4- 0 runEpochs
for j t- 0 to numBatches do output <-
for + 0 to numLayers do
if layer, is Hover Spike layer then
ui + output
L# + LH + H(w)
w/oth
17 Divide input by threshold Ext(clamp(z,
Ext compuuuEpppaettailloiii.veeeeeee
E min = 0, max
1)))
[t 2 Ext - 1, 01 < Ext] t O:
// Output spiking act avtivition
output +-
end else
output layer,
end
end
LCE + λи Lu
9C computeGradients(W.. L)
BC computeGradients(e" 46 .)
Boch
update WeddtteWighssAndTTTeshllll
0c
8c
SW Du
end
for I +- 0 numLayers do
if layer, is Hoyer Spike layer then
ui + output
< ///// th
0 Ext] 1, 01|2: < Ext] + output o
end
output = layer, (oulpul)
end
end end
17


Reviews:
[{"review_id": "s54Lne8iaJX", "reviewer": "Reviewer_qbCE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new method to train spiking neural networks (SNNs) with only one timestep. The method is based on a new regularization term (\"Hoyer\"), and a new threshold adjustment method (\"Hoyer spike\", Eq 4).\n\n", "summary_of_the_review": "A potentially promising method, but some aspects are unclear.", "strength_and_weaknesses": "STRENGTHS:\n\nThe accuracy they reach with only one timestep is impressive, better than the previous state-of-the-art.\n\nWEAKNESSES:\n\nThe theory is unclear:\n\n* \"we estimate the value of the Hoyer extremum as He(ul) =\"\nI think what follows is not the value of H but the value of the element of u_l at the extremum.\n\n* a SNN with one timestep is actually a somewhat degenerated case because there is no temporal integration. It boils down to a vanilla feed-forward artificial neural network, with Heaviside as the activation function. This also corresponds to the first neuron model proposed by McCulloch and Pitts and 1943, also known as threshold gates. This should be discussed.\n\n* according to Table 5 what matters is the Hoyer spike, not the Hoyer regularization. This raises a question: could Hoyer spike be used alone, without the Hoyer regularization?\n\n* does the approach extend to multistep SNNs?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bfrkSi9ghtL", "reviewer": "Reviewer_pFTU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to train one-time-step SNNs with a Hoyer regularizer and Hoyer spike layer. The proposed Hoyer spike layer uses adaptive threshold based on the Hoyer extremum of membrane potentials, and a Hoyer regularization for membrane potentials is added in the loss function. Experiments on static image classification and object detection tasks demonstrate competitive performance and energy efficiency.", "summary_of_the_review": "In summary, this paper lacks some significant explanations for the methods despite the good performance, and there are some questions about the experiments that should be tackled. Besides, the paper should have more focus on the more similar binary or quantized neural networks.", "strength_and_weaknesses": "Strengths:\n\n1. This paper conducts extensive experiments on static image tasks, including classification and detection, with different network structures, and considers many aspects such as accuracy, energy efficiency, and training and inference time. The effect of quantization of weights is also considered.\n\n2. This paper considers comparisons with AddNNs and BNNs and reports promising results.\n\nWeakness:\n\n1. The “one-time-step SNN” in the paper is actually an ANN with binary activations. It does not consider temporal dynamics of neuron models, so it should not belong to “spatio-temporal computing paradigm” as mentioned in the first sentence of the abstract. It cannot handle temporal inputs such as speech or DVS data, which may possibly be more suitable for SNN with temporal processing ability. So this paper is more close to binary or quantized ANNs compared with general SNNs. The introduction, related work, and comparisons should have more focus on those ANNs than SNNs, and some introduction such as the dynamics of the LIF neuron model in Section 2.1 almost has no relation with models in this paper.\n\n2. The motivation and the effect to introduce Hoyer regularizer and Hoyer spike layer are not clear enough. As introduced in Section 2.2, Hoyer regularizer is first introduced to induce sparsity. Then adding this regularizer for membrane potentials will encourage membrane potentials towards 0, implying that spikes are encouraged to be sparser. However, in Section 3.1, it is said that “it is crucial to reduce the value of the threshold to generate enough spikes for better network convergence”, which is *contradictory* to the regularizer that encourages fewer spikes. Why to combine these two opposite components and why it can work? Besides, there is no explanation for why “Hoyer extremum” is introduced to adjust the threshold and why it can work to balance the threshold and include proper data distribution as shown in Figure 1(a), why Eq. (4) considers clip function, why combining this “Hoyer extremum” with the trainable threshold can be better than the single trainable threshold? There lack significant explanations for the methods.\n\n3. Some descriptions are imprecise. For example, in Section 3.1, “This is because if a neuron does not emit a spike, the pre-synaptic weight connected to it cannot be updated”. What does “pre-synaptic weight” mean? I suppose the authors mean that the synaptic weight will not be updated if the pre-synaptic neuron does not emit a spike when gradients of $w_{ij}$ from neuron $i$ to $j$ are calculated by $g_{u_j}*o_i$, where $g_{u_j}$ is the gradient for $u_j$. \nAnd in Section 3.3, “Note that unlike existing multi-time-step SNN architectures that avoid BN due to difficulty in convergence”. This is not up-to-date since most recent SNN works use BN to achieve high performance [1,2,3,4].\n\n4. The energy results in Table 6 are strange. As introduced in the “Effect on Quantization” paragraph, transforming the full-precision ACs to 2-6 bit ACs will lead to 4.8-15.2 reduction in compute energy. Why in Table 6, there are more than 400-1000 times reduction? Not to mention that only convolutional layers are quantized as said in the paragraph.\n\n5. The comparison with BNN and AddNN in Table 7 is not clear enough. The reported accuracy of the method is based on full-precision (also note that the full-precision results in Table 6 is inconsistent with Table 4 and Table 7) but the reported energy is based on 2-bits quantization, which is inconsistent. And how is the energy of BNN and AddNN calculated? Is AddNN also quantized? As for the proposed methods, even with quantization, it is unclear to me why the energy can be much lower than BNN with only pop-count operations. I think pop-count operations are much more efficient than ACs. These details should be reported and explained.\n\n6. The reference style in the paper is with poor readability. The authors had better distinguish parenthetical and narrative types, i.e. \\citep and \\citet in the latex.\n\n[1] Zheng, H., Wu, Y., Deng, L., Hu, Y., & Li, G. (2021). Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 12, pp. 11062-11070).\n\n[2] Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., & Gu, S. (2021). Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 23426-23439.\n\n[3] Deng, S., Li, Y., Zhang, S., & Gu, S. (2021). Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In International Conference on Learning Representations.\n\n[4] Meng, Q., Xiao, M., Yan, S., Wang, Y., Lin, Z., & Luo, Z. Q. (2022). Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12444-12453).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Orc5KI4H0gV", "reviewer": "Reviewer_YzPx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an SNN model with one timestep only. The key to the model is the use of the Hoyer extremum as a spiking threshold. The Hoyer regularization is a technique previously used for DNN regularization. The proposed SNN corresponds to a DNN with binary activation functions. The backprop pipeline includes the gradient of the threshold function (nondifferentiable), which is approximated to a simple boxcar function with a sufficient width (0-2) to cover the range of inputs. The performance of the proposed model was evaluated in terms of classification accuracy (on CIFAR 10/ImageNet and VOC2007) and computational efficiency. Given the use of binary activation functions, accuracy degradation was inevitable compared with the DNN counterpart. Instead, the authors place emphasis on computational efficiency that may be enhanced given the use of binary activations that may avoid FP multiplications, this is very arguable though. ", "summary_of_the_review": "The proposed model (SNN with one timestep) corresponds to a DNN with binary activations. Therefore, the model cannot leverage the rich dynamics inherently given to SNNs, and thus the proposed model is not able to process time-varying event stream. Given this reason, I found marginal significance in this work. As well, Novelty is limited given that the Hoyer regularization concept was proposed previously. Regarding technical aspects of the proposed model, the loss in accuracy due to the use of binary activations is obvious, and the computational efficiency estimation is also very arguable.", "strength_and_weaknesses": "Strength:\\\nThe proposed model was verified on various datasets (CIFAR10, ImageNet, VOC2007) and compared with previous models including DNNs, BNNs, SNNs in a systematic manner.\n\nWeaknesses:\n\n1. SNN comes into its own when encoding time-varying data, so that it should be able to extract spatiotemporal patterns of event stream through time. Therefore, SNN should be defined in a time domain. The authors proposed time-independent SNNs given the use of a single timestep, which are not SNNs as a matter of fact given the key to SNNs, i.e., dynamic models. Therefore, the advantages of the time-independent SNN over DNNs are not sure. In fact, the model used in this work is a DNN with binary activations rather than SNN. \n\n2. The CE comparison between the proposed model and DNN in Eq. 14 is not fair. The proposed model includes additional operations that are not addressed in the CE calculation. They include the division in Eq. 3 and comparison with the threshold in FP in Eq. 4. Note that the comparison with the FP number is much more expansive than the comparison in ReLU, which can simply be done by seeing the sign bit of an input. Moreover, I wonder if FP multiplications are avoided by using binary activations (0/1 or 0.0000…/1.0000…) because this depends on hardware used. I recommend the authors to specify the hardware considered for the proposed model.\n\n3. Where is the multiplier $\\lambda_H$ in Eq.6?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CzH2QePYgui", "reviewer": "Reviewer_ujG1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a training framework (from scratch) for one- time-step SNNs that uses a novel variant of the recently proposed Hoyer regularizer. By estimating the threshold of each SNN layer as the Hoyer extremum of a clipped version of its activation map, the approach not only downscales the value of the trainable threshold, but also shifts the membrane potential values away from the threshold. The approach outperforms existing spiking, binary, and adder neural networks in terms of accuracy-FLOPs trade-off for complex image recognition tasks.\n\n\n%%%%%%%%%%%%%\n\nUpdate:\nThe authors have addressed part of my initial concern about bio-plausibility and limitation of BNN. Yet, I don't think the proposed method is that novel. So I change my score to 5. ", "summary_of_the_review": "The paper fail to clarify the problem importance, and the proposed method is limited novel. So I vote for a reject.", "strength_and_weaknesses": "Strength\n\n1. The paper is well-written and easy to follow.\n2. The experiments show the method with low-latency.\n\nWeaknesses\n1. The problem importance is unclear. One-time-step SNNs is simply binary network. The lack of dynamics make it less biologically plausible.\n2. Limited novelty. The proposed method is simply training with HOYER regularizer, while similar method has been proposed in [1]\n\n\n[1]Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. In International Conference on Learning Representa- tions, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9O0zQcem5Rs", "reviewer": "Reviewer_hEby", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an interesting training algorithm for training SNNs from scratch with only one timestep.", "summary_of_the_review": "I am not very convinced by the novelty and true efficiency advantage of this method since many comparisons and experiments are limited.", "strength_and_weaknesses": "Very comprehensive results\n\nSimple yet effective idea\n\nSince the authors use a regularization technique, I am wondering if the authors can shed light on how their method differs from previous temporal BN methods proposed by prior works that have shown accuracy improvement while decreasing the total timesteps [1, 2]. \n\n[1]Kim, Y., & Panda, P. (2020). Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, 1638. \n\n[2] Zheng, Hanle, et al. \"Going deeper with directly-trained larger spiking neural networks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\nThere is a plethora of works today on SNN algorithmic training- precisely talking about how we can get improved accuracy with less timesteps. But, I am more concerned by the fact that in such large-scale settings, are SNNs going to be actually advantageous? The authors show some simplistic energy estimation results which is grossly approximate. For true energy estimation, they have to consider memory access and data access energy which turns out to expend a lot of computations in SNNs given their repeated time-wise computation. In a recent work [3], true energy estimation on a systolic accelerator precisely shows that SNNs are not very advantageous over ANNs because repeated timestep computations will lead to redundant access of weights and membrane potentials is going to further add to energy unless we really improve the sparsity rate. Can the authors kindly comment on this - and it may be worthwhile for authors to include a discussion n the relevance of using more mainstream tools for energy estimation rather than just doing analytical modeling of FLOPS count?\n\n [3] Yin, Ruokai, et al. \"SATA: Sparsity-Aware Training Accelerator for Spiking Neural Networks.\" arXiv preprint arXiv:2204.05422 (2022).\n\nComing to my next point, there is a recent work [4] that explores sparse SNNs using lottery ticket hypothesis or NAS [R5]  to truly take advantage of SNNs energy efficiency over ANNs. Can the authors comment on how their model in terms of parameter count compares to these sparse SNN models which in fact show SOTA accuracy on CIFAR10,100 with very low timestep count? \n\n[4] Kim, Youngeun, et al. \"Lottery Ticket Hypothesis for Spiking Neural Networks.\" arXiv preprint arXiv:2207.01382 (2022).\n\n[R5] Kim, Youngeun, et al. \"Neural architecture search for spiking neural networks.\" arXiv preprint arXiv:2201.10355 (2022).\n\nFinally, I think it is well known that SNNs will be more suited to DVS or event based tasks as compared to standard digital camera recognition models. Recent works have shown superiority of SNNs over ANNs on these neuromorphic datasets [5, 6, 7]. Can the authors run their model on one of these datasets and compare to [5,6,7]? \n\n[5] Li, Yuhang, et al. \"Neuromorphic Data Augmentation for Training Spiking Neural Networks.\" arXiv preprint arXiv:2203.06145 (2022).\n\n[6] Kim, Youngeun, and Priyadarshini Panda. \"Optimizing deeper spiking neural networks for dynamic vision sensing.\" Neural Networks 144 (2021): 686-698. \n\n[7] Kim, Y., Chough, J., & Panda, P. (2022). Beyond classification: directly training spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineering.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject