DIET-SNN: A Low-Latency Spiking Neural Network with Direct Input Encoding & Leakage and Threshold Optimization
Authors: Nitin Rathi
Kaushik Roy
Venue:  (2021)
Keywords: Spiking neural networks
threshold optimization
leak optimization
input encoding
deep convolutional networks

Abstract: Bio-inspired spiking neural networks (SNNs), operating with asynchronous binary signals (or spikes) distributed over time, can potentially lead to greater computational efficiency on event-driven hardware. The state-of-the-art SNNs suffer from high inference latency, resulting from inefficient input encoding, and sub-optimal settings of the neuron parameters (firing threshold, and membrane leak). We propose DIET-SNN, a low latency deep spiking network that is trained with gradient descent to optimize the membrane leak and the firing threshold along with other network parameters (weights). The membrane leak and threshold for each layer of the SNN are optimized with end-to-end backpropagation to achieve competitive accuracy at reduced latency. The analog pixel values of an image are directly applied to the input layer of DIET-SNN without the need to convert to spike-train. The first convolutional layer is trained to convert inputs into spikes where leaky-integrate-and-fire (LIF) neurons integrate the weighted inputs and generate an output spike when the membrane potential crosses the trained firing threshold. The trained membrane leak controls the flow of input information and attenuates irrelevant inputs to increase the activation sparsity in the convolutional and linear layers of the network. The reduced latency combined with high activation sparsity provides large improvements in computational efficiency. We evaluate DIET-SNN on image classification tasks from CIFAR and ImageNet datasets on VGG and ResNet architectures. We achieve top-1 accuracy of 69% with 5 timesteps (inference latency) on the ImageNet dataset with 12x less compute energy than an equivalent standard ANN. Additionally, DIET-SNN performs 20-500x faster inference compared to other state-of-the-art SNN models.

Under review as a conference paper at ICLR 2021
DIET-SNN:
A LOW-LATENCY SPIKING NEURAL NETWORK WITH DIRECT INPUT ENCODING & LEAKAGE AND THRESHOLD OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Bio-inspired spiking neural networks (SNNs), operating with asynchronous binary signals (or spikes) distributed over time, can potentially lead to greater computational efficiency event-driven hardware. The state-of-the-art. SNNs suffer from high inference latency, resulting from inefficient input encoding, and sub-optimal settings of the neuron parameters (firing threshold, and membrane leak). We propose DIET-SNN, a low-latency deep spiking network that is trained with gradient descent to optimize the membrane leak and the firing threshold along with other network parameters (weights). The membrane leak and threshold for each layer of the SNN are optimized with end-to-end backpropagation to achieve competitive accuracy at reduced latency. The analog pixel values of an image are directly applied to the input layer of DIET-SNN without the need to convert to spike-train. The first convolutional layer is trained to convert inputs into spikes where ceaky-integrate-ame annd-fir (LIF) neurons integrate the weighted inputs and generate output spike when the membrane potential crosses the trained firing threshold. The trained membrane leak controls the flow of input information and attenuates irrelevant inputs to increase the activation sparsity in the convolutional and dense layers of the network. The reduced latency combined with high activation sparsity provides large improvements in computational efficiency. We evaluate DIET-SNN on image classification tasks from CIFAR and ImageNet datasets on VGG and ResNet architectures. We achieve top-1 accuracy of 69% with 5 timesteps (inference latency) on the ImageNet dataset with 12x less compute energy than an equivalent standard ANN. Additionally, DIET-SNN performs 20 500x faster inference compared to other state-of-the-ar SNN models.
INTRODUCTION
In recent years, a class of neural networks inspired by the event-driven form of computations in the brain has gained popularity for their promise of low-power computing (Painkras et al. 2013 Davies et al. 2018). Spiking neural networks (SNNs) first emerged in computational neuroscience as an attempt to model the behavior of biological neurons (Mainen & Sejnowski 1995). They were pursued for low-complexity tasks implemented on bio-plausible neuromorphic platforms. At the same time in standard deep learning, the analog-valued valued artificial neural networks (ANNs) became the de-facto model for training various computer vision and natural language processing tasks (Krizhevsky et al., 2012 Hinton et al.) 2012).. The skyrocketing performance and success of multi-layer ANNs came at a sigmificant power and energy cost (Li et al. (0166). Recently, major chip maker Nvidia estimated that 80 900 of the energy cost of neural networks at data centers lies inference processing (Freund 2019). The tremendous energy costs and the demand for edge intelligence on battery-powen devices have shifted the focus on exploring lightweight energy-efficiee inference models for machine intelligence. To that effect, various techniques such weight pruning (Han al. 2015), model compression (He et al. 2018), and quantization methods (Chakraborty et al. (020) are proposed to reduce the size and computations in ANNs. Nonetheless, the inherent one-shot analog computation in ANNs requires the expensive operation of multiplying two real numbers (except when both weights and activations are 1-bit (Rastegari al. 2016)). In contrast, SNNs inherently compute and

---

Under review as a conference paper at ICLR 2021
transmit information with binary signals distributed over time, providing a promising alternative for power-efficient machine intelligence.
For a long time, the success of SNNs was delayed due to the unavailability of good learning algorithms. But in recent years, the advent of supervised learning algorithms for SNN has overcome many of the roadblocks surrounding the discontinuous derivative of the spike activation function. Since SNNs receive and transmit information through spikes, analog values need to be encoded into spikes. There are a plethora of input encoding methods like rate coding (Diehl et al. 2015; Sengupta et al.) 2019), temporal coding (Comsa et al. 2020), rank-order coding (Kheradpisheh & Masquelier] 2222). and other special coding schemes (Almomani et al.) 2019). Among these, rate- coding has shown competitive performance on complex tasks (Diehl et al. 2013 Sengupta et al.) [211 Lee et al. (019)
while others are limited to simple tasks like learning the XOR function and classifying digits from the MNIST dataset. Also, dynamic vision sensors (DVS) record the change in image pixel intensities and directly convert to spikes that can estimate optical flow (Lee et al., (020) and classify hand gestures (Shrestha & Orchard 2018). In rate coding, the analog value is represented by the rate of firing of the neuron. In each timestep, the neuron either fires (output 'I') or stays inactive (output '0"). The number of timestepy determines the discretization error in the representation of the analog value by spike-train. This leads to adopting a large number of timesteps for high accuracy at the expense of high inference latency (Sengupta al., (019). The two other parameters that crucial for SNNs are firing threshold of the neuron and membrane potential leak. The neuron fires when the membrane potential exceeds the firing threshold and the potential is reset after each firing. Such neurons are usually referred to as integrate-and and-fire (IF) neurons. The threshold value is very significant for the correct operation of SNNs because high threshold will prevent the neuron from firing (dead-neurgn problem), and a lower threshold will lead to unnecessary firing, affecting the ability of the neuron to differentiate between two input patterns. Another neuron model, maaky-nttegrat--- -adde---nnreeee (LIF), introduces leak factor that allows the membrane potential to keep shrinking over time (Gerstner & Kistler 2002). Most of the recent work on supervised learning SNNs has either employed the IF or the LIF neuron model (Dichl et al. 2015 (Senuppta et al. 2019 Lee et al. 2019 Rathi et al. 2020 Han et al. 2020). Some proposals adopt kernel-based spike response models (Huh & Sejnowski
simple datasets and do not scale for deep networks. The leak provides an additional knob that can potentially be used to tune SNNs for better energy -efegy-efficienc. However, there has not been any exploration of the full design space of optimizing the leak and the threshold to achieve better latency (or energy) and accuracy tradeoff. Research, network that can limit the capabilities of SNNs (Lee et al., 2019) Rathi et al. 2020). The firing thresholds are also fixed (Lee et al.) 2019) or selected based on some heuristics (Sengupta et al.) 2019) Rueckauer et al. (017)). Some recent works employ leak/threshold optimization but their application is limited to simple datasets (Fang, 2020 Yin et al. 2020). The current challenges SNN models are high inference latency and energy, long training time, and high training costs in terms of memory and computation. Most of these challenges arise due to in-efficient input encoding, and improper methods of selecting the membrane leak and the threshold.
2018|Boht et al. 2000), but for the most part, these approaches show limited performance
far, has been mainly focused on using fixed leak for the entire
To address these challenges, this paper makes the following contributions:
We propose gradient descent based training method that Iearns the correct membrane leak and firing threshold for each layer of a deep spiking network via error-back proopaaattoooioooonn The goal is jointly optimize the neuron parameters (membrane leak and threshold) and the network parameters (weights) to achieve high accuracy at low inference latency. The tailored membrane leak and threshold for each layer leads to large improvement in activation sparsity and enfrgy-efficienc..
We train the first convolutional layer to act as the spike-generaton whose spike-rate is function of the weights, membrane leak, and threshold. This also eliminates the need for a generator function (and associated overheads) used in other coding schemey
WWall-clock time for 'timestep' is dependent the number of computations performed and the underlying hardware (Frady 2020). In simulation, timestep the time taken to perform 1 forward pass
FFrr rate-core coding, Poisson generator is used to convert the analog values to spike-train (Diehl et al. 2015 The encoder generates random numbers every timestep and compares it with the analog values to produce the spikes.
2

---

Under review as a conference paper at ICLR 2021
To evaluate the effectiveness of the proposed algorithm, we train SNNs on both VGG (Simonyan & Zisserman 2014) and ResNet (He et al. (016) architectures for
CIFAR (Krizhevsky al.] 2009) and ImageNet (Deng et al. 2009) datasets. DIET-SNN achieves similar accuracy as ANN with 6 18x less compute energy. The performance is achieved inference lateney of timesteps compared to 100 2000 timesteps for
state-of-the-a SNN models.
The source code implemented in Pytorch framework is available in the supplementary materials.
2 BACKGROUND AND RELATED WORK
The development of efficient learning algorithms for deep SNNs is an on-going research challenge. There has been significant amount of success with recent supervised learning algorithms (Bellec
20196] Han et al. (2020) that can be broadly classified as conversion algorithms (Cao et al. 2015 Rueckauer al. | 2011 Sengupta et al.) 2019) and spike-based backpropagation algorithms (Bellec et al. [0188] Lee et al. 2019). There are also some variants of bio-plausible algorithms that employ feedback alignment to update the weights. These algorithms apply random weights (Lillicrap et al., (016)) or fixed weights (Samadi et al.] (017)) as the feedback weight during backpropagationn The success of these algorithms is limited to simple tasks. Therefore, we focus our discussion ANN-SNN conversion and backpropagation algorithms that are more suitable for complex tasks and are scalable to deep networks.
et al. 2018: Wu et al., 2018 Sengupta et al.) 2019:; Wu et al.
ANN-SNN Conversion: ANN-SNN conversion is the most successful method of training rate-coded deep SNNs (Cao et al., 2015 Dichl et al. 2015 Rueckauer et al.) 2017; Sengupta et al.) 2019 Han jtt al. 2020). In this process, ANN with ReLU neurons is trained with standard backpropagation with some restrictions (no bias, average pooling, no batch normalization), although some works have shown that some of the restrictions can be relaxed (Rueckauer al., (017). Next, SNN with IF neurons and iso-architecture as ANN is initialized with the weights of the trained ANN. The underlying principle of this process is that a ReLU neuron can be mapped to an IF neuron with minimum loss. The mapping is possible for SNNs that operate on rate-coded inputs (Dichl (0.5) Sengupta et al., 2219) or on direct input encoding (Rueckauer et al.) (017). The major bottleneck of this method is to determine the firing threshold of the IF neurons that can balance the accuracy-latency tradeoff. Generally, the threshold is computed as the maximum pre-activation of the IF neuron resulting in high inference accuracy at the cost of high inference latency (2000- - 2500 timesteps) (Sengupta et al., 2019). In recent work, the authors showed that instead of using the maximum pre-activation, certain percentile of the pre-activation distribution reduces the inference latency (100- 200 timesteps) with minimal accuracy drop (Lu & Sengupta 2020). These heuristic techniques of determining the firing threshold lead to sub-optimal securacy-latenge tradeoff. Additionally, ANN-SNN conversion has a major drawback: the absence of the timing information. The quintessential parameter 'time' is not utilized in the conversion process which leads to higher inference latency.
Error Backpropagation in SNN: ANNs have achieved success with gradient-based training that backpropagates the error signal from the output layer to the input layer. It requires computing a gradient of each operation performed in the forward pass. Unfortunately, the IF and the LIF neuron does not have continuous derivative. The derivative of the spike function (Dirac delta) is undefined at the time of spike and '0' otherwise. This has hindered the application of standard backpropagation in SNN. There have been many proposals to perform gradient-based training in SNNs (Huh & Sejnowski 2018 Lee et al. 2019) - among them, the most successful is surrogate-gradient based optimization (Nefici et al.] 2019). The discontinuous derivative of the IF neuron is approximated by a continuous function that serves as the surrogate for the real gradient. SNNs trained with surrogate- gradient perform backpropagation through time (BPTT) to achieve high accuracy and low latency (100 timesteps), but the training is very compute and memory intensive in terms of total training iterations compared to conversion techniques. The multiple-iteration training effort with exploding memory requirement for backpropagation has limited the application of this method to simpler tasks on shallow architectures (Lee et al. 2019).
In standard backpropagation, the feedhack weight is W where Wi is the weight used in the forward pass
3

---

Under review as a conference paper at ICLR 2021
Hybrid SNN Training: In recent work, the authors proposed hybrid mechanism circumvent the high training costs of backpropagation as well as maintain low inference latency (100 250 timesteps) (Rathi et al. 2020 The method involves both ANN-SNN conversion and error- citoo-backrropggatiin A trained ANN is converted to an SNN as described earlier and the weights of the converted SNN are further fine-tuned with surrogate gradient and BPTT. The authors showed a faster convergence (< 20 epochs) in SNN training due to the precursory initialization from the ANN-SNN conversion. This presents practically feasible method train deep SNNs with limited resources, which is otherwise challenging with only backpropagation from random initialization (Lee et al. 2019). Hybrid training tries achieve the best of both worlds: high accuracy and low latency. But still employs rate coding, fixed membrane leak, and fixed threshold, and therefore, the latency-accuracy tradeoff can be improved further.
In this work, we adopt the hybrid training method to train the SNNs. We start with ANN-SNN conversion and select the threshold for each layer as 95 percentile of the pre-activation distribution. The pixel intensities are directly applied in the input layer during the threshold computation. This serves as the initial model that is further trained to optimize the membrane leak and threshold.
ALGORITHM FOR TRAINING DIET-SNN
Direct Input Encoding: The pixel intensities of an image are applied directly to the input layer of the SNN at each timestep (Rueckauer et al. 2017 Lu & Sengupta 2020). The first convolutional layer composed of LIF neurons acts as both the feature extractor and the accumulates the weighted pixel values and generates output spikes. This is similar to rate-coding, but the spike-rate is function of the weights, membrane leak, and threshold that are all learned by gradient-descent.
which
Neuron Model: We employ the LIF neuron model described by
ii = Ara + Σ wiyoy v20
(1)
o 6-1 [1, otherwise
if if
1-1
and
(2)
0/
where is the membrane potential, A is the leak factor with a value in [0 - 1), w is the weight connecting pre-neuron and post-neuron i, is the binary spike output, is the firing threshold, and I represents the timestep. The first term Equation denotes the leakage in the membrane potential, the second term integrates the weighted input received from pre-neuron, and the third term accounts for the reduction in potential when the neuron generates an output spike. After the spike, a soft reset is performed where the potential is reduced by threshold instead of resetting to (Han et al.) 2020). The threshold governs the average integration time of input, and the leak regulates how much of the potential is retained from the previous timestep. Now, derive the expressions compute the gradients of the parameters at all layers. The spatial and temporal credit assignment is performed by unrolling the network in time and employing BPTT.
Output layer: The neuron model in the output layer only accumulates the incoming inputs without any leakage and does not generate an output spike and is described by
ut 1-1
WioL-1
(3)
where u, is a vector containing the membrane potential of N output neurons, is the number of classes in the task, W, is the weight matrix connecting the output layer and the previous layer, and o.-1 is a vector containing the spike signals from layer (1 1). The loss function is defined on us at the last timestep T. We employ the cross-entropy loss and the softmax is computed on u. The symbol T' is used for timestep and not to denote the transpose of a matrix.
81
s(uf)
8i
N
(4)
k=
SN.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Top-1 classification accuracy
Architecture
ANN
ANN-SNN
Weight Optimization
DIET-SNN
Timesteps (T)
CIFARIO
VGG6
90.80%
86.19%
89.24%
89.42%
5
90.05%
VGG16
93.72%
73.52%
91.68%
92.70%
93.44%
ResNer20
92.79%
47.26%
90.29%
91.78%
92.54%
CIFAR100
VGG16
71.82%
46.54%
65.83%
69.67%
5
ResNer20
64.64%%
31.40%
62.95%
64.07%
ImageNet
VGG16
70.08%
24.58%
64.32%
69.00%
5
aL
L L=
Σ yilog(ss) (s:),
duf
y
(5)
where is the vector containing the softmax values, L is the loss function, and y is the one-hot encoded vector of the true label or target. The weight update is computed as
W W, nWW,
(6)
AW Σ
OL
Σ
ÖL du ÔL
Σ duj
= (s - 4) of
(7)
OW,
du OW. dul
OW:
0L
OL du
(8 W)Wr
(8)
0o duf
0
where 1) is the learning rate.
Hidden layers: The neurons in the convolutional and fully-connected layers are defined by the LIF model
ui = A t-1
Wio - 1:oi-1 1, if if
(9)
uf - and of {o: otherwise
(10)
where A (vr) a real value representing leak (threshold) for all neurons in layer All neurons in layer share the same leak and threshold value. This reduces the number of trainable parameters and we did not observe any significant improvement by assigning individual threshold/leak to each neuron. The weight update is calculated as
aL
ÖL do dzt Out
aL do dof 0z
AW
Σ
Σ
Σ
of
(11)
OW,
dof dz du, OW:
11
0ol/0s the discontinuous gradient and we approximate it with the surrogate gradient (Bellec et 2018
dof = 7 max - |z/l) and aof do 0z Oof duf dz dut
(12)
0z in where y is a constant denoting the maximum value of the gradient. The threshold update is then computed as
dzf
nAw
(13)
Σ AL
aL do dz
ØL dof do dz
( (11o us
Aul
Σ
Σ
(14)
0v: do dz de
(11)2
And finally the leak update is computed
aL
aL dot dul
OL dot u 1-1
X = A and A
Σ
Σ
Σ
(15)
oo du ad
Dof du
4 EXPERIMENTS
5

---

computations and no Under review as a conference paper at ICLR 2021
Table 2: DIET-SNN compared with other SNN models
Model
Method
Architecture
SNN Accuracy
Timesteps
CIFARTO
Sengupta et al.
(2019)
ANN-SNN
VGG16
91.55%
2500
Rueckauer al.
2017
ANN-SNN
4 Conv, 2 FC
90.85%
400
Rathi et al. 2020)
Hybrid
VGGI6
92.02%
200
Loc al.
2019
Backprop
VGG9
90.45%
100
Wu et al.
2019b
Backprop
CIFARNet
90.53%
12
Wu et al.
2019a)
Backprop
CIFARNet
90.98%
Zhang & (2020)
Backprop
CIFARNet
91.41%
This work
DIET-SNN
CIFARNet
91.59%
This work
DIET-SNN
VGG16
92.70%
5
CIFARIO0
Han et al. (2020)
ANN-SNN
VGGI6
70.09%
768
Rathi et al. (2020)
Hybrid
VGG11
67.87%
125
Lu Sengupta 2220
ANN-SNN
VGG15
63.20%
62
This work
DIET-SNN
VGG16
69.67%
5
ImageNet
Sengupta et al.
(2019)
ANN-SNN
VGGI6
69.96%
2500
Han et al. (2020
ANN-SNN
VGG16
71.34%
768
Rueckauer al.
2017
ANN-SNN
VGG16
49.61%
400
Rathi et al. 2020)
Hybrid
VGG16
65.19%
250
Lu Sengupta (2020)
ANN-SNN
VGG15
66.56%
Wu et al. (2019a)
Backprop
AlexNet
50.22%
10
This work
DIET-SNN
VGG16
69.00%
The three-step DIET-SNN training pipeline (Fig. begins with training an ANN without the bias term and batch-normalization to achieve minimal loss during ANN-SNN conversion (Diehl et al.
Train ANN
(RelU, no bias term, no batch normalization, average pooling)
2015 Sengupta et al. [219 Rathi et al. 2020j. Dropout (Srivastava et al. 2014) is used the regularizer and the dropout mask is unchanged during all timesteps of an input sample (Rathi et al. 2020). Average-pooling is used to reduce the feature map size in VGG architectures, whereas for ResNets, a stride of 2 is employed to reduce the feature size. Supplementary material contains the complete architecture details, training hyperparameters, and dataset descriptions. Next, the trained ANN is converted SNN with IF neurons. The threshold for each layer is computed sequentially as the 99.7 percentile Figure 1: Training pipeline of the pre-activation distribution at each layer. The pre-activation for each neuron the weighted sum of inputs Σ, 9jthi received by the neuron. During threshold computation, the leak in the hidden layers is set to unity, and the input layer employs direct input encoding. Finally, the converted SNN bacpropagatooo to optimize the weights, the membrane leak, and the firing thresholds of each layer as described by the equations in Section 3 We evaluated the performance of these networks on VGG and ResNet architectures for CIFAR and ImageNet datasets. Column-2 in Table shows the ANN accuracy; column-3 shows the accuracy after ANN-SNN conversion with T timesteps; column-4 shows the accuracy when only the weights in SNN are trained with spike-based backpropagation; column-5 shows the accuracy when the weights, threshold, and leak are jointly optimized (DIET-SNN). The performance of DIET-SNN compared to current state-of-thee of-the-art SNNs is shown in Table 2 DIET-SNN shows 5 100x improvement in inference latency compared to other spiking networks. The authors in Wu et al. (2019b) achieved convergence in 12 timesteps with special input encoding layer, but the extra overhead may affect the overall latency and energy.
ANN conversion (IF neuron, direct input encoding. threshold balancing)
(LIF neuron, direct input encoding, optimize weights, leaks, and thresholds)
trained with
ENERGY EFFICIENCY
In ANN, each operation computes dot-product involving one floating sooating-point (FP) multiplication and one FP addition (MAC), whereas, in SNN, each operation is only one FP addition due to binary spikes. The computations in SNN implemented on neuromorphic hardware are event- event-driven (Davies) al. 2018 Frady al. 2020). Therefore, in the absence of spikes, there

---

Under review as a conference paper at ICLR 2021
Table 3: ANN vs DIET-SNN compute energy. Each operation in ANN (SNN) consumes 4.6pJ (0.9pJ). The input layer in DIET-SNN is non-spiking, so it's energy is same as ANN. Column-5 shows the ratio of Hoperations in input layer to the total #operations the network.
Architecture (timesteps)
Normalized #OPANN
Normalized #OPsNN(b)
#OP layer Total #op
ANN/ DIET-SNN
Dataset
(c)
Energy a
c.4.6+ (1466-11-))-40.3 +)600.
VGG6 (T=5)
CIFARTO
1.0
0.14
0.029
18
VGG16 (T=5)
CIFAR10
1.0
0.39
0.005
12.4
VGG16 (T=5)
CIFAR 100
1.0
0.40
0.005
12.1
VGG16 (T=5)
ImageNet
1.0
0,41
0.006
11.7
ResNet20 (T=5)
CIFARIO
1.0
0.76
0.013
6.3
ResNer20 (T=5)
CIFARI00
1.0
0.72
0.013
6.6
active energy is consumed. We computed the energy cost/operation for ANNs and SNNs in 45nm
CMOS technology. The energy cost for 32-bit ANN MAC operation (4.6p.)) is 5. x more than SNN addition operation (0.9pJ) (Horowitz) 2014). These numbers may vary for different technologies, but generally, in most technologies, the addition operation is much cheaper than the multiplication operation. In ANN, the number of operations/layer is defined by
Kiw X kn Cin Bout X Wont Couts
Convolution layer
#OPANN
(16)
ffn fouts
Fully-connected layer
where kiw (kt.) is kernel width (height), Can (Cout) the number of input (output) channels, hout (Wout) is the height (width) of the output feature map, and fin (fou) is the number of input (output) features. The number of operations/layer in iso-architecture SNN is specified by
#OPsNN Spike Rate; #OPANN
(17)
SpikeRate #TotalSpikes over all inference timesteps
(18)
#Neurons:
where Spike Rate; is the total spikes in layer over all timesteps averaged over the number of neurons in layer I. A spike rate of 1 (every neuron fired once) implies that the number of operations for ANN and SNN are the same (though operations are MAC in ANN while addition in SNNs). Lower spike rates denote more sparsity in spike events and higher energy-efficieme VGG16 during inference around 1.6 (Fig. R(a).. indicating that DIET-SNN more energy-efficien than ANN. In deeper layers, the leak decreases while the threshold increases (Fig. (b)) resulting in lower spike rates and better energy-effien mnergy-efficieny..
The average spike rate for
Table 3 shows the compute energy comparison of ANN and DIET-SNN. The energy for ResNet is more than VGG because more than 50% of the total operations in ResNet occurs in the first 3 layers where the spike rate is high. The layerwise spike rate in ResNet is provided in the supplementary section. The standard ResNet architecture was modified with initial 3 plain convolutional layers to minimize the accuracy loss during ANN-SNN conversion (Sengupta al. 2019). In Lu & Sengupta (2020) an average spike rate of 2.35 for VGG16 on CIFARI00 was reported with 62% accuracy. The maximum spike rate of 20 reported for VGG16 architecture on CIFAR 10 dataset (Rathi et al. 2020). DIET-SNN performs considerably better in all metrics compared to these models and achieves better compute energy than ANN on complex tasks like CIFAR and ImageNet with similar accuracy. We did not consider the data movement cost in our evaluation as it is dependent on the system architecture and the underlying hardware implementation. Although we would like to mention that in SNN the membrane potentials have to be fetched at every timestep, in addition to the weights and activations. Many proposals reduce the memory cost by data buffering (Shen et al.) (017). trading computations for memory (Chen et al.] 2016a), and data reuse through efficient dataflows (Chen et al. 2016b). All such techniques can be extended SNNs to address the memory cost. The training of SNNs is still cause of concern for energy-efficiency because it requires several days, even on high-performance GPUs. The hybrid approach and DIET-SNN alleviate the issue by reducing the number of training epochs and the number of timesteps. but further innovations in both algorithms and accelerators for SNNs are required to reduce the training cost.
EFFECT OF DIRECT INPUT ENCODING AND THRESHOLD/LEAN OPTIMIZATION

---

Under review as a conference paper at ICLR 2021
Average Spuke Kate
Theeshod
Thaeshold toiiiig
Lank canekeellletttiiiii toinkng
CIFARI0; 0.39, CIMAR100 0.40, ImsggeNet: 0:41
Low high,
Theshold & Lonk
threshold lennls
ICFARR1 (1-5)
hightr spausity
SpikeTabe
MCRRRDD (T:s)
mimageNet
0.5
13 Linoar
Convolution
Consolution
(a) Spike Rate
(b) Leak and Threshold after training
Figure 2: (a) Layerwise spike rate for VGG16 during inference over entire test-set. Average spike rate is calculated as total #spikes/# uneurone. An average spike rate of 1.65 indicates that every neuron fired on average 1.65 times for each image over all timesteps. (b) Layerwise leak and threshold for VGG16 on CIFARI00 dataset. The threshold before training represents the values obtained from ANN-SNN conversion process. The leak before training is unity for all layers.
Table | shows the effect on accuracy (under (so-timesteps) by training threshold/leak along with direct input encoding. In this section, analyze the impact of input encoding, threshold and leak on the average spike rate and latency (under iso-accuracy).
130
Anerage Syloe
300
We train four different spiking networks: (a) SNN
with IF neuron and Poisson rate encoding; (b) SNN with IF neuron and direct input encoding; (c) threshold optimization added to (b): (d) SNN with LIF neuron, analog encoding, and threshold/eeak optimization (DIET-SNN). The SNNs are trained to achieve similar accuracy for VGG16 on CIFARIO 10. The networks (d)-(d) achieved an accuracy of 90% 92.41%. 92.37%, and 92.70% respectively. The network with Poisson input encoding required 150 timesteps with average spike rate of 26 (Fig. 3) By replacing Poisson encoding with direct input encoding (proposed work), the latency spike--rate) improved to 25 timesteps (1.94). As mentioned in Section | the information in rate-coded SNNs is encoded in the firing rate of the neuron. In Poisson encoding. the firing rate is proportional the pixel value, whereas in direct input encoding, the SNN learns the optimal firing rate by training the parameters of the first convolutional layer. This reduces the number of timesteps required to encode the input. Next, the addition of threshold optimization reduces the latency (spike-rate) to 15 timesteps (1.47). In SNNs with IF neurons, a neuron's 's activity is dependent on the ratio of the weights and the neuron's 's threshold. Therefore, training only weights should be sufficient, however, training threshold along with weights provides additional parameter for optimization and leads to lower latency as shown in network (c) compared to (b). Finally, the addition of the leak parameter reduces the latency (spike-rate) to 5 timesteps (0.39). The leak and threshold together eliminate the excess membrane potential and suppress unnecessary firing activities that improve the latency and the spike-rate. The compute energy compared to ANN (Table 3) for the networks (d)-(d) are 0.2, 2.6, 3.4, and 12.4. respectively. Therefore, the co-optimization of weights, leak, and threshold along with direct input encoding provide SNN with improved latency and low conrgy-consumption.
25
Potssoc ennoting
Irrst eecociing
Figure 3: Effect of employing direct input encoding, threshold and leak optimization
CONCLUSIONS
SNNS that operate with asynchronous discrete events can potentially solve the energy issue in deep learning. To that effect, we presented DIET-SNN, an energy-efficin spiking network that trained to operate with low inference latency and high activation sparsity. The membrane leak and the firing threshold of the LIF neurons are trained with error- wrrorbbckkropggatiion along with the weights of the network to optimize both accuracy and latency. We initialize the parameters of DIET-SNN, taken from a trained ANN, to speed-up the training with baske backpropagation. The image pixels are applied directly as input to the network, and the first convolutional layer is trained to perform the spike- siteeeeeeeatiinn operation. This leads to high activation sparsity in the convolutional and dense layers of the network. The high sparsity combined with low inference latency reduces the compute energy by 2 - 8x compared to an equivalent ANN with similar accuracy. DJET-SNN achieves similar accuracy as other state-of-the- SNN models with 5 100x less number of timesteps.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ammar Almomani, Mohammad Alauthman, Mohammed Alweshah, O Dorgham, and Firas Albalas. comparative study on spiking neural network encoding schema: implemented with cloud computing. Cluster Computing, 22(2):419-4333 2019.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-lear in networks of spiking neurons. In Advances in Neural Information Processing Systems, pp. 787-797, 2018.
Sander M Bohte, Joost N Kok, and Johannes A La Poutré. Spikeprop: backpropagation for networks of spiking neurons. In ESANN, pp. 419442442 419, 2000.
Yonggiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54-66 :6.666,66666 2015.
Indranil Chakraborty, Deboleena Roy, Isha Garg, Aayush Ankit, and Kaushik Roy. Constructing energy-efficient mixed-precision neural networks through principal component analysis for edge intelligence. Nature Machine Intelligence, pp. 1-13, 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:/604.060 067744 2016a.
Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Eyeriss: A spatial architecture for energy- energy-efficient dataflow for convolutional neural networks, ACM SIGARCH Computer Architecture News, 44(3): 367-379, 2016b.
Julia M Comsa, Thomas Fischbacher, Krzysztof Potempa, Andrea Gesmundo, Luca Versari, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8529-8533 IEEE, 2020.
Mike Davies, Narayan Srinivasa, Tsung-Har Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-aceuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference on Neural Networks (IJCN), pp. 1-8. IEEE, 2015.
Wei Fang. Leaky integrate- fire spiking neuron with learnable membrane time parameter. arXiv preprint arXiv:2007.05788 2020.
Paxon Frady, Garrick Orchard, David Florey, Nabil Imam, Ruokun Liu, Joyesh Mishra, Jonathan Tse, Andreas Wild, Friedrich T Sommer, and Mike Davies. Neuromorphic nearest-neighbor search using intel's pohoiki springs. arXiv preprint arXiv:2004. 12691, 2020.
Karl Freund. URL
Google cloud doubles down on nvidia gpus for inference, 2019.
goog el doubloolleeeeoooo le dwwn- wwww--on--viidie. http://ewoo.ooooooooooooooooooor
4745312fe6792
Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snns: Residual membrane potential neuron for enabling deeper high-ace accuracy and low-latency spiking neural networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2020.

---

Under review as a conference paper at ICLR 2021
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0011 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 784-800, 2018.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patriek Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6): 2012.
Mark Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14.
IEEE, 2014.
Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks. In Advances in Neural Information Processing Systems, pp. 1433-1443, 2018.
Saeed Reza Kheradpisheh and Timothee Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Chankyu Lee, Syed Shakib Sarwar, and Kaushik Roy. Enabling spike-based backpropagation in state-of-the-art deep neural network architectures. arXiv preprint arXiv: UrXivv/903.000 1903.06379, 2019.
Chankyu Lee, Adarsh Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Danillidis, and Kaushik Roy. Spike-flownet: Event-based optical flow estimation with energy-efficien hybrid neural networks. arXiv preprini arXiv:2003.06696. 2020.
Da Li, Xinbo Chen, Michela Becchi, and Ziliang Zong. Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus. In 2016 IEFE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications
pp. 477-484. IEEE, 2016.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 1-10, 2016.
Sen Lu and Abhronil Sengupta. Exploring the connection between binary and spiking neural networks. Frontiers in Neuroscience, 14:535, 2020. ISSN 1662-453X. doi: 10.3389/fmins.... 200..0000....0.00.5.... URL https: www. wwontierrsssore org/ httgg//rtccee//ee 3389/ Inins./innns..mmm 2020.0022000 00535
Zachary F Mainen and Terrence Sejnowski. Reliability of spike timing in neocortical neurons.
Science, 268(5216):1503-15000 1995.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks. arXiv preprint arXiv:1901.09944 2019.
Eustace Painkras, Luis A Plana, Jim Garside, Steve Temple, Francesco Galluppi, Cameron Patterson, David R Lester, Andrew D Brown, and Steve B Furber. Spinnaker: A 1-w 18-core system-on-chip for massively-pane -arallll neural network simulation. IEEE Journal of Solid- State Circuits, 48(8): 1953-1953, 2013.
10

---

Under review as a conference paper at ICLR 2021
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-522-22 Springer, 2016.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2020. URL /
net net/Eorum?icee Ftte:/oomnnddddllppr..ee
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience, 11:682, 2017.
Arash Samadi, Timothy P Lillicrap, and Douglas B Tweed. Deep learning with dynamic spiking neurons and fixed feedback weights. Neural computation, 29(3):578-602, 2017.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13, 2019.
Yongming Shen, Michael Ferdman, and Peter Milder. Escher: A cnn accelerator with flexible buffering to minimize off-chip transfer. In 2017 IEEE 25th Annual International Symposium Field-Prope Progrgrrrraa Custom Computing Machines (FCCM), pp. 93-100. IEEE, 2017.
Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, pp. 1412-1421, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for surge-scale image recognition. arXiv preprint arXiv: 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for efficient and rapid inference on deep spiking neural networks. arXiv, pp. arXiv-1907, 2019a.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-tempora backpropagation for training nigh-performance spiking neural networks. Frontiers in neuroscience, 12, 2018.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019b.
Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with tumeeple- spiking recurrent neural networks. arXiv preprint 0rXiv:2005.11633 2020.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint arXiv:2002. 10085, 2020.


Reviews:
[{"review_id": "qcwMMgJ2ctJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\nI appreciate the experimental results demonstrated on challenging datasets like CIFAR100 and ImageNet.\n\nWeakness\n(1) There are two existing papers emphasizing direct training SNN with extremely low latency [1][2]. Therefore, the authors should comment more on how the proposed method is different or is better compared to these two papers (they use a smaller number of time steps.). In addition, I hope the authors to show the performance comparison with these two references. In the experimental results, the paper compares performance with [1]. However, the network size is significantly different. I think a comparison of the same network size can help to demonstrate the effectiveness of the proposed method.\n\n(2) To my understanding, the method proposed in this paper has nothing different compared to the existing BPTT method with the surrogate gradient. I hope the authors can claim clearly the novelty of this paper. For example, how is the proposed method different or better from [3] and [4]. As far as I know, the only difference is that the threshold and leaky parameters are also trained in this paper. However, this can also be easily done in other existing methods like [2][3][4]. There's no doubt their performance can also be improved since it introduces more tunable parameters.\n\n(3) Tuning threshold and leaky parameters may be unbiologically plausible.\n\n\n(4) What is the weight optimization method? As shown in Table 1, it seems the weight optimization contributes more to the performance than the proposed method.\n\n\n[1] Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., & Shi, L. (2019, July). Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 1311-1318).\n[2] Zhang, W., & Li, P. (2020). Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks. arXiv preprint arXiv:2002.10085.\n[3] Wu, Y., Deng, L., Li, G., Zhu, J., & Shi, L. (2018). Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12, 331.\n[4] Shrestha, S. B., & Orchard, G. (2018). Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems (pp. 1412-1421).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ja-dZ2Bnac", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Spiking Neural Network (SNN) training method that jointly optimizes input spike encoding parameters, spiking neuron parameters (membrane leak and voltage threshold), and weights in an end-to-end fashion using gradient descent. Compared to SNNs with only weight optimization, the SNN trained by the proposed method significantly decreases the inference latency and results in high activation sparsity with minimal accuracy decrease.\n\nHighlights:\n\n1. Energy-efficiency is currently the main advantage of SNN. Two primary factors that determine the energy-efficiency of an SNN are inference latency and activation sparsity. The method proposed in the paper directly targets both factors, and the intuitions are very clear.\n\n2. The results show the joint training approach directly contributes to the improvement of SNN's inference latency and activation sparsity (as shown in Fig. 2 and Fig. 3 in the paper). It's also interesting to see the method can learn low membrane leak and high voltage threshold at the same time for later layers of VGG16, which significantly improve activation sparsity.\n\n3. The paper proposes a new spike generation function for LIF neurons in equation (2) of section 3. The function places the voltage threshold in the computational graph of backpropagation, making training voltage threshold possible in an elegant way.\n\nConcerns:\n\n1. The reviewer’s main concern with the paper is its limited novelty. The joint end-to-end training of neuron parameters has already been proposed in a recent paper ([Bojian Yin et al. 2020]). Moreover, in addition to gradient-based training, there are also local learning methods like intrinsic plasticity that tune the neuron parameters based on the activity of the SNN (for example, [Wenrui Zhang et al. 2019], [Anguo Zhang et al. 2019]). The paper needs to better position the proposed method with respect to these existing related works.\n\n2. Table 3 shows the computation overheads as a result of the direct encoding layer is around 1%. However, the large number of encoded spiking activities can slow down the inference speed when the SNN is deployed on currently available neuromorphic chips (for example, Intel's Loihi or IBM's TrueNorth) since these chips lack an efficient solution for injecting external spikes. It will be great if the paper can show how the DIET-SNN performs with different numbers of convolution channels in the encoding layer and examine if using fewer input neurons for encoding will hurt the accuracy.\n\n3. The proposed method uses backpropagation through time (BPTT) to train the SNN. Many recent papers that train SNN using spike-train level learning methods also achieve low latency and sparse synaptic activities (for example, [Jibin Wu et al., 2019], [Yingyezhe Jin et al., 2018]). The paper lacks experiments for comparing the performance, inference latency, and activation sparsity with these existing methods.\n\n4. The ablation study in section 6 uses IF neurons for the first 3 experiments and LIF neurons for the last experiment which seems unfair. The ablation study should show that DIET-SNN can do better than an SNN using LIF neurons with fixed (preset) membrane leak and voltage threshold. The current ablation study cannot rule out the possibility that an SNN with fixed (preset) low membrane leak and high voltage threshold performs as well as or even better than DIET-SNN in terms of accuracy and activation sparsity.\n\n5. The ablation study in section 6 shows the performance of only one inference timesteps value for each SNN. To show a complete picture of how the performance changes with the latency, the paper needs to train each SNN with the same group of latency values (for example, all 4 SNN trained with 150, 35, 25, and 20 timesteps).\n\n6. In section 6, the paper claims that optimizing the threshold has similar consequences as optimizing the weights. The co-optimization of them can lead to faster convergence and lower latency. However, there are no experiments to support this argument. The reviewer suggests conducting additional experiments for this.\n\n\nI want to thank the author for addressing my concerns. Many of my concerns are resolved. I have updated my rating after the discussion. I agree with the authors that the novelty of the submission lies in scaling up the existing methods. However, I'm not sure if this is enough for the paper's scientific significance required by this conference. \n\n\nBojian Yin et al. 2020, Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks\n\nWenrui Zhang et al. 2019, Information-theoretic intrinsic plasticity for online unsupervised learning in spiking neural networks\n\nAuguo Zhang et al. 2019, Fast and robust learning in Spiking Feed-forward Neural Networks based on Intrinsic Plasticity mechanism\n\n Jibin Wu et al., 2019, A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks\n\nYingyezhe Jin et al., 2018, Hybrid macro/micro level backpropagation for training deep spiking neural networks", "extra_scores": {}, "extra_text": {}}, {"review_id": "TA2k9tx-HI8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Tile: training SNNs for visual categorization with few timesteps\n\nPROS\n* high accuracy with only 20-25 timesteps\n\nCONS\n* nothing really new\n\nThe authors train convolutional SNNs for image classification, using surrogate gradient learning (SGL). They combine 4 mechanisms:\n\n1) Hybrid SNN Training. First train an ANN with the same architecture, and use the resulting weights as initial values for SGL, to accelerate convergence. This has already been proposed in (Rathi et al., 2020).\n2) Direct Input Encoding. Instead of converting the image RGB values into spike trains using Poisson rate coding, the authors feed the analog RGB values in the first convolutional layer, which treats them as input current, and emits spikes using the LIF neuron model. This allows using fewer time steps, since Poisson rate coding imposes long time windows to estimate rates and average out noise. But again, this is not new. It's been done for example in (Rueckauer et al., 2017; Lu & Sengupta, 2020), which they cite.\n3) Leak timescale training. The time scale of the leak is an important parameter in SNNs. It can be trained by SGL, just like the weights. Again this is not new. See for example (which should be cited):\nFang W (2020) Leaky Integrate-and-Fire Spiking Neuron with Learnable Membrane Time Parameter.\nYin B, Corradi F, Bohté SM (2020) Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks. arXiv.\nZimmer R, Pellegrini T, Singh Fateh S, Masquelier T (2019) Technical report: supervised training of convolutional spiking neural networks with PyTorch. arXiv.\n4) Threshold training. Again this is not new. For example Zimmer et al 2019 (ref above) did it already. Also, training both the weights and the thresholds seems redundant, since only the ratio between weights and threshold matters. About that the author say:\n\"Intuitively, the effect of optimizing either weights or thresholds in SNNs with\nIF neurons should have similar consequences. But the threshold affects the activity of each neuron, whereas the weights are shared among multiple neurons.\"\nI don't understand this sentence. Weights are shared only in the convolutional layers. Plus earlier the authors said that the threshold is the same for all neurons of a given layer.\n\nSo in short, the authors successfully combine known approaches, but do not propose anything new at a conceptual or theoretical level. In my opinion this paper is mostly an engineering effort. That being said, it seems that no one can claim a better accuracy on CIFAR and ImageNet using this nb of timesteps (or fewer).\n\nMINOR POINTS:\n\n* How much longer is convergence if the authors skip the ANN training, and start the SGL training from random weights?\n* p2: \"The length of timesteps\" -> \"The number of timesteps\"\n* \"The neurons in the convolutional and linear layers are defined by the LIF model\" I guess the authors meant \"dense\" or \"fully connected\" instead of \"linear\", since the LIF is not linear\n* I liked a lot the \"iso-accuracy\" analysis of section 6. But I suggest the authors do the same for their energy analysis of section 5. The SNNs consume less than the ANNs (Table 3) but are also less accurate (Table 1).\n \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript presents a training method for Spiking Neural Networks (SNN). The method jointly optimizes input spike encoding parameters, spiking neuron parameters (membrane leak and voltage threshold), and weights in an end-to-end fashion using gradient descent. SNNs are very interesting for energy-efficient implementations of neural networks. Their energy efficiency strongly depends on inference latency (SNNs compute in time, unlike feed-forward ANNs) and activation sparsity. 

All reviewers acknowledged that the approach directly improves inference latency and activation sparsity on large convolutional models at very good performance levels.

The main concern of all reviewers was the limited conceptual novelty. The paper combines some known techniques (hybrid SNN training, direct input encoding, training of neuron parameters like leak time constant and threshold) and scale the setup up to large networks and datasets (e.g. ImageNet).

In summary, the paper presents impressive results, but the conceptual innovation is missing. 



Decision: Reject