Accelerating Training of Deep Spiking Neural Networks with Parameter Initialization
Authors: Jianhao Ding
Jiyuan Zhang
Zhaofei Yu
Tiejun Huang
Venue: ICLR 2022 Submitted (2022)
Keywords: spiking neural networks
back-propagationthrough time
parameter initialization

Abstract: Despite that spiking neural networks (SNNs) show strong advantages in information encoding, power consuming, and computational capability, the underdevelopment of supervised learning algorithms is still a hindrance for training SNN. Our consideration is that proper weight initialization is a pivotal issue for efficient SNN training. It greatly influences gradient generating with the method of back-propagation through time at the initial training stage. Focusing on the properties of spiking neurons, we first derive the asymptotic formula of their response curve approximating the actual neuron response distribution. Then, we propose an initialization method obtained from the slant asymptote to overcome gradient vanishing. Finally, experiments with different coding schemes on classification tasks show that our method can effectively improve training speed and the final model accuracy compared with traditional deep learning initialization methods and existing SNN initialization methods. Further validation on different neuron types and training hyper-parameters has shown comparably good versatility and superiority over the other methods. Some suggestions are given to SNN training based on the analyses.

Under review as a conference paper at ICLR 2022
ACCELERATING TRAINING OF DEEP SPIKING NEURAL NETWORKS WITH PARAMETER INITIALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Although spiking neural networks (SNNs) show strong advantages in terms of information encoding, power consumption, and computational capability, the un- derdevelopment of supervised learning algorithms is still a hindrance for training SNNs. We consider weight initialization to be a pivotal issue for efficient SNN training. It greatly influences gradient generation when using the method of back- propagation through time in the initial training stage. Focusing on the properties of spiking neurons, first derive asymptotic formula for their response curve, approximating the actual neuron response distribution. Then, we propose an ini- tialization method based the slant asymptote overcome gradient vanishing. Finally, experiments with different coding schemes in classification tasks on the MNIST and CIFAR10 dataset show that method effectively improve the training speed and the model accuracy compared with traditional deep learning initialization methods and existing SNN initialization methods. Further validation on different neuron types and training hyperparameters shows comparably good versatility and superiority over other methods. Some suggestions are given for SNN training based on the analyses.
INTRODUCTION
To date, deep learning has contributed to outstanding performance on various tasks (Hinton et al. 2012] [Hosu & Rebedea, 2016 He al.] [0166] LeCun et al.) (015)). Most ofate-of-the-art models are based on analog neurons. Each analog value conveys activation; thus, the inference processes of analog neural networks (ANNs) consist of massive matrix manipulations. To overcome this, spiking neural networks (SNNs) are increasing attracting from researchers (Maass) 1997 Maass & Markram 2004 Yang et al. 2019). The spiking neurons in an SNN emit sparse 0,1 spikes when activated. Compared with existing ANNs, SNNs are considered to encode spatiotemporal patterns and have better power efficiency. Hence, implementations of SNNs are highly appealing in power- limited scenarios. In addition, SNNs have exhibited computational capabilities powerful those of conventional artificial neural networks (Lee et al., 2016).
Despite SNNs' powerful capabilities, these networks lack efficient supervised learning algorithms. The discrete spikes prevent the direct use of backpropagation (BP) training. Recently proposed
supervised algorithms have mainly focused on two methodologies: ANN-SNN conversion and BP through time (BPTT) with surrogate functions. Compared to ANN-SNN conversion (Dichl et al.
2015 Sengupta al.) 2019 Deng & Gu 202T Ho & Chang 2020 Ding el al. 2021), BPTT
more friendly to the based datasets (Lee et al. 2016; Zheng et al. 2021 Neftei et al,,
2019). However, /u al. 2018) has reported that BPTT only feasible to shallow architectures. Therefore, BPTT still cannot be regarded as mature SNN training algorithm.
We ascribe the immaturity of BPTT training to the influence of the gradients. One common reason gradient vanishing and explosion. This is natural since SNNs can be regarded neural networks (RNNs) (He et al. 2020 Fang et al. 2020). In the wisdom of RNN methodology, researchers have managed to provide simple solution by rethinking weight initialization (Le al.) 2013). The success of RNN training inspires consideration of whether there may also be a simple solution for SNNs. Another reason early gradient generation, which has rarely been mentioned in the previous literature. Most surrogate functions produce gradients when the membrane potential in the neighborhood of the threshold. Some produce gradients only when spikes fired (Lee
type of recurrent

---

Under review as a conference paper at ICLR 2022
Layer-1
Layer-2
Layer:3
Layer-4
Layer-5
Training Lass Mcan Firing Rate
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
50
100
150
200
350
100
50
150
200
100
0.06
0.06
0.03
0.03
0.00
0.00
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
Iterations
Iterations
(a)
(b)
Figure 1: Loss curves and mean firing rate curves of a 5-layer SNN, initialized using the methods proposed by GGlorot Bengio (2010) (a) and (b). Figure (a) shows an example of inadequate initialization. All losses start to decrease as the firing rates increase.
et al. 2020a). Thus, the flow of the gradients strongly depends on the number of firing neurons and will influence the optimization of the network. A small experiment can help illustrate how early gradient generation affects training. As shown in Fig. T with an inadequate weight initialization method, the neurons receive inputs that are too small and cannot fire initially, leading stall of the training process because gradients fail to be produced. Until some neurons fire, the loss does not begin decrease, and the neurons must suffer latter uncoordinated training schedule. Therefore, in general, good weight initialization is also important for efficient SNN training.
At present, initialization for SNN training mainly achieved by following and adapting methods originally developed for ANNs. However, unlike the rectified linear unit (ReLU) activation function conventionally used in ANNs, spiking neurons respond very differently, and recent improvements in models have rarely focused on the unique properties of spiking neurons. In this paper, we investigate the theoretical response of spiking neurons to develop a better SNN initialization strategy. The main contributions of this paper are summarized follows:
We use iterative systems to model first-order integrate-and-fing neurons and theoretically investigate the response curve. A slant asymptote derived to approximate the actual neuron response.
We examine the difficulty of training deep SNNs and develop an initialization method based on the slant asymptote. The proposed initialization method relies on adequate scaled weights and nonzero biases to enable early gradient training.
Experiments with rate coding and event coding in classification tasks show that our method can effectively improve the model training speed and accuracy. In addition, empirical anal- yses suggest that our asymptote initialization method offers versatility and superiority over vast range of training hyperparameters, such different neuron types, surrogate func- tions, and optimizers.
2 THEORETICAL ANALYSIS OF SNNS
2.1 SPIKING NEURONS AND SURROGATE FUNCTIONS
With the development of neuroscience, many neuron models have been proposed (Hodgkin & Hux- ley 1952 Gerstner et al. 2014), commonly in the form of differential systems. However, in the field of computing, analyses are often based on first-order differential models because of computational efficiency. Typically, leaky and nonleaky integrate-and-firg (LIF and IF) models can be formulated in terms of the dynamics of the membrane potential v as shown in Eq. a) and (b).
T dw(()
-(((t)) Ureset) Ri(t) (a) 7 time constant.
da(t) = ((())
de
2

---

Under review as a conference paper at ICLR 2022
Parameter setting examples for first-order linear integrate-and- models expressed in the form given in Eq.
Table 1:
Model
LIF model in Eq. [a)
Abbottt 1999)
1/T
R/T 1/C
IF model in Eq. ()) Abbott 1999)
Diehl et al. (2015); Sengupta el al. 2019
Wu et al. (201 Zheng al. 202
Tdecoy O
Panda ct al. 2020
By utilizing the Euler method, we can convert these dynamic systems into an iterative expression that computers prefer (Eq. [ Note that Vreset is usually 0 in the literature; hence, it is neglected in latter analyses. When an element of v' exceeds the firing threshold 6, the corresponding neuron will emit a spike of and membrane potential will then be reset to Without loss of generality, we use the following equations to describe the iterative systems for first-order integrate-and-fire neurons.
u = kv' + Ai
o u' > 0 u+ u (1-o')
(2)
In Eq. 2 1: and À are hyperparameters that control the neuron type and firing pattern (k: (0. 1))- When k is set to O, the iteration of the membrane potential over time is canceled, and the network not standard SNN. Many neuron models used in the existing literature SNN training can be regarded as the model given in Eq. 2 with different settings of the two parameters. Table illustrates some of these settings.
In Eq. 2 o' = u' > B is the non -inddffeereniable part. It can also be rewritten as H(u' 8) where (-) is the Heaviside step function. (x) 0 when S and (x) when 0. The heaviside function has a gradient of O almost everywhere. Surrogate function methods first emerged in network quantization cope with non-differentiable problems (Mairal 2013 Hubara iet al. 2016). Accordingly, four surrogate functions (sigmoid, tanh, hardtanh, arctan) are presented this paper, parameterized by the parameter a, as listed in Table|A3 These functions the shifted and scaled version of the original functions, shown in Fig. A7 the appendix. The parameter can be adjusted to rescale the output gradient. The gradient functions are typically symmetric w.r.t. an input of zero. An exception has been proposed by Lee et al. (2020a), such that only the production of spikes can trigger nonzero gradients.
2.2 RESPONSE OF SPIKING NEURONS AND ASYMPTOTES
It is not easy establish the relationship between the current input and the output based iterative equation. Considering that the difference between the iterative equation and the differential equation small when slepsize and that the differential equation can be integrated to obtain the response curve, Theorem gives the response of first-order integrate- and--nd--ii -ire neurons.
Theorem 1. For a first-order integrate- ande--nd--iee neuron as expressed in Eq. (u' = ke + Xi', e (0, 1)). the model takes constant current i' i input and produces spikes output. Let the discrete output firing rate w.r.t. i be expressed as f f= F. where T is the total inference time and N the spike count. Then, the input-output response curve (f-i curve) is estimated as follows:
If k 1:
O,
i 0, X
B
< 0 i iz
0<6
i
X
1,
If E (0.
X
(3)
0,
i 0,
(k: 1)/ In(1 [1
(k-1)0 2
0 1
0
k-
1,
i2 x
3

---

Under review as a conference paper at ICLR 2022
1.0
Thcoretical response
Theoretical response
Asymptootc
Asymptole
0.8
Nouron resporse
Neuron respons:
Firing Rate
0.6
0.4
0.2
0.0
1.0
-0.5
0.0
0.5
-1.0
0.0
1.0
2.0
3.0
Inpul (4)
Inpot (b)
Figure 2: Theoretical response curves after integration (solid lines), slant asymptotes (dashed lines), and actual neuron responses (circular dots) for the first-order linear integrate-and- model with (a) k and (b) i E (0,1).
The solid lines in Fig. 2 represent the response curves when k 1 and & (0, 1) in Theorem I Fig. also presents scatter plots of the actual response of a spiking neuron given the corresponding parameters (marked with circular dots). The actual neuron responses match the theoretical f curve for most inputs. Consider the region in which f E (0, 1). When E (0. 1). it can be intuitively found that the f-i curve, which has non-differentiable point, seems to exhibit slant asymptote. Theorem 2 manifests the existence of this slant asymptote and provides a uniform expression for it. In Fig. 2 this asymptote approximately crosses the actual neuron response distribution. In this case, the slant asymptote of f-i can better describe the relationship between the expected input current and the output firing rate.
Theorem 2. For a first-order integrate-and-fire neuron as expressed in Eq. (u' = ko + Ai k E (0,1), 1}), the slant asymptote of the f-i curve satisfying f (0, 1) exists and be uniformly ex- pressed as follows:
x 0
k k-
Sa
3 WEIGHT INITIALIZATION FOR SNN TRAINING
The gradient of the weights is related to the spike output. To better illustrate this, given an L-layered SNN, we denote the input current, membrane potential, intermediate variable, and spike output of the (th layer by g/,t, w/-,, u. and o respectively. The variables for the neurons one layer satisfy Eq, 2 gi. is the weighted input consisting of the previous layer's spike output (i W'o +b'). and the first layer takes the encoded spike 0.* as input. When we backpropagate the loss & through time T, the gradients w.r.t. the weights of the 1th layer can be expressed shown in Eq. 5 An inappropriate of-1,1 such as an output of zero, will lead to a poor gradient.
De
T
de Oulie Oil.t
T
O=
Σ 1=1
Σ
(o' 1o'
(5)
OW
Juis Jil OW!
Dull
Therefore, the main concern how to produce spikes to trigger the learning process. Thankfully, the clipped version of the slant asymptote can replace the analysis of complex iterative functions, allowing us to focus on the scale the parameters:
Z = W b'
k
6' clip(
2 0,1)
where 3t and of are the expected current and firing rate vector, respectively, of the 7th layer with n, neurons and W and are the corresponding parameters. Note that W is an n, x n--1 matrix. X k and 0 represent the neuron parameters.
First, all weights in the SNN are initialized to be independent and identically distributed (i.i.d.). Let us assume that the input current variances are the same. Then, we have:
Var(t)= n1-1 Var(W'o'
4

---

Under review as a conference paper at ICLR 2022
0.20
Mean firing rate
0.15
0.10
0.05
0.05
0.00
0.00
Layer index
Layer Index
(a) MNIST 5-layer network
(b) CIFARIO VGG9
Figure 3: The illustration of the impact of initialization methods the overall spike activity before training. The networks are randomly initialized using different initialization methods.
where 1", W and 6' denote random variables of elements in 2 w and o'-1, respectively. Considering Var( W'o'=1), Var(W' Var(W")EE(('' of zero mean; then:
|) + Var(o"")IE((// let W, be
Var(i) n4 Var Var (E('''
(8)
In the function ό = i 1=1 0, 1). region and the linear response region. When b' is initialized #(1-k) (1 1.2. ,L). zero- k(1-k)
9(1-k) 2x
the transition point between the nonspiking
2X
centered W x ô produces symmetric distribution around zero. With b' initialized as 2
B[(6'-1)2] Var )
12 Var (ar(i--))
02
By plugging (1) into Eq. we can obtain:
Var(i))
2 0z nh Var Var(W') ('')
X2
(10)
To produce spikes to maintain (6())) which related to Var according to Eq. Therefore,
determine the distribution of W follows (for 1 = 2,3, ... D):
Var(i)
1 122
2 82 n-1 Var(W")) =1
(11)
Var(i
Accordingly, W is initialized from Gaussian distribution N (0, > where nt-1 is the number of input neurons in layer I. As done in other initializers, we apply the initialization to the first layer for simplicity.
In previous work on ANNs, there is need for the neurons to accumulate input. Thus, the biases are often set to zero in an ANN, For SNNs, because the response asymptote indicates a nonzero non- differentiable point, it is natural for the biases in SNN to be initialized to that nonddifferentiable value. We validate the impact to the mean firing rate our initialization method Fig. B Our method can effectively trigger spikes when compared with ANN-based methods.
RELATED WORK
From the perspective of the loss landscape, a good weight initialization is often located in region with low loss values (Li et al. (0088). However, training deep neural networks is never an easy thing. After trails of layerwise training of deep nets (Hinton 2007 Bengio et al. (006)), Glorot Glorot & Bengio
2010 provided theoretical view the prevention of exploding and vanishing gradients. They estimated the standard deviation of the weights from the number layer neurons and initialized the biases to zero. He et al. (015) extended their hypothesis to piecewise linear the growing prevalence of ReLU activation. Both methods are widely used in deep ANNs. There also works for RNN. Saxe et al. (2014) examined the dynamies of learning and derived random
cater to
5

---

Under review as a conference paper at ICLR 2022
Table 2: Network performance comparison on the MNIST and CIFARIO datasets with different initialization methods. A hyphen in the table indicates invalid training in given setting.
MNIST (A: = 0.8, = 0.2, 0 = 1.0)
Event coding (N-MNIST)
Constant coding
Poisson coding
Retina-like coding
Glorot's 99.18
99.30
99.04
99.36
He's
99.18
99.26
99. 14
99.35
Wu's
99.18
99.28
99.24
99.29
Lee's 99.18
99.27
99.14
99.33
Ours
99.27
99.35
99.20
99.39
CIFARIO (k: = 0.8, X = 0.2, 0 = 1.0)
Event coding CIFARIO-DVS
Constant coding
Poisson coding
Retina-like coding
Glorot's
81.59
76.69
73.75
He's
82.95
81.86
76.36
66.9
Wu's
83,90
82.23
78.84
67.7
Lee's 's 83.05
81.03
73.07
Ours
87.62
83.06
82.98
68.2
orthogonal initial conditions for deep nonlinear networks. Le al. (2015) used an identity matrix to initialize the RNN weight matrix and thus maintain the gradients.
Recent SNN studies have also included some initialization methods. For example, one can regard ANN-SNN conversion as initialization method (Rathi et al.) (2020). In addition, [ee et al. (2016) set the weights and thresholds using a manually tuned hyperparameter. They also proposed a com- plex BP error normalization procedure. Wu et al. (2018) suggested normalizing weights sampled from standard uniform distribution, with no further explanation. [ee et (2020b) followed the recommendation of He et al. (2015) with manually selected constant K.
With the similar motivation of this work, many recent studies have focused on regularizing the spiking response. Zheng et al. (202T) Kim & Panda (2020) proposed batch normalization along channel axes and time axes, respectively, for BPTT; both of which are layerwise normalization methods. In addition, Kim & Panda (021) utilized a global unsupervised firing rate normalization.
The above methods lack direct attention to neuronal dynamics. SSnnupta et al. (2019), however, used dynamics of simple IF neuron proposing Spike-Norm for ANN-SNN conversion. In contrast, our work aims to improve the training for general first-order integrate-and-fig neurons by analyzing neuronal dynamics.
5 EXPERIMENTAL VALIDATION
5.1 EXPERIMENTAL SETTING AND DATASETS
To verify the training acceleration performance and the model performance against various other initialization methods, we conducted experiments on classification tasks involving different codings al.) 2009).
for the MNIST dataset (Lecun al. 1998) and the CIFARIO dataset (Krizhevsky Before being fed into an SNN, images should be converted into spike trains. We implemented constant coding, Poisson coding, retina-like coding, and event coding for both datasets. Poisson coding and constant coding are popular rate coding schemes for ANN-SNN conversion (Rueckauer al., 2017, [Snnuppta et al.) 2019). Retina-like coding mimics the integration process of bipolar cell (Zhu et al. 2019). The N-MNIST dataset was produced by converting MNIST into spike events with dimensions of size 34 x 34 using saccades (Orchard et al. 2015).. The CIFARIO-DY. dataset comprises 10000 128 x 128 event streams converted from CIFARTO using dynamic vision sensor (DVS) Li et al. 2017).
Previous work has revealed that the accuracy of an SNN is sensitive to the choice of hyperparameters such as the neuron parameters (Fang et al., (020).. Therefore, we fixed the neuron parameters for given dataset, as shown in Table 2 and compared method with four other prevalent choices

---

Under review as a conference paper at ICLR 2022
(Glorot & Bengio 2010), He al. (2015). Lee et al. (2016) and Wu et al. (2018) ). An analysis of the hyperparameter sensitivity is presented in Sec. 6 For MNIST, we tramed a 5-layer SNN. For the CIFARIO dataset, trained VGG-9 model. For the CIFARIO-DVS dataset, added one pooling layer and a fully-connected layer into the VGG-9 model. Time step in forwarding process is set to constant value of 20. All the models were optimized using the Adam optimizer. The detailed training hyperparameters are shown in the appendix. Additionally, we conducted experiments for larger datasets, i.e., CIFAR 100 and TinylmageNer Appendix for the detailed descriptions and comments on experiments on larger datasets.
compare to Batch Normalization (BN). See
5.2
RESULTS
N-MNIST
0.10
0.10
0.35
0.10
Train Loss
Traina Losss
B.65
Train
8.66
Train
0.00
18
VSl Accurney
0.59
Vsl Aesurrey
0..90
VSI Accurncy
0.00
6.30
0,90
Val Asgursyy
Aucurssy
Aucurssy
60.90
000
E.50
00
9,30
120
90
Iterations
Dierations
Demtiion
Derstions
Figure 4: Training loss and validation accuracy curves on the MNIST, N-MNIST, CIFARIO, and CIFAR 10-DVS datasets. The MNIST and CIFARIO 10 datasets are encoded with the Poisson distribu- tion. The first two rows show the loss and accuracy curves in the earliest 5 and 20 epochs. The third row shows the validation accuracy curves throughout the whole training process.
In general, Fig. 4 and Table show that good parameter initialization can not only accelerate train- ing but also lead to some performance improvement. On MNIST, the different initialization methods achieve similar performance in validation accuracy with all four encoding schemes listed in Table and our method performs slightly better for three of them. On CIFARIO, our method achieves per- formance improvement, showing accuracy increases of 4.67%, 1.20%, 6.62%, and 0.5% compared with He's method. As seen from the first row of Fig. 4 our method enables the model to generate early gradients in fewer iterations and cause the loss to drop faster. In more detail, method al- lows the model to obtain a proper parameter distribution, which helps every layer coordinate with the input and output better adapt SNN training. This advantage enables network training to begin quickly, within a few iterations. In comparison, the other methods are slower than ours.
For MNIST, our method reaches 98% validation accuracy after only epoch, while other methods reach this accuracy after about 2 or 3 epochs of training. For N-MNIST, our method reaches 98% accuracy after 2 epochs, while the others reach after epochs. For CIFARIO, our method first achieves 60% accuracy at approximately the 4th epoch, whereas Wu's, He's, Lee's, and Glorot's s methods reach this level later, respectively. For CIFARIO-DVS, our method and those of Wu and He reach 60% accuracy in order, while training fails under Lee's and Glorot's methods.
Due to the simplicity of MNIST, networks taking input encoded using each of the four schemes perform similarly. For CIFAR16 10,, model trained with constant coding achieves the highest accuracy because this data format is almost the same as the original data. Poisson coding performs worse than constant coding due to its more complicated coding rules. Retina-like coding leads to considerable degradation, probably due to the loss of some texture information. The results show that this coding scheme influences the other four initialization methods CCFARIO-DVSS) are sparse and complicated, and the amount of data is much smaller than that in CIFAR10 10, which increases the training difficulty and leads to performance degradation.
than it does ours. The event data

---

Under review as a conference paper at ICLR 2022
6 ANALYSIS OF EMPIRICAL RESULTS
To study whether the initialization strategy is sensitive to the hyperparameters of the model and the training process, we conducted many experiments. In these experiments, we used the MNIST dataset and the same 5-layer network as in Sec. 5 Training was performed for 3 epochs with minibatch size of 64.
6.1 PERFORMANCE ANALYSIS ON DIFFERENT NEURONS
We performed an analysis of different neuron types, that is, different neuron parameter settings.
While was fixed 1.0, A: sampled from 0.1 to 1.0, and À sampled from 0.1 to 4.0. Fig. A88 compares the training speeds in this grid of experiments. For most (k, X) settings, our method shows obvious acceleration relative the other methods. In particular, our method can be used to effectively train networks for which training cannot be successfully started using other initialization methods.
1-100, 0.5
K- 0.9, -1-00 0.1
R- 0.8, - 0.2
K- 0.5, 0.5
k 1.0, 110
0.2, Á 0.8
1.0
247
Ierattios
303030
30
2323
15
0.54
234
0.54
61
234
17
0044
332
-0
21
3392
-0
2392
1.0
3.0
Ours
He's
Gioret's
Wi's
Lee's
(a) Non-leaky (b) 10
(c) 5
(d)
2
(c) Non-leaky
(f) Fast leaky
Figure 5: Validation accuracy (red polylines) and training iterations (colored bar charts, where the value indicates the number of iterations at which the accuracy first reaches the expected level of 60%) for first-order integrate-ande -an-atte--n--ieeee fire neurons with different (k, A) settings and initialization methods. Note that 0 is set to 1.0.
We choose six representative groups to better illustrate this shown in Fig. For leaky neurons with different values, with decreasing т. the training speed and accuracy of all initialization meth- ods improve. In this case, our method maintains high accuracy and achieves training acceleration of more than 11x compared to the baseline Glorot initialization. For nonleaky neurons, as shown in panels (a) and (e), the difference among methods is obvious. As À decreases, our method enables faster training. For neurons that leak quickly (k = 0.2). Glorot's 's method performs poorly terms of the start-up training speed but achieves comparable accuracy our method. Among the faster methods, our method achieves the best accuracy.
Besides, instead of fixing 0, we plot the heatmap in Fig. A9 after sampling 0 from (0.5. 1, 4) This figure illustrates that our method is suitable for almost every chosen parameter and can help address the stalling training caused by extremely large threshold due to the compensation of the initial biases.
6.2 PERFORMANCE ANALYSIS WITH DIFFERENT OPTIMIZERS AND SURROGATE FUNCTIONS
We selected 7 and test the sensitivity the training hyperparameters. Two main factors that influence the gradient update are the optimizer and the surrogate function. Accordingly, Fig. 6 shows the training curves with different initialization methods, and the accuracy is marked in the figure in the order of the legends. Generally, if training can be successfully started via the gradient update with a given hyperparameter, the difference in accuracy not huge. However, we can still observe an accuracy advantage with our method. For example, for (0.8, 0.2, 1.0), the accuracy of Wu's s and Lee't methods is 1.8% lower than ours. In addition, roughly, the initialization methods can be roughly arranged in the following order by the training speeds observed from the experiments: ours > Wu's > He's ' > Lee's > Glorot's s. Moreover, better training observed in experiments with smaller 7.
8

---

Under review as a conference paper at ICLR 2022
k 0.8, A 02,00-0200
A O.S.A= 9- 1.0
8-1
D.S.A 9-5,
(210
(130,994 (2)0.980
Vil (210.990
(330973 (43097) (50,002
(430.900
Adam
(50,960 (500977 (50.95
3309964 (4)0.87 (530987
(ir--a)
0.55
1.35
Acc (DM.114 (2RLI14
Val Apc: 09e (200.566 876
VIE Acc (10,995 (200978
Val. Acc: (110 430
5 1.00
(330.114 (40.114 (50.829
(36911 (410.008 (5)0.959
(390.982 (4)09799 (500987
(310.667 6410.900 9538988
1:50
Val Acc (1,0,00 1200,00
Va 2202250
Ace (10,996 (20.57
933 G3
(30.096 (530008 (530
632260 (400 (33993)
ANGI
(30.911 (410978 (5,9985
(3,0.558 (4)0,900 (510.599
(x-1.8)
< 1075
0.50
2 1:00
(190 (630957
Val (190.59 1210.967
Val Acc. 1177777 1200.522
083 (491978 (30.933
Adalrral
(3,0.095 (4,0.995 (549955
Acc 10R (20DOO
Acc (1) C3 13000 (4)0.098 (510959
100
150 Derations
50
150
270
100
250
Aterationa
Iterations
Eterations
Gowevill
Gew
Bredi
We
(a) Performance with optimizers
(b) Performance with surrogate functions
Figure 6: Training accuracy and validation accuracy using different training hyperparametens, dif- ferent (k, X, 0) settings and different initialization methods. The performance with four different optimizers (Adam, SGD, ASGD, AdaGrad) is shown in (a). The performance with four different surrogate functions (sigmoid, tanh, arctan, hardtanh) is presented in (b).
Regarding the optimizers, SGD(Ir=0.5) and ASGD((n=20)) fail to activate fast early training for other initialization methods, yet methods still works. Compared with 10-3,1 =00..00.9)), AdaGrad((r=1e-2) seems less sensitive to the choice of the initialization methods.
With different surrogate functions, the order of the methods is maintained. Hardtanh produces coarse surrogate gradients and suppresses training under the other four methods. Surrogate functions that produce more fine- menegrained gradients all successfully activate training. Sigmoid(a=2.0) has some advantages in terms of training speed, but it is slightly inferior to tanh(a=2.0) and arctanh(a=1.0) in training accuracy.
DISCUSSIONS AND LIMITATIONS
The large number of experiments conducted allows to provide suggestions for SNN training. A good combination of hyperparameters is essential. Our initialization method can speed up training in most cases and adapt more difficult neural parameters. In addition, we recommend the of adaptive optimizers, as it helps accumulate the gradients. Finally, the arctan and tanh surrogate functions are suggested because they offer high accuracy and relatively high training speeds.
Apart from the advantages, our method still has some limitations in terms of neuronal dynamics. Although analyzing the response of spiking neurons and asymptotes is feasible in most cases, the theoretical results will be hampered if the input current carries more noise. The noise will have at least two impacts [Gerstner al. 2014). One is that the neurons will fire even when the mean input is below zero. The other is that noise in the input current will cause the membrane potential to drift away from the theoretical asymptote. Furthermore, the noise will increase when we extend to wider and deeper networks. Therefore, for large datasets such as ImageNet, the asymptote should take the current noise into account.
8 CONCLUSION
In this work, we aim to enhance the efficient BPTT training of SNNs by proposing an initializa- tion method that is consistent with the response of spiking neurons in initial training. Our method bridges the spiking neuron response with the wisdom of traditional deep learning training, which may have influence future research topics such as ANN-SNN conversion with LIF neu- rons or other SNN training methods. Our experimental results validate that our initialization method shows superiority, versatility, and robustness vast training hyperparameter space.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Larry F Abbott. Lapieque's introduction of the integrate-and-fir model neuron (1907). Brain research bulletin, (005-6)3333-04 1999.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems, pp. 153-160. MIT Press, 2006.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations (ICLR), 2021.
Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2015.
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural networks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI) pp. 2328 2336, 2021.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. CoRR, abs/2007. 005785, 2020.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), volume 9 of JMLR Proceedings, pp. 249-256, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance imagenet classification. In IEEE International Conference Computer Vision (ICCV), pp. 1026-1034. IEEE, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778. IEEE Computer Society, 2016.
Weihua He, Yujie Wu, Lei Deng, Guoqi Li, Haoyu Wang, Yang Tian, Wei Ding, Wenhui Wang, and Yuan Xie. Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences. Neural Networks, 132:108-13 2020.
Geoffrey Hinton. Learning multiple layers of representation. Trends in Cognitive Sciences, 11(10): 428 434, 2007.
Geoffrey Hinton, Li Deng, Dong Yu. George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82 27,2227722222 2012.
Nguyen-Dong Ho and Ik-Joon Chang. TCL: an ANN-1o-SNN conversion with trainable clipping
layers. CoRR, 00500000804009 2020.
Alan L Hodgkin and Andrew Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology, 117(4)5000544, 1952.
Alexa-le Hosu and Traian Rebedea. Playing Atari games with deep reinforcement learning and human checkpoint replay. CoRR, abs/1607 050777577 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in Neural Information Processing Systems, pp. 4107-4115 2016.
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 1448-566 PMLR. 2015.
Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. arXiv preprint arXiv:2010.01729, 2020.
Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Quoc Le, Navdeep Jaitly, and Geoffrey Hinton. A simple way to initialize recurrent networks of rectified linear units. CoRR, abs/ abs/1504.000 1504,0094,, 2015.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-ban based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 ():211)):78-221 2324,, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Chankyu Lee, Adarsh Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy. Spike-flownet: Event-based optical flow estimation with eeeegy-efficient hybrid neural networks. In European Conference on Computer Vision (ECCV), volume 12374, pp. 366-382. Springer, 2020a.
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Fron- fiers in Neuroscience, 14, 2020b.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10:508, 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances in Neural Information Processing Systems, pp. 6391-6401, 2018.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. CIFARIO-DVS: An event- stream dataset for object classification. Frontiers in Neuroscience, 11:309, 2017.
Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models.
Neurul Networks, 10(9):1659-165 1997.
Wolfgang Maass and Henry Markram. On the computational power of circuits of spiking neurons. Journal of computer and system sciences, 69(4):593-61) 2004.
Julien Mairal. Optimization with first-order surrogate functions. In Proceedings of the 30th Inter- national Conference on Machine Learning (ICML), volume 28, pp. 783-791,
Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 1-63, 2019.
Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 9:437, 2015.
Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accu- rate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Newroscience, 14, 2020.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference Learning Representations (ICIR), 2020.
11

---

Under review as a conference paper at ICLR 2022
Bodo Rueckauer, version of continuous- valued deep networks to efficient event-driven networks for image classifi- cation. Frontiers in Neuroscience, 11:682, 2017.
Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Con-
Andrew M. Saxe, James L.. McClelland, and Surya Ganguli. Exact solutions the nonlinear dy- namics of learning in deep linear neural networks. In International Conference on Learning Representations (ICLR), 2014.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience, 13:95, 2019.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
Zheyu Yang, Yujie Wu, Guanrui Wang, Yukuan Yang, Guoqi Li, Lei Deng, Jun Zhu, and Luping Shi. DashNet: hybrid artificial and spiking neural network for high-speed object tracking. CoRR, abs/1909.12942. 2019.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-traine larger spiking neural networks. Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), pp. 11062-11070, 2021.
Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspireds sampling method for visual texture reconstruction. In IEEE International Conference on Multimedia and Expo (ICME), pp. 1432-1437. IEBE, 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 APPENDIX FOR THEORETICAL ANALYSIS
Here, we provide theoretical proofs of the theorems in Sec. 2 These are the foundation of the analyses in this paper. Additionally, Fig. A7 presented here to illustrate the surrogate functions and symmetric gradients.
2.5
Heaviside
Dirac
2.0
Sigmoid(a=2)
2.0
Sigmoid(a=2)
Tanh(a=2)
Tanh(a=2)
Outpuu
1.5
Hardtanh(a=1)
Gradient
1.5
Hardtanh(a=1)
Arctan(a=1)
Arctan(a=1)
1.0
1.0
0.5
0.5
0.0
0.0
-2
0
-2
0
Input (a)
Input (b)
Figure A7: Surrogate functions and corresponding gradients.
Table A3: Surrogate functions for the non-differentiable model (a is the parameter).
Surrogate function h
h(x,a) a)
11xx, a)
Sigmoid-like Tanh-like
1/(1 + e-ax)
h(r, a) h(z, a))
taah ar) +
aa (1 tanh(ar)? a Heaviside 20 - (x])
Hardtanh- Arctan-like
clip(ax + 2 0,1)
(arctan(Taz) +
a/(1 + (Taxx)²
Theorem 1. For a first-order integrate-and-fii neuron as expressed in Eq. (u' = kv + Xi', k E (0, 1)), the model takes constant current discrete output firing w.r.t. be expressed f = T where T the total inference time and N is the spike count. Then, the input-outpy output response curve (f-i curve) is estimated as follows:
input and produces spikes as output. Let the
If k = 1:
0,
i 0, Xi,
0<
i 0
1. i2 8
If k (0,
(AI)
0,
i 0,
(k:
1)/ In
[1
+
(1=1)) Ai
0 1
1,
Proof. From Eq. 2 the iterative functions are first converted into differential equations. Here, use the scalar vanable form because we are interested in the time interval between the resting state and the firing state for constant input. Our goal is to determine the time 1" of firing (u(t") !)
13

---

Under review as a conference paper at ICLR 2022
with (()) = O. The discrete output firing rate f T N spiking interval
then be approximated the inverse of the
du
dt = (A: 1)u +
(A2)
One can use integration to obtain the expression for given constant input. When k = 1, the integral is merely an accumulation of the scaled input. Considering u(0) = 0, we have the following expression:
u(t) Ai-
(A3)
Then, with u(t*)= =0, it is calculated that ta Thus, when k = 1,
(A4)
N
For i 2 f. the neuron constantly fires, and reaches its upper bound of When 0, the neuron accumulates negative imput and produce zero spikes, O. When & (), Eq. A2 is an ordinary differential equation. By separating the integration variables u and t, we obtain:
du
dt.
(k: - 1)u + Ai
Integrating both sides of the equation yields
In((1 k)uu - Xi) = (k 1)
(1 k)u Xi C2
where and O' are the constants of integration. Considering u(0) 0, have C' thus,
-Xi, and
u(l) = Ai k 10---11-00 (1-k)().
(A5)
Then, by solving u(t") we obtain t" 1 1Inn1 + (k-1)0) Hence,
(k 1)/ In(1 + (k 1)0
(A6)
N t
Ai
When - / 1. Thus, when X - f is clipped to
Theorem 2. For a first-order integrate-and-fic neuron as expressed in Eq. 2 (u' ko + Xi", k (0, 1)), the slant asymptote of the f-i curve satisfying J E (0, 1) exists and can be uniformly ex- pressed follows:
X 7
k
Ja
(A7)
0
Proof. From Theorem 2 the relationship between and f satisfying f E (0, 1) is given as follows:
f 5- {
d (k- (k-1)/mm(1 + ()-1)0
k ==1, 1, k E (0, 1).
(A8)
Suppose that the asymptote is formulated as fa = K i B. Then, for k = 1, K }.. and B = 0. For (0, 1), if slant asymptote exists, then lim;-1zo & K and lim;- Ki] - B.
14

---

Under review as a conference paper at ICLR 2022
k
lim
lim
iln(1
((N-1]
Ar
lim
1-000
In(Ai + O(k - 1)) In(Ai)
lim
(L L'Hossital''s Rule)
(k+6(k-1) Ai
Ai + Ock: - 1)
lim
Oi
(L'Hospital's Rule)
Thus, a slant asymptote exists for k E (0, 1), and K for all ls E (0, 1]. For (0, 1).
k
B lim
8-000
[In(1 +
k(--1)
0
Ai
k
$ Dilln(Xi + 0(k 1)) [ni)]]
lim
(-+00
In(Ai + 0(k: - 1)) - (ni)))
2 In(Ai + 0(k - 1)) - 3 x-+0(k-1) In(Ai) +
Di
lim
(I''Hosiitll's Rule)
(-700
>
A In(Ar) - In(Ai + 0(k: 1)) + 11+0(k-1)
lim (-000
0
A(A++0(R- 13) 4A(k-
X 1
lii
(2i+0(k 1]}²
(L L'Hossiitl''s Rule)
1-000
0
(x++0(k- 1)}2 + $
22
A
82 (k: 1)
lim
1+00 0 2A0(k - 1)i + 0² (k- 1)²
X [lk 1)
(I''Hospital'' Rule)
2X
1
2
When &o = 1, B = 0. Thus, fa 3/i =
R-1 is the uniform slant asymptote for all k E (0, 1].
A.2 APPENDIX FOR EXPERIMENTAL RESULTS
The detailed hyperparameters are presented in Table A4 Note that the data in CIFARIO-DVS and N-MNIST take the form of address event representation (AER). To split the events into T slices that match the input streams of our network, we utilized the technique referenced in Appendix F of Fang ct al. (2220) to preprocess the data.
Performance Analysis under Different Neuron Configurations
Eq. shows that the neuron types and firing patterns are controlled by & and A, which may vary the biological brain. To cover many of the neuron models used in the existing neuroscience literature as possible, we discretely sampled & from 0.1 to 1.0 and X from 0.1 to 4.0. In each experiment, compared four other methods with initialization method in two aspects: training speed and accuracy. The four methods considered for comparison were proposed by Glorot & Bengio (2010), He et al. (2015). Lee et al. (2016) and Wu et al. (0018).
15

---

Under review as a conference paper at ICLR 2022
Table A4: Detailed hyperparameter settings in the experimental validation. designed to fit the different shapes of input data.
The networks were
MNIST
N-MNIST
CIFARIO
CIFARIO-DVS
Input size
28 x 28
Tx x 32
32 32
Tx128 128
Timestep
20
20
20
20
conv X 3 64
LIF neuron conv 3 x 128 LIF neuron pool 2
conv 3 x 128
LIF neuron
cony 3 x 128
conv 3 x 3 x 128
LIF neuron
LIF neuron
pool 2
conv 3 x 3 x 128
conv 3 X 3 x 32
conv 3 x 3 x 32
conv 3 x 3 x 256
LIF neuron
LIF neuron pool 2 x 2
LIF neuron
LIF neuron
pool 2 2
pool 2 x 2
conv 3 x 3 x 256
conv x 3 x 128
conv 3 x 32
conv 3
32
LIF neuron
LIF neuron
LIF neuron pool x 2 fc 500 LIF neuron fc 100
LIF neuron
conv pool 2 x
pool 2 2
Network
pool x fe 500
x
conv x 3 128
LIF neuron
LIF neuron pool 2
LIF neuron fc 100
conv 3 x x 256
LIF neuron
conv 3 x 3 x 128
LIF neuron
LIF neuron fe 10
conv 3 x 3 x 256
LIF neuron
fc 10
LIF neuron
pool 2 2
pool 2 x
1024
fc 1024
LIF neuron dropout fc 1024
LIF neuron dropout fc 10
LIF neuron dropout
fc 10
Epoch
50
50
150
150
Batch size
32
32
32
16
LR
0.001
0.001
0.001
0.001
Optimizer
Adam(0.9,0.99)
Adam(0.9.0.99)
Adam(0.9,0.99)
Adam(0.9.0.99)
Multistep
Multistep
Scheduler
Cosine(T =50))))))
Cosine(Tmax=
(T=(50,90,130)
(T=(50,90.130)
Surrogate
Arctan(a=2.0)
Aretan(a=2.0)
Arctan(a=2.0)
ArcTan(a=2.0)
The exact iteration step at which the training accuracy first reached certain expected level (60%) was recorded to represent the training speed. In A8 green bars represent our method, while bars in other colors represent other methods. It is intuitive that our method reaches the expected accuracy faster or at least at the same performance level for most neuron types compared to other methods; this is more obvious when falls in the range of (=o0, 0.6) or (2.0, -.)). particular, there are no valid records at all using other methods in cases in which À is set 0. 0.2. 8.0, and so However, even these cases, the training accuracy of our method still reaches the expected level in very short time.
In addition to our method's superiority in terms training speed, we measured its performance in terms of the validation accuracy to check its robustness classification tasks under various configurations. In Fig.A9 the classification accuracies of five initialization methods under different neuron configurations shown heatmaps. Our method can be used effectively and efficiently
16

---

Under review as a conference paper at ICLR 2022
Table A5: Optimizers and their parameters used test the versatility.
Optimizer Learning rate
Other Parameters
Adam
0.001
betas=(0.9, 0.99)
SGD
0.5
momentum=0.9 lambda=0.000001, alpha=0.75
ASGD
2.0
AdaGrad
0.01
train SNNs with almost all neurons types, while other methods fail train networks with some neuron types.
To further analyze whether the proposed method exhibits versatility for various choices of optimizers and surrogated funtions, eight representative groups of (k, X, 0) were selected: (0.3, 0.1. 1.0), (0.8, 0.2, 1.0), (0.8, 0.5, 1.0), (0.2, 0.8, 1.0), (1.0, 1.0, 1.0), (0.5, 4.0, 1.0), (0.6, 0.3, 4.0) and (0.4, 0.4, 4.0).
Ours
He's
Gloret's
Wu's
Lec's
2-87
9.0
+
W
1+10
Figure A8: Training iterations at which the accuracy first reaches the expected level (60%) for first- order integrate- interrate-and-in and-fire neurons with different (k:, A) settings and initialization methods. Note that 0 is set to 1.0. The total number of iterations of 3 training epochs is 2811. 1. A value of 281 in this figure means that in the corresponding setting, the network cannot be trained to reach the expected training accuracy level.
Versatility Across Optimizers
During model training, different optimizers may be used to minimize the loss function. To test the versatility of our method across various optimizers, we chose four classic and prevalent optimizers for model training, i.e., Adam, SGD, ASGD, and AdaGrad. The detailed optimizer parameters are listed Table A5
we show the training curves during the first 500 iterations illustrate the results more detail. As shown this figure, when SGD or ASGD is chosen to optimize the network, our method demonstrates the training in the start-up phase with the accuracy reaching a good level within 100 iterations in most cases. For example, when A = 0.8, X 0.5, and 0 = 1.0, Wu's and He's methods allow training to begin early but with slower speed, achieving an accuracy of approximately 90% after 500 iterations, while Glorot's and Lee's methods show much worse performance. When Adam or AdaGrad is chosen to optimize the network, despite the similar performance all methods some cases (for example the 3rd-5th rows in the figure), our method outperforms the others in accelerating
We present the accuracy of training in Table A7 In Fig. A10
advantage in accelerating
17

---

Under review as a conference paper at ICLR 2022
Ours
Glorot's
He's
Wu's
Lee's
0.3
k
0.5
θ 0.5
0.9
0.3
k
0 == 1
0.9
0.3
k
0=
0.7
0.9
0.3
0.5
=
0.7
0.1 0.5
0.1
0.5
0.1
0.5
0.1
0.5
A
A
0.0
0.2
0.4
0.6
0.8
1.0
Figure A9: Training accuracy heatmaps for first-order integrate-and-fing neurons with different (k,, A, 0) settings and initialization methods. The color represents the validation accuracy in the given setting.
the training process. In addition, our weight initialization for SNNs enables valid training in some difficult cases in which training totally fails with other initialization methods, such as (k 0.4, A 0.4, 0.4).
Versatility Across Surrogate Functions
Various surrogate functions may be chosen for SNN training w.r.t different needs and tasks. As listed in Table A3 four surrogate functions (sigmoid, tanh, hardtanh, and arctan) are popularly used BPTT. As described in Sec. 2.J] the input subjected to the Heaviside function to perform the forward pass, while the backward gradient is computed using surrogate functions. We trained the 5-layer SNN using surrogate functions consistent with those Table A3 by adjusting the parameter
The validation accuracy after epochs of training is presented in Table A7 and the training curves from the first 220 iterations with the four surrogate functions are shown in Fig. AI]] The curves in green show our method's effectiveness. Our method reaches acceptable accuracy within relativelt few iterations (fewer than 50 in most cases), while in comparison, the decline in the network loss starts more slowly under other initialization methods, taking parameter sets of (0.8, 0.2, 1.0), (0.8, 0.5, 1.0), (0.2, 0.8, 1.0) and (0.5, 4.0, 1.0) examples. Especially when & and À have small values or has a large value, other initialization methods cannot generate effective gradients for BP, leading to training failure. For almost all sets of surrogate functions and neuronal parameters, enables neuron firing and early generation of network gradients. Fig. All reveals that our method enables models start training faster and to end training with considerably greater classification accuracy.
method
A.3 COMPARISON TO BATCH NORMALIZATION (BN)
As Eq. [I implies, the variances of input current from the adjacent layers have been scale to be identical when using initialization method, which can be viewed as a kind of normalization of current before the start of training. The difference lies in the term of validity. Normalization methods such Kim & Panda 2020), Kim & Panda (021) normalizing current throughout training. To clarify this, we examined the validation accuracy with or without normalization methods on CIFAR 100 and TinyImageNet datasets. We chose vanilla BN (Toffe & Szegedy 2015) and BNTT
18

---

Under review as a conference paper at ICLR 2022
Adam
SGD
ASGD
AdaGrad
0.5
$ 1.
5 0.4
66
6.9
g 6.4
2 6.5
1.
0.5
0.9
0.3
1=04
0.3
He
Tierations
Tlerations
Merations
Eteration
Figure A10: Training accuracy obtained using four different optimizers, different (k, X) settings and different initialization methods.
Sigmoid(a-2.0)
Tanh(a-2.0)
Arctan(a-1.0)
Hardmahh(a-1.0)
0.3
11
0
0.3
11
0.0
1.0
0.3
11
0.0
0.9
111 0
04
0.9
11
0.
0.9
5 9.6
111 1
09
5 9.6
111 50
0.6
11
100
200
2001
100
Figure All: Training curves obtained using four different surrogate functions, different (k, O) settings and different initialization methods.
(batch normalization through time) Kim & Panda 2020 for comparison. When the network is without BN, dropout is implemented in place of BN. (T 0.99, À 1.0) is the default setting iriKim Panda 2020). The validation accuracy curve is plotted in Fig. A12 The use BNTT make
the effect of initial lization insignificant it works throughout training. Table A8 shows that with BN
19

---

Under review as a conference paper at ICLR 2022
Table A6: Validation accuracy achieved when training on MNIST for 3 epochs using four different optimizers, different (k; X, 0) settings and different initialization methods. A hyphen in this table indicates invalid training in a given setting.
k 0.3, Glorot's
0.1, 0 1.0
k 0.8,
0.2,
1.0
Ours
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
Adam
0.979
0.982
0.972
0.972
0.973
0.971
SGD
0.776
0.929
ASGD
0.636
0.887
AdaGrad
0.966
0.878
0.898
0.876
0.740
0.975
0.959
0.967
0.956
0.957
k 0.8,
0.5,
0 1.0
== 0.2,
À 0.8,
1.0
Ours
Glorot's
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
Adam
0.984
0.980
0.986
0.983
0.984
0.984
0.978
0.988
0.982
0.989
SGD
0.975
0.921
0.959
0.964
0.945
0.968
0.950
0.956
0.884
ASGD
0.958
0.165
0.910
0.918
0.866
0.951
0.905
0.926
0.612
AdaGrad
0.984
0.978
0.984
0.984
0.982
0.977
0.978
0.986
0.985
0.986
k 1.0,
1.0,
0 1.0
k: 0.5, À 4.0,
1.0
Ours
Glorot's
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
Adam
0.987
0.986
0.987
0.986
0.986
0.986
0.987
0.984
0.985
0.986
SGD
0.985
0.982
0.985
0.983
0.985
ASGD
0.986
0.981
0.986
0.981
0.984
0.981
AdaGrad
0.986
0.984
0.986
0.985
0.986
0.983
0.977
0.984
k: = 0.6,
0.3,
0 4.0
h: 0.4,
À 0.4, 4.0
Ours
Glorot's
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
Adam
0.980
0.984
SGD
0.895
0.923
ASGD
0.841
0.876
AdaGrad
0.968
0.882
0.889 0.910
0.743
0.972
0.906
0.924
0.926
0.752
or BNTT, the gaps the accuracy of different initialization methods are not obvious. But when the network is without BN, method can also work under (0.5, 0.5) configuration. Therefore, not stating that our initialization method can outperform some well-designed learning algorithms. Instead, we believe our method is useful supplement to make SNNs trainable on larger datasets.
20

---

Under review as a conference paper at ICLR 2022
Table A7: Validation accuracy achieved when training on MNIST for 3 epochs using four different surrogate functions, different (ki, X, 0) settings and different initialization methods. A hyphen in this table indicates invalid training a given setting.
= 0.3, 0.1,
0 1.0
0.8,
À 0.2,
1.0
Ours
Glorot's
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
(ardtanh(a=1.0))
0.982
0.310
0.985
0.974
0.980
0.980
0.977
Arctan(a=1.0)
0.983
0.984
0.972
0.980
0.980
0.976
Tanh(a=2.0)
0.984
0.986
0.976
0.979
0.981
0.978
Sigmoid(a=2.0)
0.984
0.985
== 0.8,
0.5,
0 1.0
0.2, À 0.8,
1.0
Ours
Glorot's
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
Hardtanh(a==100)
0.987
0.985
0.987
0.984
0.986
0.987
0.983
0.990
0.986
0.988
Arctan(a=1.0)
0.988
0.983
0.986
0.987
0.989
0.988
0.986
0.991
0.987
0.991
Tanh(a=2.0)
0.989
0.987
0.987
0.988
0.990
0.988
0.984
0.990
0.989
0.991
Sigmoid(u=2.0)
0.988
0.986
k == 1.0,
1.0,
1.0
0.5,
À 4.0,
1.0
Ours
Glorot's
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
Hardtanh(a=1100
0.988
0.988
0.988
0.984
0.989
0.985
0.987
0.980
0.984
0.986
Arctan(c=1.0)
0.986
0.987
0.986
0.981
0.984
0.987
0.987
0.973
0.973
0.978
Tanh(a=2.0)
0.988
0.988
0.988
0.990
0.987
0.988
0.983
0.985
0.984
0.980
Sigmoid(a=2.0)
0.987
0.986
0.987
0.887
0.989
0.199
0.197
0.176
k
0.6, À 0.3,
4.0
0.4, X 0.4,
4.0
Ours
Glorot's
He's
Wu's
Lee's
Ours
Glorot's
He's
Wu's
Lee's
(ar))
0.979
0.983
Arctan(c=1.0)
0.917
0.953
Tanh((=2.0)
0.968
0.976
Sigmoid(a=2.0)
0.887
0.951
Aceuracy comparison with different normalization technique training with VGG11 on large datasets. A hyphen in this table indicates invalid training given setting.
Table A8:
Dataset
(k, A)
Technique
Initialization method
Glorot's
He's
Wu's
Lee's
Ours
CIFAR 100
(0.5,
w/o BN. Dropout
54.94
CIFAR100
(0.99.1.0)
w/o BN. Dropout
54.03
54.46
53.79
CIFARIOG
(0.99,
BN
57.18
57.19
56.45
56.64
57.02
CIFAR 100
(0.99.
w/ BNTT
65.89
65.55
64.91
65.65
65.44
TinylmageNet
(0.5.0.5)
w/o BN
44.16
TinyImageNet
(0.99.
BN
45.97
47.18
47.60
45.03
47.51
TinyImageNer
10.99,
w/ BNTT
57.03
57.09
56,90
57.19
56.80
21

---

Under review as a conference paper at ICLR 2022
TinyInageNa eegee Ape
CIFARIOO Acc
30
30
20
10
Gland's
Gloe
25
50
75
100
125
150
175
200
20
40
80
100
Epochs
Epochs
Figure A12: BNTT Training curves obtained using different initialization methods under the (k=0.99, x=1.0) configuration.
22


Reviews:
[{"review_id": "wFh0s3O1D2", "reviewer": "Reviewer_xJ5p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors have presented interesting results. The discrepancy between forward spike activation function and backward surrogate gradient function during backprop restricts training capability in SNNs. Modern deep learning relied on tricks like ReLU, initialization to balance forward/backward variance of activations (Glorot/He), skip connections etc. to overcome the vanishing gradient problem. The weight initialization approach in this work aims to do similar things. \n\nWhile the authors have compared with recent works, I would like to bring the attention of authors to recent works from a group that authors have cited in their work, that use batch norm and threshold initialization as a way to mitigate the gradient training issue. I feel these two works are closely related to the authors work and both yield better performance results with very interesting SNN dynamics advantages like low latency etc. Since the authors have missed out comparing their work to these to works which are more in line with what authors are trying to do, I am a little concerned about the sanity and the novelty of this work.\n \n[R1] Kim, Y., & Panda, P. (2020). Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. arXiv preprint arXiv:2010.01729.\n[R2] Kim, Y., & Panda, P. (2021). Optimizing deeper Spiking Neural Networks for Dynamic Vision Sensing. Neural Networks.\n\nIn [R1], Kim et al. present a temporal batchnorm technique to improve the gradient vanishing/explosion. I feel this work is very related to [R1]. So, it will be prudent for the authors to comment and show a comparsion. Further, in [R1] , the authors are able to train large-scale datasets from scratch like CIFAR100, TinyImagenet with a very low latency on rate-coded inputs. I am wondering if the author's method is scalable to non-trivial datasets beyond CIFAR10, MNIST (which are easy to get good accuracy with).\n\nBased on this, my next question is, can authors comment how weight initialization impacts the overall spike activity, latency of processing of the network.\n\nIn [R2], Kim et al. propose a threshold initialization technique to improve the trainign of SNNs for DVS datasets and they looked at interesting datasets like N-Caltech (more complex than N-MNIST) as well as DVS-CIFAR10, DHP etc. Again, since the authors have looked into small scale datasets, it makes me question the scalability of their work. I think the authors should comment on this and also highlight how their work is better or orthogonal or complementary to these two works.\n\n\n", "summary_of_the_paper": "This paper enhances the efficient BPTT training of SNNs by proposing an initialization method to match the response of spiking neurons in initial training. Their method bridges the spiking neuron response to the wisdom of traditional deep learning training, which may have an influence on future research like ANN-to-SNN conversions with LIF neurons or other SNN training methods. The authors conduct experiments on CIFAR10, MNIST, and neuromorphic datasets to show the efficacy of their technqiue. One of the main contribution is their theorteical analysis of iterative systems to model first-order integrate-and-fire neurons and investigate the response curve.\n", "summary_of_the_review": "The results are interesting. I feel the authors should make a more thorough investigation of recent work that highlight similar ideas and yield better results on more complex and large-scale datasets. I am giving a rating of 5, but I am willing to chnage my score if the authors can convince me of the scalability and the novelty of their approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jyV8bO5wxU", "reviewer": "Reviewer_gWVF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem of inefficiencies in trainign SNN models and presents a compelling argument for the inefficient SNN initialization and demonstrate that improving it so that neurons are responding from training onset can increase both convergence and test accuracy.\n\nWhile the paper is technically sound and offers a substantial result, the presentation is quite  convoluted and could be substantially streamlined and would benefit dramatically from a read by someone fluent in English. Just to mention a few issues, the introduction section is overly verbose and does not seem to be always to the point. Many statements are non-informative, like \"neurons respond in a suitable region...\" or \"producing proper amount of spike\" where \"suitable region\" and \"proper amount\" is never defined. Incorrect use of terms abound (non-differential => non-differentiable etc) that can seriously obscure the authors' intentions.\n\nMinor issues:\nFigure 2: Dashed lines referenced but not displayed; what is the difference; why the actual neuron responses (circles) are not shown for the full range of inputs?\n\n\n", "summary_of_the_paper": "The authors tackle the problem of improving spiking neural net training with better initialization. They observe that error backpropagation efficiency in SNNs depends on availability of responding neurons and show that traditional surrogate backprop methods are inefficient as they take many iterations to move weights to the regime supporting neuronal responses. \nThe authors approximate the SNN neuron model (LIF, IF etc) I/O function using a piecewise-linear  iterative expression and show that clipping the output to (0,1) and normalizing the variance of the random weight initialization to insure that neurons produce substantial responses even with initial random weights results improves SNN training substantially. The improvement is to a large extent due to early onset of convergence. The authors demonstrate the superiority of their approach on MNIST, N-MNIST, DVS-MNIST and CIFAT datasets and present extensive experiments using different optimization algorithms and response functions. \n", "summary_of_the_review": "While the paper presents an important and technically sound contribution, the quality of presentation dramatically reduces the value of the manuscript.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EqD7RNmXetN", "reviewer": "Reviewer_ZLvg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper proposes a weight initialization method to enhance the efficient BPTT training of SNNs. The derivation and validation of the weight initialization method appears to be sound. It is interesting to see the authors relate the spiking neuron response to the traditional deep learning training. This work may also inspire research like ANN-to-SNN conversions with LIF neurons.\nWeaknesses:\n(1) The related work section should be introduced in more detail. Is there any other study about weight initialization of SNN? \n(2) Why is the weight initialization more effective on the CIFAR10 than the MNIST?\n(3) I suggest to demonstrate the effect of the initialization method on more complex dataset..\n", "summary_of_the_paper": " This paper focuses on the parameter initialization problem of training SNNs. The authors derive the theoretical response of spiking neurons, and propose an initialization method based on slant asymptote, which can overcome the gradient vanishing problem. The results show that the proposed method can effectively improve training speed and accuracy.", "summary_of_the_review": "Overall, it is a nice paper with solid theoretical basis. It has the potential to convince researchers to adopt SNNs and exploit learning of deep SNN. I consider to raise my score if the authors can resolve the concerns above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1O5JTrVoVLB", "reviewer": "Reviewer_LCr2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Whereas previous work on Deep SNN has often borrowed techniques from standard ANNs, this paper goes beyond that and develops SNN-specific methods that can accelerate Deep SNN training and lead to better accuracies. This is achieved from theoretical analysis of SNN neuron models and network dynamics, and the results seem to indicate that the method is generally superior to non-SNN-specific methods under a variety of parameters and datasets. If this method generalizes to larger tasks and networks, this could become a new standard for SNN training, and therefore a significant contribution to the SNN literature.\n\nStrenghts:\n+ derivation of the weight initialization method from theory\n+ convincing results of experimental validation on a variety of datasets\n+ comparison to a number of standard initialization methods from the ANN world,which do not have SNN-specific adaptations\n+ possibly very broad application field, could become a new standard\n\nWeaknesses:\n- occasional unclear writing and grammar errors which should be fixed\n- tested only on MNIST and CIFAR variants, it is unclear whether the method generalizes to larger networks\n\nRecommendation:\n- I would suggest including at least one larger experiment that shows that the weight initialization scales to larger networks, and is not an effect of the relatively easy datasets that were considered", "summary_of_the_paper": "The paper investigates the effects of weight initialization for deep spiking neural networks (SNNs) on training speed and final accuracy. The main finding is that popular initialization methods for conventional ANNs and RNNs do not match the specific dynamics of SNNs, in particular they ignore the need to have sufficient firing early on to generate good gradients. The paper derives a theoretical first-order approximation on the response curve, and from it a weight initialization scheme that initializes the weights in a region where neurons fire from the first epoch on, but also avoid explosion or reduction of activities throughout the network. In various experiments on standard and spiking variants of MNIST and CIFAR, the initialization scheme shows good performance and fast convergence for different neuron types (e.g. non-leaking vs. fast leaking), and different encoding schemes. For CIFAR10 the results are significantly better than for other initialization schemes.", "summary_of_the_review": "The presented method could become important for further studies on Deep SNNs, as a replacement of poorly fitting weight initialization schemes borrowed from ANNs. It remains to be shown that the method scales to more difficult tasks and larger networks. If the method still works there, this could be a paper with very high impact.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper derives a new parameter initialization for deep spiking neural networks to overcome the vanishing gradient problem.

During the review, concerns were expressed about how well the method would scale to larger neural networks. It was also questioned how this parameter initialization technique compares with a recently proposed batch normalization technique, especially when training larger neural network on more challenging datasets. There were also concerns raised about the readability of the paper.

I commend the authors for improving the readability of their paper in their revision. I also commend them for taking the time to implement the comparisons requested by the reviewers. These new comparisons revealed that batch normalization and its recently proposed variant were superior to the initialization method on its own, and that the initialization proposed in the paper did not significantly improve performance when paired with batch norm [[1](https://openreview.net/forum?id=T8BnDXDTcFZ&noteId=yIAPcSbUAQ0)]. The authors also acknowledged based on the new results, that their proposed parameter initialization scheme appears to fail to scale to more complex datasets and networks, especially relative to competing methods, which invalidates a key claim that their approach can "accelerate training and get better accuracy compared with existing methods" [[2](https://openreview.net/forum?id=T8BnDXDTcFZ&noteId=j12fwayWEb)].

The recommendation is to reject the paper in its current form.
Decision: Reject