Sequence Approximation using Feedforward Spiking Neural Network for Spatiotemporal Learning: Theory and Optimization Methods
Authors: Xueyuan She
Saurabh Dash
Saibal Mukhopadhyay
Venue: ICLR 2022 Poster (2022)
Keywords: spiking neural network
spatiotemporal processing
feedforward network

Abstract: A dynamical system of spiking neurons with only feedforward connections can classify spatiotemporal patterns without recurrent connections. However, the theoretical construct of a feedforward spiking neural network (SNN) for approximating a temporal sequence remains unclear, making it challenging to optimize SNN architectures for learning complex spatiotemporal patterns. In this work, we establish a theoretical framework to understand and improve sequence approximation using a feedforward SNN. Our framework shows that a feedforward SNN with one neuron per layer and skip-layer connections can approximate the mapping function between any arbitrary pairs of input and output spike train on a compact domain. Moreover, we prove that heterogeneous neurons with varying dynamics and skip-layer connections improve sequence approximation using feedforward SNN. Consequently, we propose SNN architectures incorporating the preceding constructs that are trained using supervised backpropagation-through-time (BPTT) and unsupervised spiking-timing-dependent plasticity (STDP) algorithms for classification of spatiotemporal data. A dual-search-space Bayesian optimization method is developed to optimize architecture and parameters of the proposed SNN with heterogeneous neuron dynamics and skip-layer connections. 

Published as a conference paper at ICLR 2022
SEQUENCE APPROXIMATION USING FEEDFORWARD SPIKING NEURAL NETWORK FOR SPATIOTEMPORAL LEARNING: THEORY AND OPTIMIZATION METHODS
Xueyuan She, Saurabh Dash & Saibal Mukhopadhyay School of Electrical and Computer Engineering Georgia Institute of Technology, Atlanta, GA 30332, USA (xshe6, s8ash38 (eatte).@aahhhhhcc. sai saibal mu mmkkoppadyyay@eee.eeteeee edu
ABSTRACT
A dynamical system of spiking neurons with only feedforward connections can classify spatiotemporal patterns without recurrent connections. However, the the- oretical construct of a feedforward spiking neural network (SNN) for approximat- ing a temporal sequence remains unclear, making challenging to optimize SNN architectures for learning complex spatiotemporal patterns. In this work, estab- lish a theoretical framework to understand and improve sequence approximation using a feedforward SNN. Our framework shows that a feedforward SNN with one neuron per layer and skip-layer connections can approximate the mapping function between any arbitrary pairs of input and output spike train on compact domain. Moreover, we prove that heterogeneous neurons with varying dynam- ics and skip-layer connections improve sequence approximation using feedfor- ward SNN. Consequently, we propose SNN architectures incorporating the pre- ceding constructs that are trained using supervised backpropagation grggationnthrough--a through-time (BPTT) and unsupervised spiking-timing-depple timing-dependent plasticity (STDP) algorithms for classification of spatiotemporal data. A dual--serrh--spaee Bayesian optimiza- tion method is developed to optimize architecture and parameters of the proposed SNN with heterogeneous neuron dynamics and skip-layer connections.
INTRODUCTION
Spiking neural network (SNN) (Pouulak & Kasinski 2011 uses biologically inspired neurons and synaptic connections trainable with either biological learning rules such as spike-timing- dependent plasticity (STDP) (Gerstner & Kistler 2002) or statistical training algorithms such as backpropagation-thhone thr.iiinnnttrouu---iitee (BPTT) (Werbos) 1909. The SNNs with simple leaky integrate-and- fire (LIF) neurons and supervised training have shown classification performance similar to deep neural networks (DNN) while being energy efficient (Kim et al.) 20206] Wu et al.) 2019)] Srinivasan & Roy 2019). One of SNN's main difference from DNN is that the neurons tems with internal states evolving over time, making it possible for SNN to learn temporal patterns without recurrent connections. Empirical results on feedforward-onle SNN models show good per- formance for spatiotemporal data classification, using either supervised training (Lee et al.) 2016)] Kaiser et al. 2020 Khoei al. 2020), or unsupervised learning (She et al. 2021), However, while empirical results are promising. lack of theoretical understanding of sequence approximation using SNN makes challenging to optimize performance on complex spatiotemporal datasets.
dynamical sys-
In this work, we develop a theoretical framework for analyzing and improving sequence approxi- mation using feedforward SNN. We view feedforward connection of spiking neurons as a spike propagation path, hereafter referred to as a memory pathways (She et al. 2021). that maps an input spike train with an arbitrary frequency to an output spike train with a target frequency. Conse- quently, argue that an SNN with many memory pathways can approximate temporal sequence of spike trains with time-varying unknown frequencies using a series of pre-defined output spike trains with known frequencies. Our theoretical framework aims first establish SNN's ability to map frequencies of input/output spike trains within arbitrarily small error; and next, derive the basic principles for adapting neuron dynamics and SNN architecture to improve sequence approximation.

---

Published as a conference paper at ICLR 2022
The theoretical derivations are then investigated with experimental studies feedforward SNN for spatiotemporal classifications. We adopt the basic design principles for improving sequence approx- imation to optimize SNN architectures and study whether these networks can be trained to improve performance for spatiotemporal classification tasks. The key contributions of this work are:
We prove that any spike- wwweennee---siik--eeeee sequence mapping functions compact do- main can be approximated by feedforward SNN with one neuron per layer using skip-layer connections, which cannot be achieved if skip-layer connection used.
We prove that using heterogeneous neurons having different dynamics and skip-layer con- nection increases the number of memory pathways a feedforward SNN can achieve and hence, improves SNN's capability to represent arbitrary sequences.
We develop complex SNN architectures using the preceding theoretical observations and experimentally demonstrate that they can be trained with supervised BPTT and unsuper- vised STDP for spatiotemporal data classification.
We design dlual-search-ppee space option for Bayesian optimization process to sequentially op- timize network architectures and neuron dynamics of a feedforward SNN considering het- erogeneity and skip-layer connection improve learning and classification of spatiotem- poral patterns.
We experimentally demonstrate that our network design principles coupled with the dual-search- space Bayesian optimization improve classification performance on DVS Gesture (Amir et al.) 2017), N-caltech (Orchard et al., 2015), and sequential MNIST. Results show that the design prin- ciples derived using our theoretical framework for sequence approximation can improve spatiotem- poral classification performance of SNN.
2 RELATED WORK
Prior theoretical approaches analyze SNN often focus the storage and retrieval of precise spike patterns (Amit & Huang 2010: Brea et al.
2013).
There are also works that consider SNN for
Binas et al. 2016) and works that analyze the dynamics of SNN (Zhang et al. 2019 Barrett et al. 2013). Those are different topics from the approximation of cec---ccuencett--spi-ee sequence mappings functions. SNN that incorporates ex- citatory and inhibitory signal is shown for its ability to emulate sigmoidal networks (Maass) 1997) and is theoretically capable of universal function approximation. Feedforward SNN with specially designed spiking neuron models (Tannella & Back 2001 Torikai et al., 2008) have been demon- strated for function approximation, while for networks using LIF neurons, function approximation has been shown with only empirical results (Farsa et al. 2015). On the other hand, existing works that have developed efficient training process for SNN and demonstrated classification performance comparable deep learning models, have mostly used simpler and generie LIF neuron models (Lee) ctt al. 2016 Kaiser et al. 2020 Kim et al. 2020b) Wu et al. 2019 Sengupta et al.) 2019) Safa et al.] Han et al. 20220. Therefore, this paper develops the theoretical basis for function approxima- tion using feedforward SNN with LIF neurons, and studies applications of the developed theoretical constructs improving SNN-based spatiotemporal pattern classification.
solving optimization problems (Chou et al.
2018
2021
The effectiveness of heterogeneous neurons (She et al. 202T) and skip-layer connections (Srinivasan & Roy 2019 Sengupta et al. 2019) in SNN has been empirically studied in the past. However, no theoretical approach has been presented to understand why such methods improve learning of spike sequences, and how optimize SNN's architecture and parameters to effectively exploit these de- sign constructs. It is possible to search for the optimal SNN configurations through optimization algorithms, but the large amount of hyper-parameten for spiking neurons and network structure cre- ates a high-dimensional search space that is long and difficult to solve. Bayesian optimization (Snoek) ctt al. 2012) uses collected data points to make decision on the next test point that could provide im- provement, thus accelerates the optimization process. Prior works (Parsa et al.) 2019: Kim et al.) 2020a) have shown that SNN performance can be effectively improved with Bayesian optimization. While those works consider single or few neuron parameters, the dual-search- space Bayesian optimization proposed in this work optimizes both network architecture and neuron parameters effi- ciently by separating the discrete search spaces from the continuous search spaces.
2

---

Notations Neuron Delay Ind is the time for spike from pre-synaptic neuron to arrive at its post- Published as a conference paper at ICLR 2022
Input spikes
Neuron Dynamic d.
spikes
One neuron per layer per dynamic
spikes
02 spikes
ng
Neuron Dynamic
n2 spikes
0g
Layer number: m
nd
(a)
(b)
Figure/ (a) time-varying input spike sequence received by two memory pathways: neuron membrane potential plots show the different response from the neurons to the given input. (b) A minimal multi-ncuron-dyyamii (mMND) network with m layers and n neuron dynamics.
APPROXIMATION THEORY OF FEEDFORWARD SNN
3.1 DEFINITIONS AND NOTATIONS
Definition Neuron Response Rate y For spiking neuron with membrane potential at Greset and input spike sequence with period fin 7 is the number input spike n needs to reach Uth-
Definition 2 Memory Pathways For feedforward SNN with layers, memory pathway is defined as a spike propagation path from input to the output layer. Two memory pathways are considered distinct if the set of neurons contained in them is different.
Definition 3 Minimal
titi---uuron-dynnttt (mMND) Nenwork densely connected network in which each layer has an arbitrary number of neurons that have different neuron parameters. All synapses from one pre-synaptic neuron have the same synaptic conductance.
synaptic neurons, as shown in Figure a[). For a feedforward SNN with m layers, a skip-layer connection be defined with source layer and target layer pair (l,, 4).. The output feature map from source layer is concatenated to the original input feature map of the target layer. For the analysis of spike sequence in temporal space, the notation of T max and Tiii defined positive real numbers such that Tmax
Tmin- >> 0 is the eтror of approximation.
Figure [ shows two memory pathways receiving an input spike sequence with time-varying pe- riods. As the neurons have different dynamics, the two memory pathways have different response the input spike sequence. An example of mMND network with m layers and n neuron dynam- ics is shown in Figure [[).. SNN with multilayer perceptron (MLP) structure can be considered scaled-up mMND network with multiple neurons for each dynamic. A network with convolutional structure can be considered scaled-up mMND network with duplicated connections in each layer. We analyze the correlation of network capacity and structure based on mMND networks. The de- sign of neuron heterogeneity can also be implemented in MLP-SNN and Conv-SNN described in Section The analysis for network capacity can be extended to those networks according to their specific layer dimensions.
3.2 MODELING OF SPIKING NEURON
SNN consists of spiking neurons connected by synapses. The spiking neuron model studied in this work is leaky integrate-and-fir (LIIF) as defined by the following equations:
du Tm
= a + Rm 2i U = Ureeet: o > Uthreshold
(1)
dt
3

---

Published as a conference paper at ICLR 2022
Rm is membrane resistance, Tm = Rn Cn is time constant and C. is membrane capacitance. is the resting potential. is the sum of current from all input synapses that connect to the neuron. A spike is generated when membrane potential t cross threshold and the neuron enters refractory period r, during which the neuron maintains its membrane potential at Ureset- The time takes for pre-synaptic neuron send spike to its post-synaptic neurons is bnd- Neuron response rate y is a property of a spiking neuron's 's response to certain input spike sequence. We show how the value of y can be evaluated below.
Remark For any input spike sequence, each individual spike can be described with Dirac delta function 6(t tA) where t' is the time of the i-th input spike. Consider membrane potential of a spiking neuron receiving the input before reaching spiking threshold, with initial state at t = 0 with V Ureset, solving the differential equation ① leads to (Gerstner 1995):
(t) UreaetC Ton a(1 C Fire +
Rm Tm
Ee
S 5(t - t dt
Here, G is the conductance of input synapse connected to the neuron, which is trainable. From ( there exists a value of u such that on (t"-)) < Uthreshold and Unm(t)) >= l'threholdd By evaluating for u given neuron parameters and input spike sequence, the neuron response rate y can be found.
3.3 APPROXIMATION THEOREM OF FEEDFORWARD SNN
To develop the approximation theorem for feedforward SNN, we first aim to understand the range of neuron response rate that can be achieved. We show with Lemma ttha for any input spike sequence with periods in a closed interval, it is possible to set the neuron response rate y to any positive integer. Based on this property, we show with Theorem thha by connecting list of spiking neurons with certain 7 sequentially and inserting skip-layer connections, network with spike period mapping function P(t) can be achieved to approximate the target spike sequence. To understand whether this capability of feedforward SNN relies on skip-layer connections, we develop Lemma 2
to prove that skip-layer connections are indeed necessary. In subsection 3.4 we investigate the correlation between approximation capability and network structures by analyzing the cutoff property of spiking neurons, which can change the network's connectivity. In our analysis, we focus on two particular designs: heterogeneous network (Lemma 4 and skip-layer connection (Lemma 5 and show their impact on the number of distinct memory pathways in a network. All lemmas are formally proved in the appendix.
Lemma For any input spike sequence with period ten in range [Tmins Tmauly there exists spik- ing neuron n. with fixed parameters Uth: Gresets R. and Tma such that by changing synaptic conductance G, is possible to set the neuron response rate "In to be any positive integer.
Proof Sketch. (Formal proof in Appendix A Given an input spike sequence, the highest possible membrane potential decay Av for any input tin [Tmmn [ can be derived as a function of neuron parameters. We show is possible to make Ae tend to zero by setting the neuron parameters. Since the decay of v can be negligible, In can be set to any positive integer by changing G.
Theorem For any imput and target output spike sequence pair with periods (Lin; lout) E [Tmin: Tmaxx] mina m. there exists hainimllllayerssize network with skip-layer connections that has memory pathway with output spike period function P(t) such that [P(tin) - touel < e.
Proof Sketch. (Formal proof in Appendix B} With skip-layer connections, there can be multiple memory pathways in a minimal-layer--ee network as neurons can be either included or skipped through. Hence it is possible create memory pathways with different delay times for each input spike in a network. By connecting the output of those memory pathways to a common neuron n', spike sequence of any arbitrary period fint such that line <= tin can be generated within e. By setting " > the output from n' receiving input spike sequence with line is cout = 7 Lint- Hence it is possible to achieve a network with output spike period P(t) such that [((tn)) - toutl <.
Lemma 2 With skip-layer connection, there does not exist minimal-layend layer-size network that has output spike period function P(t) such that for any input and target output spike sequence pair with Lout)
periods (Jins lout) E [T minn: T ] x [T minnn T T.naz). |P((.n)
4

---

Published as a conference paper at ICLR 2022
Proof Sketch. (Formal proof in Appendix C A minimal-layer-Sa network without skip-layer nection has only one memory pathway. For a particular input spike sequence with period tin, dif- ferent output period P(t.)) can be achieved by changing ? of neurons in the memory pathway. We show that there exists two output spike periods P(laa) and P'(Im).. ). such that P((in) - P (lin) is a constant value independent of network neuron configurations, and there can be P"(tin) such that P(tm) < (tin) < (ttm). Therefore, for any minimal-layer-sizz network, there exists tout within the range of (P(tin,, P (tin)) such that [P(tm) toua < e does not hold true.
3.4 NETWORK STRUCTURE AND MEMORY PATHWAYS
Based on Theorem 1 it is possible to approximate an input/output spike sequence mapping func- tion using a minimal-layer-sizze network with specific configuration, which can be considered as a memory pathway. Since any bounded continuous function on compact domain can be approxi- mated to arbitrary accuracy using a piece-wise constant function, and it is possible to use a memory pathway approximate each of the constant function, with increasing number of distinct memory pathways, a feedforward SNN can achieve approximation of continuous functions with less error. In this subsection, show that with two structural designs: heterogeneous network i.c. network having neurons with different dynamics and adding skip-layer connections, a feedforward SNN has the capability to achieve more distinct memory pathways.
Cutoff Frequency of a Memory Path neuron parameters with Lemma which is proved in Appendix D
We first show the correlation of cutoff period and spiking
Lemma 3 A spiking neuron has cutoff period we Tm In Vreset
above which input spike
Ureset a+ Rmm
sequence cannol cause the spiking neuron spike.
Remark From Lemma it can be observed that the cutoff period we of neuron can be configured to any positive real number by changing the neuron parameters and synaptic conductance G. Further, with fixed G, we be configured any positive real number by changing the neuron parameters. Neurons that are in cutoff change the spike propagation path in a network as they send no output spikes. This creates different memory pathways without changing the connections in a network.
Heterogeneous Network If an mMND network has the same parameters for all neurons in each layer, the majority of the neurons are included in the same memory pathway, leading to the upper bound of number of distinct memory pathways to be limited. With Lemma g we show the rela- tionship between the upper bound of the number of distinct memory pathways and the number of different neuron dynamics in an mMND network.
Lemma 4 For an mMND network with m layers and A2....mm number of different neuron dynamics each layer, the least upper bound of the number of distinct memory pathways is I[ X
Proof Sketch. (Formal proof in Appendix D For mMND network, it is possible have neurons with different We in each layer, which creates X; number of different neuron activation states for layer i. Across all network layers, the highest possible number of different neuron activation states is therefore the product of each layer. Since neurons in cutoff do not propagate spikes, they can be removed from a memory pathway. This leads of memory pathways.
I X as the least upper bound of the number
Compared to network with homogeneous neuron parameters, in which the upper bound of number of distinct memory pathways is Ams Lemma 4 indicates that heterogeneous network increases the maximum achievable number of distinct memory pathways in feedforward SNN.
Skip-layer Connection We show that adding skip-layer connection increases the upper bound of the number of memory pathways in a network with Lemma
Lemma For an mMND network with m layers and {)i, .m. different neuron dynamics in each layer and a skip-layer connection made between layer l and ln. S.I. a, b E {1, 2, 2,.... and
(b a) > the least upper bound of the number of memory pathways is I ) + (II X T X;)
5

---

Published as a conference paper at ICLR 2022
(a) BPTT Training
(b) STDP Training
Skip-taper connection
Skip layar
Memooyy
Dymamic
Dynamis
Impar
Lincor
Scurcs
Layer
Lineer
Dynamic
Multi-neuron- Pouti-mmrron-yymmmmc Layers
Layers
infered Syrapce
Memory
Module
Memory Module
Synappo
Dynamic
Dynamic
Synopse
Dynamic
Dynamic
Dynamic
Memory
Langet
Meduul
Module
Laye
Dynamoc
Dynamic
Learner
Learner Module
Figure 2: (a) The proposed network with BPTT training, each multi-neuron-dynamic layer contains neuron dynamics from di dm (b) The proposed network with STDP training.
Proof Sketch. (Formal proof in Appendix D) Compared to the network considered in Lemma by adding a skip-layer connection, there additional possible neuron activation states in the net- work that result from the cutoff of neurons in layers between la and l4- Without layers between la and i, in the spike propagation path, the least upper bound of the number of memory pathways is increased by the maximum number of distinct memory pathways in network that has layers (h, 12, ..., lay Lb, 1,.,,..., (m) connected sequentially.
SNN ARCHITECTURE DESIGN USING APPROXIMATION THEORY
In this section, we discuss design of SNN architectures inspired by the developed approximation theory for feedforward SNN, with more details in Appendix
Network Template for BPTT Training For BPTT training, the network template is shown in Fig- aa) The heterogeneity neuron dynamics implemented by using the multi-neuron-de neuron-dynamic layers, which can either be convolutional or fully connected. The mulii-neuron-dynamm different neuron parameters for spiking neurons each neuron dynamic module, which contains certain number of feature maps for convolutional layers certain number of neurons for fully- connected layers. There are two types of synapses between layers: transferred synapses marked as black dashed arrows and learned synapses marked red solid arrows. The conductance of learned synapses is optimized by the BPTT algorithm during training, and the transferred synapses have the same conductance as the learned synapses from the same pre-synaptic neuron. During forward pass, neurons in each layer receive the same input features and respond differently based on their neuron dynamics generate different output features. During back-propagation, only conductance of the learned synapses are updated. Skip-layer connection is implemented with the output spike matrix from source layer concatenated skip-layer connection has the same implementation the regular connection between consecutive layers, with both learned and transferred synapses (Figure a)). The last layer of the network is fully-connected layer with homogencous dynamic to generate prediction labels.
layers use
the original input spike matrix of the target layer. The
Network Template for STDP Learning For networks trained with STDP, the template is shown in Figure b). Each layer contains a leamner module and memory module. Learner modules use homogeneous neuron dynamic that is suitable for STDP learning, and memory modules consist of neurons with different dynamics. There also types of synapses: transferred synapses and learned synapses. Between two layers, memory modules are connected with transferred synapses and memory modules are connected to learner modules with learned synapses. Leaner modules be-

---

Published as a conference paper at ICLR 2022
tween layers are not directly connected. STDP training proceeds as layer-by-layer process. During training of the first layer, conductance of synapses connecting neurons in memory module to neu- rons in learner module is learned with STDP using all training data without supervision. Then, the learned conductance is transferred to the transferred synapses in the same layer. During training of the second layer, layer 1 memory module perceives the same input features and generates different output features with the heterogeneous neurons. This lay-by-layer process is repeated until the layer before the final layer finishes learning. The final linear layer is then fine-tuned using stochastic gradient descent (SGD) based on spike frequency array from the last multi-neuron-dyeee dynamic layer generated based on the labeled data. Skip-layer connection is implemented by connecting the mem- ory module of the source layer to the target layer. The connections are made with the two types of synapses and follow the same training process as the consecutive layers.
-pac--serc Bayesian Optimization Bayesian optimization uses Gaussian process to model the distribution of an objective function, and an acquisition function to decide points to evaluate. For data points in target dataset z E X and the corresponding label y E Y. an SNN with network structure and neuron parameters W acts as a function (x)) that maps input data æ to predicted label y. The optimization problem in this work is defined as
miny.wP where P =
Σ TEX
Cly, fv,w(x))
V contains the number of layers Nayerss the number of memory dynamics Naynmaic and skip- layer connection configuration variables Nakip, Lstart and Lend, each controlling the number of skip-layer connections, the first layer and last layer to implement skip-layer connections. All of the values are discrete. W contains the values for a, Tm and R., in (I), which are continuous. We separate the discrete and continuous search spaces by implementing dual-search-space optimiza- tion process, where V is first optimized with fixed, manually tuned neuron parameters. After an optimal structure is found. W are optimized for the selected V. Details on the configurations of the optimization process are listed in the appendix. To achieve Bayesian optimization with constraints, we implement modified expected improvement (EI) acquisition function similar to the one shown by Gardner (Gardner et al.) (2014), which uses Gaussian process to model the feasibility indicator due to its high evaluation cost. In this work, since the constraint function can be explicitly defined, we use a feasibility indicator that is directly evaluated. The modified EI function is defined as:
LWW) = (W) max (0, P(W) P(W+))
(4)
where W is the network configuration containing W and V. W+ is the test point that provided the best result. A(W) is the explicitly defined indicator function that takes the value of when all constraints are satisfied and otherwise.
EXPERIMENTS
5.1 EXPERIMENT SETTINGS
Datasets tested in the experiment include the DVS Gesture (Amir et al. (017). which is human gesture dataset captured by DVS cameras, and the N-Caltech101 (Orchard et al. (205)). which is an event-based version of the Caltech101 dataset. The proposed method is also tested for MLP-style SNN on the sequential MNIST dataset presented row-by-row. We also vary the amount of labeled data used during training ranging from using 100% labeled data for training down to 10% labeled data (30% for N-Caltech101 during training. Note, during STDP training networks always use the entire but un-labeled training dataset; however, only the fraction of the labeled data is used for su- pervised fine-tuning of the last layer. Comparison is made for DVS Gesture and N-Caltech101 with prior works including ConvLSNN, which is a combination of convolutional SNN and recurrent SNN with long and short-term neurons trained with BPTT (Salaj et al.) 2020). DECOLLE (Kaiser et al.) 2020), which uses surrogate gradient to train a convolutional feedforward SNN, HATS (Sironi et al. 2018), which implements time surfaces and SVM for classification and H-SNN (She et al.) 2021 which uses STDP to train a convolutional SNN with two neuron dynamics. Additional function approximation experiments discussed in Appendix E
7

---

Published as a conference paper at ICLR 2022
N-Cabech101 BPTT Training
N-Callcch 101
70
Single seareeeerch-paaas
Single Sngle-searchepees
Dual search-spase
Dustaearch-seeee
Valitation Enor
stage
Moldationn Emoo
stage
40
40
10
20
30
40
60
60
10
30
50
63
80
Evaluations
Evaluations
Figure 3: Validation error over optimization iterations for the proposed dual--earch-space Bayesian optimization compared to the normal single-search-space Bayesian optimization.
5.2 EFFECT OF DUAL-SEARCH-SPAAE SPACE BAYESIAN OPTIMIZATION
We compare the proposed stal--seacch-space Bayesian optimization with regular Bayesian optimiza- tion using single search space for network validation error over runs. The result from the N- Caltech101 dataset is shown in Figure 3 It can be observed that the two optimization approaches achieve similar minimum validation error after convergence. By separating the search spaces, the proposed optimization process reaches convergence faster than regular single-search-sye search-spae opti- mization. It is also worth noting that, between the two stages in the optimization process for BPTT training, the first stage accounts for more reduction validation error than the second stage. This indicates that optimizing network structure causes neuron parameters, which potentially due to the reason that network structure more heavily fects the number of memory pathways in the network than STDP training where learning behavior is sensitive to the dynamic of spiking neurons, the reduction of validation
impact to BPTT training than optimizing
parameters. On the other hand, for
is more equally shared between the optimization stages. Over the 5 runs, among all network configurations achieved after the dual-search-spaa compare the configuration with the lowest number of trainable parameters against baseline models. The specific configurations for the optimized networks are listed in Table T It can be observed that for BPTT algorithm, the optimized networks have more layers than the STDP trained networks, and the optimal values found for neuron parameters are highly distinct for the two training methods.
optimization converges, we
Table I: Configuration of optimized network models
Conv. Layer Number
Skip-layer Connection
Number of Different
Neuron Parameters
Network
Neuron Dynamics and a
Tm
R.,
BPTT, Gesture
9
(2,7)
4, 1-24.-17,-122999
120
340
BPTT, N-Caltech
12
(2,5), (5,8), (5,8). (8,11)
5. 5.6623-16,114-111388
70
300
STDP, Gesture
6
(2,4), (4.6)
(.22.244.11))9)
110
260
STDP, N-Caltech
(3.5). (5.7)
6, (211.119.117.113.9..)
140
240
5.3
ABLATION STUDIES
To investigate the effect of using multiple neuron dynamics, we apply the same dual-search-space Bayesian optimization process for networks that have homogeneous neuron dynamic for the same
Table Ablation studies of optimized networks
Model
Hetcrogeneity
Skip-layer
DVS Gesture
N-Caltech101
S-MNIST
Homogeneous-BPTT
N
Y
95.0
65.3
95.5
No-SKip-layer-BPTT
Y
96.5
63.5
94.8
This Work-BPTT
98.0
71.2
97.3
Homogeneous- STDP
STDP
N
Y
91.3
37.0
94.3
No-skip-layer-STDD
93.1
51.9
95.5
This STP
96.6
58.1
96.1
8

---

Published as a conference paper at ICLR 2022
Table 3: Accuracy (%) for DVS Gesture (top) and N-Caltech101 (bottom)
Labeled Data % In Training
Parameter Number
Model
100%
50%
30%
10%
ConvLSNN Salaj al. 2020
97.1
95.3
92.0
84.3
2.9M
DECOLLE (Kaser et al.
2020
97.5
95.0
91.2
83.9
1.3M
(Fang et al. 2021
97.8
HAT' SSron el 2018
95.2
94.1
91.6
83.7
H-SNN She ct al.
2021
96.2
95.8
93.7
88.2
0.74M
This Work-STDP Training
96.6
96.0
94.1
91.2
0.81M
This Work-BPTT Training
98.0
95.3
91.1
82.4
1.IM
Labeled Data % In Training 100%
Parameter Number
Model
70%
50%
30%
ConvLSNN Salaj
et al. 2020
63.1
58.7
51.3
45.4
3.0M
DECOLLE K Kaiser ct al
2020
66.9
61.9
56.2
50.6
2.0M
HATS Sirom et al. 2018
64.2
61.0
54.3
48.8
H-SNN She
2021
42.8
41.9
37.0
34.6
1.7M
This Work-STDP Training
58.1
57.8
57.2
54.6
1.4M
This Work-BPTT Training
71.2
65.4
56.0
52.5
1.7M
number of evaluations as the proposed design. Similarly, to study the contribution to performance gain from skip-layer connections, the Bayesian optimization process is used for network templates without skip-layer connections. The optimization process runs for the same number of evaluations as the proposed design. From the results shown in Table 2 it can be observed that compared to baselines, the proposed networks achieve the best accuracy for all datasets. Specifically, when ho- mogeneous network is used, the performance of STDP trained network noticeably lower than the proposed method for DVS Gesture and 101.. For BPTT training, using heterogeneous network and skip-layer connection show different level of benefit for each dataset. For sequential MNIST which has less complexity, the improvement from using heterogeneous neurons and skip- layer connections is not as significant.
5.4 COMPARISON WITH PRIOR WORKS
DVS Gesture As shown in Ttole with 100% labels, the proposed network trained with BPTT demonstrates higher accuracy than all tested networks without using the most trainable parameters. The proposed network trained with STDP has slightly lower accuracy than some baselines when 100% labels are used; for reduced-label training it outperforms all other networks.
N-Caltech101 As shown all baselines with both 70% and 100% training labels and also has less trainable parameters than most baselines. The un-supervised learning models i.e., H-SNN and the proposed network with STDP, show considerably lower performance (more than what was observed for DVS Gesture) than supervised ones when 100% labels are available; However, the proposed network with STDP shows better performance than H-SNN, and outperforms all networks when available labels below 50%.
the proposed network trained with BPTT outperforms
CONCLUSION
We develop theoretical basis understand and optimize the ability of feedforward SNN to approximate temporal sequence. We analytically show how heterogeneity and skip-layer connec- tions can improve sequence approximation with SNN, and empirically demonstrate their impact on spatiotemporal learning. It is well-known in neuroscience that, heterogeneity (De Kloet & Reul, 1987) and irregular connectivity (Eickhoff al. 2018) are intrinsic properties of human brains. Our analysis shows that incorporating such concepts within artificial SNN is beneficial for designing perforrrfeme SNN for classification of spatiotemporal data.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This material is based on work sponsored by the Army Research Office and was accomplished The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government.
under Grant Number 191INF-19-1-04
Authors would like to thank Drs. Mark Mclean, Chris Purdy, Fernando Camacho, and James Keiser from Laboratory for Physical Sciences for many helpful discussions this work.
REFERENCES
Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event-based gesture recognition system. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7243-7252, 2017.
Y. Amit and Yibi Huang. Precise capacity analysis in binary networks with multiple coding level inputs. Neural Computation, 22:660-688, 2010.
David G Barrett, Sophie Denève, and Christian K Machens. Firing rate predictions in optimal bal- anced networks. Advances in Neural Information Processing Systems, pp. 1538-1546. Citeseer, 2013.
Jonathan Binas, Giacomo Indiveri, and Michael Pfeiffer. Spiking analog vlsi neuron assemblies as constraint satisfaction problem solvers. In 2016 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 2094-2097, 2016. doi: 111111//AAA2221..65555
Johanni Brea, Walter Senn, and Jean-Pascal Pfister. Matching recall and storage in sequence learning with spiking neural networks. Journal of neuroscience, 33(23):9565-955) 9575, 2013.
Chi-Ning Chou, Kai-Min Chung, and Chi-Jen Lu. On the algorithmic power of spiking neural networks. arXiv preprint arXiv:1803.10375. 2018.
ER De Kloet and JMHM Reul. Feedback action and tonic influence of corticosteroids on brain function: concept arising from the heterogeneity of brain receptor systems. Psychoneuroen- docrinology, 12(2):83-108 105, 1987.
Simon B Eickhoff, BT Thomas Yeo, and Sarah Genon. Imaging-based parcellations of the human brain. Nature Reviews Neuroscience, 19(11)6672-666 2018
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2661-2671, 2021.
Edris Zaman Farsa, Soheila Nazari, and Morteza Gholami. Function approximation by hardware spiking neural network. Journal of Computational Electronics, 14(3):707-716, 2015.
Jacob Gardner, Matt Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John P Cunningham. Bayesian optimization with inequality constraints. In ICML, volume 2014, pp. 937-945, 2014.
Wulfram Gerstner. Time structure of the activity in neural network models. Physical review E, 51 (1):738, 1995.
Wulfram Gerstner and Werner M Kistler. Mathematical formulations of hebbian learning. Biological cybernetics, 87(5-6):0444155 2002.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
10

---

Published as a conference paper at ICLR 2022
Nicolangelo lannella and Andrew D. Back. A spiking neural network architecture for nonlin-
ear function approximation. Neural Networks, 14(6):933 939, 2001. ISSN 0893-6080. doi:
URL https:/ hts:://wwwwt /wwww www.scciencdiirett.
6
Jacques Kaiser, Hesham Mostafa, and Emre Neftci. Synaptic plasticity dynamics for deep con- tinuous local learning (decolle). Frontiers in Neuroscience, 14:424, 2020. ISSN 1662-453X. doi: 10. 10.3389/fmims. 2020000444 URL htt.ss//ww...oo...eeshhrrcrrrr.e
338 2389/fnins 2020. 00424
Mina A Khoei, Amirreza Yousefzadeh, Arash Pourtaherian, Orlando Moreira, and Jonathan Tapson. Sparnet: Sparse asynchronous neural network execution for energy efficient inference. In 2020 2nd IEEE International Conference Artificial Intelligence Circuits and Systems (AICAS), pP-
256-260. IEEE, 2020.
Seijoon Kim, Scongsik Park, Byunggook Na, Jongwan Kim, and Sungroh Yoon. Towards fast and accurate object detection in bio-inspired spiking neural networks through bayesian optimization. IEEE Access,
2020a.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficien object detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 11270-11277, 2020b.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks us-
ing backpropagation. Frontiers in Neuroscience, 10:508, 2016. ISSN 1662-453X. doi: 10. 22899//iins.2166.0000
URL fattps://wwx www.::wwwwoottiriiieem http:/ii....//atttt////.......
Enins, 2016. 00508
Wolfgang Maass. Fast sigmoidal networks via spiking neurons. Neural computation, 9(2)279-334, 1997.
Garrick Orchard, Ajinkya Jayawant, Gregory K. Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Newroscience, 9:437, 2015. ISSN 1662-453X.
doi: 10.3389/fnins.2015 500437. URL https://www/ www.::ffooontrrii...
org/article/10.1 3389/fnins 2015..0437
Maryam Parsa, J Parker Mitchell, Catherine D Schuman, Robert M Patton, Thomas E Potok, and Kaushik Roy. Bayesian-based hyperparameter optimization for spiking neuromorphic systems. In 2019 IEEE International Conference on Big Data (Big Data), pp. 4472-4477 IEEE, 2019.
Filip Ponulak and Andrzej Kasinski. Introduction to spiking neural networks: Information process- ing, learning and applications. Acta neurobiologiae experimentalis, 71(4):409-43 433, 2011.
Ali Safa, Federico Corradi, Lars Keuninckx, Ilja Ocket, André Bourdoux, Francky Catthoor, and Georges GE Gielen. Improving the accuracy of spiking neural networks for radar gesture recog- nition through preprocessing. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Darjan Salaj, Anand Subramoney, Ceca Kraišniković, Guillaume Bellec, Robert Legenstein, and Wolfgang Maass. Spike-frequency adaptation provides a long short-term memory networks of spiking neurons. bioRxiv, 2020. doi: 10. 10.1100//000005... 0815113. URL https::////. www. biorxiv.
2020. 05 11. 081513
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Xueyuan She, Saurabh Dash, Dachyun Kim, and Saibal Mukhopadhyay. A heterogeneous spiking neural network for unsupervised learning of spatiotemporal patterns. Frontiers in Neuroscience, 14:1406, 2021.
A. Sironi, M. Brambilla, N. Bourdis, X. Lagorce, and R. Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In 2018 JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1731-1740, 2018 00186.
doi: 10.1109/CVPR.2018.
11

---

Published as a conference paper at ICLR 2022
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Gopalakrishnan Srinivasan and Kaushik Roy. Restocnet: Residual stochastic binary convolutional spiking neural network for memory-efficiet neuromorphic computing. Frontiers in neuroscience, 13:189, 2019.
Hiroyuki Torikai, Atsuo Funew, and Toshimichi Saito. Digital spiking neuron and its learning for approximation various spike-trains. Neural Networks, 21(2):140-149 149, 2008. ISSN 0893-6080. doi: https://doi.org/ 10.g///////// 100 12.045. URL ht https: /www scienced: rect.
/ttp:///0000///0000////633300000 Advances in Neural Networks Re-
search: IJCNN '07.
Paul J Werbos. Backpropagation through time: what it does and how do it. Proceedings of the IEEE, 18510)1 1990.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019.
Shao-Qun Zhang, Zhao-Yo Zhang, and Zhi-Hua Zhou. Bifurcation spiking neural network. arXiv preprint arXiv:1909.083..0 2019.
12

---

Published as a conference paper at ICLR 2022
A SNN DYNAMICS
Remark For sequentially connected neuron list with m neurons all with y = and neuron delay ndd input spike at time leads the neuron list to generate output spike at time t mtnd
Remark For any imput sequence with period fan to a spiking neuron with response rate y such that if refractory period is set < tyns the neuron can exit refractory period before the next spike arrives.
For Lemma For any imput spike sequenee with period ti in range [Tiins T Tazz there exists spiking neuron n with fixed parameters Veh, Ureset, a, Rm and Tim such that by changing synaptic conduc- tance G, it is possible set the neuron response rate 7n be any positive integer.
Proof. For given input spike sequence period tim. consider the maximum possible membrane मे
< 0 and de increases with higher Hence, the maximum decay of v is reached when initial membrane potential v(t 0) - E2h and the neuron decays for period tin Tmez- The decayed membrane potential (f = mmax) can be derived by solving the differential equation
potential decay that can be reached within period of lin- From (1), when 0, du
[ for u(t) = Uth. at t = 0:
v T mar)
Uth
Truse
aE
Tume
(5)
is possible to have spiking neuron with Rm, a and Tm such that Av, defined as
Tmax
Tman Tim
Ao = v(t = Tmaz) - v(t Uthe
Tim
ae
+ a Uth
(6)
tends to zero. With this configuration, since the the highest possible decay of membrane potential is negligible, for any target 7, it possible set G such that
Uth Ureset
G
7
The proof is complete.
Note, input sequence that has no spike has tim tending to infinity, thus violating the bounded con- straint mentioned in Lemma
B PROOF OF THEOREM
Theorem For any input and target output spike sequence pair with periods (tens tout) E [Tmin T Tmasl [T'in T ma there exists minimal-layer-size network with skip-layer connections that has memory pathway with output spike period function P(t..) such that P((in) fout < c.
Proof. For any given Cout)) first consider the condition where tin > fout- is possible to construct a minimal-layer-sizze network N connecting m spiking neurons with neuron response rate 7 = 1 sequentially, denoted as m-tuple of neurons {mi, n2,.... Nmm.. Since any configuration of skip-layer connection with source layer and target layer pair (I,, 42), such that I, E [1, m 2). and 4 [L. + 2, ml), can be added, it is possible to add a (m - 2)-tuple of skip-layer connections
Sa = {(i, m)
(1,2,3 ..... m. -2]
(8)
Denote the synaptic conductance for all the skip-layer connections as a (m 2)-tuple
Sq {G+l
G
Ggl
G
(9)
For any tout < tin it is possible to find k-tuple of synaptic conductance
G i lout and i
2
Sg (G Gal, Gal
(10)
End
13

---

Published as a conference paper at ICLR 2022
Set synaptic conductance in Sqat \ SS/. to O. Then set the conductance of synapse connecting 7m-11 and 7m to In such way, The output spikes from network N have period
tout Ind
P(tsn) =
End
(11)
For given it is possible choose lnd such that And < 2e, therefore satisfying (P((in) boue < m can be chosen as
maa
Timnn
m78
(12)
Ind
or equivalently:
T maa
T min
m 2e
(13)
Since Fmar-Tmae is finite, is finite.
For tin < touts using as described above, is possible to achieve output spike with period within of any period in (0, t. For given tout, assume the configuration in neuron list N has output spike interval i' such that kt'mn = taut, where & is a positive integer. From Lemmal] [ it is possible set G for neuron Rm++1 such that its neuron response delay satisfies Tu k for input spike period t A new network, denoted as N', can be formed by connecting 7m to the output of
N. N' has output spike with period P(tm) kt' bout. Hence, to reach the given e, it requires neuron list N to have output spike interval tine such that
Pint L'neel < To.
(14)
Since l is finite, 14
be achieved.
For tin >= touts it possible to configure network N' such that tine satisfies |tant - toutl < e, and the value of 7/m+1 set to hence P((tn)) toutl can be achieved. The proof is complete.
C PROOF LEMMA
Lemma 2 With no skip-layer connection, there does exist minimal-layer-siin network that has output spike period function P(t:n) such that for any input and target output spike sequence pair with periods (lin, loux) E [Tminy Tmaz] [Tmin; Tmar]. P(ti.) Loutl E.
Proof. A minimal-layer-size network N with m layers can be denoted as a m-tuple of neurons (n1,72, n2,....,mm } connected sequentially. Since no skip-layer connection exists, there is only one distinct memory pathway that contains all neurons {π1, R2, }.
Denote the set of neuron response rate corresponding to each neuron in N as
r {71, 12, 72, 7m
(15)
For given input sequence with lin denote the timing of the first spike din consider the output spike sequence for network with
7 = 1
(16)
The first output spike from N has timing jout t mtnds and the second output spike has timing Jout tin tim + mt It can be easily derived that the period of the output spike sequence is 2
P(In)) lin
(17)
14

---

Published as a conference paper at ICLR 2022
Also consider the output spike sequence for network with
7j for any j (1,2,3,4..... m) and "/ i (f1,2,3,4 m) (5)
(18)
Following the same process, the period of the output spike sequence is
P(li)) = 2/
(19)
Since the smallest increase to any 76 is by I, there is no set of values for such that the network output spike sequence has period P(fmn) satisfying Lin P(tin) 26 Since within the range (tin, ), there exists values of tout such that [P(tin) touel < does not hold. The proof is complete.
D MEMORY PATHWAYS IN SNN
In this section we analyze the increase to the least upper bound of the number of memory pathways network by using heterogeneous networks and skip-layer connections.
Lemma A spiking neuron has cutoff period We Tm In Uneaet -0
above which input spike
Breeet a Rum
sequence cannot cause the spiking neuron to spike.
Tm
Proof. Consider 2 since membrane potential increases at time e and decays otherwise, solving for I == L and the equation can be expanded:
Um (t') Uresct im + aa( ett)
Rm d-st
Ge
+
Run
Ge + ... +
Run
G
(20)
Tm
Tm
Tm
For input with frequency f. titl consecutive 4 provides:
At
y.
subtracting membrane potential values at two
(+]
1611 Li
Rm Tym
Um Uy (E++1) Um (t')
Bresetle e
a(e
e
Ge
(21)
setting time of the first input spike t to zero leads
1' Ac Acm = e ((e 1) - a) +
Rm
At fm
Ge
(22)
Tm
As : O. and the term ((e (()rae Arm does change with time, em either strictly increasing, staying the same or decreasing with higher t'. This indicates that, when Aum < 0 the post-synaptic neuron can never spike regardless of how many pre-synaptic spikes receives. Arm S when input spike period 4in satisfies
A/
Rm Ge does not depend on t`, the polarity of Tm
Ureset
ti 2 Ti In
(23)
+ Rm G Fm
Ureset
Therefore, the cutoff period of neuron is
Wr Tm In(
Gresct -
(24)
Rm G Tm
Ureset
The proof is complete.
15

---

Published as a conference paper at ICLR 2022
In the following proof, consider cutoff frequency, fo 1 of spiking neurons. From (4 we can be easily derived that fe can be configured to any positive real number by changing the neuron parameters.
Lemma 4 For mMND network with layers and {Y1 Ag ... number of different neuron dynamics in each layer, the least upper bound of the number of memory pathways is I A,
Proof. Denote the set of neurons in layer 7 with distinct neuron dynamics
Sy = -nni, n22 72. ..., n'de
(25)
Since the network is mMND, |S Ay. Denote the of cutoff frequency corresponding each neuron in S as
F. = (fe fe fo"
(26)
Since neurons in S can have different neuron parameters, from Lemma it is possible to set the parameters such that all entries of Ft. are distinet. Hence, there exists permutation such that
Sutto <
Se
(27)
Denote the input spike frequency to layer I as fin. neuron nł is part of a valid memory pathway in the network
fo S f:n
(28)
Therefore, for input spike frequency fl, 2 fe (00) all neurons in St can be part of a valid memory
pathway of the network. For input spike frequency such that fc fe < fin be part valid memory pathway the network.
fe (aactt neurons can
For any input to the network Jin E [Fmmn; Fa denote the number of ways different neurons in S can be part valid memory pathways as kg. The total number of distinct memory pathways IC in the network is the product of ki of all layers:
m
ky
(29)
Since for all layers, 0 < hi S At.
T
<
II X
(30)
Consider that, for any layer E 11, m.., the input fi is bounded by [P PPhsss Ft
Fmazl: kr
A
can be achieved by setting fo (-()) and fo po0D) such that:
fc 20
pl min
and fo
F. Fhaa
(31)
Hence the bound is tight for (30) The proof is complete.
Lemma For mMND network with m layers and {}1 ... different neuron dynamics in
each layer and a skip-layer connection made between layer I, and I, s.t. a, E (1.2,..m) and (b a) 1, the least upper bound of the number of memory pathways I X; + ([I X; -I X1)
16

---

Published as a conference paper at ICLR 2022
Proof. Denote the mMND network with skip-layer connection between layer la and layer io as P. Denote the set of all neurons in P as
Sp = {n): m22, ng. ... ni, n mg:....
(32)
where n. is neuron in layer 4, and nē neuron in layer l2, etc. The activation state ong of neuron can be denoted with binary values 0 and 1 with On? representing that ne receives input frequency that is above its cutoff frequency f²
The set of all possible neuron activation states O in P that generates non-zero network output feature vector can be partitioned into two subsets denoted as O and Og.
Set Oa contains all states where the input frequency fth to any layer la such that a < k < b satisfies
f < fe ViE f1,2,3.... }x}
(33)
Set Og contains all the remaining neuron activation states in O, where all layers receive input fre- quency higher than cutoff frequency of at least one neuron in that layer.
For all the states in OA. no spike signal is sent from layer 1 to layer b. since least one layer between la and I, generates no output. Hence, output from P is not affected if connections between layer and 1,+1, such that i E {a, a + 1.... b- 1). are removed. Network P is therefore equivalent to network p' that has layers (4,, 12..., la I., 4411,-.,44 Im connected sequentially.
According to Lemma 4 it can be derived that the least upper bound of the number of distinct memory pathways in P is II A 11 A.
Hence, for all states in set O., the least upper bound of the number of distinet memory pathways in P is also I[ X; layer of the skip-layer connection is already accounted for when considering layer la. the least upper bound of the number of distinct memory pathways is the same as network P' that has no skip-layer connection, which is I X; according to Lemma
II A₁- For all states in set Og. since the activation of neurons in the source
For the set of memory pathways from states in OA, denoted as MA, and the set of memory pathways from states in OB, denoted as MB. the number of memory pathways of network is |M MB).
From Lemma 5 the bound is tight for MAl S I d II= A and for [M S I A, It also satisfies that MT nM since all elements in MA have (m (b a - 1)) layers and all elements in MB have m layers. Hence the bound is tight for
m
MA U M
II
A
II A)
(34)
b=b
The proof is complete.
E TIME-VARYING FUNCTION APPROXIMATION
time-varying function f(t) can be approximated using piece-wise constant function, such as illus- trated in Figure 4 It is therefore possible to approximate the time-varying function with feedfor- ward SNN by approximating each of the constant function with a memory pathway. In this section, we test the approximation capability of feedforward SNN for time-varying functions using this prin- ciple. The target functions to approximate have the form of:
x
f(x)
(35)
Here, x is variable; n and m are function parameters. For discrete -ime simulation of the network, approximate the target function with
Lin and ((x) bout
(36)
17

---

Published as a conference paper at ICLR 2022
Memory Pathway
60
Memory Pathway
50
Set of memory
pathways feedforward
f(t)
30
Piece wine approximation
20
10
6
8
10
12
14
16
18
20
Figure 4: Using set of memory pathways to map a piece-wise constant function for approximating une-varying function.
where fin is the input spike period and fout is the output spike period. We test approximation performance of small-scale feedforward SNN with fully-connected layers, skip-layer connec- tions {(2, 5), (3, 5)} and 4 neuron dynamics. The network is trained with BPTT to minimize mean squared error (MSE) loss between the spike period of network output lous and target spike period bout-
To evaluate the network's approximation performance, we construct 6 networks with the same struc- ture as discussed above, and change each to have different trainable parameter numbers by scaling the layer size. For baseline comparison, networks with layers, no skip-layer connection and using homogeneous neurons, configured to have the same trainable parameter numbers as the proposed networks, are tested. All networks are trained with BPTT. The loss function measures the difference between the network output spike trains and the target spike trains with MSE. Two sets of function parameters: (yn 1, n = 3.3) and (m = 2,
2.1) are tested for the target functions (xx) on domain [3, 10]. The resulting MSE losses for different network scales are shown at the bottom of Figure 5 can observed that the proposed networks can approximate target functions with less eπror than the bascline networks at all network scales. The smallest tested networks have rela- tively high losses while performance increases quickly with more trainable parameters. The rate of performance improvement decreases when trainable parameter number is above 4000.
To understand the impact of the target function parameters to approximation performance of SNN, test baseline and the proposed network with 4167 trainable parameters for different pairs of
0.2
aet, (x) Baseline network
riag
Proposed Network
[4] Baseliee
0.15
[8] Proposed network
MSE Loss
0.1
0.05
1000
2000
3000
4000
5000
6000
7000
8000
Parameter Number
Figure 5: MSE loss vs. number of trainable parameters for the function approximation experiments.
18

---

Published as a conference paper at ICLR 2022
Loss (log)
-4.0
-6.5
-1.2
0,8
-44
3.5
3.7
0.8
1.2
2.0
211
2.3
2.7
2.9
(a)
Baseline Network
Proposed Network
1.8c-1
3.4e:3
-.6
-16
-1.2
0.4
-0.4
0.4
0.4
0.8
0.0
16
16
0.7e-1
18e-3
3.7
8
(b)
Figure 6: (a) MSE loss (log) of baseline network (top) and the proposed network (bottom) for approximating functions with different parameters m and n. (b) Heat plots of MSE loss for approx- imating functions with different parameters and
function parameters m and n. The resulting MSE loss is shown with a heat map in Figure 6 can be observed that for all m and value pairs, the proposed network achieves lower loss than the baseline network. Another observation that, there is no clear correlation between the value of m and approximation error. On the other hand, for higher values of n, the approximation error generally increases for both baseline and the proposed network.
F NETWORK SPIKING ACTIVITY
To investigate neuron spiking activity in the proposed network, for the function approximation task as described Appendix E with parameters (m 1, 2.3) and randomly selected approxi- mation point f(r = 19), we plot the timing of neuron spikes at the last layer over training epochs in Figure 7 (a). can be observed that the network initially generates spikes at widely distributed tim- ings. After the first 50 training epochs spike timing starts to converge and remains stable at around 200 epochs. The final output spike period is l'o = 48, which matches the target output period.
Next, we investigate spiking activity of the BPTT trained network described in Section 5.4 for N-Caltech101 classification task. Three different test data points are presented the network, and the spikes from neurons in the first depth of layer 8 and neurons in the final layer, recorded and shown in Figure (b). Here, (i), (iii) and (v) are from layer 8; (ii), (iv) and (vi) are from the final layer. (i) and (ii) from the observation of a data point with class label "5"; (iii) and (iv) from the observation of another data point also with class label "5"; (fii) and (iv) are from the observation of data point with class label "1".
can observed that neurons in layer exhibit similar activity in (i) and (iii) as the network is presented with data points from the same class. (ii) and (iv) show that most spiking activities
19

---

Published as a conference paper at ICLR 2022
300
250
200
150
0
50
100
150
200
250
300
Epoch
(a)
8000
(ii)
Neuron Index
8000
Neuron Index
40
4000
20
2000
20
40
60
80
100
120
20
40
80
80
100
120
(iii)
8000
(iv)
60
Neuron Index
6000
Neuron Index
4000
20
2000
20
40
60
80
100
120
20
40
60
80
100
120
8000
(vi)
60
60000000
0000 000000 40000
Neuron Index
6000
Neuron Index
40
4000
20
2000
800
20
40
80
100
120
20
40
60
80
100
120
(b)
Figure 7: (a) Raster plot for function approximation experiment; is the simulation time step. (b) Raster plots for the proposed SNN trained for N-Caltech101 with BPTT; is the simulation time step: (i), (iii) and (v) are from layer 8; (ii), (iv) and (vi) are from the final layer.
20

---

In the case of Published as a conference paper at ICLR 2022
from the neuron with index 5, indicating correct prediction for the two data points. When data point from a different class is presented, spiking activity in (v) shows different patterns than those in (i) and (iii). In the final layer, neuron with index generates the most spikes, leading to correct prediction. However, for this data point, neuron with index 58 is also generating a considerable number of spikes, indicating that the network is more likely to mis-classify this data point, compared to the other two tested data points.
G IMPLEMENTATION OF HETEROGENEOUS CONV-SNN
Multi-neuron-dynant (MND) networks with convolutional layers can be considered a scaled-up version of the mMND network, where each neuron dynamic now contains a matrix of neurons that receive input features from different spatial locations. To implement heterogeneous, Conv-SNN, first consider a regular Conv- SNN layer with no heterogeneity. The spiking neuron matrix has dimension (W, H, D), where D is the depth of the layer. The convolution filter has dimension {C, wr h, D} where the number of input channels and D is the number of output channels.
Based on this, heterogeneous layer 1 can be constructed, by concatenating heterogeneous neu- ron matrices with the same W and H along layer depth. The resulting spiking neuron matrix has dimension {W, H, AD), where À is the number of neuron dynamics. Hence, layer depth i E (k + k + 2, k 3.... + D} have the same neuron parameters, where S k: S À 1 an integer representing the index of neuron dynamies. The convolution filter has the same dimension as the regular Conv-SNN layer: {C, w, h, D). which shared by layer depth with different values. During forward pass of this layer, convolution operation is applied to generate trix with dimension [W, H, D}. For each value of k. neurons in depth (k+ 1, k+ 2, k + 3,.., k + +d) are simulated based on the neuron parameters for such neuron dynamic index & using the signal ma- trix as input. For the next convolutional layer l' receiving input from layer I, the filter dimension is {AD, w W, D'3 with AD input channels.
input signal ma-
H DETAILS ON THE BAYESIAN OPTIMIZATION PROCESS
During the first stage of the dual-search-space optimization process, the parameters optimize include:
Naager, Lstart, Lend, Nakipp Ndgramic. all of which are positive integers. Specifically, Niager is the number of convolutional layers. For skip-layer connection, there are three configura- tion parameters to optimize: starting layer Lstarts which is the source layer of the first skip-layer connection; ending layer Trnns which is the target layer of the last skip-layer connection; skip-layer connection number N Nakipp. which defines how many connections to implement. The source layer of the Nakip skip-layer connections are placed evenly between Lstart and Lend, each with skip length equal
Lend Latart Nakip
Lend
Lstart
Lend Listart
#
Nakip
Nakip
the value of Lend is reduced to the maximum value that satisfies
Lend
Letart
Lend
Lestart
Nakip
Nakip
For heterogeneity, the number of different dynamic Néynamic in all layers is optimized jointly. The constraints for the parameters are defined as:
Ntayer [4, 15)
21

---

Published as a conference paper at ICLR 2022
73
Va Validation Accuracy (%))
8
11
70 Land
Lyoo
Figure 8: Visualization of the learned distribution from Bayesian optimization: N-Caltech101 vali- dation accuracy vs. skip-layer configurations.
Listart < (Ntayer - (Lstart + 1) < Lend Niayer
0 S Nekip I (Lend - Latars)/2
and
N Ndynamic E [0]
The manually configured neuron parameters are, Tm 100 and R,, 300 for all neuron dynamics, and a [-30, 55 is distributed evenly for each neuron dynamic. Figure 8 shows visualization of the learned distribution from the Bayesian optimization process. Here, mapping from the search space of skip-layer connection configurations to the validation accuracy of I-Caltech is shown. The highest validation accuracy is reached at Latart and Lend 11.
Due to the exponential increase of search space with the number of neuron dynamics in each layer, is highly inefficient to search for every neuron parameters of each dynamic. In the second stage of the optimization process, choose to apply Bayesian optimization for the parameter of each neuron dynamic separately, while Tm and R., are optimized jointly with the same values shared by all neuron dynamics. values are taken the precision of 10 and Tm and R,, values taken to
the precision of 1 The constraints are a E [-30, 5]. Tm E [50, 200) and Rm E [200, 400). The value of bnd for all networks is set The parameters of each optimized network shown in Table T Note, the skip-layer connections are listed as source and target layer pairs.
22


Reviews:
[{"review_id": "Z7b1-u4Fyg2", "reviewer": "Reviewer_oV8W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major Points\n\nThe paper is well written, the motivation of the authors is clear, and the framework presented is easy to understand. The theory part is written in a clear mathematical manner, with a theorem-proof structure, which I really like. But I have one main point of criticism that seems very important to me, while the entire theory section provides insights and interesting ideas the provided results in sec. 5 Experiments does not hold up to this. Certainly, a benchmark comparison on common data sets with other methods is very important, but it’s not adequate for the hypotheses presented in the theory part. The main argument of this paper (at least for me) is that a feedforward SNN with many memory pathways can approximate a mapping between a temporal sequence of spike trains with time-varying unknown frequencies to a pre-defined output spike trains with known frequencies, where the number of memory pathways can be increased by optimizing the skipping connections and heterogeneities in the network. This connection to the universal approximation theorem should be more supported by quantitative experiments. My suggestion to investigate the computational power of the optimised SNN would be to see how big the memory effect is as well as to exam the complexity of the mapping the SNN can approximate. One way to address this would be to train the SNN to learn a mapping from $f(x)=x $ to $g(x)=(x^n)/f(x-m)$ for different combinations of $m=1,\\dots, M$ and $n=1,\\dots, N$, e.g. $M=N=10$.  The resulting reconstruction errors could then be considered as a function of m and n as well as the number of network parameters. If all this should provide very good reconstructions, the mapping function could be chosen in a more complicated design.\nA next point: On page 3 “[…] as for MLP-SNN and Conv-SNN network the analysis can be extended according to the specific layer dimensions.” At this point more details or explanations would be desirable I only understood this part after I had a look at the source code.\n\nPage 7: Please provide a formal definition for the objective function, which metric is optimised?\n\nPage 7: “[…] first optimized with fixed, manually selected neuron parameters.” How were these parameters elected or on what basis, I assume random values should not work?\n\nPage 8 Fig 3.: I would suggest using a different symbol for the stage 1 and stage 2 optimisation. Further, I agree with the authors that the dual search converges faster, but it seems that the single search achieves a marginally better validation error. In order to better assess the advantage of faster convergence, how long does an iteration step take and what hardware was used?\n\nMinor Points\nPage 5: “Since any continuous bounded function on a compact interval […]”. For the sake of completeness, I would suggest adding the definition for a compact interval. How were these parameter selected?\n\nPage 5 and other use $\\left( … \\right)$, e.g. Lemma 3\n\nPage 7: This is really optional: Would it be possible to show some projections of the search space from the architecture search space? Maybe in the appendix or even the code-section.\n\nPage 8: It’s a matter of taste: I would suggest that the y axis in figure 3 has the same range in both plots. then the plots can be compared directly with each other without having to pay attention to the axis section.\n", "summary_of_the_paper": "In this publication, the authors present an interesting framework for approximation the mapping between sequences by feedforward spiking neural networks (SNN), providing insights on the computational capabilities of feedforward SNNs for the approximation of the mapping of input to output spike trains. And how these are influenced by hyperparameters such as the network architecture, heterogenic properties of neurons and skipping connections. The authors also provide an ansatz how these hyperparameters can be optimised in a two-steps Bayesian optimisation fashion. The performance of the approach is demonstrated by several spatiotemporal classification tasks, on datasets like DVS Gesture, N-Caltech 101 and sequential MNIST. ", "summary_of_the_review": "The paper is well written, the motivation of the authors is clear, and the framework presented is easy to understand. The theory part is written in a clear mathematical manner, with a theorem-proof structure, which I really like. The theoretical results provide new insights and should give new impulses to other scientists in this field as well as in related fields, e.g.: for improved applications as well as building theories.\n\nMy main criticism is that the good theoretical results are not sufficiently well supported by numerical experiments, which could further consolidate the good theory results. I hope to provide a possible motivation for supplementing the results in my review. However, these would have to be provided in order to recommend a publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Oah-2eHLUAQ", "reviewer": "Reviewer_himw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Theoretical understanding of SNNs in terms of approximation, temporal memorability, optimization, and generalization is a fundamental and challenging problem for SNNs. This paper takes a step in this direction. The motivation is interesting and significant.\n\nDespite the interest of the overall goal of the research, this paper suffers from several limitations,  which make it improper for acceptance. There are two main reasons for rejection, (1) the writing of this paper, especially theoretical conclusions,  needs to be improved for proofreading; (2) the claimed contributions cannot be confirmed or supported by the theory and experiments.\n\nIn detail,\n1. Definitions 1-5 are too redundant. It's not necessary to state these informal definitions in a formal environment. Please be clear and simple.\n2. What's the difference between lemmas and theorems presented by this paper? Is it just because the conclusions of the theorems are more important or more dazzling than those of the lemmas?\n3. Can the authors provide a formal description and proof for their theoretical results? The current can only be called a heuristic description.\n4. What's G in Eq. (2)? Is it trainable in practice? It works just like a normalizer, and thus, can be replaced by $R_m / \\tau_m e^{-1/ \\tau_m}$? In fact, the legality of rate coding has already been discussed.\n5. Besides, there have been great efforts on the theoretical understanding of SNNs, such as [1-4].\n6. Notice that lemmas 4 and 5 only investigate the upper bound, not the least upper bound. So one cannot conclude that using heterogeneous and skip connections can improve the sequence approximation of feedforward SNNs.\n\nThe experiment is intriguing and could lead to something new. However, I am unable to follow the author's suggestions, such as how to build an experiment based on past beliefs. Furthermore, don't the optimization techniques discussed in Section 4 already exist? Both BPTT and STDP are trivial for skip connection and feedforward architecture [5]. In order to show the effectiveness of the proposed network, it is not sufficient to use only accuracy as the evaluation indicator. It's better to display the spot rasters of the memory and learner modules, just like what shows in [2] and [6].\n\nI felt a sense of fragmentation when reading this paper. It is very much like a suture monster. I would suggest that the authors use rigorous proof to re-write clearly what the conclusions of the theory are? Where is the proof of innovation? And explain clearly what is the innovation of the experimental model or algorithm? Why is it effective?\n\n[1] On the Algorithmic Power of Spiking Neural Networks. 2019. \n[2] Bifurcation Spiking Neural Network. 2019.\n[3] Firing rate predictions in optimal balanced networks. 2013. \n[4] Spiking Analog VLSI Neuron Assemblies as Constraint Satisfaction Problem Solvers.2015.\n[5] Error-Backpropagation in Temporally Encoded Networks of Spiking Neurons. 2000.\n[6] Slayer: Spike layer error reassignment in time. 2018.", "summary_of_the_paper": "This paper presents some theoretical understanding of sequence approximation using feedforward SNNs. The main conclusions are two folds: (1) a feedforward SNN with one neuron per layer and skip-layer connections can approximate the mapping rate-coding function; (2) a feedforward SNN constructed by heterogeneous neurons with varying dynamics and skip-connections can improve sequence approximation. Besides, the authors provide the DSBO to optimize the architecture and parameters of their proposed SNNs.\n\n", "summary_of_the_review": "In summary, I find the social impact of this work, but it still needs to be refined carefully. If there were an option for 'Weak Reject', I would select that. Since there is no such option, I recommend rejection given the many suggestions.\n\n______________\nAfter reading the rebuttal and other reviewers' comments, I consider arising my score to 6, although there are still many inappropriate points in the current version.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FqJQ7RnPkCy", "reviewer": "Reviewer_EDAS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* very good accuracy\n\nCons:\n* many doubts need to be cleared about the theory\n* the connection between the theory and the experiments is unclear\n\nThe theory is unclear:\n* Does it only apply to sequences with constant inter-spike intervals (ISI), as Fig 1a suggests? The fact that they focus on the spike counts (gamma) also suggests that.\nIf true, this is a strong loss of generality, which should be properly acknowledged. Also, I don't understand how the theory can help with DVS datasets, in which the ISI are highly variable.\nIf false, then Fig 1 should show variable ISIs, and the authors should justify why they can summarize a spike train by its spike count.\n* Lemma 1 seems wrong: what if no input spike? Then changing G has no effect on the response...\n* Does the theory work in continuous or discrete-time? The equations suggest continuous time. But a timestep is mentioned on page 4. Plus BPTT works in discrete-time.\n\nThe connection between the theory and the experiments is not clear to me. Does the theory only suggests heterogeneous neurons and skip connections, but then the training is done with normal BPTT (or STDP)?\nDoes the theory also justify the use of learned and transferred synapses? BTW, these concepts are obscure to me. Isn't it equivalent to having a set of weights that is shared between neurons (like in a convolutional layer)?\n\n\nMINOR POINTS:\n* Eq 1: you should say that a is the resting potential\n* Eq 2: don't use t for the integration variable\n* Table 1 should specify the number of timesteps\n* Table 3 should include: https://arxiv.org/abs/2007.05785\n", "summary_of_the_paper": "The authors introduce a theory on how to learn arbitrary input spike train - output spike train mappings, using a feedforward SNN with heterogeneous neurons and skip connections.\nThen this theory is used to train a deep SNN using either BPTT or STDP. The SNN is then evaluated on IBM DVS Gesture, N-Caltech101. The accuracy that they get with BPTT is beyond the SOTA.", "summary_of_the_review": "Potentially interesting given the accuracy, but unclear", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide a theory for training feed-forward spiking neural networks (SNNs) on input-to-output spike train mappings. They utilise for this heterogenous neurone and skip connections.

The resulting method is tested on DVS Gesture, N-Caltech 101 and sequential MNIST.
It achieved very good performance. The reviewers agreed that the results are interesting and significant.

In the initial reviews, the reviewers pointed out some doubts about the theory and clarity of writing.
These doubts and objections were addressed in the revision and the reviewers were quite satisfied with that.

In conclusion, the manuscript presents interesting results for SNNs with a solid theory and very good experimental results. All reviewers vote for acceptance.
Decision: Accept (Poster)