Sequence Approximation using Feedforward Spiking Neural Network for Spatiotemporal Learning: Theory and Optimization Methods

1 INTRODUCTION . Spiking neural network ( SNN ) ( Ponulak & Kasinski , 2011 ) uses biologically inspired neurons and synaptic connections trainable with either biological learning rules such as spike-timingdependent plasticity ( STDP ) ( Gerstner & Kistler , 2002 ) or statistical training algorithms such as backpropagation-through-time ( BPTT ) ( Werbos , 1990 ) . The SNNs with simple leaky integrate-andfire ( LIF ) neurons and supervised training have shown classification performance similar to deep neural networks ( DNN ) while being energy efficient ( Kim et al. , 2020b ; Wu et al. , 2019 ; Srinivasan & Roy , 2019 ) . One of SNN ’ s main difference from DNN is that the neurons are dynamical systems with internal states evolving over time , making it possible for SNN to learn temporal patterns without recurrent connections . Empirical results on feedforward-only SNN models show good performance for spatiotemporal data classification , using either supervised training ( Lee et al. , 2016 ; Kaiser et al. , 2020 ; Khoei et al. , 2020 ) , or unsupervised learning ( She et al. , 2021 ) . However , while empirical results are promising , a lack of theoretical understanding of sequence approximation using SNN makes it challenging to optimize performance on complex spatiotemporal datasets . In this work , we develop a theoretical framework for analyzing and improving sequence approximation using feedforward SNN . We view a feedforward connections of spiking neurons as a spike propagation path , hereafter referred to as a memory pathways ( She et al. , 2021 ) , that maps an input spike train with an arbitrary frequency to an output spike train with a target frequency . Consequently , we argue that an SNN with many memory pathways can approximate a temporal sequence of spike trains with time-varying unknown frequencies using a series of pre-defined output spike trains with known frequencies . Our theoretical framework aims to first establish SNN ’ s ability to map frequencies of input/output spike trains within arbitrarily small error ; and next , derive the basic principles for adapting neuron dynamics and SNN architecture to improve sequence approximation . The theoretical derivations are then investigated with experimental studies on feedforward SNN for spatiotemporal classifications . We adopt the basic design principles for improving sequence approx- imation to optimize SNN architectures and study whether these networks can be trained to improve performance for spatiotemporal classification tasks . The key contributions of this work are : • We prove that any spike-sequence-to-spike-sequence mapping functions on a compact domain can be approximated by feedforward SNN with one neuron per layer using skip-layer connections , which can not be achieved if no skip-layer connection is used . • We prove that using heterogeneous neurons having different dynamics and skip-layer connection increases the number of memory pathways a feedforward SNN can achieve and hence , improves SNN ’ s capability to represent arbitrary sequences . • We develop complex SNN architectures using the preceding theoretical observations and experimentally demonstrate that they can trained with supervised BPTT and unsupervised STDP for classification on spatiotemporal data . • We design a dual-search-space option for Bayesian optimization process to sequentially optimize network architectures and neuron dynamics of a feedforward SNN considering heterogeneity and skip-layer connection to improve learning and classification of spatiotemporal patterns . We experimentally demonstrate that our network design principles coupled with the dual-searchspace Bayesian optimization improve classification performance on DVS Gesture ( Amir et al. , 2017 ) , N-caltech ( Orchard et al. , 2015 ) , and sequential MNIST . Results show that the design principles derived using our theoretical framework for sequence approximation can improve spatiotemporal classification performance of SNN . 2 RELATED WORK . Prior theoretical approaches to analyze SNN often focus on the storage and retrieval of precise spike patterns ( Amit & Huang , 2010 ; Brea et al. , 2013 ) . There are also works that consider SNN for solving optimization problems ( Chou et al. , 2018 ; Binas et al. , 2016 ) and works that analyze the dynamics of SNN ( Zhang et al. , 2019 ; Barrett et al. , 2013 ) . Those are different topics from the approximation of spike-sequence-to-spike-sequence mappings functions . SNN that incorporates excitatory and inhibitory signal is shown for its ability to emulate sigmoidal networks ( Maass , 1997 ) and is theoretically capable of universal function approximation . Feedforward SNN with specially designed spiking neuron models ( Iannella & Back , 2001 ; Torikai et al. , 2008 ) have been demonstrated for function approximation , while for networks using LIF neurons , function approximation has been shown with only empirical results ( Farsa et al. , 2015 ) . On the other hand , the existing works that has developed efficient training process for SNN and demonstrated classification performance comparable to deep learning models , have mostly used simpler and generic LIF neuron models ( Lee et al. , 2016 ; Kaiser et al. , 2020 ; Kim et al. , 2020b ; Wu et al. , 2019 ; Sengupta et al. , 2019 ; Safa et al. , 2021 ; Han et al. , 2020 ) . Therefore , this paper develops the theoretical basis for function approximation using feedforward SNN with LIF neurons , and studies applications of the developed theoretical constructs in improving SNN-based spatiotemporal pattern classification . The effectiveness of heterogeneous neurons ( She et al. , 2021 ) and skip-layer connections ( Srinivasan & Roy , 2019 ; Sengupta et al. , 2019 ) in SNN has been empirically studied in the past . However , no theoretical approach has been presented to understand why such methods improve learning of spike sequences , and how to optimize a SNN ’ s architecture and parameters to effectively exploit these design constructs . It is possible to search for the optimal SNN configurations through optimization algorithms , but the large number of hyper-parameters for spiking neurons and network structure creates a high-dimensional search space that is long and difficult to solve . Bayesian optimization ( Snoek et al. , 2012 ) uses collected data points to make decisions on the next test point that could provide improvement , thus accelerates the optimization process . Prior works ( Parsa et al. , 2019 ; Kim et al. , 2020a ) have shown that SNN performance can also be effectively improved with Bayesian optimization . While those works consider a single or a few neuron parameters , the dualsearch-space Bayesian optimization proposed in this work optimizes both network architecture and neuron parameters efficiently by separating the discrete search spaces from the continuous search spaces . t 3 APPROXIMATION THEORY OF FEEDFORWARD SNN . 3.1 DEFINITIONS AND NOTATIONS . Definition 1 Neuron Response Rate γ For a spiking neuron n with membrane potential at vreset and input spike sequence with period tin , γ is the number of input spike n needs to reach vth . Definition 2 Memory Pathways For a feedforward SNN with m layers , a memory pathway is defined as a spike propagation path from input to the output layer . Two memory pathways are considered distinct if the set of neurons contained in them is different . Definition 3 Minimal Multi-neuron-dynamic ( mMND ) Network A densely connected network in which each layer has an arbitrary number of neurons that have different neuron parameters . All synapses from one pre-synaptic neuron have the same synaptic conductance . Notations Neuron Delay tnd is the time for a spike from pre-synaptic neuron to arrive at its postsynaptic neurons , as shown in Figure 1 ( a ) . For a feedforward SNN with m layers , a skip-layer connection can be defined with source layer and target layer pair ( ls , lt ) . The output feature map from source layer is concatenated to the original input feature map of the target layer . For the analysis of spike sequence in temporal space , the notation of Tmax and Tmin are defined as positive real numbers such that Tmax > Tmin . ϵ > 0 is the error of approximation . Figure 1 ( a ) shows two memory pathways receiving an input spike sequence with time-varying periods . As the neurons have different dynamics , the two memory pathways have different response to the input spike sequence . An example of mMND network with m layers and n neuron dynamics is shown in Figure 1 ( b ) . SNN with multilayer perceptron ( MLP ) structure can be considered a scaled-up mMND network with multiple neurons for each dynamic . A network with convolutional structure can be considered a scaled-up mMND network with duplicated connections in each layer . We analyze the correlation of network capacity and structure based on mMND networks . The design of neuron heterogeneity can also be implemented in MLP-SNN and Conv-SNN as described in Section 4 . The analysis for network capacity can be extended to those networks according to their specific layer dimensions . 3.2 MODELING OF SPIKING NEURON . SNN consists of spiking neurons connected with synapses . The spiking neuron model studied in this work is leaky integrate-and-fire ( LIF ) as defined by the following equations : τm dv dt = a+RmI − v ; v = vreset , if v > vthreshold ( 1 ) Rm is membrane resistance , τm = RmCm is time constant and Cm is membrane capacitance . a is the resting potential . I is the sum of current from all input synapses that connect to the neuron . A spike is generated when membrane potential v cross threshold and the neuron enters refractory period r , during which the neuron maintains its membrane potential at vreset . The time it take for a pre-synaptic neuron to send a spike to its post-synaptic neurons is tnd . Neuron response rate γ is a property of a spiking neuron ’ s response to certain input spike sequence . We show how the value of γ can be evaluated below . Remark For any input spike sequence , individual spike can be described with Dirac delta function δ ( t− ti ) where ti is the time of the i-th input spike . For the membrane potential of a spiking neuron receiving the input before reaching spiking threshold , with initial state at t = 0 with v = vreset , solving the differential equation ( 1 ) leads to : v ( t ) = vresete − tτm + a ( 1− e− t τm ) + Rm τm e− t τm ∑ i G ∫ t 0 δ ( t− ti ) e t τm dt ( 2 ) Here , G is the conductance of input synapse connected to the neuron , which is trainable . From ( 2 ) , there exists a value of u such that vm ( tu−1 ) < vthreshold and vm ( tu ) > = vthreshold . By evaluating ( 2 ) for u given neuron parameters and input spike sequence , the neuron response rate γ can be found . 3.3 APPROXIMATION THEOREM OF FEEDFORWARD SNN . To develop the approximation theorem for feedforward SNN , we first aim to understand the range of neuron response rate that can be achieved . We show with Lemma 1 that for any input spike sequence with periods in a closed interval , it is possible to set the neuron response rate γ to any positive integer . Based on this property , we show with Theorem 1 that by connecting a list of spiking neurons with certain γ sequentially and inserting skip-layer connections , approximation of spikesequence mapping functions can be achieved . To understand whether this capability of feedforward SNN relies on skip-layer connections , we develop Lemma 2 to prove that skip-layer connections are indeed necessary . In subsection 3.4 we investigate the correlation between approximation capability and network structures by analyzing the cutoff property of spiking neurons , which can change the network ’ s connectivity . In our analysis , we focus on two particular designs : heterogeneous network ( Lemma 4 ) and skip-layer connection ( Lemma 5 ) , and show their impact on the number of distinct memory pathways in a network . All lemmas are formally proved in the appendix . Lemma 1 For any input spike sequence with period tin in range [ Tmin , Tmax ] , there exist a spiking neuron n with fixed parameters vth , vreset , a , Rm and τm , such that by changing synaptic conductance G , it is possible to set the neuron response rate γn to be any positive integer . Proof Sketch . ( Formal proof in Appendix A ) Given an input spike sequence , it is possible to derive the highest possible membrane potential decay ∆v for any input tin ∈ [ Tmin , Tmax ] as a function of neuron parameters . We show that it is possible to make ∆v tends to zero by configuring the neuron parameters . Since the decay of v can be negligible , γn can be set to any positive integer by changing G. Theorem 1 For any input and target output spike sequence pair with periods ( tin , tout ) ∈ [ Tmin , Tmax ] × [ Tmin , Tmax ] , there exist a minimal-layer-size network with skip-layer connections that has memory pathway with output spike period function P ( t ) such that |P ( tin ) − tout| < ϵ . Proof Sketch . ( Formal proof in Appendix B ) With skip-layer connections , there can be multiple memory pathways in a minimal-layer-size network as neurons can be either included or skipped through . Hence it is possible to create memory pathways with different delay times for each input spike in a network and by connecting the output of those memory pathways to a common neuron n′ , spike sequence of any arbitrary period tint such that tint < = tin can be generated within ϵ . By setting γn′ > 1 , the output from n′ receiving input spike sequence with tint is tn ′ out = γ · tint . Hence it is possible to achieve a network with output spike period P ( t ) such that |P ( tin ) − tout| < ϵ. Lemma 2 With no skip-layer connection , there does not exist a minimal-layer-size network that has output spike period function P ( t ) such that for any input and target output spike sequence pair with periods ( tin , tout ) ∈ [ Tmin , Tmax ] × [ Tmin , Tmax ] , |P ( tin ) − tout| < ϵ . Proof Sketch . ( Formal proof in Appendix C ) A minimal-layer-size network without skip-layer connection has only one memory pathway . For a particular input spike sequence with period tin , different output period P ( tin ) can be achieved by changing γ of neurons in the memory pathway . We show that there exists two output spike periods P ( tin ) and P ′ ( tin ) , such that P ( tin ) − P ′ ( tin ) is a constant value independent of network or neuron configurations , and there can be no P ′′ ( tin ) such that P ( tin ) < P ′′ ( tin ) < P ′ ( tin ) . Therefore , for any minimal-layer-size network , there exists tout within the range of ( P ( tin ) , P ′ ( tin ) ) such that |P ( tin ) − tout| < ϵ does not hold true .

Summary: The authors introduce a theory on how to learn arbitrary input spike train - output spike train mappings, using a feedforward SNN with heterogeneous neurons and skip connections. Then this theory is used to train a deep SNN using either BPTT or STDP. The SNN is then evaluated on IBM DVS Gesture, N-Caltech101. The accuracy that they get with BPTT is beyond the SOTA.