Sequence Approximation using Feedforward Spiking Neural Network for Spatiotemporal Learning: Theory and Optimization Methods

1 INTRODUCTION . Spiking neural network ( SNN ) ( Ponulak & Kasinski , 2011 ) uses biologically inspired neurons and synaptic connections trainable with either biological learning rules such as spike-timingdependent plasticity ( STDP ) ( Gerstner & Kistler , 2002 ) or statistical training algorithms such as backpropagation-through-time ( BPTT ) ( Werbos , 1990 ) . The SNNs with simple leaky integrate-andfire ( LIF ) neurons and supervised training have shown classification performance similar to deep neural networks ( DNN ) while being energy efficient ( Kim et al. , 2020b ; Wu et al. , 2019 ; Srinivasan & Roy , 2019 ) . One of SNN ’ s main difference from DNN is that the neurons are dynamical systems with internal states evolving over time , making it possible for SNN to learn temporal patterns without recurrent connections . Empirical results on feedforward-only SNN models show good performance for spatiotemporal data classification , using either supervised training ( Lee et al. , 2016 ; Kaiser et al. , 2020 ; Khoei et al. , 2020 ) , or unsupervised learning ( She et al. , 2021 ) . However , while empirical results are promising , a lack of theoretical understanding of sequence approximation using SNN makes it challenging to optimize performance on complex spatiotemporal datasets . In this work , we develop a theoretical framework for analyzing and improving sequence approximation using feedforward SNN . We view a feedforward connections of spiking neurons as a spike propagation path , hereafter referred to as a memory pathways ( She et al. , 2021 ) , that maps an input spike train with an arbitrary frequency to an output spike train with a target frequency . Consequently , we argue that an SNN with many memory pathways can approximate a temporal sequence of spike trains with time-varying unknown frequencies using a series of pre-defined output spike trains with known frequencies . Our theoretical framework aims to first establish SNN ’ s ability to map frequencies of input/output spike trains within arbitrarily small error ; and next , derive the basic principles for adapting neuron dynamics and SNN architecture to improve sequence approximation . The theoretical derivations are then investigated with experimental studies on feedforward SNN for spatiotemporal classifications . We adopt the basic design principles for improving sequence approx- imation to optimize SNN architectures and study whether these networks can be trained to improve performance for spatiotemporal classification tasks . The key contributions of this work are : • We prove that any spike-sequence-to-spike-sequence mapping functions on a compact domain can be approximated by feedforward SNN with one neuron per layer using skip-layer connections , which can not be achieved if no skip-layer connection is used . • We prove that using heterogeneous neurons having different dynamics and skip-layer connection increases the number of memory pathways a feedforward SNN can achieve and hence , improves SNN ’ s capability to represent arbitrary sequences . • We develop complex SNN architectures using the preceding theoretical observations and experimentally demonstrate that they can trained with supervised BPTT and unsupervised STDP for classification on spatiotemporal data . • We design a dual-search-space option for Bayesian optimization process to sequentially optimize network architectures and neuron dynamics of a feedforward SNN considering heterogeneity and skip-layer connection to improve learning and classification of spatiotemporal patterns . We experimentally demonstrate that our network design principles coupled with the dual-searchspace Bayesian optimization improve classification performance on DVS Gesture ( Amir et al. , 2017 ) , N-caltech ( Orchard et al. , 2015 ) , and sequential MNIST . Results show that the design principles derived using our theoretical framework for sequence approximation can improve spatiotemporal classification performance of SNN . 2 RELATED WORK . Prior theoretical approaches to analyze SNN often focus on the storage and retrieval of precise spike patterns ( Amit & Huang , 2010 ; Brea et al. , 2013 ) . There are also works that consider SNN for solving optimization problems ( Chou et al. , 2018 ; Binas et al. , 2016 ) and works that analyze the dynamics of SNN ( Zhang et al. , 2019 ; Barrett et al. , 2013 ) . Those are different topics from the approximation of spike-sequence-to-spike-sequence mappings functions . SNN that incorporates excitatory and inhibitory signal is shown for its ability to emulate sigmoidal networks ( Maass , 1997 ) and is theoretically capable of universal function approximation . Feedforward SNN with specially designed spiking neuron models ( Iannella & Back , 2001 ; Torikai et al. , 2008 ) have been demonstrated for function approximation , while for networks using LIF neurons , function approximation has been shown with only empirical results ( Farsa et al. , 2015 ) . On the other hand , the existing works that has developed efficient training process for SNN and demonstrated classification performance comparable to deep learning models , have mostly used simpler and generic LIF neuron models ( Lee et al. , 2016 ; Kaiser et al. , 2020 ; Kim et al. , 2020b ; Wu et al. , 2019 ; Sengupta et al. , 2019 ; Safa et al. , 2021 ; Han et al. , 2020 ) . Therefore , this paper develops the theoretical basis for function approximation using feedforward SNN with LIF neurons , and studies applications of the developed theoretical constructs in improving SNN-based spatiotemporal pattern classification . The effectiveness of heterogeneous neurons ( She et al. , 2021 ) and skip-layer connections ( Srinivasan & Roy , 2019 ; Sengupta et al. , 2019 ) in SNN has been empirically studied in the past . However , no theoretical approach has been presented to understand why such methods improve learning of spike sequences , and how to optimize a SNN ’ s architecture and parameters to effectively exploit these design constructs . It is possible to search for the optimal SNN configurations through optimization algorithms , but the large number of hyper-parameters for spiking neurons and network structure creates a high-dimensional search space that is long and difficult to solve . Bayesian optimization ( Snoek et al. , 2012 ) uses collected data points to make decisions on the next test point that could provide improvement , thus accelerates the optimization process . Prior works ( Parsa et al. , 2019 ; Kim et al. , 2020a ) have shown that SNN performance can also be effectively improved with Bayesian optimization . While those works consider a single or a few neuron parameters , the dualsearch-space Bayesian optimization proposed in this work optimizes both network architecture and neuron parameters efficiently by separating the discrete search spaces from the continuous search spaces . t 3 APPROXIMATION THEORY OF FEEDFORWARD SNN . 3.1 DEFINITIONS AND NOTATIONS . Definition 1 Neuron Response Rate γ For a spiking neuron n with membrane potential at vreset and input spike sequence with period tin , γ is the number of input spike n needs to reach vth . Definition 2 Memory Pathways For a feedforward SNN with m layers , a memory pathway is defined as a spike propagation path from input to the output layer . Two memory pathways are considered distinct if the set of neurons contained in them is different . Definition 3 Minimal Multi-neuron-dynamic ( mMND ) Network A densely connected network in which each layer has an arbitrary number of neurons that have different neuron parameters . All synapses from one pre-synaptic neuron have the same synaptic conductance . Notations Neuron Delay tnd is the time for a spike from pre-synaptic neuron to arrive at its postsynaptic neurons , as shown in Figure 1 ( a ) . For a feedforward SNN with m layers , a skip-layer connection can be defined with source layer and target layer pair ( ls , lt ) . The output feature map from source layer is concatenated to the original input feature map of the target layer . For the analysis of spike sequence in temporal space , the notation of Tmax and Tmin are defined as positive real numbers such that Tmax > Tmin . ϵ > 0 is the error of approximation . Figure 1 ( a ) shows two memory pathways receiving an input spike sequence with time-varying periods . As the neurons have different dynamics , the two memory pathways have different response to the input spike sequence . An example of mMND network with m layers and n neuron dynamics is shown in Figure 1 ( b ) . SNN with multilayer perceptron ( MLP ) structure can be considered a scaled-up mMND network with multiple neurons for each dynamic . A network with convolutional structure can be considered a scaled-up mMND network with duplicated connections in each layer . We analyze the correlation of network capacity and structure based on mMND networks . The design of neuron heterogeneity can also be implemented in MLP-SNN and Conv-SNN as described in Section 4 . The analysis for network capacity can be extended to those networks according to their specific layer dimensions . 3.2 MODELING OF SPIKING NEURON . SNN consists of spiking neurons connected with synapses . The spiking neuron model studied in this work is leaky integrate-and-fire ( LIF ) as defined by the following equations : τm dv dt = a+RmI − v ; v = vreset , if v > vthreshold ( 1 ) Rm is membrane resistance , τm = RmCm is time constant and Cm is membrane capacitance . a is the resting potential . I is the sum of current from all input synapses that connect to the neuron . A spike is generated when membrane potential v cross threshold and the neuron enters refractory period r , during which the neuron maintains its membrane potential at vreset . The time it take for a pre-synaptic neuron to send a spike to its post-synaptic neurons is tnd . Neuron response rate γ is a property of a spiking neuron ’ s response to certain input spike sequence . We show how the value of γ can be evaluated below . Remark For any input spike sequence , individual spike can be described with Dirac delta function δ ( t− ti ) where ti is the time of the i-th input spike . For the membrane potential of a spiking neuron receiving the input before reaching spiking threshold , with initial state at t = 0 with v = vreset , solving the differential equation ( 1 ) leads to : v ( t ) = vresete − tτm + a ( 1− e− t τm ) + Rm τm e− t τm ∑ i G ∫ t 0 δ ( t− ti ) e t τm dt ( 2 ) Here , G is the conductance of input synapse connected to the neuron , which is trainable . From ( 2 ) , there exists a value of u such that vm ( tu−1 ) < vthreshold and vm ( tu ) > = vthreshold . By evaluating ( 2 ) for u given neuron parameters and input spike sequence , the neuron response rate γ can be found . 3.3 APPROXIMATION THEOREM OF FEEDFORWARD SNN . To develop the approximation theorem for feedforward SNN , we first aim to understand the range of neuron response rate that can be achieved . We show with Lemma 1 that for any input spike sequence with periods in a closed interval , it is possible to set the neuron response rate γ to any positive integer . Based on this property , we show with Theorem 1 that by connecting a list of spiking neurons with certain γ sequentially and inserting skip-layer connections , approximation of spikesequence mapping functions can be achieved . To understand whether this capability of feedforward SNN relies on skip-layer connections , we develop Lemma 2 to prove that skip-layer connections are indeed necessary . In subsection 3.4 we investigate the correlation between approximation capability and network structures by analyzing the cutoff property of spiking neurons , which can change the network ’ s connectivity . In our analysis , we focus on two particular designs : heterogeneous network ( Lemma 4 ) and skip-layer connection ( Lemma 5 ) , and show their impact on the number of distinct memory pathways in a network . All lemmas are formally proved in the appendix . Lemma 1 For any input spike sequence with period tin in range [ Tmin , Tmax ] , there exist a spiking neuron n with fixed parameters vth , vreset , a , Rm and τm , such that by changing synaptic conductance G , it is possible to set the neuron response rate γn to be any positive integer . Proof Sketch . ( Formal proof in Appendix A ) Given an input spike sequence , it is possible to derive the highest possible membrane potential decay ∆v for any input tin ∈ [ Tmin , Tmax ] as a function of neuron parameters . We show that it is possible to make ∆v tends to zero by configuring the neuron parameters . Since the decay of v can be negligible , γn can be set to any positive integer by changing G. Theorem 1 For any input and target output spike sequence pair with periods ( tin , tout ) ∈ [ Tmin , Tmax ] × [ Tmin , Tmax ] , there exist a minimal-layer-size network with skip-layer connections that has memory pathway with output spike period function P ( t ) such that |P ( tin ) − tout| < ϵ . Proof Sketch . ( Formal proof in Appendix B ) With skip-layer connections , there can be multiple memory pathways in a minimal-layer-size network as neurons can be either included or skipped through . Hence it is possible to create memory pathways with different delay times for each input spike in a network and by connecting the output of those memory pathways to a common neuron n′ , spike sequence of any arbitrary period tint such that tint < = tin can be generated within ϵ . By setting γn′ > 1 , the output from n′ receiving input spike sequence with tint is tn ′ out = γ · tint . Hence it is possible to achieve a network with output spike period P ( t ) such that |P ( tin ) − tout| < ϵ. Lemma 2 With no skip-layer connection , there does not exist a minimal-layer-size network that has output spike period function P ( t ) such that for any input and target output spike sequence pair with periods ( tin , tout ) ∈ [ Tmin , Tmax ] × [ Tmin , Tmax ] , |P ( tin ) − tout| < ϵ . Proof Sketch . ( Formal proof in Appendix C ) A minimal-layer-size network without skip-layer connection has only one memory pathway . For a particular input spike sequence with period tin , different output period P ( tin ) can be achieved by changing γ of neurons in the memory pathway . We show that there exists two output spike periods P ( tin ) and P ′ ( tin ) , such that P ( tin ) − P ′ ( tin ) is a constant value independent of network or neuron configurations , and there can be no P ′′ ( tin ) such that P ( tin ) < P ′′ ( tin ) < P ′ ( tin ) . Therefore , for any minimal-layer-size network , there exists tout within the range of ( P ( tin ) , P ′ ( tin ) ) such that |P ( tin ) − tout| < ϵ does not hold true .

Summary: In this publication, the authors present an interesting framework for approximation the mapping between sequences by feedforward spiking neural networks (SNN), providing insights on the computational capabilities of feedforward SNNs for the approximation of the mapping of input to output spike trains. And how these are influenced by hyperparameters such as the network architecture, heterogenic properties of neurons and skipping connections. The authors also provide an ansatz how these hyperparameters can be optimised in a two-steps Bayesian optimisation fashion. The performance of the approach is demonstrated by several spatiotemporal classification tasks, on datasets like DVS Gesture, N-Caltech 101 and sequential MNIST.