DCT-SNN: Using DCT to Distribute Spatial Information over Time for Learning Low-Latency Spiking Neural Networks

1 INTRODUCTION . Deep Learning networks have tremendously improved state-of-the-art performance for many tasks such as object detection , classification and natural language processing ( Krizhevsky et al. , 2012 ; Hinton et al. , 2012 ; Deng & Liu , 2018 ) . However , such architectures are extremely energyintensive ( Li et al. , 2016 ) and hence require custom architectures and training methodologies for edge deployment ( Howard et al. , 2017 ) . To address this , Spiking Neural Networks ( SNNs ) have emerged as a promising alternative to traditional deep learning architectures ( Maass , 1997 ; Roy et al. , 2019 ) . SNNs are bio-plausible networks inspired from the learning mechanisms observed in mammalian brains . They are analogous in structure to standard networks , but perform computation in the form of spikes instead of fully analog values , as done in standard networks . For the rest of this paper , we refer to standard networks as Analog Neural Networks ( ANNs ) to distinguish them from their spiking counterparts with digital ( spiking ) inputs.The input and the correspondingly generated activations in SNNs are all binary spikes and inference is performed by accumulating the spikes over time . This can be visualized as distributing the one step inference of ANNs into a multi-step , very sparse inference scheme in the SNN . The primary source of energy efficiency of SNNs comes from the fact that very few neurons spike at any given timestep . This event driven computation and the replacement of every multiply–accumulate ( MAC ) operation in the ANN by an addition in SNN allows SNNs to infer with lesser energy . This energy benefit can be further enhanced using custom SNN implementations with architectural modifications ( Ju et al. , 2020 ) . ( Li et al. , 2017 ) have released a spiking version of the CIFAR-10 dataset based on inputs from neuromorphic sensors . IBM has designed a noncommercial processor ‘ TrueNorth ’ ( F. Akopyan et al. , 2015 ) , and Intel has designed its equivalent ‘ Loihi ’ ( Davies et al. , 2018 ) , that can train and infer on SNNs , and Blouw et al . ( 2019 ) have shown SNNs implemented on Loihi to be two orders of magnitude more efficient than an equivalent ANN running on GPU for keywork spotting . However , a major challenge still to be addressed is that the accumulation of spikes over timesteps results in a higher inference latency in SNNs . Energy efficiency at the cost of too high a latency would still hamper real-time deployment . Consequently , reduction of timesteps required for inference in SNNs is an active field of research . One of the factors that significantly affects the number of timesteps needed is the encoding scheme that converts pixels into spikes over the timesteps . Currently , the most common encoding scheme is Poisson spike generation ( Rueckauer et al. , 2017 ) , where the spikes at the input are generated as a Poisson spike train , with the mean spiking rate proportional to the pixel intensity . This scheme does not encode anything meaningful in the temporal axis and each timestep is the same as any other . Moreover , networks trained using this scheme suffer from high inference latency ( Rueckauer et al. , 2017 ) . Temporal coding schemes such as phase ( Kim et al. , 2018 ) or burst ( Park et al. , 2019 ) coding have been introduced to better encode temporal information into the spike trains , but they still incur high latency and require a large number of spikes for inference . Another related temporal method is time-to-first-spike ( TTFS ) coding ( Zhang et al. , 2019 ; Park et al. , 2020 ) . They limit the number of spikes per neuron but the high latency problem still persists . Relative timing of spikes to encode information has been used in Comsa et al . ( 2020 ) , but the results are only reported for simple tasks like MNIST and its scalability to deeper architectures such as VGG and more complex datasets like CIFAR remains unclear . In this paper , we propose a novel encoding scheme to convert pixels into spikes over time . The proposed scheme utilizes a block-wise matrix multiplication to decompose spatial information into a weighted sum of basis , and then reverses the transform to allow reconstruction of the input over multiple timesteps . These bases , taken one per timestep , modulated by the weights from the forward transform are then presented to the spike generating layer . The spike generator sums the contribution of all bases seen until the current timestep , as shown in Figure 1 . Though any invertible matrix can be utilized as the transform , the ideal transform follows the properties of energy compaction and orthonormality of bases as outlined in Section 3.1 . We motivate Discrete Cosine Transform ( DCT ) as the ideal choice , since it is data independent , with orthogonal bases ordered by their contribution to spectral energy . Each timestep gets the information corresponding to a single base , starting from the zero frequency component at the first timestep . Each subsequent step refines the input representation progressively . At the end of the cycle , the entire pixel value has passed through the spike generating neuron . Thus , this methodology successfully distributes the pixel value over all the timesteps in a meaningful manner . Choosing the appropriate dimensions of the transform provides a fine grained control on the number of timesteps used for inference . We use the proposed scheme to learn DCT-SNN , a spiking version of an ANN and show that it cuts down the timesteps needed to infer an image taken from CIFAR-10 , CIFAR-100 and TinyImageNet datasets from 100 to 48 , 125 to 48 and 250 to 48 , respectively , while achieving comparable accuracy to the state-ofthe-art Poisson encoded SNNs . Additionally , ordering the frequencies bases being input at each timestep provides a principled way of trading off accuracy for a reduced number of timesteps during inference , if desired , by dropping the least important ( highest frequency ) components . To summarize , the main contributions of this work are as follows , • A novel input encoding scheme for SNNs is introduced wherein each timestep of computation encodes distinct information , unlike other rate-encoding methods . • The proposed encoding scheme is used to learn DCT-SNN , which is able to infer with 2-14X lower timesteps compared to other state-of-the-art SNNs , while achieving comparable accuracy . • The proposed technique is , to the best of our knowledge , the first work that leverages frequency domain learning for SNNs on vision applications . • To the best of our knowledge , this is the first work that orders timesteps by significance to reconstruction . This provides an option to trade-off accuracy for faster inference by trimming some of the later frequency components , which is non-trivial to perform in other SNNs . 2 RELATED WORKS . Learning ANNs in the frequency domain . Successful learning for vision tasks in the frequency domain has been demonstrated in ANNs in several works . These utilize the DCT coefficients directly available from JPEG compression method ( Wallace , 1992 ) without performing the decompression steps . Conventional CNNs were used with DCT coefficients as input for image classification in Ulicny & Dahyot ( 2017 ) and Rajesh et al . ( 2019 ) . Ehrlich & Davis ( 2019 ) proposed a model conversion algorithm to apply pretrained spatial domain networks to JPEG images . Wavelet features are utilized in Williams & Li ( 2016 ) to train CNN-based classifiers . However , these methods suffer a small accuracy degradation compared to learning in spatial domain . DCT features were used effectively for large scale classification and instance segmentation tasks in Xu et al . ( 2020 ) . Although such frequency domain approaches have proved fruitful in ANNs , it is unexplored in SNNs despite the conversion of spatial bases of the image to temporal bases in the frequency domain being intuitively related to distributing the analog pixel values in ANNs to spikes over time in SNNs . There exist three prominent line of works for training SNNs , namely using spike-timing-dependent plasticity rules ( STDP ) , ANN-SNN conversion and training from scratch . While STDP-based local learning ( Diehl & Cook , 2015 ; Xu et al. , 2020 ) is more bio-plausible , scaling such algorithms beyond MNIST type of tasks has been challenging . So , the following discussion focuses mainly on conversion and backpropagation based works . ANN-SNN Conversion . The most common approach of training rate-coded deep SNNs is to first train an ANN and then convert it to an SNN for finetuning . ( Diehl et al. , 2015 ; Sengupta et al. , 2019 ; Cao et al. , 2015 ) . Usually , the ANNs are trained with some limitations to facilitate this , such as not using bias , batch-norm or average pooling layers , though some works are able to bypass these constraints ( Rueckauer et al. , 2017 ) . To convert ANNs to SNNs successfully , it is critical to adjust the threshold of Integrate-and-Fire ( IF ) / Leaky IF ( LIF ) neurons properly . The authors in Sengupta et al . ( 2019 ) recommend computing the layerwise thresholds as the maximum pre-activation of the neurons . This results in high accuracy but incurs high inference latency ( 2000-2500 timesteps ) . Alternatively , Rueckauer et al . ( 2017 ) choose a certain percentile of the pre-activation distribution as the threshold , reducing inference latency and improving robustness . The difference between these works and ours lie in the significance we attach to the timesteps . Backpropagation from Scratch and Hybrid Training . Another approach to training SNNs is learning from scratch using backpropagation , which is challenging due to the non-differentiability of the spike function at the time of spike . Surrogate gradient based optimization ( Neftci et al. , 2019 ) has been utilized to circumvent this issue and implement backpropagation in SNNs effectively ( Lee et al. , 2020 ; Huh & Sejnowski , 2018 ) . Surrogate gradient based backpropagation on the membrane potential at only a single timestep was proposed in Zenke & Ganguli ( 2018 ) . Shrestha & Orchard ( 2018 ) compute the gradients using the difference between the membrane potential and the threshold , but only demonstrate on MNIST using shallow architectures . Wu et al . ( 2018 ) perform backpropagation through time ( BPTT ) on SNNs with a surrogate gradient defined on the membrane potential as it is continuous-valued . Overall , SNNs trained with BPTT using such surrogategradients have been shown to achieve high accuracy and low latency ( ∼100-125 timesteps ) , but the training is very compute intensive compared to conversion techniques . Rathi et al . ( 2020 ) propose a combination of both methods , where a pre-trained ANN serves as initialization for subsequent surrogate gradient learning in the SNN domain . This hybrid approach improves upon conversion by reducing latency and speeding up convergence . However , this is orthogonal to the encoding scheme and can be used to improve the performance of any rate-coded scheme . In this work , we adopt the hybrid training method to train the SNNs . The key distinction of our method lies in how the pixel values are encoded over time , which is described next . 3 ENCODING SCHEME . An ideal encoding scheme to convert pixel values into spikes over time should capture relevant information in the temporal statistics of the data . Additionally , the total spike activity over all the timesteps at the input neuron should correspond to the pixel intensity . Our encoding scheme deconstructs the image into a weighted sum of basis functions . We invert this transform to reconstruct the image over time steps . Each basis function , taken one per timestep and modulated by the weights from the deconstruction is input to an Integrate-and-Fire ( IF ) neuron , which accumulates the input over timesteps and fires when accumulation crosses its threshold .

Summary: The scheme proposed breaks down the information in a block of an image into orthogonal basis functions (DCT is used) to make a progressively better reconstruction of the original image block with the addition of more basis functions used (like an nth order Taylor expansion).  The increasing spatial frequency components are known to be perceptually less sensitive (they need to include this) in images, so the low freq components can be presented first.  Each freq component is encoded into spikes sequentially, thereby staging the more perceptually important information first, with less important info coming later.  This reorders the presentation of information to allow a tradeoff of image quality with time/latency.