DCT-SNN: Using DCT to Distribute Spatial Information over Time for Learning Low-Latency Spiking Neural Networks
Authors: Isha Garg
Sayeed Shafayet Chowdhury
Kaushik Roy
Venue:  (2021)
Keywords: Spiking Neural Networks
Input Encoding
Low Latency
Discrete Cosine Transform
Temporal Information
Frequency Domain

Abstract: Spiking Neural Networks (SNNs) offer a promising alternative to traditional deep learning frameworks, since they provide higher 
computational efficiency due to event-driven information processing. SNNs distribute the analog values of pixel intensities into binary spikes over time. However, the most widely used input coding schemes, such as Poisson based rate-coding, do not leverage the additional temporal learning capability of SNNs effectively. Moreover, these SNNs suffer from high inference latency which is a major bottleneck to their deployment. To overcome this, we propose a scalable time-based encoding scheme that utilizes the Discrete Cosine Transform (DCT) to reduce the number of timesteps required for inference. DCT decomposes an image into a weighted sum of sinusoidal basis images. At each time step, a single frequency base, taken in order and modulated
by its corresponding DCT coefficient, is input to an accumulator that generates spikes upon crossing a threshold. We use the proposed scheme to learn DCT-SNN, a low-latency deep SNN with leaky-integrate-and-fire neurons, trained using surrogate gradient descent based backpropagation. We achieve top-1 accuracy of 89.94%, 68.3% and 52.43% on CIFAR-10, CIFAR-100 and TinyImageNet, respectively using VGG architectures. Notably, DCT-SNN performs inference with 2-14X reduced latency compared to other state-of-the-art SNNs, while achieving comparable accuracy to their standard deep learning counterparts. The dimension of the transform allows us to control the number of timesteps required for inference. Additionally, we can trade-off accuracy with latency in a principled manner by dropping the highest frequency components during inference.

Under review as a conference paper at ICLR 2021
DCT-SNN: USING DCT TO DISTRIBUTE SPATIAL INFORMATION OVER TIME FOR LEARNING Low- LATENCY SPIKING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Spiking Neural Networks (SNNs) offer a promising alternative traditional deep learning frameworks, since they provide higher computational efficiency due to event-driven information processing. SNNs distribute the analog values of pixel intensities into binary spikes over time. However, the most widely used input coding schemes, such as Poisson based rate-coding, do not leverage the additional temporal learning capability of SNNs effectively. Moreover, these SNNs suffer from high inference latency which is a major bottleneck to their deployment. To overcome this, propose scalable time-based encoding scheme that utilizes the Discrete Cosine Transform (DCT) to reduce the number of timesteps required for inference. DCT decomposes image into a weighted sum of sinusoidal basis images. At each time step, a single frequency base, taken in order and modulated by its corresponding DCT coefficient, is input to an accumulator that generates spikes upon crossing threshold. We use the proposed scheme to learn DCT- SNN. a low-latency deep SNN with beaak-integrate-ande surrogate gradient descent based backpropagation. We achieve top-1 accuracy of 89.94% 68.3% and 52.43% on CIFAR-IO, CIFAR- and TinylmageNet, respectively using VGG architectures. Notably, DCT-SNN performs inference with 2-14X reduced latency compared to other state-of-the-art SNNs, while achieving comparable accuracy to their standard deep learning counterparts. The dimension of the transform allows us to control the number of timesteps required for inference. Additionally, we trade-off accuracy with lateney in principled manner by dropping the highest frequency components during inference.
neurons, trained using
INTRODUCTION
Deep Learning networks have tremendously improved state-of-the-ar performance for many tasks such as object detection, classification and natural language processing (Krizhevsky et al., 2012]
Hinton et al. 2012 Deng & Liu 2018). However, such architectures are extremely energy- intensive (Li et al. 2016) and hence require custom architectures and training methodologies for edge deployment (Howard al. 2017). To address this, Spiking Neural Networks (SNNs) have emerged as a promising alternative to traditional deep learning architectures (Maass) 1997| Roy et al. 2019). SNNs are bio-plausible networks inspired from the learning mechanisms observed in mammalian brains. They are analogous in structure to standard networks, but perform computation in the form of spikes instead of fully analog values, as done in standard networks. For the rest of this paper, we refer to standard networks as Analog Neural Networks (ANNs) to distinguish them from their spiking counterparts with digital (spiking) inputs. The input and the correspondingly generated activations in SNNs are all binary spikes and inference is performed by accumulating the spikes over time. This can be visualized distributing the one step inference of ANNs into multi-step. very sparse inference scheme in the SNN.
The
The primary source of energy efficiency of SNNs comes from the fact that very few neurons spike at any given timestep. This event driven computation and the replacement of every multiply-accumulate (MAC) operation in the ANN by an addition SNN allows SNNs to infer with lesser energy. This energy benefit can be further enhanced using custom SNN implementations with architectural modifications {Ju et al. 2020). (Li et al. 2017) have released a spiking version of the CIFAR- 10 dataset based inputs from neuromorphic sensors. IBM has designed commercial processor 'TrueNorth' (F. Akopyan et all 2015 ). and Intel has designed its equivalent "Lohhi (Davies et al. 2018 ). that train and infer on SNNs, and Blouw al. (019 have shown

---

Under review as a conference paper at ICLR 2021
SNNs implemented on Loihi to be two orders of magnitude more efficient than an equivalent ANN running on GPU for keywork spotting. However, a major challenge still to be addressed is that the accumulation of spikes over timesteps results in higher inference latency in SNNs. Energy efficiency at the cost of too high a lateney would still hamper real-time deployment. Consequently, reduction of timesteps required for inference in SNNs is active field of research. One of the factors that significantly affects the number of timesteps needed is the encoding scheme that converts pixels into spikes over the timesteps. Currently, the most common encoding scheme is Poisson spike generation (Rueckauer et al., 2017), where the spikes at the input are generated as a Poisson spike train, with the mean spiking rate proportional to the pixel intensity. This scheme does not encode anything meaningful in the temporal axis and each timestep is the same as any other. Moreover, networks trained using this scheme suffer from high inference latency (Rueckauer et al. 2017). Temporal coding schemes such as phase (Kim et al. 2018) or burst (Park et al. (019) coding have been introduced to better encode temporal information into the spike trains, but they still incur high latency and require large number of spikes for inference. Another related temporal method is time-to-fin to-iiss-ssspeke (TTFS) coding (Zhang et al.] 2019 Park et al.) 2020). They limit the number of spikes per neuron but the high latency problem still persists. Relative timing of spikes to encode information has been used in Comsa et al. (2020), but the results are only reported for simple tasks like MNIST and its scalability deeper architectures such as VGG and more complex datasets like CIFAR remains unclear.
In this paper, we propose a novel encoding scheme to convert pixels into spikes over time. The proposed scheme utilizes block-wise matrix multiplication to decompose spatial information into weighted sum of basis, and then reverses the transform to allow reconstruction of the input over multiple timesteps. These bases, taken one per timestep, modulated by the weights from the forward transform are then presented to the spike generating layer. The spike generator sums the contribution of all bases seen until the current timestep, as shown in Figure | Though any invertible matrix can utilized as the transform, the ideal transform follows the properties of energy compaction and orthonormality of bases as outlined in Section 3.1. We motivate Discrete Cosine Transform (DCT) as the ideal choice, since it is data independent, with orthogonal bases ordered by their contribution to spectral energy. Each timestep gets the information corresponding to single base, starting from the zero frequency component at the first timestep. Each subsequent step refines the input representation progressively. At the end of the cycle, the entire pixel value has passed through the spike generating neuron. Thus, this methodology successfully distributes the pixel value over all the timesteps in meaningful manner. Choosing the appropriate dimensions of the transform provides a fine grained control on the number of timesteps used for inference. We use the proposed scheme to learn DCT-SNN. a spiking version of an ANN and show that cuts down the timesteps needed to infer an image taken from CIFAR-10. 10, CIFAR-100 and TinyImageNet datasets from 100 to 48, 125 to 48 and 250 to 48, respectively, while achieving comparable accuracy to the state-of- the-art Poisson encoded SNNs. Additionally, ordering the frequencies bases being input at each timestep provides a principled way of trading off accuracy for a reduced number of timesteps during inference, if desired, by dropping the least important (highest frequency) components.
To summarize, the main contributions of this work are as follows,
A novel input encoding scheme for SNNs is introduced wherein each timestep of computation encodes distinct information, unlike other rate-encoding methods.
The proposed encoding scheme is used to learn DCT-SNN, which is able to infer with 2-14X lower timesteps compared to other -f-te-of-thhearr SNNs, while achieving comparable accuracy.
The proposed technique is, to the best of our knowledge, the first work that leverages frequency domain learning for SNNs on vision applications.
To the best of our knowledge, this is the first work that orders timesteps by significance to reconstruction. This provides an option to trade-off accuracy for faster inference by trimming some of the later frequency components, which non-trivial to perform in other SNNs.
2 RELATED WORKS
Learning ANNs in the frequency domain. Successful learning for vision tasks in the frequency domain has been demonstrated in ANNs in several works. These utilize the DCT coefficients directly available from JPEG compression method (Wallace 1992) without performing the decompression
2

---

Under review as a conference paper at ICLR 2021
steps. Conventional CNNs were used with DCT coefficients input for image classification in Ulieny & Dahyot (0177) and Rajesh et al. 2019). Ehrlich & Davis (2019) proposed a model conversion algorithm apply pretramed spatial domain networks to JPEG images. Wavelet features are utilized in Williams & Li (2016) to train CNN-based classifiers. However, these methods suffer a small accuracy degradation compared to learning in spatial domain. DCT features were used effectively for large scale classification and instance segmentation tasks in Xu et al. (2020). Although such frequency domain approaches have proved fruitful in ANNs, is unexplored in SNNS despite the conversion of spatial bases of the image to temporal bases in the frequency domain being intuitively related to distributing the analog pixel values in ANNs to spikes over time in SNNs. There exist three prominent line of works for training SNNs, namely using tenke--iming-deeeddettt plasticity rules (STDP), ANN-SNN conversion and training from scratch. While STDP-based local learning (Diehl & Cook 2015 Xu et al.] (020) is more bio-plausible, scaling such algorithms beyond MNIST type of tasks has been challenging. So, the following discussion focuses mainly on conversion and backpropagation based works.
ANN-SNN Conversion. The most common approach of training rate-coded deep SNNs is to first train an ANN and then convert it to an SNN for finetuming. (Diehl et al.) 2015 Sengupta et al., 2019 Cao et al. 2015). Usually, the ANNs are trained with some limitations to facilitate this, such as not using bias, batch-norm or average pooling layers, though some works are able to bypass these constraints (Rueckauer et al. 2017). To convert ANNs to SNNs successfully, it is critical to adjust the threshold of Integrate-and- -ire (IF) / Leaky IF (LIF) neurons properly. The authors in Sengupta iet al. (2019) recommend computing the layerwise thresholds as the maximum pre-activation of the neurons. This results in high accuracy but incurs high inference latency (2000-2500 timesteps). Alternatively, Rueckauer et al. (2017) choose a certain percentile of the pre-activation distribution as the threshold, reducing inference latency and improving robustness. The difference between these works and ours lie in the significance we attach to the timesteps.
Backpropagation from Seratch and Hybrid Training. Another approach to training SNNs is learning from scratch using backpropagation, which is challenging due to the non-differentiabilic of the spike function at the time of spike. Surrogate gradient based optimization (Neftci et al.) 2019) has been utilized to circumvent this issue and implement backpropagation in SNNs effectively (Lee et al.] 2020: Huh & Sejnowski] 2018). Surrogate gradient based backpropagation on the membrane potential at only a single timestep was proposed in Zenke& & Ganguli (2018), 8). Shrestha & Orchard (2018) compute the gradients using the difference between the membrane potential and the threshold, but only demonstrate MNIST using shallow architectures. Ww et al. (2018) perform backpropagation through time (BPTT) on SNNs with a surrogate gradient defined on the membrane potential as it is continuous-valued. Overall, SNNs trained with BPTT using such surrogate- gradients have been shown to achieve high accuracy and low latency (~100-125 timesteps), but the training very compute intensive compared to conversion techniques. |Rahi al. (2020) propose a combination of both methods, where pre-trained ANN serves as initialization for subsequent surrogate gradient learning in the SNN domain. This hybrid approach improves upon conversion by reducing latency and speeding up convergence. However, this is orthogonal to the encoding scheme and can be used to improve the performance of any rate-coded scheme. In this work, we adopt the hybrid training method to train the SNNs. The key distinction of our method lies in how the pixel values are encoded over time, which is described next.
ENCODING SCHEME
An ideal encoding scheme to convert pixel values into spikes over time should capture relevant information in the temporal statistics of the data. Additionally, the total spike activity over all the timesteps at the input neuron should correspond to the pixel intensity. Our encoding scheme deconstructs the image into weighted sum of basis functions. We invert this transform to reconstruct the image over time steps. Each basis function, taken one per timestep and modulated by the weights from the deconstruction is input to Integrate-and-Fiin (IF) neuron, which accumulates the input over timesteps and fires when accumulation crosses its threshold.
3.1 A GENERIC 1-D TRANSFORM TO DISTRIBUTE PIXELS OVER TIME
1-D transformation. For simplicity, we first consider one dimensional transform over the entire input pixel space. Let us consider a single D-dimensional image, X e RIXD We transform this image using transform matrix T into a new coordinate system, where T E RDD, The transformed
3

---

Under review as a conference paper at ICLR 2021
X(1)
1:
Y:
Y
Y:
V:
X
t=3
T
T22
T-1
X(3)
t=4
X(4)
t=5
Y:
X = X
Forward Transform: Deconstruction Xf
Reverse Transform: Time-Spread Reconstruction Xf
X())
Yn Tni
x(1)= X
X Y
Figure 1-D Encoding Scheme: On the left we show the Forward Transform. T represents the transform matrix that takes the input X into intermediate coordinate system, resulting in representation Y. On the right, we show the Inverse Transform that uses Y to reconstruct X over time. Here T-1 T'. The input image X is reconstructed progressively each timestep by summing the basis vectors Tn modulated by the corresponding coefficient yo over all previous timesteps. Since there 5 bases shown here, X requires 5 timesteps for reconstruction.
vector, XT, where Y IIXD, contains the coefficients of the image in the new coordinate system. This is shown pictorially for d = in Figure [ Assume that 7 is full rank matrix and let us consider T-1 E DxD, the inverse transformation matrix that takes us back into the original coordinate system. For reasons clarified shortly, assume that T is orthonormal matrix with its inverse equal to its transpose, 7 = TV. The forward transform represents deconstruction of the input X into weighted shown in Figure I These bases are referenced by T,, n = 2....... If we input one basis function per timestep to SNN, we get intermediate representations of the input timestep t, X by modulating the t-th basis by its corresponding weight from the forward transform, summed over all previous timesteps. Summing over all bases allows us to reconstruct X. Mathematically.
of basis vectors, represented by the of T-1 columns of T as
(
Σ Tr and ( =,
The analog value of (*) is the input the spike generator at each timestep and is converted to spikes using IF neurons as shown in Figure Hence, we have successfully distributed the input X our chosen bases. In the next section, we discuss the desirable properties of the bases for deconstructing X.
over timesteps, with each timestep carrying information
Desirable Properties of the Basis Vectors. The columns of the transform matrix T contain the bases to deconstruct X. Since we use one basis per timestep, we want each base to offer non-interfering information about X. This captured by the orthonormality constraint on T. Orthonormal columns avoid cancellation of information between timesteps, and relate the forward and reverse transforms by transpose operation. The second constraint T is that the bases be ranked by a measure of the information they carry. This allows each basis function to successively refine the representation per timestep. the carlier timesteps, with fine-grained information added by the later steps. This ordering of bases allows us to drop bases in a principled manner to trade off accuracy for latency during inference.
desirable to have the bulk of the information focused in
Transforms that Satisfy Constraints. There are two widely used transforms that satisfy these properties: the DCT transform (Ahmed et al.) 1974) and the Karhunen-Locs transform (Dony et al. 2001), also known Principal Component Analysis (PCA). DCT decomposes image into a linear combination of sinusoidal frequencies, ranked by spectral energy. PCA uses the eigenvectors of the covariance matrix of the inputs the bases, ranked by the amount of variance
4

---

Under review as a conference paper at ICLR 2021
Figure 3: 8 x 8 2-D DCT bases on the left and PCA bases for CIFAR-I0 on the right. The DCT bases are ranked in zig-zag fashion starting from top left the bottom right and the PCA bases are ranked in significance from left to right and top to bottom.
Figure 2: Spike generation with 1-D DCT basis functions input per timestep (shown vertically). The neuron spikes when the accumulated value crosses a threshold.
they explain. DCT is commonly used in JPEG Compression and PCA in dimensionality reduction, by approximating the later components. However, PCA results in dataset dependent bases, whereas the DCT bases are pre-determined, avoiding extra computation. The I-D DCT transform uses the following equation to take the pixel values En into DCT coefficients X₃ using sinusoidal bases.
N-1
X
Fn COS
+
k= 0.....,,A
N
m=00
Conversion of pixels denoted by Say to DCT coefficients Fuv for M N block is shown by:
{
= =
22
else
M-1 N-1
Fuv
2
C.C,
Σ
cos
um
2x
+
COS
DTT
2y + 2N
(3)
VMN
2M
==0 y=0
The sinusoidal bases can be entered as the columns of a transformation matrix T. The forward transform is then computed Y TXT' and the reverse transform is computed X T'YT. comparison of the results for DCT, PCA, a random orthonormal transform without ranked bases, and random non-orthonormal non-ranked transform is shown in Table For the rest of the paper, we use the dataset agnostic DCT. Additionally, the conversion of spatial to temporal frequency in DCT lends itself intuitively the concept of distributing spatial information in ANNs into spikes over time.
3.2 2-D DISCRETE COSINE TRANSFORM
We extend the scheme to 2-D. The 2-D DCT is just the 1-D DCT applied first along the width channel and then along the length channel. Images are high dimensional, resulting in large transformation matrices T. This is undesirable since the number of DCT bases (or the dimension of T") dictates the number of timesteps required to reconstruct the image. To tackle this, similar to JPEG compression scheme, of size n x n and perform 2-D DCT on these blocks, getting n² ordered frequency components. We replace the n X pixel block with the equivalently reshaped frequency coefficients. requires n² timesteps for perfect reconstruction of the pixel block. Small values of n allow us to reconstruct the images by summing over only few basis images. In standard JPEG compression, 8 x 8 blocks are used, resulting in the 64 basis images shown in Figure 3 Similarly, the bases obtained from PCA on the training dataset of CIFAR- 10 are also shown. Empirically, we find that block sizes of 4 4 converge the best accuracy with the lowest number of timesteps. We usually need to run full cycles to achieve convergence to best accuracy, amounting to x x 3 48 timesteps. In each of the 3 cycles, repeat the 16 DCT coefficients and bases, to allow time for spike propagation to the deeper layers. This is discussed in further detail in Section 4. Unlike the JPEG compression scheme, improve accuracy utilize overlapped DCT scheme, where the
first convert the image from RGB YCbCr domain, and then take blocks
n X block
5

---

Under review as a conference paper at ICLR 2021
Table 1: Accuracy of VGG5 on CIFAR-10, with DCT block size 4x4
Table Accuracy(%) with timesteps indicated in parenthesis. -p and -d represent training with pixels and DCT coefficients, respectively
VGG9
VGG11
VGG13 TinyImageNet
Configuration
CIFAR-10
CIFAR-100
Transform Accuracy (%)
ANN-p
91.3
69.7
56.9
Matrix
4
ANN-d
90.4
66.4
Random
64.7
53.
Unranked
83.3
SNN-p
90.1 (175)
88.9 (100)
67.8 (125)
53 (250)
Orthonormal
SNN-d
88.2 (100)
65.1 (125)
44.6 (250)
PCA 83.8
89.94 (48)
68.3
52,43 (125)
DCT
83.5
DCT-SNN
(48)
51,45 (48)
ANN without batchnorm and maxpool to facilitate conversion ANN with batchnorm and maxpool
blocks of pixels overlap. This is equivalent to performing convolution with a kernel size of 4 and a stride of 2, and increases our input dimensions by 4x. To counter this, we add an additional 2 x average pooling layer before the linear layers.
3.3 CONVERSION FROM ANN AND THRESHOLD SELECTION FOR SPIKES
In our scheme, the SNN trains on intermediate pixel representations, and hence we utilize an ANN trained with pixel intensities (rather than DCT coefficients) for initialization. The threshold of the IF neuron at the spike generator significantly affects the timesteps required for spike propagation to deeper layers. This IF neuron, as shown in Figure 2, receives the bases modulated by the DCT coefficients and accumulates them over timesteps, firing when the accumulation crosses the threshold. We allow for both positive and negative spikes to account for the positive and negative cycles of the sinusoidal bases. Similar to the hidden layer neurons, the threshold is chosen a percentile of the accumulation the spike generator neurons. We obtained best results by using 6.5 and 93.5 percentile of the accumulation thresholds for negative and positive spikes, respectively.
4 EXPERIMENTS AND RESULTS
We implement DCT-SNN by incorporating the proposed encoding scheme with ceroogtte-gradien based learning using LIF neurons. Starting with pretrained ANN, we copy the weights to the SNN and select the 99.9 percentile of the pre-activation distribution at each layer as its threshold. The details of the learning methodology and hyperparameters of training are given in appendix section A.1 and A.2. The implementation is provided as part of the supplementary material.
Choice of Transformation. We first analyze the performance of DCT-SNN trained different choices of transformation matrices (denoted as T in section 3.1). Table shows the results for VGG5 network trained on CIFAR-10. With random T, the network does converge but with much lower accuracy than the ANN. Next, to avoid interference between different bases, we use random orthonormal T. Table [ shows that the accuracy improves by 20% compared non- orthonormal case. However, this choice of I does not perform energy compaction. Ranking bases by their contribution reconstruction allows to trade off accuracy for latency during inference. To incorporate this, we experiment with the transformation matrix generated by performing PCA on x 4 blocks of the inputs from the training dataset. While this satisfies both the desired properties and gives the best performance, it is data-dependent transform. Therefore, we utilize the fixed DCT matrix, and find that it performs at par with PCA, while additionally being data-agnostic. For all subsequent analysis, use DCT as the choice of transformation.
Effeet of Block Size and Overlap. Having chosen DCT to determine the bases of our encoding scheme, we tune the block size and stride. The results are shown in Fig. 4 "DCT-x' denotes network trained on inputs transformed with DCT of blocksize x, and 'ov' refers to overlap among the DCT blocks. Reducing the blocksize from 16 to 4 improves accuracy consistently. Moreover, since a blocksize of x requires x² timesteps to pass one information of cycle, smaller blocksizes benefit from lesser requirement of timesteps per cycle. The results different block sizes with

---

Under review as a conference paper at ICLR 2021
accuracy for varying DCT configurations
Accuracy vs Timesteps for Polsson Encoding
95
9913
90
100
Accuracy (%)
85
36.9
$9.9
Accuracy (96)
90
80
80.7
60
75
76.0
40
70
65
20
PolssonnSNN
0
16x16 (256)
BKB (64)
BxB (128)
4x4 (4B)
4yáov (4B)
40
60
100
120
140
160
180
SNN Accuracy
-ANN Accuracy
Timesteps
Figure 4: Accuracy(%) for VGG9 on CIFARIO
Figure 5: Accuracy (% ) for VGG9 on CIFARIO with varying timesteps
with varying DCT blocksize (timesteps)
timesteps required in parenthesis are shown in Fig. 4
We empirically find that DCT-2 is unable to converge, and that overlapped version of DCT-4 with a stride of 2 outperforms all other cases. Hence, we utilize this scheme for all further experiments.
Number of Cycles for Information Propagation. The next design parameter is the number of timesteps per forward pass. The performance of DCT-SNN trained with different timesteps is shown in Fig. S In the scheme DCT-4, one full cycle amounts to 16 timesteps. The network converges to 89. 99.94 accuracy with 48 timesteps and 88.41% accuracy with just 32 timesteps. Since performance saturates after 3 cycles (48 timesteps), we choose 48 as the number of timesteps to train DCT-SNN on CIFAR-10 and CIFAR-I00. However, for deeper networks and larger datasets, larger timesteps might yield further improvements, shown in Table 2 for Tinylmagenet with VGG13. Notably, the accuracy of Poisson Poisson-encoded encoded networks drops severely below 45 timesteps (Fig. 5. whereas DCT- SNN suffers minimal drop with 28 timesteps. In particular, Poisson does not converge for 32 timesteps or lesser, whereas we achieve less than 2% accuracy drop at 32 timesteps.
Results on CIFAR and Imtgee. The experimental results using the proposed scheme for DCT-SNN performs comparably to SNNs
CIFAR and IinylmageNet datasets are shown in Table 2 trained with Poisson-encoded pixels, but requires lesser than half the timesteps. We also compare our results with SNNs trained on Poisson-encoded DCT coefficients and show that our method performs better, presumably due the reconstruction of pixels over time. To demonstrate the scalability of the proposed algorithm, we apply it to the Tinylmagenet dataset. SNNs with Poisson-encoe encoded pixels require 250 timesteps to converge, whereas our method can converge comparable accuracy in 125 timesteps. Allowing for a 1% drop in accuracy, our method converges with even 48 timesteps.
Performance Comparison. We collect reported results for different sate-of-the-ar SNNs and compare our performance in Table DCT-SNN performs better than or comparably to the reported accuracy of the these methods, while achieving lower latency for inference. [Wu et al. (2018) report CIFARIO results with 30 timesteps for a shallow network with 2 convolutional and 2 fully-connected layers with 50.7% accuracy. We implement the same net with DCT-SNN and achieved 68.1% accuracy with 28 timesteps. Next, compare with methods that expose analog pixel intensities directly to the first convolutional layer, instead of spikes. In a subsequent work, /w et al.) (2019b) achieve 90.53% accuracy on CIFAR-10 using just 12 timesteps and 2 fully connected layers. After each conv layer, the binary activations go through a channel- wise normalization (termed as neunorm in the paper). This makes the binary activations essentially analog, as can be seen from Eqn. (9) and (10) in Wu et al. (2019)). It is unclear whether the efficiency of 12 timesteps is arising due to their encoding scheme, because of the proposed channel- wise normalization resulting in analog computation (MAC instead of just accumulation) at each layer, or their voting scheme based on slass-wise populations. We believe that the analog nature of computation at each layer makes this network closer to ANNs than SNNs, resulting in the significant reduction in timesteps, especially since their network is shown to converge good accuracy in a single timestep.
network with 5 convolutional
We also compare with two works that train ANNs on DCT coefficients. Ehrlich & Davis (0019)
report ANNs with 72.5% and 38.5% accuracy on CIFAR-10 and CIFAR-100, respectively and Ulieny& & Dahyot 2017) report 86.35% accuracy on CIFAR-10. 10. DCT-SNN reaches upto ~ 90% in Table 2 achieved by performing DCT blocks of size 4 x with an overlap of 2, unlike the standard JPEG scheme of x with no overlap. However, this does
accuracy for CIFAR-IO,
not train in the frequency domain unlike Ehrlich & Davis 2019
since after passing the modulated
7

---

Under review as a conference paper at ICLR 2021
Table Comparison of DCT-SNN to other reported results. SGB denotes Murrogate-Gradicn Based backprop, Hybrid denotes pretrained ANN followed by SNN fine-tuning, TTFS denotes Time- To- First-Spike scheme, TL denotes tandem learning and (xC, yL) denotes an architecture with x Conv layers and y Linear layers.
Reference
Datasct
Training
Architecture
Accuracy(%)
Timesteps
(Hunsberger & Eliasmith
2015
CIFARIO
Conversion
2C. 2L
82.95
6000
(Cao et al.
2015
CIFARIO
Conversion
3C, 2L
77.43
400
Sengupta ct al. 2019
CIFARIO
Conversion
VGG16
91.55
2500
(l.ce ct |
2020)
CIFARTO
SGB
VGG9
90.45
100
[Rueckauer et al.
2017)
CIFARIO
Conversion
4C. 2L
90.85
400
(Rathi et al.
2020
CIFARIO
Hybrid
VGG9
90.5
100
(Park ct al.
2020
CIFAR10
TTFS
VGGI6
91.4
680
(Park et al.
2019
CIFARIO
Burst-coding
VGG16
91.4
1125
(im et al.
2018
CIFARIO
Phase-coding
VGG16
91.2
1500
Wu
2018
CIFARIO
SGB
2C, 2L
50.7
Wu et al.
2019b
CIFARIO
SGB
5C,2L
90.53
12
Wu et al.
2019a
CIFARIO
TL(LIF)
5C, 2L
89.04
This work
CIFAR10
DCT-SNN
VGG9
89.94
48
(Lu & Sengupta
2020)
CIFAR100
Conversion
VGG15
63.2
62
|Rathi et al.
2020)
CIFAR100
Hybrid
VGG11
67.9
(Park al.
2020
CIFARI00
TIFS
VGG16
68.8
680
(Park et al.
2019
CIFAR100
Burst-coding
VGG16
68.77
3100
(Kim et al.
2018
CIFAR100
Phase-coding
VGG16
68.6
8950
This work
CIFARI00
DCT-SNN
VGG11
68.3
48
This work
Tinyl Finylmagenet
DCT-SNN
VGG13
52.43
125
bases through the network, have passed the equivalent information of the input image in the pixel domain. To verify that our method is trainable via backpropagation from scratch, we trained VGG9 SNN from scratch for CIFAR-IO, which gives 84.9% accuracy with 48 timesteps. We show more detailed comparison with other encoding schemes in Appendix section A.5.
Accuracy-Lateney Trade-off. The ranking of bases in our scheme allows us drop the least significant components, In Figure 6, we show the effect the ranking of bases has on accuracy by performing inference on VGG9 DCT-SNN trained on CIFAR-10 for 48 timesteps using all 16 frequencies. A minimum of 16 timesteps (1 cycle) are required for spike propagation to the deeper layers, and hence any configuration with timesteps lesser than 16 cannot infer correctly. We provide 2 cycles of inputs the test data. The first cycle uses all 16 components, and the next adds successively higher frequencies. Due to the fact that the bulk the information is contained in the earlier timesteps, are able to get good accuracy (73.9% out of 88.6% with just the first bases. Successive components add more refined information, and therefore the accuracy saturates, evident from Figure 6 To the best of our knowledge, this is the first work that demonstrates a principled tradeoff between inference accuracy and latency on a trained network. Results for networks trained with limited frequencies order of inputting frequencies is shown in Appendix section A.6.
shown in Appendix A.3. The effect of changing the
Computational Efficiency. The floating-point (FP) MAC operations in ANN replaced by FP additions in SNN. The cost of a MAC operation (4.6pJ) is 5. compared to an addition (0.9pJ) (Horowitz) 2014) in 45mm CMOS technology. The expressions representing the computational cost in the form of operations per layer in an ANN, # ANNooss are given in Appendix A.4. The number of operations per layer in an equivalent DCT-SNN are related to # #ANNops by the layer's spike-rate.
# #DCT-SNN ops, spike rate, x #ANN
where spike rate, is the average number of spikes per neuron per image over all timesteps in layer L. The layerwise spike rates for CIFAR-10 and CIFAR- 100 using DCT-SNN are shown in Fig. The overall average spike rate considering all layers for both cases is well below 5.1 (relative cost of MAC to addition), indicating the energy benefits DCT-SNN over the corresponding ANN. For the DCT-SNN, there additional MAC operations for 2 full precision matrix multiplications in
8

---

Under review as a conference paper at ICLR 2021
Limiting Frequencles in Cycle2 During
VGG9 CIFAR-10 DCT-SNN
100
Inference
Spike Rate
10
A Average Spike Rate: 3.35
80
5
Accuracy(39
C1-16;
C1-16;
C1=16;
C1=16;
C2-8
C2-12
C2-16
C2=4
VGG11 CIFAR-100 DCT-SNN
20
C1-16;
Spike Rate
10
C2-0
Average Spike Rate 2.94
15
20
25
30
35
5
Total Timesteps
Figure 6: Accuracy-latency tradeoff during
FCI FC2
inference; VGG9 trained on CIFAR-IO with all 16 frequencies for 48 timesteps. During inference, cycle1 uses all 16 frequencies, cycle2 uses limited, ordered frequencies.
Layer
Figure 7: DCT-SNN layerwise spike rate. and FC denote Conv and Fully Connected layers.
the preprocessing step (the forward and reverse transforms). We denote these as Encoderape- This computation is needed for only one cycle (16 timesteps), since the other 2 cycles just repeat the same bases and coefficients, The overhead negligible when compared to the number operations over all the layers across all timesteps. We compute the energy benefits of DCT-SNN over ANN, as
EANN
Σn #ANNopsLL ops.L. * 4.6
EDCT-SNN # Encoderops 4.6 + Et #DCT-SNN ops.LL * 0.9
The values of a are 1.52 and 1.74 for VGG9-CIFARIO and VGGLI-CIFARIOO, respectively, demonstrating that DCT-SNN provides improved energy efficiency over its ANN counterpart. Similar to Park et al. (2020), the cost of memory access has not been considered in this evaluation, since it depends on the hardware architecture and system configurations. This efficiency metric a can be further enhanced by reducing the encoder overhead by approximating the DCT transformation with integer transform (IT).. IT can be performed with just shift and add operations. We implemented IT- ba-ed encoding for a VGG9 SNN on CIFAR-10 and obtained 89.2% accuracy with 48 timesteps.
CONCLUSION
Bio-plausible SNNs offer a promising energy-efficient alternative to ANNs. The inputs SNNs are available as spikes over time, instead of analog pixel values. The SNN derives efficiency from the sparsity of spikes per timestep, combined with their event-driven computation. However, the spike distribution over timesteps causes SNNs to have high inference latency. The most widely used Poisson based rate encoding scheme does not encode any meaningful information into the temporal axis of SNNs, and requires a large number of timesteps for inference. To address this, we propose a new encoding scheme that can be used to distribute spatial pixel information over timesteps in ordered fashion. The proposed scheme utilizes a lincar transform in the form of an invertible matrix, with columns that serve as basis of representation distribution. The input pixels are reconstructed over time by summing these bases modulated by the coefficients from the intermediate representation. At each step, we feed in the product of one of these bases and the corresponding DCT coefficient, by order of their contribution, cycle through all bases, the neuron receives the total pixel value spread over all timesteps. The ideal properties of the bases are orthonormality to avoid interference with each other, and ordering by their contribution to reconstruction of the pixel intensities. The choice of the transformation matrix are thus PCA and DCT. We chose DCT due to the frequency bases being dataset-agnostic. We get best performance with 2-D DCT on 4 x 4 blocks of the input, resulting in 16 basis frequencies. We show that passing these 16 bases through the SNN cyclically few times gives DCT-SNN comparable accuracy their ANN counterparts, with less than half the number of timesteps required to learn other sfate-of-the-- art SNNs. Additionally, ranking these bases differentiates one timestep from another. Consequently, and still infer, albeit with lower accuracy. This principled trade-off between inference accuracy and latency is promising direction for deploying SNNs on edge devices.
integrate-and-firg neuron at the input layer. As we
able drop the least important bases (and therefore, timesteps)
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Nasir Ahmed, T. Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):90- 93, 1974.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-leam in networks of spiking neurons. In Advances in Neural Information Processing Systems, pp- 787-797, 2018.
Michael Beyeler, Nikil D Dutt, and Jeffrey I. Krichmar. Categorization and decision-making in neurobiologically plausible spiking network using a stdp-like learning rule. Neural Networks, 48: 109-124, 2013.
Peter Blouw, Xuan Choo, Eric Hunsberger, and Chris Eliasmith. Benchmarking keyword spotting efficiency on neuromorphic hardware. In Proceedings of the 7th Annual Neuro-inspired Computational Elements Workshop, pp. 1-8, 2019.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54-66, 2015.
F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y. Nakamura, P. Datta, G. Nam, B. Taba, M. Beakes, B. Brezzo, B. Kuang, R. Manohar, W. P. Risk, B. Jackson, and D. S. Modha. Truenorth: Design and tool flow of 65 mw 1 million neuron programmable neurosynaptic chip. IEEE Transactions on Computer-Aides Design of Integrated doi: N0.1109/CCAD22055.24177 22444996.
Circuits and Systems, 34( 34(10):1537-1557, 2015
Iulia M Comsa, Thomas Fischbacher, Krzysztof Potempa, Andrea Gesmundo, Luca Versari, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8529 8533. IEEE, 2020.
Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018
Li Deng and Yang Liu. Deep learning in natural language processing. Springer, 2018.
Peter U Dichl and Matthew Cook. Unsupervised learning of digit recognition using spike-timing dependent plasticity. Frontiers in computational neuroscience, 9:99, 2015.
Peter U Dichl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Cho Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference Neural Networks (IJCNN), pp. 1-8. ieee, 2015.
Dony al. Karhunen-loeve transform. The transform and data compression handbook, 1:1-34, 2001.
Max Ehrlich and Larry S Davis. Deep residual learning in the jpeg transform domain. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3484-3493, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 292,
Mark Horowitz.
1.1 computing's 's energy problem (and what International Solid-Stal Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14. IEFE, 2014.
do about it). In 2014 IEEE
10

---

Under review as a conference paper at ICLR 2021
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.0486/1 2017.
Dongsung Huh and Terrence Sejnowski. Gradient descent for spiking neural networks. In Advances in Neural Information Processing Systems, pp. 1433-1443, 2018.
Eric Hunsberger and Chris Eliasmith. Spiking deep networks with lif neurons. arXiv preprint urXiv:151000882. 2015.
Xiping Ju, Biao Fang, Rui Yan, Xiaoliang Xu, and Huajin Tang. An fpga implementation of deep spiking neural networks for low-power and fast classification. Neural Computation, 32(1):182- 204, 2020.
Saeed Reza Kheradpisheh and Timothée Masquelier. S4nn: temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems, 30(6): 2050027, 2020
Jaehyun Kim, Heesu Kim, Subin Huh, Jinho Lee, and Kiyoung Choi. Deep neural networks with weighted spikes. Neurocomputing, 311:373-386, 2018
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097- 1105, 2012.
Chankyu Lee. Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Frontiers Neuroscience, 14, 2020.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. CifarlO-dvs: an event-stream dataset for object classification. Frontiers neuroscience, 11:309, 2017
Da Li. Xinbo Chen, Michela Becchi, and Ziliang Zong. Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus. In 2016 IEEE international conferences on big data and cloud computing (BDCloud), social computing and networking (SocialCom), sustainable computing and communications (SustainCom) (BDCiinnd Seaaaa-oooCoo-Ssssiiiiii pp. 477-484. IEEE, 2016.
Sen Lu and Abhronil Sengupta. Exploring the connection between binary and spiking neural networks. arXiv preprini 2000:44002.1004 2020.
Wolfgang Maass, Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659-1671, 1997.
Emre Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning spiking neural networks. IEEE Signal Processing Magazine, 36:61-63, 2019.
Seongsik Park, Seijoon Kim, Hyeokjun Choe, and Sungroh Yoon. Fast and efficient information transmission with burst spikes in deep spiking neural networks. In 2019 56th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2019.
Seongsik Park, Seijoon Kim, Byunggook Na, and Sungroh Yoon. T2fsnn: Deep spiking neural networks with time-to-fistt-siike coding. arXiv preprint 1rX1v:2003.11741, 2020.
Bulla Rajesh, Mohammed Javed, Shubham Srivastava, et al. Det-compenn: A novel image classification network using jpeg compressed det coefficients. In 2019 IEEE Conference on Information and Communication Technology, pp. 1-6. IEEE, 2019.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2020. htLhttps: //tp:////////e/i.ie////
net:forrumidd=alxssee

---

Under review as a conference paper at ICLR 2021
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- based machine intelligence with neuromorphic computing. Nature, 575(7784):07-611,, 2019.
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu. Michael Piciffer, and Shih-Chii Liu. Conversion of continuous-vai valued deep networks to efficient event-driven networks for image classification. Frontiers neuroscience, 11:682, 2017.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems, pp. 1412-1421, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-158,, 2014.
Andrew Stephan, Brian Gardner, Steven Koester, and Andre Gruning. Supervised learning in temporally-code spiking neural networks with approximate backpropagation. arXiv preprint arXiv:2007. 132966 2020.
Matej Ulieny and Rozenn Dahyot. On using cnn with det based image data. In Proceedings of the 19th Irish Machine Vision and Image Processing conference IMVIP, 2017.
G. K. Wallace. The ipeg still picture compression standard. IEEE Transactions on Consumer Electronics, 38())xxviiixxxx) 1992.
Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 15500115 1990.
Travis Williams and Robert Li. Advanced image classification using wavelets and convolutional neural networks. In 2016 15th IEEE international conference on machine learning and applications (ICMLA), pp. 233-239. IEEE, 2016.
Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for efficient and rapid inference on deep spiking neural networks. arXiv, pp. arXiv 1907, 2019a.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019b.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. Spatio-tempont backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Qi Xu, Yu Qi, Hang Yu, Jiangrong Shen, Huajin Tang, and Gang Pan. Csnn: An augmented spiking based framework with inceptroniinceptio.. In IJCAI, pp. 1646-16665 2018.
Qi Xu, Jianxin Peng, Jiangrong Shen, Huajin Tang. and Gang Pan. Deep covdensesnn: A
hierarchical event-driven dynamic framework with spiking neurons in noisy environment. Neural Networks, 121:512-519, 2020.
Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1740-1749, 2020.
Qiang Yu, Huajin Tang, Kay Chen Tan, and Haizhou Li. Rapid feedforward computation by temporal encoding and learning with spiking neurons. IEEE transactions on neural networks and learning systems, 24(10):1539-1555 2013.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 30(6):1514 15441 2018.
Lei Zhang, Shengyuan Zhou, Tian Zhi, Zidong Du, and Yunji Chen. Tdsnn: From deep neural networks to deep spike neural networks with temporal-coding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 1319-1326, 2019.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 OVERALL TRAINING METHODOLOGY
A.1.1 SPIKING NEURON MODEL
In this work, we employ the bio-plane dio-plausible Leaky Integrate and Fire (LIF) model Hunsberger & Eliasmith 2015), which is described by-
dU
Tm di -U - Urest) + RI, U S V4h
(5)
where U denotes the membrane potential, / is the input current representing the weighted summation of spike-inputs, Tm indicates the time constant for membrane potential decay, R represents membrane leakage path resistance, Via is the firing threshold and Urest is resting potential. The discretized version of Eqn. 5qn. implemeeted in our work given as-
m = Au!- + Σ mijof vano
(6)
1, if u Uth
(7)
0, otherwise
where u the membrane potential, subscripts i and represent the post and pre-neuron, respectively, t denotes timestep, À is the leak constant= the i-th and j-th neuron, o is the output spike, and Vra is the firing threshold. As evident from Eqn. [6 whenever crosses this threshold, it is reduced by the amount UtA, implementing soft-reset. We implement the proposed DCT-SNN using the model described above with our encoding scheme, the code for our work submitted as part of the supplementary material.
Wij represents the weight of connection between
A.1.2 SURROGATE-GRADIESS BASED LEARNING
To train deep SNNs, temporal as well as the spatial credit assignment of errors. Spatial credit assignment is achieved by spatial error distribution across all layers, while for temporal credit assignment, we unroll the network in time and employ backpropagation through time (BPTT) (Werbos 1990). The output layer neuronal dynamics is given as-
surrogate- gradient based backpropagation which performs both the
suroogate-rradiett
m = u
(-1 + Σ Wijoj.
(8)
here the u;s correspond to the membrane potential of i-th neuron of final (L-th) layer. The final layer neurons do not spike, rather we just accumulate their potential over time for classification purpose. These accumulated outputs are passed through a softmax layer obtain the class- wise probability distribution and then the cross cross-entropy between the true output and the network's s predicted distribution is used as loss for backpropagation. The governing equations are-
= Σ yllo((2),
(9)
e
(10)
ΣN 3
where is the loss function, y denotes true output, z is the prediction, T the total number of timesteps and N is the number of classes. The derivative of the loss to the membrane potential of the neurons in the final layer is given as-
aL
ouf
= 2; V/:
(11)
and the weight updates at the output layer are done as-
Wij.1. Wij.L. A.Wigg
(12)
13

---

Under review as a conference paper at ICLR 2021
ÖL Ouf
dL
Σ Ous
Awg. Σ
Σ
(13)
Iw
du! dufz,
dug
aw
where 7) is the learning rate, and wiy.r. is the weight between i-th neuron at layer L and j-th neuron at layer 1-1 at timestep Since the output layer neurons are non-spiking, the non-differentiabiiiy not an issue here. On the other hand, the hidden layer parameter update is given by-
Awyj = E ØL
ÖL d Oul,k E
h 2,3, 2.
(14)
, k. du,, h Owij,
where of, is the spike-generating function (Eqn. 7), k is layer index. We approximate the gradient of this function w.r.t. its input using the linear surrogate grandeetr (Bellec et al. 2018) as-
do =ymar (0,
Mah [}.
(15)
du
where y is hyperparameter chosen as 0.3 in this work.
Algorithm Procedure of spike-based backpropagation learning for an iteration.
Input: pixel-based mini-batch of input (X) target (Y) pairs, total number of timesteps (T), number of layers (L), pre-trained ANN weights (W), membrane potential (U), membrane leak constant (A), array of layer-wise firing thresholds (Vrn), det block-size b, number of freq. component to train with f
Initialize: U; O, v/= =,.....,
M- generate det macoded-inpute for the current mini-batch data for * b timesteps // Forward Phase
for + to do
0 = M[4%/]]; here M[x] denotes the det-encoded inputs sampled from frequency index for < 2 to L do
// membrane potential integrates weighted sum of ppike-inputs
Uf = - +W 0,11 Veh then
if // membrane potential exceeds Via, a neuron fires spike
0 = U U: ViA
else
else, membrane potential decays exponentially = U = X Uf
end if end for
// final layer neurons do not fire - + W₁ 0
end for /calcclate loss, Los-ccross Backward Phase
for /, + 7 to 1 do do
for 1 T, // evaluate partial derivatives of loss with respect to weight by unrolling the network over time
aLoss a0t aut A AW
o0; अU. aw!
end for
end for
A.1.3 WEIGHT INITIALIZATION AND THRESHOLD BALANCING
A key component in successful implementation SNNs is proper initialization of weights and thresholds. As mentioned in section 2 of the main manuscript, first pre- mre-train analogous ANN and copy the weights to the SNN for finetuning. It critical to balance the layerwise neuronal thresholds to achieve satisfactory performance in SNNs. One approach is choose the maximum
14

---

Under review as a conference paper at ICLR 2021
20
80
24
Timesteps
78
87.1
60
50
32
87.5
36
77.6
86.7
88.6
89.1
Frequencies Retained
Figure 8: Training Accuracy with Limited Frequencies.
input to the neurons computed over all timesteps at each layer as the threshold at that corresponding layer (Sengupta al., 2019). However, empirically, have found this scheme to be unstable (training did not converge in some cases due to spike-vanishing in the deeper layers), hence we select the 99.9 percentile of the pre-activation distribution at each layer be that layer's 's threshold. Again, such threshold balancing has been argued to be more robust (Rueckauer et al. 2017). Notably, the threshold computation has to be performed one layer at time and sequentially from first layer to the end. Having initialized the SNN as discussed above, we perform the surrogate- aurooate-gradient based learning, the details of which is depicted in Algorithm 1. In addition, next we also provide the experimental details in appendix section A.2.
A.2 EXPERIMENTAL DETAILS
A.2.1 DATASETS AND MODELS
We perform our experiments on VGG9 for CIFAR1O dataset, VGGI1 for CIFAR100 and VGG13 for Tinylmagener Some comparisons with other encoding schemes are done using VGG5.
A.2.2 TRAINING PARAMETERS
For all datasets, we follow some standard data augmentation techniques such as padding by 4 pixels on each side, and a 32 x 32 crop is randomly sampled from the padded image its horizontally flipped version (with 0.5 probability of flipping). While testing, the original 32 x 32 images are used. Both training and testing data are normalized using 0.5 mean and standard deviation for all channels. For training the ANNs, we use cross-entropy loss with stochastic gradient descent optimization (weight decay=0.0001, mometum=0.9). In the ANN domain, VGG5, VGG9 and VGG11 are trained for a total of 300 epochs, with an initial learning rate of 0.1, which is divided by 10 at each 100-th epoch. VGG13 with TinyImagenet is trained with similar learning rate schedule, but with initial learning rate of 0.01. The ANNs are trained with some architectural constraints avoid significant loss during subsequent ANN-SNN conversion (Diehl et 2015 Sengupta et al. 2019). The ANNs do not have bias terms as it increases the difficulty of threshold balancing. Again, Patch-normalization is not used, rather dropout (Srivastava et al.) 2014) used as the regularizer and constant dropout mask is used across all timesteps while training in SNN domain. Furthermore, average-pooling is used to reduce the feature map size since max-pooling causes significant information loss in SNNs (Diehl et al.] 2015). During training the ANN, the weights are initialized using Xavier initialization (Glorot & Bengio 2010). After conversion, for training in the SNN domain, networks trained for 20-30 epochs with cross-entropy loss and adam optimizer (weight decay=0.0005). Initial learning rate is kept at 0.0001, which is halved every 5 or 6 epochs. The leak constant À is chosen 0.9901 for all simulations.
15

---

Under review as a conference paper at ICLR 2021
A.3 TRAINING WITH CURTAILED FREQUENCIES
We show the effect of training with limited frequencies in Figure g The frequencies are repeated cyclically until the specified number of timesteps. For instance, the accuracy for 8 frequencies given 3 times each (24 timesteps) is 69.7%. We note that during training, there is no benefit to dropping frequencies at iso-latency requirements.
A.4 COMPUTATIONAL COST
The equations for calculating the number of operations in particular layer of an ANN are given by ki x ky X Cin x hout X Wout X Couts
Conv layer
#ANNops ops
Ahn Noutt
Linear layer
where New (kin) denote filter width (height), Can (Cout) is number of input (output) channels, houe (out) is the height (width) of the output feature map, and nin (Nout) is the number of input (output) nodes.
A.5 PERFORMANCE COMPARISON WITH DIFFERENT ENCODING SCHEMES
A.5.1
PERFORMANCE COMPARTSON WITH TEMPORAL ENCODING SCHEMES ON MNIST
Table 4: Accuracy of various temporal encoding schemes on MNIST
Reference
Accuracy($5.
Timesteps
Kheradpisheh & Masquelier
2020
97.4
256
(Comsa et al.
2020)
97.9
not reported
Stephan et al.
2020
85
10
Yu al. 2013)
78
100
Xu et al.
2018)
87
not reported
Beyeler et al.
2013)
91.6
500
This work
98.54
16
This work
86.7
2
This work
97.3
5
To further compare the performance of the proposed DCT-SNN encoding scheme with recent temporal methods on MNIST, we implement it on a shallow network with just 1 hidden layer consisting of 784-100-10 neurons (all fully-connected). The results reported in Table 4 As can be seen, our methods outperforms these recent temporal methods in terms of accuracy and also converges at much lower timesteps.
A.5.2 PERFORMANCE COMPARISON WITH DIFFERENT ENCODING SCHEMES ON CIFAR
To further compare our results with other reported works in terms of timesteps required at iso- accuracy level, we depict the inference curve across different timesteps in Fig. The figure is adopted from Fig. 6 of Park et al.] (2020) and demonstrates the results of "T2FSNN" encoding scheme, which a temporal encoding scheme and other rate and temporal encoding schemes such as "Rate" (Rueckauer et al. 2017). "Phase" (Kim et al.) 2018), and "Burst" coding (Park et al.) 2019. The left graph in Fig. ii is recreated for CIFAR- 10, and shows ~ 200 timesteps for the fastest convergence among these encoding methods, in contrast, we achieve 90% accuracy in just 48 timesteps, saturating far earlier than any of these methods. From Fig. 6 of Park et al. (2020) can tell that the best version of "T2FSNN" first reaches 90% roughly 240 steps, "Burst" at 300, "Phase" " at 425, and "Rate" at 1200 timesteps, showing that we reduce latency by orders of magnitude, resulting in convergence much fewer timesteps. The network is slightly different here,
16

---

Under review as a conference paper at ICLR 2021
CIFAR-10 Accuracy vs Time steps
CIFAR-100 Accuracy vs Time steps
60
Accuracy (96)
60
Plase
Bust
30
Busst
Phace
100
125
150
175
200
500
1000
1500
2000
3000
Time step
Time step
Figure 9: Accuracy versus Latency curve for various coding schemes, the values for TTFS Park
al. 2020 Phase [Kim et al. 2018). Burst (Park al. 2019 and rate (Rueckauer 2017
have been adopted from Park et al. (2020).
ours is VGG9 and the network used in Park et al. 2020 is VGG16, but in opinion, that affects final convergence accuracy more than it affects orders of magnitude of timesteps for inference. Similarly, we exceed 68% accuracy in 48 timesteps when training VGGI1 on CIFAR100. The graph on the right in Fig. / shows the convergence statistics for VGG16 on CIFARI00 using "T2FSNN", "Burst". "Phase" and "Rate". The best version of "T2FSNN" reaches 68% roughly at 500 steps, "Burst" at 1500, "Phase" at 2000, and "Rate" does not go above 60% in even 3000 timesteps.
A.6 TRAINING WITH INTERLEAVED AND INTERMITTENT FREQUENCIES
In this section, we analyze the effect of training with (a) interleaved and (b) intermittent frequencies, instead of 16 frequency components given in a cyclic order.
For the interleaved case, instead of giving input as frequencies 0,1,2, .5,,,,,....15,,,11, 15 we input them as 1,1,0,1,,,1,,,,22,, .2..22......1....., 15,15,555 So, one specific basis repeated for subsequent timesteps before giving the next frequency input. Our original cyclic scheme gave the best reported accuracy of 89.94% and the interleaved scheme only achieved 79.7%. similar cyclic vs interleaving experiment was done with the frequencies limited top 8, repeated for 3 cycles (24 timesteps). The cyclic scheme achieved 69.7% and the interleaved achieved 53.73%. We believe this drop is due the resetting of membrane potential it fires between timesteps, causing temporal dependency to be incorporated between different timesteps and interleaving cannot leverage this dependency. Additionally, since the carlier DCT coefficients contain most of the energy, the spikes at the later timesteps start dying out with interleaved frequencies.
Next, we tried giving intermittent frequencies such as 102.5.7.9.00.122115 given cyclically for cycles (48 timesteps) and got 84.4%, an expected drop from 89.9% with all frequencies for 3 cycles (equivalent 48 timesteps) since we are only giving partial information for reconstruction.
As an additional experiment to re-emphasize that our ordering is beneficial, we give only the top 8 frequencies for the same number of cycles as the previous experiment cycles, 48 timesteps) and get 87.2%, which is 3% better than the scheme with intermittent frequencies, validating the importance of ordering timesteps.
17


Reviews:
[{"review_id": "wABm5UlB8_R", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an encoding method based on the Discrete Cosine Transform (DCT) for Spiking Neural Network (SNN). The key idea is to decompose an image into different frequency components and feed them to the SNN sequentially. Compared to the Poisson coding method used in most SNN studies, the proposed encoding method significantly decreases the latency that the SNN needs for image classification while having minimal accuracy decease. \n \nHighlights:\n\n1. The idea of using DCT for input spike encoding is novel and has great potential. One of the problems that prevent the SNN from using fewer inference timesteps is the ineffectiveness of encoding input information. Using DCT, the method can potentially filter out less important information and more effectively encode the information in limited timesteps (as shown in Fig. 6 and Fig. 8 in the paper).\n\n2. The paper doesn't directly learn in the frequency domain generated from DCT. Instead, it reverse transforms the DCT result back to the spatial domain and spreads it into different timesteps of the SNN. By doing so, the spike encoding gives more importance to the low-frequency information in the image. This is desirable because low-frequency information is more important than high-frequency information in the image for classification.\n \nConcerns:\n\n1. The paper lacks experiments to show that DCT directly contributes to the decrease of timesteps for classification. Although comparisons with earlier SNN works that use Poisson encoding are shown, there is a lack of comparison with any SNN methods that directly convert pixel values into spikes using IF neurons and threshold selection. Thus, the existing experiments are not sufficient to exclude the possibility that the latency decrease is not due to DCT. The reviewer suggests conducting additional experiments for this.\n\n2. While the proposed method only focuses on the input encoding of SNN, many recent papers target new training methods (such as [Jibin Wu et al, 2019], [Sen Lu et al, 2020]) that also result in significant latency decrease of SNN for image classification. The paper lacks experiments to compare the performance with these more recent results. The reviewer suggests conducting additional experiments for this.\n\n3. The paper claims that the proposed method has better performance than ANNs trained on DCT coefficients. However, this is not a fair comparison since their encodings are different. The ANNs trained on DCT coefficients (such as [Max Ehrlich et al, 2019]) follow the same procedure as JPEG compression. The encoding uses non-overlapping 8x8 blocks, and the ANNs directly learn from the JPEG transformed domain. If the paper wants to compare with these ANNs, it needs experiments using the same encoding input.\n\n4. The example in Fig. 1 for the reverse transformation is not the same as the source code (spike_model_vgg9_submit.py: Line 233). In section 3.1, the paper performs inverse transform by doing an element-wise multiplication between the transformed vector Y and each frequency basis in the transformation matrix, and claims the same method generalizes to the 2D case. However, the source code performs inverse transformation using the particular element in the transformed matrix Y (it's now a matrix but not a vector) corresponding to the specific frequency basis. Thus, the explanation in the paper contradicts the implementation. The reviewer thinks the example given in Fig. 1 is mathematically incorrect.\n\n5. The proposed method spreads the reverse transformed image into different timesteps, and each timestep corresponds to a particular frequency. However, the paper doesn't explore other possible approaches for spreading the information. For example, the encoding method can use multiple subsequent timesteps for a particular frequency or only present intermittent frequencies. The lack of in-depth analysis of the proposed method possibly prevents the paper from fully exploring the potential of the use of DCT encoding for SNN.\n\nMinor Comments:\n\n1. What is the meaning of \"ov\" in Fig. 4? The reviewer thinks it means \"overlap\". However, it needs to be explained in the text or figure caption.\n\n2. In Table 2, it's not clear whether DNN-d uses DCT coefficients or the reverse transformed image. If the DNN-d uses the DCT coefficient, is there any change to the ConvNet since the DCT destroys the block's spatial relationships?\n\n3. In Table 2, the SNN-d results for TinyImageNet are missing. Is there any reason for that?\n\n\nJibin Wu et al, 2019, A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks\n\nSen Lu et al, 2020, Exploring the Connection Between Binary and Spiking Neural Networks\n\nMax Ehrlich et al, 2019, Deep residual learning in the jpeg transform domain\n\n Since most of my primary concerns are resolved, I have updated my rating based on the revised version.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jbLUg-MjuOS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1.\tA novel coding scheme is proposed based on Discrete Cosine Transform (DCT) for efficient information expression in place of conventional Poisson distribution method. The required time-steps are 2-14x reduced compared with other conversion-SNNs or hybrid trained SNNs.\n2.\tDCT is data-independent while performing at par with PCA.\nCons:\n1.\tThe experimental results are not convincing enough.\n2.\tCorrectness Problem. I am afraid that the descriptions about the reconstruction of input image is wrong. The reverse transform should be a matrix multiplication instead of Hadamard product of the coefficients and the basis. For example, in Fig 1., X=Y_1*T_1+Y_2*T_2+…+Y_5*T_5. It results in conceptual errors in Fig1 & Fig2.\n3.\tThe specific equation of DCT should be with the discussion of the desirable properties and constraints in the section of encoding scheme to provide a clear picture of the method. \nAbout the experimental results:\n1.\tCould the authors try to provide some explanations for why DCT is able to outperform Poisson method or directly exposing the original image to the input spike neuron, since DCT’s reverse transform is a reconstruction of the original image over time?\n2.\tFewer time-steps with relatively lower accuracy is kind of confusing. I would like that the authors could further show the required time-steps for reaching strictly equal (or better) accuracy results with other SNN works (especially those directly trained). It is because that the trade-off between accuracy and the number of time-steps is natural in SNNs. For example, Rathi et al. (2020) could increase their VGG16’s performance from 91.13% to 92.02% by adding 100 time-steps on CIFAR-10. \nRathi, Nitin, et al. \"Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation.\" arXiv preprint arXiv:2005.01807 (2020).\n\n3.\tI notice that you cite Wu's article in related works, but there is a lack of comparison to it in later experiment part. In my opinion, the results of the paper and its sequel on time-steps for training SNNs from scratch are worth-noticing. I would like a more comprehensive and fair comparison of results.\nWu, Yujie, et al. \"Spatio-temporal backpropagation for training high-performance spiking neural networks.\" Frontiers in neuroscience 12 (2018): 331. \nWu, Yujie, et al. \"Direct training for spiking neural networks: Faster, larger, better.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019. \n\nClarity: The paper is fairly well-written.\nOriginality: DCT is a widely used transformation technique in signal processing and data compression, but this paper creatively explores it as a coding scheme in SNNs.\nSignificance: The proposed coding scheme might provide a new resolution to the high inference latency bottleneck in SNNs.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "e2Eu0Sbdc0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors applied this algorithm into several datasets such as CIFAR 10, CIFAR 100 and TinyImagenet, they argued the image classification accuracies onto these datasets were comparable. The experiments were sufficient but not well designed. The reported experimental results show some novelty and good performance.\n\nIn the early stage of the spiking neural networks, the encoding methods are very important, especially for the training process. As the authors said that the rated based encoding method brings much more time latency which is time consuming. Therefore, the topic of this work is critical. However I have some worries about the proposed methods, from my point of view, this work is just combing the DCT and ANN-SNN method, the novelty is significantly limited, but the idea is interesting. Then, the experimental results reported by this paper were not well designed. The authors argued that the rated based encoding methods are time consuming, but they just did not compare the much more temporal encoding methods in Table 3. And for me, CIFAR10, CIFAR 100 and MNIST are in the same quantity level, which means that you used a VGG net is waste of resource (VGG is too deep). I even did not know what parts of the final results works, the deep CNN based network architecture? The DCT encoding method? Or the ANN-SNN methods. Actually, ANN-SNN method is not a typical bio-inspired way to construct a SNN, I prefer to see you adopt the proposed method into a Tempotron or STDP based learning rule not a surrogate-gradient based rule. The computational efficiency is nice; especially the authors calculated the spike rate of each single layer, but if you just argued the proposed the method is energy consumption, you should at least consider the ANN training process, it is not a single trade-off between inference accuracy and latency. I have run the code from authors provided, the reproducibility is reliable.  \n\nAlso there are some writing errors, such as thy->they, -s, etc. and reference missing, such as these important works:\n1.\tAn FPGA Implementation of Deep Spiking Neural Networks for Low-Power and Fast Classification.\n2.\tDeep CovDenseSNN: A hierarchical event-driven dynamic framework with spiking neurons in noisy environment \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DkON7Q4s4_o", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The scheme proposed breaks down the information in a block of an image into orthogonal basis functions (DCT is used) to make a progressively better reconstruction of the original image block with the addition of more basis functions used (like an nth order Taylor expansion).  The increasing spatial frequency components are known to be perceptually less sensitive (they need to include this) in images, so the low freq components can be presented first.  Each freq component is encoded into spikes sequentially, thereby staging the more perceptually important information first, with less important info coming later.  This reorders the presentation of information to allow a tradeoff of image quality with time/latency.\n\nI think the solution proposed is well-founded and will indeed mitigate the latency problem for spiking neural networks.  However, this feels a bit more like an engineering solution to a specific problem rather than a new concept.  I do like the injection of methods from other fields like image/video compression; it often feels that the deep learning field rediscovers things that have been uncovered years ago in other fields.  I see that as the main value of the paper in addition to helping to make spiking neural networks a POSSIBLE viable solution to edge deployment.  \n\nSection 1: I don’t think it’s a strongly supported claim that deep learning architectures are unsuitable for edge deployment.  There are plenty in deployment and there are new processors (Movidius, Mythic, etc) that can handle these computations for real-time applications.  I’d suggest a softer language there.  This does weaken the motivation for the paper though.\n\nSection 1, second paragraph:  typo: Thy -> The\n\nSection 3.2: On constraints for the transforms.  Did the authors consider Integer Transform (IT)?  This is used in MPEG/AVC.  It is a reversible transform that is an integer simplification of the DCT.  Given that the point of the paper is to decrease latency and computing requirements for edge deployments, this could help.\n\nSection 3.2: The authors do a good job of sweeping performance for different block sizes.\n\nFigure 5:  Isn’t it an obvious result that more time steps are required for Poisson vs DCT?  There simply aren’t enough bins to sum over to have a result until a certain point.  \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a method of encoding inputs to a spiking neural network (SNN) using the discrete cosine transform (DCT). The goal is to create a more energy and time efficient means of doing inference with SNNs. The authors provide a description of the method, then show accuracy results on a variety of standard benchmarks. They also compare to a number of other methods for ANN and SNN based inference in the literature. Altogether, they show that their method allows for accurate inference using fewer spikes than other approaches, which can potentially reduce the energy used for inference.

The paper is fairly clearly written, and the results well articulated. The reviewers had a number of concerns, most notably related to questions of (1) clarity about the actual benefits of this approach, and (2) fair comparisons to other models. Altogether, the authors did try to address the reviewers comments, and at least one reviewer increased their score. 

However, the actual scores for this paper remained very close to the acceptance threshold, and the first point was difficult to rebut without a lot more added to the paper. Ultimately, this paper is using a classic signal processing strategy to improve SNN run time, and the reviewers asked for some reason as to why that is desirable/novel. The author's answer was effectively that SNNs provide promise for low-energy edge computing, and their method could make SNNs for edge computing even more efficient.  This is potentially of interest for edge computing, but the paper could do a lot more to demonstrate that. Specifically, some consideration of how this would actually operate on spiking chips or a more robust estimate of energy efficiency than that given at the end of section 4 would be required to make this paper a clear accept. Notably, the paper does not demonstrate that this technique could be used to significantly reduce the energy requirements for spiking chips, relative to other SNNs, just that this is more energy efficient than ANNs, which is already known for other spiking neural network approaches. Given this, and the scores relative to other papers at ICLR, a "reject" is recommended. However, the AC notes that this was a difficult decision, and this paper was right at the threshold.
Decision: Reject