Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings
Authors: Ilyass Hammouamri
Ismail Khalfaoui-Hassani
Timothée Masquelier
Venue: ICLR 2024 poster (2024)
Keywords: Spiking Neural Networks
Delays
Neuromorphic Computing
Speech Recognition
Primary area: applications to neuroscience & cognitive science
TL;DR: A new method to learn delays with backprop in deep spiking neural networks

Abstract: Spiking Neural Networks (SNNs) are a promising research direction for building power-efficient information processing systems, especially for temporal tasks such as speech recognition. In SNNs, delays refer to the time needed for one spike to travel from one neuron to another. These delays matter because they influence the spike arrival times, and it is well-known that spiking neurons respond more strongly to coincident input spikes. More formally, it has been shown theoretically that plastic delays greatly increase the expressivity in SNNs. Yet, efficient algorithms to learn these delays have been lacking. Here, we propose a new discrete-time algorithm that addresses this issue in deep feedforward SNNs using backpropagation, in an offline manner. To simulate delays between consecutive layers, we use 1D convolutions across time. The kernels contain only a few non-zero weights – one per synapse – whose positions correspond to the delays. These positions are learned together with the weights using the recently proposed Dilated Convolution with Learnable Spacings (DCLS). We evaluated our method on three datasets: the Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC) and its non spiking version Google Speech Commands v0.02 (GSC) benchmarks, which require detecting temporal patterns. We used feedforward SNNs with two or three hidden fully connected layers, and vanilla leaky integrate-and-fire neurons. We showed that fixed random delays help and that learning them helps even more. Furthermore, our method outperformed the state-of-the-art in the three datasets without using recurrent connections and with substantially fewer parameters. Our work demonstrates the potential of delay learning in developing accurate and precise models for temporal data processing. Our code is based on PyTorch / SpikingJelly and available at: https://github.com/Thvnvtos/SNN-delays

Published as a conference paper at ICLR 2024
LEARNING DELAYS IN SPIKING NEURAL NETWORKS USING DILATED CONVOLUTIONS WITH LEARNABLE SPACINGS
llyass Hammouamri CerCo UMR 5549
Ismail Khalfaoui-Hassan
Artificial and Natural Intelligence Toulouse Institute (ANITI)
CNRS Université Toulouse, France i1 yass. hammouamr @cnrs. fr
Université de Toulouse, France
sma khal facui -hassan B-niv-tsee.. fr
Timothée Masquelier
CerCo UMR 5549
CNRS Université Toulouse, France t timothee masqueliergcns.....
ABSTRACT
Spiking Neural Networks (SNNs) are a promising research direction for build- ing power-efficient information processing systems, especially for temporal tasks such as speech recognition. In SNNs, delays refer to the time needed for one spike to travel from one neuron to another. These delays matter because they influence the spike arrival times, and it well-known that spiking neurons re- spond more strongly coincident input spikes. More formally, it has been shown theoretically that plastic delays greatly increase the expressivity in SNNs. Yet, efficient algorithms learn these delays have been lacking. Here, we propose a new discrete- time algorithm that addresses this issue in deep feedforward SNNs using backpropagation, in an offline manner. To simulate delays between con- secutive layers, we use ID convolutions across time. The kernels contain only few non-zero weights one per synapse whose positions correspond the delays. These positions are learned together with the weights using the recently proposed Dilated Convolution with Learnable Spacings (DCLS).. We evaluated method on three datasets: the Spiking Heidelberg Dataset (SHD), the Spik- ing Speech Commands (SSC) and its non-spiking version Google Speech Com- mands v0.02 (GSC) benchmarks, which require detecting temporal patterns. We used feedforward SNNs with two or three hidden fully connected layers, and vanilla leaky antegrate-and-iin neurons. We showed that fixed random delays help and that learning them helps even more. Furthermore, our method outper- formed the state-of-the-art in the three datasets without using recurrent connec- tions and with substantially fewer parameters. Our work demonstrates the tential delay learning developing accurate and precise models for temporal data processing. Our code is based on PyTorch SpikingJelly and available at: ptttp::/gittuue htt.huc.oooomeeeccocmmmmooooeeeo wwwpvttsvvvvoooooeoooooo SNN-deellp delays
INTRODUCTION
Spiking neurons are coincidence detectors (König et al.) 1996 Rossant et al.) 2011): they respond more when receiving synchronous, rather than asynchronous, spikes. Importantly, it the spike ar- rival times that should coincide, not the spike emitting times - these times are different because prop- agation is usually not instantaneous. There is delay between spike emission and reception, called delay of connections, which vary across connections. Thanks to these heterogencous delays, neurons can detect complex spatiotemporal spike patterns, not just synchrony patterns (Izhikevich) 2006) (see Figure T).

---

Published as a conference paper at ICLR 2024
In the brain, the delay of connection corresponds to the sum of the axonal, synaptic, and den- dritic delays. It can reach several tens of milliseconds, but it can also be much shorter (1 ms or less) (Izhikevich] 2006). For example, the axonal delay can be reduced with myelination, which an adaptive process that is required to learn some tasks (see [Bowers] (2017) for a review). In other words, learning in the brain not be reduced to synaptic plasticity. Delay learning is also important.
A certain theoretical work has led the same conclusion: Maass and Schmitt demonstrated, using simple spiking neuron models, that a SNN with k adjustable delays can compute much richer class of functions than a threshold circuit with adjustable weights (Maass & Schmitt 1999).
Finally, on most neuromorphic chips, synapses have programmable delay. This is the case for Intel Loihi (Davies et al. 2018). IBM TrueNorth (Akopyan al. 2015), SpiNNaker (Furber et al.) 2014) and SENeCA (Yousetzadeh et al.) 2022).
All these points have motivated us and others (see related works in the next section) to propose delay learning rules. Here, we show that delays can be learned together with the weights, using backpropa- gation, in arbitrarily deep SNNs. More specifically, we first show that there is a mathematical equiv- alence between 1D temporal convolutions and connection delays. Thanks to this equivalence, we then demonstrate that the delays can be learned using Dilated Convolution with Learnable Spacings (Khalfaoui Hassaani et al. 2023a b). which was recently proposed for another purpose, namely to increase receptive field sizes in non-spiking 2D CNNs for computer vision. In practice, the method fully integrated with PyTorch and leverages its automatic differentiation engine.
Membrane potential
Input spike trains
e Fiiing Thmeshold
d1
f1
dsa
8
f
da.
d2 s 8
82
0
$ T
dx
N₂
dg
Figure 1: Coincidence detection: we consider two neurons N1 and N2 with the same positive synap- tic weight values. N2 has a delayed synaptic connection denoted d21 of Sms, thus both spikes from spike train S₁ and S2 will reach N2 quasi-simuitancouslyy As result, the membrane potential of N₂ will reach the threshold i and N2 will emit a spike. On the other hand, Nj will not react to these same input spike trains.
2 RELATED WORK
2.1 DEEP LEARNING FOR SPIKING NEURAL NETWORKS
Recent advances in SNN training methods like the surrogate gradient method (Netcci et al. 2018
Shrestha & Orchard 2018) and the ANN2SNN conversion methods (Bu et al. 2022: Deng & Gu 2021 Han et al.) 2020) made possible to train increasingly deeper spiking neural networks. The surrogate gradient method defines a continuous relaxation of the non-smooth spiking nonlinearity: it replaces the gradient of the Heaviside function used in the spike-generating process with smooth surrogate gradient that is suitable for optimization. On the other hand, the ANN2SNN methods convert conventional artificial neural networks (ANNs) into SNNs by copying the weights from ANNs while trying to minimize the conversion error.
Other works have explored improving the spiking neurons using inspiration from biological mech- anisms techniques used in ANNs. The Parametric Leaky andgrate-and-Fin (PLIF) (Fang al.) 2021a) incorporates learnalle membrane time constants that could be trained jointly with synaptic weights. [ellec et al. (018) were the first to propose method for dynamically adapting firing
2

---

Published as a conference paper at ICLR 2024
thresholds in deep (recurrent) SNNs, Hammouamri et al. (2022 also proposes method dynam- ically adapt firing thresholds in order to improve continual learning in SNNs. Spike- Spike-Element Wise ResNet (Fang et al. 2021b) addresses the problem of vanishing/exploding gradient in the plain Spik- ing ResNet caused by sigmoid-like surrogate functions and successfully trained the first deep SNN with more than 150 layers. Spikformer (Zhou al., (023) adapts the softmax-base self-attention mechanism of Transformers (Vaswani et 2017) to a spike-based formulation. Other recent works like SpikeGPT (Zhu et al., (023) and Spikingformer (Zhou al. 2023) also proposes spike-based transformer architectures. These efforts have resulted in closing the gap between the performance
of ANNs and SNNs on many widely used benchmarks.
2.2 DELAYS IN SNNS
Few previous works considered learning delays in SNNs. Wang et al. (2019) proposed similar method to ours in which they convolve spike trains with an exponential kernel so that the gradient of the loss with respect the delay can be calculated. However, their method is used only for shallow SNN with no hidden layers.
2023 Zhang al. 20200) Taherkhani et al. (2015) Other methods like Grimaldi Perrinet 2022
also proposed learning rules developed specifically for shallow SNNs with only one layer. Hazan al. 22022) proposed to learn temporal delays with Spike Timing Dependent Plasticity (STDP) in weightless SNNs. Han et al. (2021) proposed method for delay-weight supervised learning in op- tical spiking neural networks. SSuccdo et al. (2023) proposed method for deep feedforward SNNs that uses set of multiple tixed delayed synaptic connections for the same two neurons before pruning them depending on the magnitude of the learned weights.
To the best of our knowledge, SLAYER (Shrestha & Orchard 2018) and Sun et al. [2022] 2023b a) (which are based on SLAYER) are the only ones to learn delays and weights jointly in a deep SNN. However, unless Spike Response Model (SRM) (Gerstner 1995) is used, the gradient of the spikes with respect to the delays is numerically estimated using finite difference approximation, and we think that those gradients experiments with fixed random delays (see Table and Figure 4
precise enough as achieve similar performance in
We propose control test that not considered by the previous works and that we deem necessary: the SNN with delay learning should outperform an equivalent SNN with fixed random and uniformly distributed delays, especially with sparse connectivity.
3 METHODS
3.1 SPIKING NEURON MODEL
The spiking neuron, which is the fundamental building block of SNNs, can be simulated using various models. In this work, we use the Leaky Integrate-and-F -ire model (Gerstner & Kistler 22002) which is the most widely used for its simplicity and efficiency. The membrane potential u, of the i-th neuron in layer follows the differential equation:
du: dt
- (1: (1) Ureses) RT,"((
where is the membrane time constant, Ureset the potential at rest, R the input resistance and ((( (L) the input current of the neuron at time I. addition to the sub-threshold dynamics, a neuron emits unitary spike si") when its membrane potential exceeds the threshold 19, after which it is instantaneously reset to ureset- Finally, the input current I(0h (t) is stateless and represented as the sum of afferent weights w!()) multiplied by spikes s/ s/(-1)((): (t):
(0) - w/ 5/ (c)
3

---

Published as a conference paper at ICLR 2024
We formulate the above equations in discrete time using Euler's method approximation, and using Ureset and R = т.
" (1 Juu 1 ju
40
(3)
[1 = Σ w -1
(4)
s/ @(u(t) 0)
(5)
We use the surrogate gradient method (Nffci et al. 2018 and define @'(r) _ o'xx) backward step, where a (x) the surrogate aretangent function (Fang et al. 2021a)
during the
Input spike traine
Equivalent Keme! Representation
d
S:
wi,d1
0
0
0
MTQ
I=
Time
Figure 2: Example of one neuron with 2 afferent synaptic connections, convolving K1 and K2 with the zero left-padded S₁ and S2 is equivalent to following Equation
3.2 SYNAPTIC DELAYS AS TEMPORAL CONVOLUTION
In the following, for clarity, assume one synapse only between pairs of neurons (modeled with a kernel containing only one non-zero element). Generalization to multiple synapses (kernels with multiple non-zero elements) is trivial and will be explored in the experiments.
feed-forward SNN model with delays is parameterized with R+, where the input of neuron i at layer is
(wg) E IR and D (d
(1) (((-1) d!9 w T = Σ
(6)
We model synaptic connection from neuron j in layer 1 - 1 to neuron in layer which have synpatic weight w.y and delay d') as one dimensional temporal convolution (see Figure 2
with
kernel (1() as follows:
Vn E [O.... ... Ta 1]:
( 1
[n]
w
(1)
if Ta
otherwise
where Ta is the kernel size or maximum delay + 1. Thus we redefine the input 1!"h in Equation 6 as sum of convolutions:
1 = Σ
((1-1)
We used a zero left-padding with size Tą on the input spike trains so that [[]] does correspond to Moreover, zero right-padding could also be used, but is optional; it could increase the
4

---

Published as a conference paper at ICLR 2024
expressivity of the learned delays with the drawback of increasing the processing time as the number of time-steps after the convolution will increase.
the ID version of DCLS (Khalfaoui- Hassami et al. 2023a) with a Gaussian kernel Satafaoui-Hassanne et al.] 2023b) centered at Ta
To learn the kernel elements positions (i.e., delays),
1, where d [0, Ta - II, and of standard deviation oij E IR" thus we have:
vn E [0.... Ta 1]:
(1) Wij ij c
n Ta + 1
+
[n]
exp
(1)
(9)
With
n1 Ta + 4
+
C +
exp
(10)
n=0
a normalization term and fl Float:32 precision. During training, (1) are clamped after every batch to ensure their value stays in [O. Ta-10 I]]
le
to avoid division by zero, assuming that the tensors are in
The learnable parameters of the 1D DCLS layer with Gaussian interpolation the weights way. the corresponding delays dij. and the standard deviations Øij. However, in our case, Øsj are not learned, and all kernels in our model share the same decreasing standard deviation, which will be denoted as Throughout training, we exponentially decrease as our end goal is to have sparse kernel where only the delay position non-zero and corresponds the weight.
The Gaussian kernel transforms the discrete positions of the delays into a smoother kernel (see Figure 5) which enables the calculation of the gradients BL
By adjusting the parameter a, we can regulate the temporal scale of the dependencies. A small value for a enables the capturing of variations that occur within brief time frame. In contrast, larger value of a facilitates the detection of temporal dependencies that extend over longer durations. Thus, tuning crucial to the trade-off between short-term precision and long-term dependencies.
We start with high value and exponentially reduce it throughout the training process, after each epoch, until it reaches its minimum value of 0.5 (Fig. 33. This approach facilitates the learning of distant long-term dependencies at the initial time. Subsequently, when has a smaller value, it enables refining both weights and delays with more precision, making the Gaussian kernel more similar to the discrete kernel that is used at inference time. As we will see later in our ablation study (Section 4.3) this approach outperforms a constant o.
Indeed, the Gaussian kernel is only used to train the model; when evaluating the validation or test set, it is converted discrete kernel as described in Equation by rounding the delays. This permits to implement sparse kernels for inference which very useful for neuromorphic hardware, for example, as they correspond to only one synapse between pairs of neurons, with the corresponding weight and delay.
4 EXPERIMENTS
4.1
EXPERIMENTAL SETUP
We chose to evaluate our method on the SHD (Spiking Heidelberg Digits) and SSC (Spiking Speech Commands)/GSC (Google Speech Commands v0.02) datasets (Cramer et al. (022). as they require leveraging temporal patterns of spike times to achieve a good classification accuracy, unlike most computer vision spiking benchmarks. Both spiking datasets are constructed using artificial cochlear models to convert audio speech data to spikes; the original audio datasets are the Heidelberg Dataset (HD) and the GSC v0.02 Dataset (SC) (Warden 2018) for SHD and SSC, respectively.
The SHD dataset consists of 10k recordings of 20 different classes that consist spoken digits ranging from nine in both English and German languages. SSC and GSC are much larger
5

---

Published as a conference paper at ICLR 2024
(e) Delays
( Intermediate delays
(c) delays
Final rounding
66
0.00
Figure 3: This figure illustrates the evolution of the same delay kernels for example of eight synaptic connections one neuron throughout the training process. The x-axis corresponds to time, and each kernel is of size Ta 25. And the y-axis is the synapse id. (a) corresponds to the initial phase where the standard deviation of the Gaussian or is large (2 allowing to take into consideration long temporal dependencies.
(b) corresponds to the intermediate phase, (c) taken from the final phase where a is at its minimum value (0.5) and weight tuning is more emphasized. Finally, (d) represents the kernel after converting to the discrete form with rounded positions.
datasets that consist of 100k different recordings. The task we consider on SSC and GSC is top one classification on all 35 different classes (similar to Cramer et al. (2022); Bittar & Garner (2022)), which more challenging than the original key-word spotting task on 12 classes, proposed in Warden (2018).
For the two spiking datasets, we used spatio-temporal bins to reduce the input dimensions. Input neurons were reduced from 700 to 140 by binning every 5 neurons; as for the temporal dimension, we used a discrete time-step At = 10 ms and zero right-padding to make sure all recordings in batch have the same time duration. As for the non-spiking GSC, we used the Mel Spectrogram representation of the waveforms with 140 frequency bins and approximately 100 timesteps to remain consistent the input sizes used in SSC.
We used a very simple architecture: a feedforward SNN with two or three hidden fully connected layers. Each feedforward layer is implemented using a DCLS module where each synaptic con- nection is modeled as ID temporal convolution with one Gaussian kernel element (as described in Section 3.2). followed by batch normalization, LIF module (as described in Section 3.] and dropout. Table lists the values of some hyperparameters used for the three datasets (for more details, refer the code repository).
Table I: Network parameters for different datasets
Dataset
# Hidden Layers
# Hidden size
(ms)
Maximum Delay(ms)
Dropout rate
SHD
2
256
10.05*
250
0.4
SSCIGSC
or
512
15
300
0.25
#We found that LIF with quasi-instantaneous leak 7 10.05 (since Heaviside function for SHD.
10) is better than using

---

Published as a conference paper at ICLR 2024
The readout layer consists of classes LIF neurons with infinite threshold (where Relasses is 20 or 35 for SHD and SSC/GSC, respectively). Similar to Bittar & Garner (022). the output out, for every neuron at time t is
(
out [t] = softmax(u," (()
(11)
Tichassx j=1
where (1()) [t] is the membrane potential of neuron i in the readout layer r at time t.
The final output of the model after 7 time-steps is defined
out [t]
(12)
We denote the batch size by N and the ground truth by y. We calculate the cross-entropy loss for one batch as
N N
C
log(softmax(0v. [n:])
(13)
The Adam optimizer (Kingma & Ba 2017) is used for all models and groups of parameters with base learning rates Ir_ 0.001 for synaptic weights and Ira 0.1 for delays. We used one-cycle learning rate scheduler (Smith & Topin. 2018) for the weights and cosine annealing (Loshchilov & Hutter 2017) without restarts the delays learning rates. Our work is implemented using the Py -baseh SpikingJelly(Fang et al. 2020 2023) framework.
4.2 RESULTS
We compare our method (DCLS-Delays) in Table previous works on the SHD, SSC, and GSC- 35 (35 denoting the 35 classes harder version) benchmark datasets terms of accuracy, model size, and whether recurrent connections or delays were used.
The reported accuracy of method corresponds to the accuracy the test set using the best- performing model on the validation set. However, since there is no validation set provided for SHD use the test set the validation set (similar BBttar & Garner (2022)). The margins of error are calculated at 95% confidence level using t-distribution (we performed ten and five experiments using different random seeds for SHD and SSC/GSC, respectively).
Our method outperforms the previous seate-of-the-ae accuracy on the three benchmarks (with significant improvement on SSC and GSC) without using recurrent connections (apart from the self- recurrent connection of the LIF neuron), with a substantially lower number of parameters, and using only vanilla LIF neurons. Other methods that use delays do have a slightly lower number of param- eters than do, yet outperform them significantly on SHD, while they didn't 't report any results on the harder benchmarks SSC/GSC. Finally, by increasing the number of hidden layers, we found that the accuracy plateaued after two hidden layers for SHD and three for SSC/GSC. Furthermore, we also evaluated a model (Dense Conv Delay) that uses standard dense convolutions instead of the DCLS ones. This corresponds conceptually having fully connected SNN with all possible delay values as multiple synaptic connections between every pair of neurons in successive layers. This led worse accuracy (partly due to overfitting) than DCLS. The fact that DCLS outperforms a standard dense convolution, although DCLS is more constrained and has fewer parameters, is remarkable.
4.3 ABLATION STUDY
In this section, we conduct control experiments aimed at assessing the effectiveness of our delay learning method. The model trained using our full method will be referred to as Decreasing (specifically, we use the 2L-IKC version), while Constant a will refer a model where the standard deviation a is constant and equal to the minimum value of 0.5 throughout the training. Additionally,
Our code is available http:///ggthhbo..oo////oo///////

---

Published as a conference paper at ICLR 2024
Table 2: Classification accuracy on SHD, SSC and GSC-35 datasets
Dataset
Method
Rec.
Delays
#Params
Top1 Acc.
EventPro--GeNN (Nowotny et al.
2022)
N/a
84.80 10.80++11.5 1.5%
Cuha-IIIF (Dampfhoffer et al. 2022
X
0.14M
87.80+
Adaptive SRNN Vin et al. 2021)
X
N/a
90.40%
SNN+Delays "Patino-Saucees et al.
2023
X
V
0.IM
90.43%
SHD
TA-SNN Yao et 2002T
X
X
N/a
91.08%
X
STSC-SNN Yu et 2022
X
2.1M
92.36%
Adaptive Delays (Sun el al.
2023b
X
0.IM
92.45%
D1.188-NNNDloss (Sun
2023a
X
0.14M
92.56%
Dense Conv Delays (ours)
X
2.7M
93.44%
RadLIF Bittar & Garner
2022
3.9M
94.62%
DCLS-Delays (2L-IKC)
0.2M
95.07-10.249% %
Recurrent SNN (Cramer et al.
2022
N/a 50.90 1.1%
SSC
Heter. RSNN (Perez Nieves al
2021
N/a
57.30%
SNN-CNV CNN Sadovsky al. 2023
N/a
72.03%
Adaptive SRNN (Yin et al. 202
N/a
74.20%
SpikGRU (Damphhofter al.
2022
0.28M
77.00 77.00%00.4%
RadLIF Bittar & Gamner 2022
3.9M
77.40%
Dense Conv Delays 21. (ours)
10.9M
77.86%
Dense Conv Delays 3I (ours)
19M
78.44%
DCLS-Delays (2L-1KC)
0.7M
79.77=1.099 %
DCLS-Delays (2L-2KC)
X
1.4M
80.16:10.09 %
DCLS-Delays (3L-1KC)
1.2M
%
DCLS-Delays (3L-2KC)
2.5M
80.69 %
MSAT [e 2023)
X
X
N/a
87.33%
GSC-35
Dense Conv Delays22 2L (ours)
X
10.9M
92.97%
Dense Conv Delays 3L (ours)
X
19M
93.19%
Radl II (Bittar & Gameer 2022
X
1.2M
94.51 %
DCI.SDellas (2I,-1KC)
X
0.7M
94.91 0.09 %
DCLS-Delays (2L-2KC)
EXX
1.4M
95.00-10.06 %
DCLS-Delays (3L-1KC)
1.2M
95.29 0.11 %
DCLS-Delays (3L-2KC)
X
V
2.5M
95.35 04%
nL-mKC stands for model with hidden layers and kernel count where kernel count denotes the number of non-zero elements the kernel. "Rec." denotes recurrent connections.
Fixed random delays will refer to model where delays are initialized randomly and not learned, while only weights are learned. Meanwhile, Decreasing a Fixed weights will refer to model where the weights fixed and only delays learned with decreasing a. Finally, No delays denotes a standard SNN without delays. To ensure equal parameter counts across all models (for fair comparison). we increased the number of hidden neurons in the No delays wider case, and increased the number of layers instead in the No delays deeper case. Moreover, to make the comparison even fairer, all models have the same initialization for weights and, if required, the same initialization for delays.
We compared the five different models as shown in Figure 4a or learned) significantly outperformed the No delays model both on SHD (FC) and SSC (FC); for us, this was an expected outcome given the temporal nature of these henchmarks, as achieving a high accuracy necessitates learning long temporal dependencies. However, we didn't expect the Fixed random delays model be almost on par with models where delays were trained, with Decreasing model only slightly outperforming it.
The models with delays (whether fixed
To explain this, we hypothesized that a random uniformly distributed set of delay positions will likely cover the whole temporal range. This hypothesis is plausible given the fact that the number of synaptic connections vastly outnumbers the total possible discrete delay positions for each kernel. Therefore, the number of synaptic connections within layer grows, the necessity of moving
8

---

Published as a conference paper at ICLR 2024
Decreaiin Fied we gegs
1.0
Decrecting d
Canaten:
Reed randum delays
No dehys
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0 SHD
0.0
SSC
SHD
SSC
(a) FC: Fully Connected
(b) S: Sparse connections
Figure 4: Barplots of test accuracies on SHD and SSC datasets for different models. With (a): fully connected layers (FC) and (b): sparse synaptic connections (S). Reducing the number of synaptic connections of each neuron to ten for both SHD and SSC.
delay positions away from their initial state diminishes. And only tuning the weights of this set of fixed delays is enough to achieve comparable performance to delay learning.
In order to validate this hypothesis, we conducted comparison using the same models with a sig- nificantly reduced number synaptic connections. We applied fixed binary masks to the network's synaptic weight parameters. Specifically, for each neuron in the network, we reduced the number of its synaptic connections to ten for both datasets (except for the No delays model, which has more connections to ensure equal parameter counts). This corresponds to 96% sparsity for SHD and 98% sparsity for SSC. With the number of synaptic connections reduced, it is unlikely that the random uniform initialization of delay positions will cover most of the temporal range. Thus, specific long- term dependencies will need to be learned by moving the delays.
The test accuracies corresponding to this control test shown in Figure 4b It illustrates the difference in performance between the Fixed random delays model and the Decreasing/Constant o models in the sparse case. This enforces our hypothesis and shows the need perform this control test for delay learning methods. Furthermore, it also indicates the effectiveness of our method.
In addition, we also tested model where only the delays are learned while the synaptic weights are fixed (Decreasing a Fixed weights). It can be seen that learning only the delays gives acceptable results in the fully connected case (in agreement with Grappolini & Subramoney (2023) but not in the sparse case. To summarize, it is always preferable to learn both weights and delays (and decreasing a helps). one has to choose, then learning weights is preferable, especially with sparse connectivity.
5 CONCLUSION
In this paper, we propose a method for learning delays in feedforward spiking neural networks using dilated convolutions with learnable spacings (DCLS). Every synaptic connection is modeled as a ID Gaussian kernel centered on the delay position, and DCLS is used to learn the kernel positions (i.c. delays). The standard deviation of the Gaussians is decreased throughout training, such that at the end of training, we obtain SNN model with one discrete delay per synapse, which could potentially be compatible with neuromorphic implementations. We show that our method outperforms the state-of-the--rt in the temporal spiking benchmarks SHD and SSC and the non-spiking benchmark GSC-35 while using fewer parameters than previous proposals. Finally, we also perform a rigorous control test that demonstrates the effectiveness of our delay learning method. Future work will investigate the use of other kernel functions than the Gaussian or applying our method to other network architectures like convolutional networks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This research was supported in part by the Agence Nationale de la Recherche under Grant ANR- 20-CE45-0005 BRAIN-Net. This work was granted access to the HPC resources of CALMIP su- percomputing center under the allocation 2023-(P22021J. Support from the ANR-3IA Artificial and Natural Intelligence Toulouse Institute is gratefully acknowledged. We also want to thank Wei Fang for developing the SpikingJelly framework that we used this work.
REFERENCES
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, Brian Taba, Michael Beakes, Bernard Brezzo, Jente B. Kuang, Rajit Manohar, William P. Risk, Bryan Jackson, and Dharmendra S. Modha. TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neu- rosynaptic Chip. IEEE Transactions
Computer-Aided Design of Integrated Circuits and Sys- tems, 15100)55337-1555 oct 2015. ISSN 0278-0070. doi: 10.1 109/TCAD.2011.247744..... URL http://Leeexplore ieee.iieeee... org/ /ttp///cmmmmmt/2222/2
Guillaume Bellec, Darjan Salaj. Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. In Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https::/proceediing/ www.ine.r.iccoc cttps/s.cpppppppeppppe files//aper/2018/ htlee/2220Baa5111222222224442222 6634
Alexandre Bittar and Philip N. Garner. surrogate gradient spiking baseline for speech com- mand recognition. Frontiers in Neuroscience, 16, 2022. ISSN 1662-453X. doi: 10.3389/fnins.
2022.865897, URL htthttss: hwtphtts://www.eootote http://wwww.o.iriiinnnocc http://wwwoooaaaaacacc.aaaaaaaaa 3389///1.333////
2022. 865897
Jeffrey Bowers. Parallel Distributed Processing Theory in the Age of Deep Networks. Trends in Cognitive Sciences, pp. 1-12, 2017. ISSN 13646613, doi:
URL
el httpa//w.....//////eaee////////.
Tong Bu, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang. Optimal ANN-
SNN conversion for high-accuracy and Altra-low-lateney spiking neural networks. In Interna-
tional Conference on Learning Representations, 2022. URL httpt::t/////tttt https///epeeeeeee.eommmm
Cerum?id=7B3. IJMM1K
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33(7):2744-23 2757, 2022. doi: 10.1109/TNNL. 2000.. 3044364.
Manon Dampthoffer, Thomas Mesquida, Alexandre Valentian, and Lorena Anghel. Investigating current-based and gating approaches for accurate and energy-effciient spiking recurrent neural networks. In Elias Pimenidis, Plamen Angelov, Chrisina Jayne, Antonios Papaleonidas, and Mehmet Aydin (eds.), Artificial Neural Networks and Machine Learning - ICANN 2022, pp- 359-370, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-11934-33
Mike Davies, Narayan Srinivasa, Tsung Han Lin, Gautham Chinya, Prasad Joshi, Andrew Lines, Andreas Wild, and Hong Wang. Loihi: A Neuromorphic Manycore Processor with On-Chip Learning, 2018. ISSN 02721732.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representattions 2021. htts: //openreviee ew net./ wtt.m/orrmmmmmmmmmm
oTwcXchK]
W. Fang, Z. Yu, Y. Chen, T. Masquelier, T. Huang, and Y. Tian. Incorporating learnable mem- brane time constant to enhance learning of spiking neural networks. In 2021 IEEE/CVF In- ternational Conference Computer Vision (ICCV), pp. 2641-2651, Los Alamitos, CA, USA, oct 2021a. IEEE Computer Society. doi: 10.1109/CCC449922..2.22 2026000.... URL https:
7/doi. lecccoppeersooollt..ooooo....... 201.:0110000000192222222202..220
10

---

Published as a conference paper at ICLR 2024
Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Timothée Masque- lier, Yonghong Tian, and other contributors. Spikingjelly. https://githhu com/
Eangwe1123456/ http///5///ikiiigee1.... 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In M. Ranzato, Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp. 21056-21069. Curran Associates, Inc., newripsu.r..ee
2021b. URL htttps://prccedd hwtp://s.ww.........pp..p.a..aee fi wtwes//aper//00// filc//58445553322222000888821133
Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(4)), oct 2023. doi: 10.1126/sciadv.adi 1480.
http:://ww..ccienne...roo/...... htt.:///......e.oo//.../////1///
adi 1480
Steve B. Furber, Francesco Galluppi, Steve Temple, and Luis A Plana. The SpiNNaker Project. Proceedings of the IEEE, 102(5):652 665, may 2014. doi: 10.1 10.1199/PRC.2204.220000 https: http:://eeeeeppleeeee leee org/ /tt.//mmmttt55002222222
URL
Wulfram Gerstner. Time structure of the activity in neural network models. Phys. Rev. E, 51:738- 758. Jan 1995. doi: 10.1 10.1103/PhysRev.5511.7 URL attt:p::/iinx..ass.rro///// L103/PhysRevE.51.
Wulfram Gerstner and Werner M. Kistler. Spiking Neuron Models: Single Neurons, Populations, Plasticity. Cambridge University Press, 2002. doi: (2.1017/C09978011111177777
Edoardo W. Grappolini and Anand Subramoney. Beyond weights: Deep learning in spiking neural networks with pure synaptic-delay training, 2023.
Antoine Grimaldi and Laurent U Perrinet. Learning hetero-synaptic delays for motion detection in single layer of spiking neurons. In 2022 IEEE International Conference Image Processing (ICIP), pp. 3591-3595, 2022. doi: 10. 10.11000CC446777222229999993
Antoine Grimaldi and Laurent U Perrinet. Learning heterogeneous delays in layer of spiking neurons for fast motion detection. Biological Cybernetics, 2023. doi: io/
10. 10.1007/000422-0244007---8 23-bc/
URL https::/laarrenterriitt..t.i
llyass Hammouamri, Timothée Masquelier, and Dennis George Wilson. Mitigating catastrophic forgetting in spiking neural networks through threshold modulation. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL htt.tppss//ppeeeevieee...
B. Han, G. Srinivasan, and K. Roy. Rmp-snn: Residual membranc potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In 2020 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13555-13564 Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10.1 1LL11999CVPR44220220000000111
URL https://
doi.
01357
Yanan Han, Shuiying Xiang, Zhenxing Ren, Chentao Fu, Aijun Wen, and Yue Hao. Delay-weight plasticity-based supervised learning in optical spiking neural networks. Photon. Res., 9()): B119 1119-B127,40 Apr 2021. doi: 10.1364/PRJ.417722. /R/httppp://ooo.ppiiccaaa. abst ract
Hananel Hazan, Simon Caby, Christopher Earl, Hava Siegelmann, and Michael Levin. Memory temporal delays in weightless spiking neural network, 2022.
Xiang He, Yang Li, Dongcheng Zhao, Qingqun Kong, and Yi Zeng. Msat: Biologically inspired multi-stage adaptive threshold for conversion of spiking neural networks, 2023.
11

---

Published as a conference paper at ICLR 2024
Eugene M Izhikevich. Polychronizzation: computation with spikes. Neural Comput. 18(2):245- 282. feb 2006. doi: 10.116//099966666755598888. URL htLhtt:://// doi. oog.org//o...... 01897660677509881
Ismail Caaufaauu-Hasssan Thomas Pellegrini, and Timothée Masquelier. Dilated convolution with learnable spacings. In The Eleventh International Conference Learning Representations, 2023a. URL httpss////peeeeee..eooom net. -=
Ismail Khalfaoui- Hassani, Thomas Pellegrini, and Timothée Masquelier. Dilated convolution with learnable spacings: beyond bilinear interpolation, 2023b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 2017.
König, K Engel, and Singer. Integrator or coincidence detector? The role the cortical neuron revisited. Trends Neurosci, 19(4):130-7., 1996.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017.
Wolfgang Maass and Michael Schmitt. On the Complexity of Learning for Spiking Neurons with Temporal Coding. Information and Computation, 153(1):26 1999. ISSN 08905401. doi: 10. 1006//n/oc.. I 1999.2806.
Emre Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks. IEEE SIGNAL PROCESSING MAGAZINE, 1053(5888/18), 2018.
Thomas Nowotny, James Turner, and James C. Knight. Loss shaping enhances exact gradient learning with eventprop in spiking neural networks, 2022.
Alberto PatifioSSaueeoo Amirreza Yousefzadeh, Guangzhi Tang, Federico Corradi, Bernabé Linares-Barranee and Manolis Sifalakis. Empirical study on the efficiency of spiking neural networks with axonal delays, and algorithm-harde hardware benchmarking.
2023 IEEE Interna- tional Symposimm on Circuits and Systems (ISCAS), pp. 1-5, 2023. doi: 10.1 10911//SSAS4 2023.10188 1018778
Nicolas Perez-Nieve Nieves, Vincent C. H. Leung, Pier Luigi Dragotti, and Dan F. M. Goodman. Neu- ral heterogeneity promotes robust learning. Nature Communications, 12(1):5791, 2021. ISSN 1721- doi: 10.
URL https://doi org 10. 1038/
841467-021-26023
Cyrille Rossant, Sara Leijon, Anna K Magnusson, and Romain Brette. Sensitivity of noisy neurons to coincident inputs. The Journal of Neuroscience, 31(47):17133-206, nov 2011. ISSN 1529- 2401. doi: 10. 11.22//NEUURSSC2222222 pubmed/ 22114286
11.2011. URL BtLbhtp::/// icbi. nlm. nih gov/
Erik Sadovsky, Maros Jakubec, and Roman Jarina. Speech command recognition based on con- volutional spiking neural networks. 2023 33rd International Conference Radioelektronika TRADIOELEKTRONIKA) pp. 1-5, 2023. 10109082.
doi: 10LIOORADDDOLLEKTOOOKKKKSSSSS..
Sumit Bam Shrestha and Garrick Orchard. SLAYER: Spike layer eтτor reassignment in time. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gareett (eds.), Advances in Neural Information Processing Systems 31. pp. 1428. Curran Associates, Inc., 2018. URL Sas-ssslarr-sppeeeee wwwpiik--laye-eeeeee eayer-ero gnment- in time. .mm..dff.
nips cc/paper/////cpp
Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates, 2018.
Pengfei Sun, Longwei Zhu, and Dick Botteldooren. Axonal delay as a short-term memory for feed forward deep spiking neural networks. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8932-8934 2022. doi: 10.1109/ ICASSP49222202.977711.
12

---

Published as a conference paper at ICLR 2024
Pengfei Sun, Yansong Chua, Paul Devos, and Dick Botteldooren. Learnable axonal delay in spiking neural networks improves spoken word recognition. Frontiers in Neuroscience, 17, 2023a. ISSN 1662-453X. doi: 00.3389/iins.2223.2754444
URL https:/ http://wwwwwwt..eeetceeeee
org/
Articles/1n 3t..el/s.......11/....
33899/889/f1 2023. 1275944
Pengfei Sun, Ehsan Eqlimi, Yansong Chua, Paul Devos, and Dick Botteldooren. Adaptive axonal
delays in feedforward spiking neural networks for accurate spoken word recognition, 2023b.
Aboozar Taherkhani, Ammar Belatreche, Yuhua Li, and Liam P. Maguire. DI-resume: A delay learning-based remote supervised method for spiking neurons. IEEE Transactions on Neural Networks and Learning Systems, 26(12):3137331 2015. doi: 10.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/proceedd.ogg/ wtw.://cceedoooo..enneoooo.eoooe www.:neeuiiiccooo cc/paper Satpaapeerfleeepppeeeeeee files/paper/2017/ File/3f5ee 2155ee223547deeee fbdo cccce
Xiangwen Wang, Xianghong Lin, and Xiaochao Dang. A delay learning algorithm based on spike train kernels for spiking neurons. Frontiers in Neuroscience, 13, 2019. ISSN 1662-453X. doi: 2ttp://minnn.29..00222000
URL https::/wwwt
10. 33893339//5 fnins/fnis....m 2019. 00252
Pete Warden. Speech commands: dataset for limited-vocabulle speech recognition. arXiv, 2018.
M. Yao, H. Gao, G. Zhao, D. Wang, Y. Lin, Z. Yang, and G. Li. Temporal-wise atten- tion spiking neural networks for event streams classification. In 2021 IEEF/CVF Interna- tional Conference on Computer Vision (ICCV) pp. 10201-10210, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society. doi: 10. 10110001CCC4499220000..000 URL https: 7/ /ooii
2021..2......00 01006
Bojian Yin, Federico Corradi, and Sander M. Bohte. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelli- gence, 2021.
doi: 10.101144256-021--03997.4 URL hattps://di..o/et
1038/
Amirreza Yousefzadeh, Gert-Jan van Schaik. Mohammad Tahghighi, Paul Detterer, Stefano Tra- ferro, Martijn Hijdra, Jan Stuijt, Federico Corradi, Manolis Sifalakis, and Mario Konijnenburg.
SENeCA: Scalable Energy-efficient Neuromorphic Computer Architecture. In 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp. 371-374. URL
IEEE, jun 2022. ISBN 978-1-6654-09964
doi: BLLL0//AACA5548222222290000 https: wwweepeeeeeeeeeeeee leee org/ http///mmmtt/97702533/3///
Chengting Yu, Zheming Gu, Da Li, Gaoang Wang. Aili Wang, and Erping Li. Stsc-sun: Spatio-temporal synaptic connection with temporal convolution and attention for spiking neu- ral networks. Frontiers in Neumscience, 16, 2022. ISSN 1662-453X. doi: 10.3389/ URL httpps://ww/ Prontierssii wtt.rrss....ggrtreeeeeeeeeeeee/e 3389/
fnins.2022. wtt.22002..179357. 1079357.
Enins 2022. 1079357
Malu Zhang. Jibin Wu, Ammar Belatreche, Zihan Pan, Xiurui Xie, Yansong Chua, Guoqi Li, Hong Qu, and Haizhou Li. Supervised learning spiking neural networks with synaptic delah- plasticity. Newrocomputing, 409:103-118, 2020. ISSN 0925-2312. doi: https://doi.rr//10.0000
079. URL htRhttts::// http://wwwwwwcccccieeeeeeee wtt acience/articles pi 211/0992223122034555
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and
Li Yuan. Spikformer: When spiking neural network meets transformer. In The Eleventh Interna- tional Conference Learning Representations, 2023. URL https: / http//pprre//e/eeeeeeeeeeeee rewiew. nnt./
Forum?1d=frE4fUW.
Rui-Jie Zhu, Qihang Zhao, and Jason K. Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks, arXiv preprint 2rX.v:2302.33990 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
SUPPLEMENTARY FIGURE
Input spike trains
Convolution Kernel
K2 S
f11
0G2
futtd
82
8xt
12
Eau 4-
*
Sw
Exaa
T
T2 dg
typttdy
T
Time
Ta
Time
Figure 5: Gaussian convolution kernels for N synaptic connections. The Gaussians centered the delay positions, and the area under their curves corresponds to the synaptic weights wy. On the right, we see the delayed spike trains after being convolved with the kernels. (the -1 was omitted for figure clarity).
14


Reviews:
[{"review_id": "L7MgfnQcI0", "reviewer": "Reviewer_Wbur", "summary": "In this paper, the authors use the previously published Dilated Convolution with Learnable Spacings (DCLS) method to learn delays in a deep feed-forward spiking neural network using back-propagation. They demonstrate this method on various temporal tasks such as spiking Heidelberg dataset and versions of google speech commands. The authors also demonstrate that learning delays contributes to an increase in performance in sparse networks.", "questions": "## Suggestions:\n\n- The DVS gesture recognition dataset, due to its event-based nature, might have been a really good fit for a method that learns delays.\n- Since delays use temporal information, it might have made more sense to use a loss function that made use of this (for e.g. time-to-first-spike loss)\n\n### Minor:\n- Acronyms for task names are not explained in the results section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Learning delays, and more generally, using temporal information is a very relevant topic.\n- The paper is generally well written and the experiments and setup are clearly described.\n- The improvement of performance in networks with fixed sparsity when delays are included is very interesting and this analysis is novel.", "weaknesses": "- The novel contribution of this paper over the DCLS paper is not clear. Is it just the evaluation on multiple tasks? It is very important to clarify this aspect.\n- The comparison of the model with delays versus no-delays in Sec. 4.3 may not be completely fair: Using more layers (with same number of parameters) for the no-delay network seems more comparable.\n- The statement of \"Here we show for the first time that delays can be learned together with the weights, using backpropagation, in arbitrarily deep SNNs.\" is not true. (Shrestha & Orchard 2018) do exactly that.\n- Some of the related work are incorrectly cited or not cited:\n    - The SLAYER paper (Shrestha & Orchard 2018) does train the delays along with the weights but the authors don't mention it in this context (although it is cited in a different context).\n    - dynamically adapting firing thresholds for deep (recurrent) SNNs was first proposed in (Bellec et al. 2018)\n    - Spike based transformer references should include SpikeGPT (Rui-Jie et al. 2023) and Spikingformer (Zhou, Chenlin, et al. 2023)\n\n(Shrestha & Orchard 2018) Shrestha, S.B., and Orchard, G. (2018). SLAYER: Spike Layer Error Reassignment in Time. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds. (Curran Associates, Inc.), pp. 1412–1421.\n\n(Bellec et al. 2018) Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., and Maass, W. (2018). Long short-term memory and Learning-to-learn in networks of spiking neurons. In Advances in Neural Information Processing Systems 31, pp. 787–797.\n\n(Rui-Jie et al. 2023) Zhu, Rui-Jie, Qihang Zhao, and Jason K. Eshraghian. \"Spikegpt: Generative pre-trained language model with spiking neural networks.\" arXiv preprint arXiv:2302.13939 (2023).\n\n(Zhou, Chenlin, et al. 2023) Zhou, Chenlin, et al. \"Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Network.\" arXiv preprint arXiv:2304.11954 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHNDSBss1E", "reviewer": "Reviewer_sQsE", "summary": "The paper presents a way to learn synaptic (or axonal) delays in spiking neural networks (SNNs), where the delay of each synapse is realized as a discretized kernel of temporal convolution with a single non-zero element. An evaluation of classification accuracy on three temporal datasets shows that the method works well, with the authors claiming to surpass the state of the art in those datasets.", "questions": "Could the weaknesses be addressed? Most importantly, could the paper better clarify its significance in the broader field of neural networks? Changes and additions to the text might help address the issues somewhat, but missing experimental evaluations should ideally also be performed, or other measurements of any possible advantage claimed, e.g. number of parameters, energy efficiency etc.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "When evaluated within the context of SNNs alone, the paper offers several strengths. Namely, the method is relatively original, the evidence that the method works well is rather convincing, and the impact on SNNs could be significant, given the relative ease of implementation and effectiveness. SNNs themselves interest a growing community.", "weaknesses": "The main weakness of the paper is common to many works on SNNs. Specifically, the significance, novelty, potential impact, and experimental validation are limited to the narrow field of SNNs themselves. Very rarely does an SNN paper show its advantages in the broader literature on neural networks, let alone in the real world. The present manuscript too, when evaluated in a broader scope, suffers from the same issues.\n\nMore concretely:\n- the method is only new for SNNs, but not for neural networks in general.\n- the performance is claimed to surpass the state of the art (already in the abstract), but the authors do not actually compare with the true state of the art, including non-spiking networks.\n- there is no experimental comparison with standard (i.e. less constrained) temporal convolutions.\n\nTherefore, it is unclear what the true contribution of the work is, beyond a nice conceptual analogy between temporal convolutions and synaptic delays.\n\n\nSecondary weaknesses:\n- Even within spiking networks, the work does not seem to surpass the state of the art, contrary to the authors' claims. In [1], a partly spiking neural network reached 95.6% on the GSC v0.02, where the authors report 95.35% at most. The manuscript does not cite that prior work.\n- The paper does not motivate sufficiently the choice of spiking neurons as a model. A paragraph explaining the advantages of SNNs *in comparison with the true state of the art, i.e. ANNs*, supported with citations that demonstrate them measurably, such as energy efficiency, but also rarely in other metrics such as speed of inference and training [1] and even classification accuracy [2]. Any other arguments and citations that the authors can add to support that choice would be useful.\n- The authors claim that there is no recurrency in their models, but a leaky integrate-and-fire neuron's leak membrane potential is equivalent to a self-recurrent connection. I understand what the authors mean, but, again in the spirit of appealing to the broader ICLR community and not only to the SNN niche, this should be clarified.\n\n[1] Jeffares et al., Spike-inspired rank coding for fast and accurate recurrent neural networks, ICLR 2022\n\n[2] Moraitis et al., Optimality of short-term synaptic plasticity in modelling certain dynamic environments, arXiv 2021\n\n----------------------------------\nEDIT (adding my responses here too, for public visibility):\n\n----------------------------------\nThe authors' response dedicates a large section to address points that I did not make. To correct the record I must unfortunately reply to that section too, even though it is merely a distraction.\n\nNowhere did I claim that SNNs are not important or not a legitimate research direction, or that the entire field deserves rejection. I did not dismiss the paper on the basis of it being an SNN. I did point out that some of its weaknesses are frequent in the SNN literature, but pointing that out does not make those weaknesses irrelevant to this specific review. The attempt by the authors to entirely dismiss my review based on how many SNN papers per year are published and how many good reviews the paper received is an attempt to evade my specific criticisms. Worse, the aggressive style of the authors' response, and the misconstrual of my arguments as if they were a personal matter of mine is not helpful.\n\nAgain, SNNs can certainly have important advantages, and some SNNs do have them, but a neural network merely being implemented with spiking neurons does not guarantee these benefits. An SNN paper must be evaluated as any other paper, and not merely be accepted as a significant contribution because the network is spiking.\n\nDespite this attempt to discount my comments, I continue my contribution to this process in a separate comment.\n\n----------------------------------\n\nSome important weaknesses remain.\n\n- The key method that the authors used is not new, only its application is.\n\n- The so-far evaluation does not suffice to compare with other works:\n(a) Two of the three used datasets have received very little if any attention outside of the SNN literature.\n(b) Only feedforward architectures, with only 2 or 3 layers, have been tested.\n(c) Only spiking networks have been tested, so it is unclear whether the same results could be achieved, for example, with much smaller (and thus possibly more efficient) non-spiking networks.\n\n- The paper is missing a sufficient motivation of SNNs as a model. A paragraph with the potential benefits of SNNs should be added, citing the previously demonstrated improvements in efficiency, inference speed, and even classification accuracy, but it should also explain that these benefits are not present in all SNNs by default. Examples of such references were given in my original review.\n\n\n**On the other hand**, the paper now does include a comparison with a more standard method, i.e. conventional temporal convolutions, and it does outperform it. Of course, the work already was a good contribution to the SNN field, but this addition makes it now a relatively convincing demonstration of the power of learned delays more generally, that is a also useful result for the broader ICLR community. Based on these, I am raising my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9i9GiOyu8s", "reviewer": "Reviewer_ybas", "summary": "This paper proposes to study an important problem in spiking neural networks, which involves the explicit incorporation of propagation delays between different neurons in the network. This is an important problem that has been addressed regularly in recent years, and the paper provides a novel and simple solution based on a temporal convolution parameterized by a certain precision. The delays are learned using a variant of the surrogate gradient method, and numerical simulations demonstrate the learning of delays by this method. Experimental results show very good performance on traditional community datasets (in particular, a top score in the leaderboard for the Spiking Heidelberg dataset) and also demonstrate a certain robustness of the method when certain connections are pruned.", "questions": "What is the influence of the meta-parameters on the obtained performance? The influence of the characteristic time of the membrane potential would be interesting to study, as it corresponds to a kind of regularization of spike precision.\n\nCould you comment on the fact that \"We found that a LIF with quasi-instantaneous leak τ = 10.05 (since ∆t = 10) is better than using a Heaviside function for SHD.\" ? Would such a difference matter in biology?\n\nConcerning \"We used a one-cycle learning rate scheduler (Smith & Topin, 2018) for the weights and cosine annealing (Loshchilov & Hutter, 2017) without restarts for the delays learning rates. \": Could you comment on your choice of learning rate schedulers? Would different schedulers significantly alter our results? Or does it just improve learning speed?\n\nMinor:\n- complete reference for Kingma, for Warden. There seems to be a newer one by Grimaldi for \"Learning heterogeneous delays\" instead of \"Learning hetero-synaptic delays\" - plus an additional application paper on motion detection by the same authors. \n- spacing: \"weights.Hammouamri et al. (2022)\"\nThe LaTeX formatting of the paper is excellent but could be further enhanced. In Figure 1, utilize \"N_2\", \"S_1\", and other symbols for clarity. Some citations in the text (\"Spike-Element-Wise ResNet Fang et al. (2021b) \", ...) should be enclosed in parentheses, e.g. using `citep`. Text \"reset\" appearing in equation (1) should be formatted as text, e.g.  using the `\\text{}` formatting.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper effectively introduces the problem and motivation and presents the methods clearly. A major strength of the paper is the model's relative mathematical simplicity and its successful performance in supervised classification on two datasets. Promising experimental results indicate that significant energy savings can be achieved with the application of such models to neuromorphic chips by demonstrating the network's robustness when connections are removed.", "weaknesses": "The paper's connections with related works are satisfactory; however, it could benefit from presenting neuroscientific evidence on the plasticity of neural delays in biology. Additionally, it lacks a discussion on the relationship between the model's parameters and those observed in biology. For instance, the maximum delays utilized are around 250 milliseconds (300 milliseconds for SSC), while delays used in Izhikevitch's polychronization model are around 20 milliseconds. Furthermore, the paper does not establish any predictions made by the model that can be experimentally observed in biology.\n\nThe model has several limitations, such as the use of discrete time, a forward propagation training model, or a limited number of computational layers. However, the presented performance of the network validates the decisions made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNPwefDiwT", "reviewer": "Reviewer_7RCo", "summary": "As far as we know, the plastic delays greatly increase the expressivity in SNNs. However, efficient algorithms to learn these delays have been lacking. In this manuscript, the authors propose a new discrete-time algorithm that addresses this issue in deep feedforward SNNs using backpropagation (i.e., offline manner). Then, the kernels contain only a few non-zero weights – one per synapse – whose positions correspond to the delays. Thus, these positions are learned together with the weights using the dilated convolution with learnable spacings (DCLS). The authors show us in a practical way that building deep SNNs can learn together with fixed delays and weights.", "questions": "Please look at the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The gap between theory and practice is opened up, especially an efficient fixed delays with weights learning algorithm is designed.\n2. The effects of delays can be well explained by visual examples.\n3. The anonymous open source code is shared with readers, and the detailed implementation helps inspire readers to build complex and deep SNNs.", "weaknesses": "1. The reviewers are very concerned about the innovation of the structure, despite the effort that went into achieving such a particularly efficient discrete-time learning algorithm. The reviewer noted these sentences: \"The trick is to simulate delays using temporal\nconvolutions and to learn them using the recently proposed Dilated Convolution with Learnable Spacings (Khalfaoui-Hassani et al., 2023a;b). In practice, the method is fully integrated with PyTorch and leverages its automatic-differentiation engine.\" So can we say that structurally this contribution is just a combination that happens to work. This contribution would be improved if the author could further clarify the motivation or give a more solid analysis. After all, a trick feels like an inadequate contribution.\n2. Just two datasets with similar statistic information may not seem sufficient, and it would be better if the authors had time to supplement the experiment with a new dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)