WaveSense: Efficient Temporal Convolutions with Spiking Neural Networks for Keyword Spotting
Authors: Philipp Weidel
Sadique Sheik
Venue: ICLR 2022 Submitted (2022)
Keywords: spiking
keyword spotting
temporal processing
streaming
audio
neuromorphic
wavenet
wavesense
always-on
low-power
temporal convolution

Abstract: Ultra-low power local signal processing is a crucial aspect for edge applications on always-on devices. 
Neuromorphic processors emulating spiking neural networks show great computational power while fulfilling the limited power budget as needed in this domain.
In this work we propose spiking neural dynamics as a natural alternative to dilated temporal convolutions. We extend this idea to WaveSense, a spiking neural network inspired by the WaveNet architecture.
WaveSense uses simple neural dynamics, fixed time-constants and a simple feed-forward architecture and hence is particularly well suited for a neuromorphic implementation.
We test the capabilities of this model on several datasets for keyword-spotting. The results show that the proposed network beats the state of the art of other spiking neural networks and reaches near state-of-the-art performance of artificial neural networks such as CNNs and LSTMs.

Under review as a conference paper at ICLR 2022
WAVESENSE: EFFICIENT TEMPORAL CONVOLUTIONS WITH SPIKING NEURAL NETWORKS FOR KEYWORD SPOTTING
Anonymous authors Paper under double-blind review
ABSTRACT
Ultra-low power local signal processing is a crucial aspect for edge applications on always-on devices. Neuromorphic processors emulating spiking neural networks show great computational power while fulfilling the limited power budget as needed in this domain. In this work we propose spiking neural dynamics as natural ternative dilated temporal convolutions. We extend this idea WaveSense, spiking neural network inspired by the WaveNet architecture. WaveSense uses simple neural dynamics, fixed time-constants and simple feed-forward architec- ture and hence particularly well suited for a neuromorphic implementation. We test the capabilities of this model on several datasets for keyword-spotting The results show that the proposed network beats the state of the art of other spiking neural networks and reaches near state-of-the-an performance of artificial neural networks such CNNs and LSTMs.
INTRODUCTION
Local signal processing is important component of the computational pipeline for Internet-of- Things (IoT) devices equipped with a range of sensors like audio, video, and motion sensing. A significant range of these sensors capture signals comprising temporal features. Ideally these features need to extracted by an on-board processor before decision making or relaying the pre- processed information for further computation. Processing temporal signals is often computationally challenging and requires large amounts of memory and power, especially in always-on scenarios. Neuromorphic (Mead (990) processors with spiking- neural networks have shown promise in this domain as ultra-low power compact solutions Indiveri et al.) (201T); Benjamin et al. (0114); Merolla al. 2014); Furber al. (2014): Davies (2018); Liu et (2019).
In this work we propose an elegant way of implementing temporal convolutions in spiking neural networks by leveraging their inherent synaptic and membrane dynamics. Based on this idea we propose WaveSense, a Spiking Neural Network (SNN) model suitable for efficient neuromorphic implementations while retaining high accuracy on temporal data streams. This work bridges the performance gap between Artificial Neural Networks (ANNs) and SNNs for temporal tasks. Crucially, the proposed model
accepts spike streams and not 'buffered frames' input,
requires no delays in its connectivity,
utilizes very simple spiking neuron model Leaky Integrate and Fire (LIF) neuron without the need of any additional adaptive mechanisms,
does not require recurrent connectivity (which can often be difficult to tune or train),
achieves high classification performance.
Recently several works have shown how to build efficient SNNs with accuracies equivalent to ANNs Diehl et al. (2015); Rueckauer et al. (2017). In these studies, spiking neurons are used rate mode with equivalent response curves ReLU activations, to transfer weights from pre-trained ANNs to SNNs. This approach therefore completely neglects the temporal capabilities spiking neurons. On the other hand, surrogate gradient methods enable directly training SNN using Back Propagation

---

Under review as a conference paper at ICLR 2022
Through Time (BPTT) Neftci al. (2019). (Shrestha & Orchard 2018) for instance show temporal processing on temporal gesture recognition task and show a good performance on a visual task. Similar approaches have already been investigated
audio tasks [ellee et al.] (2018); ; Wu al. (2019); Cramer et al. (2020). (Bellec et al., 2018) demonstrate classification results on the TIMIT dataset using long time constants, complex learning strategy (Deep-R) and require significant computational resources to train. (Wu et al., 2019) train SNNs for automatic speech recognition tasks in a tandem approach with an ANNs. This training pipeline integrated language model and pronunciation model which goes beyond the capabilities of the neuromorphic system. The same authors showed in an earlier study the capabilities of an SNN in combination with self-organized map to learn to recognize digits using the TIDIGIT dataset Wu et al. (2018).
(Blouw et al. 2018) demonstrate high accuracy in a audio classification task using dense networks with spectrograms inputs. This approach requires passing the frequency data of previous time steps (defined by the spectrogram time window) in every sample presentation to the network. (Kugele) et al. 2020) show that by matching ANN roll-out delays to the propagation delays in SNN, the resultant networks can demonstrate a high accuracy on vision-based spatio-temporal classification tasks. Implementation of delays in neuromorphic hardware requires additional memory resources to store and deliver spikes in a delayed fashion and could be potentially quite expensive. (Yin) et al. 2020) use Spiking Recurrent Neural Network (SRNN) architecture and demonstrate that by utilizing adaptive LIF neurons and learning the time constants, these networks can perform temporal classification tasks in sequential manner fairly well. The authors demonstrate the effective use of spiking neural networks and show a significant increase in power efficiency. Unfortunately. having fine tuned time constants in low-power neuromorphic hardware can often be challenging especially while using fixed precision numerical representations and computations.
We propose a novel network architecture for SNN that does not require buffering or delays and can directly process temporally varying streams of spiking data from event-based sensors using simple LIF neurons. Our architecture is derived from first principles and inspired by the WaveNet van den Oord et al. (2016) architecture that does not necessitate learning the time constants of the system but could be defined as the task demands. In addition we also propose an efficient training strategy and a corresponding loss-function that is suitable for streaming based models, in particular models that could be run in real-time with neuromorphic hardware.
The first key aspect of the WaveNet architecture is the use of multi-layer causal dilated convolutions. The causal refers to the use of data from the past, dilated refers to a sparse kernel and the convolution is along the time axis. Stacking such convolutional operations along multiple layers enables the network to have a long temporal memory. A second aspect is that it eliminates the need for sliding window based inference/prediction and minimizes the number of computations within the network when operating on a continuous stream of data.
The WaveNet architecture is very amenable general purpose proccesoocessssssc and micro-controllers but it still requires a reasonable amount of memory and non-linear computations such as tanh and sigmoid which are fairly complex (within the context of ultra-low power devices) and this in turn requires higher energy and power requirements. Neuromorphic technology promises to bring the energy required for these tasks down by utilizing SNNs to perform ultra-low power computation. So far, while neuromorphic devices have been demonstrated to operate at extremely low power, they have fallen short at demonstrating computational performance that is on- on-par or comparable to state-of-the-art ANNs for temporal tasks, most prominently in the audio domain.
The WaveSense model proposed here aims to bridge this gap. For the results in this work, we focus on audio tasks as spatio-temporal tasks without loss of generality. Sec. details all the methods used for audio data pre- pre-prceessing conversion to spikes and the details of the network architecture. In Sec. 3 we demonstrate the computational capability of this network over several audio datasets for key-word spotting tasks. We compare our results to the state-of-the-an SNNs and ANNs We conclude in Sec. A where discuss the implications and impact of this work and potential areas where this work could be utilized.
Most importantly all the code used to generate the reported results have been made open-source and can be found at http:: h /XXXXXXXXXXX We believe this will enable the research community to explore other avenues that could take advantage of our work.
2

---

Under review as a conference paper at ICLR 2022
2 MATERIALS AND METHODS
2.1 DILATED TEMPORAL CONVOLUTIONS
B
output OOCO COOOOO
d=8
hidden COOO
d=4
hidden
C
d=2
hidden
tau=3
d=1
input
Figure 1: Relegating the job of delays in dilated convolutions to synaptic dynamics.
Temporal convolutions enable a layer of neurons to integrate information from the past. The dilation enables convolution over large time window (or sample length) while still only using small number of parameters. Temporal dilated convolutions therefore, perform of information from different points in time, separated by the dilation parameter. This is done by storing previous activations in ANNs. Naively, the equivalent could be achieved in SNNs by utilizing synaptic transmission delays. this work, we observe that neuron and synaptic dynamics could be seen proxies for temporal convolutional processing shown in Figure Figure [ shows the contributions of each projection with a kernel size 2 when implemented with delays. Implementing synaptic transmission delays in real-time neuromorphic hardware incurs steep overhead in terms of memory and computational resources and only a limited number of neuromorphic devices support them.
We instead propose to use an appropriate set of synaptic time constants т, as shown in Figure DC. While quantitatively, these are different, qualitatively both these approaches provide the ability transform and project information in the temporal domain Sheik et al. (2012). This is the key insight we leverage design our final model.
2.2 NETWORK DESCRIPTION
We take inspiration from the work of (Coucke et al. 2018) who uses the WaveNet architecture van den Oord et al. (2016) for classification of continuous audio streams. The WaveNet architecture provides prescription for distributing temporal memory and computation across layers without repeated presentation of previous input data.
The original ANN WaveNet model comprises of few different computational building blocks. We translate each of these building blocks to SNNs as follows.
Dilated Temporal Convolutions mentioned above are implemented by synaptic projections with multiple time constants.
Rectified Linear Unit (ReLU) activations can be approximated by spiking neurons because spiking neuron can only produce spikes if the membrane potential crosses a threshold and silent otherwise Diehl et al. (2015).
Non-linear activations like tanh and sigmoid cannot be efficiently translated to SNNs. So we choose to replace these activations with SNNs activations (potentially ignoring the benefits of filtering and gating). In the original model, these two activations are preceded by two sets of weights, where in SNN only
weighted projection. (See Figure 2
Residual connections and summation (+) are realized by synaptic connection.
The WaveSense model is built upon these building blocks shown in Figure 2] It comprises of several 'blocks', each which comprises three spiking neuron layers. The first spiking layer receives inputs filtered by two separate synapses with different time constants 7, and weights. The

---

Under review as a conference paper at ICLR 2022
time constants of the slow projections in each of these blocks are chosen such that they span range of values relevant to the task. The number of blocks chosen such that the sum of all these time constants is proportional to the temporal memory demanded by the task. This layer projects to the second spiking layer. Additionally third spiking layer in each of these blocks projects to "hidden" layer followed by non-spiking 'low pass' readout layer. The output of this block is the summation of its input (residual connection) and the output of the second layer. These 'blocks' are connected in a feed forward manner. The non-spiking 'low-pass" (LP) layer simply acts low pass filter on the spikes of the 'hidden' layer. This is equivalent to the synapse of a spiking neuron (without the neuron's membrane potential or the spiking dynamics) and does not require any extra components unavailable to spiking neurons on typical neuromorphic platform. The choice of leaving the output layer to be non-spiking is to enable a smooth, continuous valued readout useful for faster learning using BPTT.
weighted
Residual
Block
Spike!
Skip
connections
Linear
Linear
Thresholded Output
Linear
Spike
Linear
LP
Spike
Linear
Multi-synapse projection
Single synapse projection
Spike!
Linear
Input
Figure 2: The WaveSense model prescribed in this work is theoretical adaptation of the WaveNet architecture vvan den Oord et al. (2016) based on first principles.
2.3
DATASETS
In order to evaluate the efficacy of the proposed model, we train and test it against several open-source publicly available audio datasets.
2.3.1 ALOHA DATASET
The Aloha dataset Blouw et al. (2018) is small collection audio samples containing the keyword Aloha' and several distractors such 'take load off". As the dataset is very small, only 2000 samples, we augmented the samples using the MUSAN noise dataset Snyder et al. (2015). For that end we standardized the sample length of each utterance the training and validation seconds and added randomly selected background noise data with a signal-to-nn -onii- ratio (SNR) of 5 dB the training data.
five
2.3.2 HEY SNIPS DATASET
The 'Hey Snips' dataset Coucke et al. (0018) for wake phrase spotting distinguishes between two classes. The positive class contains 11 100 utterances from over 2'000 speakers of the wake phrase 'Hey Snips' while the negative (or distractor) class contains over 86'000 negative examples from more than 6'000 speakers. We split the data into training-, validation- and test-set provided by
4

---

Under review as a conference paper at ICLR 2022
the authors of the dataset. We standardized the sample length of each utterance in the training and validation set to five seconds. As the dataset is already very large, no noise augmentation was needed.
2.3.3 SPEECH COMMANDS DATASET
The Speech Commands Warden (0188) describe dataset containing 35 keywords uttered in total 1000 times from over 2'600 speakers. The keywords contain the numbers 0 10, commands such as "stop", "go". "Ieft" and "right" as well as other words like "Marvin", "Sheila", etc. This dataset was initially designed for keyword spotting in a limited vocabulary and the intended experiment is to detect 10 commands (plus silence) out of all 35 keywords (12 classes in total). Nevertheless, there studies training models and showing results on all 35 classes Cramer et al. (2020). We augment the training set with noise data from the MUSAN dataset using an SNR of 5 dB just as we do for the Aloha dataset.
2.4 PRE-PROCESSING
The raw audio data is pre-processed in several stages:
Noise augmentation The training data is augmented with noise from the MUSAN noise dataset using SNR of 5dB (except for the HeySnips data).
Length standardization and pre-amplification The noise augmented waveform is cut into a standard length (dependent on the dataset) and the amplitude normalized.
Band-pass filters The audio then passed through 64 Butterworth bandpass filters 2nd order. The bandpass filters are distributed in Mel-scale between 100Hz and 8kHz.
Rectification The response of the 64 bandpass filters is rectified using full-wave rectifier.
Spike conversion and binning The output of the rectifier is applied as direct input to the membrane of 64 simplified LIF neurons resulting in rate code. The spike trains are binned into 10ms timesteps allowing multiple spikes per timestep.
Audio Data
Audio waveform
Pre-amplification
Noise
Spike conversion
Rectification
Band-pass filters
Figure 3: Data pre-processing pipeline figure.
2.5 TRAINING METHOD
In order to train the parameters of the SNN (See Sec. 2.2) BPTT. In particular we aim to be able to deploy the network in streaming mode i.e. the model receives the data stream directly generated from a sensor without any frame-based (sliding window) buffering. This requires to employ an appropriate loss function.
Often in classification task, the output class can be determined by computing cross-entropy loss the sum the outputs over the sample length for each output neuron. While this would yield good classification accuracy, the magnitude of the output trace at given point time' is not indicative of the network prediction. This is not ideal for models being run in streaming mode.
2.5.1 PRAK LOss
Typically for streaming models, a signal is predicted as belonging to a certain class when the corresponding output trace exceeds 'detection threshold'. This approach is also ideal for always-on
5

---

Under review as a conference paper at ICLR 2022
pre-processing
WaveSense
Figure 4: An example visualization of the peak loss with peak times to for channels and 1.
neuromorphic systems. We therefore design our loss function to reflect this detection mechanism and train our neural networks. We determine the peaks of the output traces and use only the activation values at the peaks to compute the cross entropy loss (see Figure 4 similar to max-over-time loss Cramer et al. (2020).
Consequently the loss computed as follows:
IIc = Σ Aelog(pe)
(1)
where Å, yields 1 if class label corresponds to the current input and 0 otherwise, Pe is the prediction probability by the neural network that current input belongs class e. calculated by a sofimax operation shown below.
Sc
Pe Σ, e8
where & are the 'logits' produced by the neural network.
For temporal tasks, the input x²' xI...T and the output (logits) of the neural network time-series over time T.
8' fx=10=
where is the transformation of the neural network, O the network parameters and s is the internal state of the network at time t. In peak-loss we pass the peak of each output trace to the softmax function. The peaks are calculated follows:
y. max(y.) = ye"
(4)
where 1* argmax(y?) is the "peak time', the time of maximal activation of output trace (see Figure
2.5.2 SPIKING ACTIVITY REGULARIZATION
The activity LIF neurons can change dramatically during the learning process. It can either lead to the absence of spikes which stalls learning in exploding activation which results in high energy utilization of the network in a neuromorphic implementation.
In order to limit the activity of these neurons and maintain sparse activity, include activity regularizer term in our loss function (Soraao et al. (2020).
Loct
(Nt spae/ (T Nneurms)))
(5)
where the activation loss Lact is dependent on the total excess number of spikes N spt produced by the network with a population size N Nneurons in response a input of length T time steps. Nf. given as:
apk is

---

Under review as a conference paper at ICLR 2022
Table 1: Aloha result model size and resource comparison. #Parameters
Publication
#Neurons
Accuracy
Blouw et al. 2018)
541
172800
95.8
This work
864
18482
98.0 = 1.1
Nt spk
= ΣΣ NOON -
(6)
is the sum of spikes from all neurons N; exceeding 1 in each time bin (+ is a heaviside function). Finally the loss function given as:
L = LCE + abeet
where a was chosen to be 0.01.
RESULTS
In order to validate and verify that sufficient information from the input is retained after pre- pre-procsssigg and conversion to spikes, we train a state-of-the-ee WaveNet classifier on the datasets considered in this work and check that can obtain high accuracy. We implement a non-spiking dilated Convolutional Neural Network (CNN) to replicate the WaveNet architecture very similar to that described in (Coucke et al. 2018 van den Oord et al., 2016) (see Section 2.2 for details).
We train this ANN on the HeySnips. Aloha and SpeechCommands datasets and compare our results to those reported in literature Coucke et al. (2018); Blouw et al. (2018): Cramer et al. (2020). The results obtained from this network are then used as baseline to evaluate the performance of the proposed SNN.
3.1 ALOIIA DATASET
In order compare our model other SNN implementations in the keyword spotting domain, we trained our WavseSense on the Aloha dataset Blouww et al. (2018). Table shows the memory resources of the proposed model comparison to the work demonstrated With an average accuracy of 98.0% with standard deviation of 1.1%, the model presented this work performs significantly better while at the same time requiring a significantly fewer parameters. The best runs of the WaveSense model yielded 99.5% accuracy which is equal to the performance of the ANN model. It is important to note that the key focus of the work by (Blouw et al. 2018) is to benchmark energy and power consumption and not model performance.
(Blouw et al. 2018).
3.2 HEYSNIPS DATASET
On the HeySnips dataset, our implementation of the WaveNet reaches an accuracy of 99.8% on the clean dataset. In (Coucke et al. 2018), the authors do not report any accuracy number but rather report the false rejection rate (FRR) 0.12% for fixed false alarm per hours (FAPH) of 0.5. In order to compare our results more accurately, we implement the same metrics; our WaveNet implementation reaches 0.95 FAPH and 0.8% FRR on the test set. These results are slightly worse than the results reported by (Coucke et al. 2018) but that is expected as we do not apply the same specific methods to improve performance such as "End-OO-Keyword labeling" " and "masking" Without those methods and without gating, the FRR reported by (Coucke al.) 2018)) drops to 0.98%. On the other hand, our WaveNet implementation reaches similar or even better results than the CNN and LSTM reported by (Coucke al. 2018). This fact shows that pre-po meeprocessinng method indeed extracts sufficient information from the input such that a neural network can reach very high accuracy. Hence, we train spiking version of the WaveNet architecture (WaveSense), described in 2.2 the same data.
In the WaveSense model we do not use any gating mechanism, a kernel size of 2 and only layers; much less compared the 24 layers and kernel size of 3 as used in the WaveNet implementation by
7

---

Under review as a conference paper at ICLR 2022
Table 2: A comparison of model performance for various datasets and network architectures.
Publication
Dataset
Accuracy ( (%)
Architecture
Coucke et al.
2018
HeySnips
FRR 0.12 FAPH 0.5
WaveNet
Coucke et al.
2018
HeySnips
FRR 2.09 FAPH 0.5
LSTM
Coucke et al.
2018
HeySnips
FRR 2.51 FAPH 0.5
CNN
This work
HeySnips
99.8 (FRR 0.8 FAPH 0.95)
WaveNet
This work
HeySnips
99.6 + 0.1 (FRR 1.0 FAPH 1.34)
SNN
Cramer et al.
2020
SpeechCommands(35))
50.9 1.1
SNN
Cramer et al.
2020
SpecchCommands(55
131
LSTM
Cramer et al.
2020
SpeochCommands(35)
17.7
CNN
Perez-Niever et al.
2021
SpeechCommands(35
57.3 0.4
SNN
This work
SpeechCommands(35)
87.6
WaveNet
This work
SpeechCommands(35)
79.6 = 0.1
SNN
[Blouw et al. 2018
Aloha
93.8
SNN
This work
Aloha
99.5
WaveNet
This work
Aloha
98.0 + 1.1
SNN
(Coucke et al., 2018). The memory in our model is still long enough as WaveSense implements the dilations using synaptic dynamies with long time constants but the number of parameters drops from 47'090 to 13'042, Despite the low number of parameters and quantization from spiking activations, the WaveSense model achieves an average accuracy of 99.6% over 11 runs (only drops by 0.2%) Our best run the WaveSense model yielded the same accuracy (of 99.8%) as our WaveNet implementation. With an FRR 1.0% and FAPH 1.34 the performance is indeed lower than the WaveNet, but is comparable to that of L.TTM and CNN as reported by (Coucke et al. 2018)
3.3 SPEECHCOMMANDS DATASET
We also trained WaveSense on the SpeechCommands dataset. We evaluated our model by training it to classify all 35 classes in the dataset. In study by (Perez-Nieves et al.) 2021). in which the authors investigate the impact of heterogenity time constants on the performance, the best model reached ~ 57.3% accuracy on the same dataset. In (Cramer et al., 2020) the best performing SNN a recurrent network which yields 50.9% accuracy of all classes. In the same study, also an LSTM and CNN are trained on the same data resulting in an accuracy of ~ 73% resp. 77.7%. The WaveSense model reaches an average accuracy of 79.6% over 11 runs (best 80.0%) which is significantly higher than the best SNN described in previous studies. Notably, WaveSense performs better than the reported LSTM and CCNN(Cranae et al. 2020).
4 DISCUSSION AND CONCLUSION
While the results demonstrated here are obtained using a fixed set of time constants, it conceivable that according the constraints of the neuromorphic hardware, an appropriate network could be trained to obtain qualitatively similar results. This holds true even for mixed-signal neuromorphic devices Indiveri al.) (2011) with programmable weights and tune-able time constants. Because the algorithm provides recipe for how to choose the time constants in the network, even if neuromorphic substrate has limited range of time constants, a number of layers with appropriate combination (sum) of time constants can always be chosen to fit the temporal task. This is in stark contrast to recurrent neural networks that often require tight balance between excitation and inhibition and long time constants Bellec et al. (0188: Yin et al. 2020)).
The choice of time constants and number of layers informed by the total temporal memory required by the task. We choose them in similar fashion to that of WaveNet with time constants increasing and such that the sum all the time constants is proportional to Trask- Typically we with factors proportionality of 2.5 is suitable with kernel size of 2. The proportionality factor is observe that the length of time after which the effect
Post Synaptic Potential (PSP) is negligible. This also translates to compact networks with fewer parameters for the same amount of temporal memory (at the same time resolution). In other words, given a network, the temporal memory of a given task can be computed as follows:
8

---

Under review as a conference paper at ICLR 2022
Tlask R3 2.5
(8)
where is the list of the layers in the WaveSense network.
While the results reported here are significantly high, we believe this can be further improved by modifying the loss function. For instance the peak loss computed only during the presence of a keyword opposed to the entire sample Coucke et al. (2018) has been shown to improve performance of such models. Furthermore, a thorough architecture search could potentially result in better combination of time constants Tm and T., number of channels, kernel sizes etc.
crucial factor in adopting a model ease of training, deployment and power efficiency. By utilizing simple LIF neurons, take full advantage of their computational efficiency Yin et al. (0200) in additional to sparse computations afforded by SNNs. While training SNNs is relatively slow on CPUs and GPUs, utilizing the Spike Response model (SRM) in combination with the SLAYER algorithm Shrestha & Orchard (2018), we are able to train at a relatively high speed. All experimental results reported in this manuscript were performed run time per experiment. We further improve upon this efficiency with a custom fork of the SLAYER implementation | The resulting models, while accurate within the SRM framework, are identical to simulations based on LIF neurons, supported by most digital neuromorphic devices. But we find that they are close approximation and quick retraining can recover the model's performance using LIF neurons.
single NVIDIA 1080 Ti with few hours of
The WaveNet architecture requires storing activations of each of its layers depending on their kernel size and dilation value: Nouf (k - 1) d + 1. In contrast, WaveSense does not buffer any spikes(activations)) from the past explicitly. Instead, the information is retained the neuron and synaptic states: Nbuf k + 1. This makes WaveSense extremely efficient in terms of memory utilization contrast to WaveNet.
The results demonstrated here show that the WaveSense architecture is suitable for audio classification tasks and show a promising performance improvement in comparison to prior state-of-the-art Audio signals, after they are pre-processed are equivalent a population of neurons producing spike patterns with complex spatio-temporal correlations. We argue therefore, that the results presented here can be extended to other modalities of sensory data such as ECG, PPG, machine vibrations DVS data.
This work we believe could contribute towards a future with ubiquitous abundance of always-on audio and other sensory devices responding to user commands. This could lead to potential misuse of the technology for surveillance. Thankfully, neuromorphic algorithms such as the one proposed here require specialized neuromorphic hardware take full advantage. the availability of such hardware could be regulated, we hope that society can benefit from this technology while protecting itself from misuse.
"xtp:///XXXXXXXXXXXX

---

Under review as a conference paper at ICLR 2022
REFERENCES
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and leaming-to-leam in networks of spiking neurons. arXiv preprini arXiv:1803.09574, 2018.
Ben Varkey Benjamin, Peiran Gao, Emmett McQuinn, Swadesh Choudhary, Anand R Chandrasekaran, Jean-Marie Bussat, Rodrigo Alvarez-lcaza, John V Arthur, Paul A Merolla, and Kwabena Boa- hen. Neurogrid: mixed-analog-digiral multichip system for large-scale neural simulations. Proceedings of the IEEE, 102(5):699-716, 2014.
Peter Blouw, Xuan Choo, Eric Hunsberger, and Chris Eliasmith. Benchmarking keyword spotting efficiency on neuromorphic hardware, 2018.
Alice Coucke, Mohammed Chlieh, Thibault Gisselbrecht, David Leroy, Mathieu Poumeyrol, and Thibaut Lavril. Efficient keyword spotting using dilated convolutions and gating, 2018.
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82-99) 2018.
Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-aceuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference neural networks (IJCNN), pp. 1-8. ieee, 2015.
Steve B Furber, Francesco Galluppi, Steve Temple, and Luis A Plana. The spinnaker project.
Proceedings of the IEEE, 102(5):652-665, 2014.
Wulfram Gerstner. A framework for spiking neuron models: The spike response model. In Handbook of Biological Physics, volume 4, pp. 469-516. Elsevier, 2001.
Giacomo Indiveri, Bernabé Linares-Barranco, Tara Julia Hamilton, André Van Schaik, Ralph Etienne- Cummings, Tobi Delbruck. Shih-Chii Liu, Piotr Dudek, Philipp Häfliger, Sylvie Renaud, al. Neuromorphic silicon neuron circuits. Frontiers in neuroscience, 5:73, 2011.
Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers Neuroscience, 14:439, 2020. ISSN 1662-453X, doi: 203/////iins.2200000..
htthhtts://wwe http::://wwwwwffoterrrrttttt frontiersin
org/article/1o.3 3389/ fnins 20200.0423
www.s.222n.nn......... 00439
Qian Liu, Ole Richter, Carsten Nielsen, Sadique Sheik, Giacomo Indiveri, and Ning Qiao. Live demonstration: face recognition on an ultra-low power event-driven convolutional neural network asic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.
Carver Mead. Neuromorphic electronic systems. Proceedings of the IEEE, 78(10):1629-1636 1990.
Paul A Merolla, John Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668 (60))666666666,00 2014.
Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks, 2019.
Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. bioRxiv, pp. 2020-12, 12, 2021.
Bodo Rueckauer, Iulia-Alexanrra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conver- sion of ciciinoou--vlllee deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience, 1 11:682, 2017.
10

---

Under review as a conference paper at ICLR 2022
Sadique Sheik and Martino Sorbaro. Project title. hhttps://sinab:: ai/ 2013.
Sadique Sheik, Martin Coath, Giacomo Indiveri, Susan Denham, Thomas Wennekers, and Elisabetta Chicca Emergent auditory feature tuning in a real-time neuromorphic vlsi system. Frontiers
in Neuroscience, 6:17, 2012. ISSN 1662-453X. doi: 000.:/8//iinn.2222200000000
URL https
www. fwontierssiii.o org wwwaaattieeeaaeeee /00 3389/ Enins. 2012 00017
Sumit Bam Shrestha and Garrick Orchard. SLAYER: Spike layer enror reassignment in time. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 1419-1428. Curran Associates, Inc., 2018. URL htttp::/papprs..rtt nips. wwwp.cc/ppppp........e..pppppppe 7415-slayrr-ppicee -aye--e--lyeeerro- wwwssigneenttinntteeeee
Cesa-Bianchi,
David Snyder, Guoguo Chen, and Daniel Povey. MUSAN: A Music, Speech, and Noise Corpus, 2015. arXiv:1510.08484%
Martino Sorbaro, Qian Liu, Massimo Bortone, and Sadique Sheik. Optimizing the energy consump- tion of spiking neural networks for neuromorphic applications. Frontiers neuroscience, 14:662, 2020.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv.1804.03209, 2018.
J Wu, Y Chua, M Zhang, Li, and KC Tan. A spiking neural network framework for robust sound classification. front. neurosci. 12 (2018), 2018.
Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. arXiv e-prints, pp. a90,, 2019.
Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with multiple- timescale spiking recurrent neural networks. In International Conference on Neuromorphic Systems 2020, pp. 1-8, 2020.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 NEURON MODEL
In this work used the Leaky Integrate and Fire (LIF) neuron model with synaptic time constant T, and membrane time constant Ty. The sub-threshold dynamics of this neuron are described below.
8(t) = - + (t) (it)==i(()\TT + Σ wys)(t)
(9)
(10)
where u is the membrane potential, ig is the synaptic current, w the synaptic weight and s is the input spike train.
In-order to optimize simulation time and computational efficiency, we make some alterations. Unlike traditional LIF which resets a resting potential upon reaching threshold 0, subtract fixed value 0 from the membrane potential. Furthermore, if the membrane potential increases beyond N8, where N is arbitrary positive integer, then this neuron produces N spikes, and proportionally No is subtracted from the membrane potential.
if of [o(t)/0). A
s() 0,
(11)
otherwise
The use of a mechanism for generating multiple spikes enables the computation to be more robust to the choice of simulation time steps. This enables to choose a relatively large time step for our simulations. In practice we observe that some neurons occasionally do produce multiple spikes. (Producing multiple spikes in single simulation time-step is not necessary if one chooses small enough time step.)
For further simulation efficiency the LIF neurons were simulated using the SRM Gerstner 2001
v(t = Σ wy(c+ ())(+)s))()))))))))))))))))))){) + (v st)
(12)
where
(t) = (-1/x)
(13)
(t) (-t/r))
(14)
v(t) (-t/r))
(15)
<(l) (c. * +
(16)
We use exponential kernels for synaptic (t) and membrane dynamics (t) and derive the PSP kernel <tt). The refractory kernel v(t) is also negative exponential kernel with the same time constant as the membrane potential. The symbol denotes convolution operation.
As spike generation is non-differentiable,, we use a surrogate gradient Neftei et al. (2019). Several profiles for the surrogate gradients have been proposed in literature. We use modified exponential kernel for the surrogate gradient function. In order to accommodate the multi-spike behavior of the neuron, we choose periodic exponential function (Figure 5) as the surrogate gradient. This function peaks as the membrane potential approaches multiples neuron spiking threshold 0. Intuitively, this gradient function maximizes the impact of parameter when the neuron is close to spiking or has just spiked and is a variant of the exponential gradient function. An extreme simplification of the periodic exponential would be Heaviside function
TThe heaviside function like ReLU has range of membrane potentials where the gradient is and could potentially prevent the network from learning at low activity levels.
12

---

Under review as a conference paper at ICLR 2022
Periodic
1000
Exponential
0.75
Heaviside
0.50
0.25
0.00
-0.25
Figure 5: The surrogate gradient of the spiking neuron as a function of the (pre-spike) membrane potential.
The simulations were run with the library 'sinabs' Sheik Sorbaro (003 using an adaptation of SLAYER (Shrestha & Orchard (018).
A.2 SIMULATION PARAMETERS
The various parameters used in the experiments described in this article are listed below.
Table 3: Parameters used for the Aloha simulations (ANN).
Parameter name
Value
p_classes
2
n m_cchanels__ in
64
channels res
16
n_ ccaanellssskip
32
hidden
32
dilations
[2, 4, 8, 2, 4, 8, 2, 4, 8]
keree_size
bias
true
'GNU AGPL v3 License
13

---

Under review as a conference paper at ICLR 2022
Table 4: Parameters used for the Aloha simulations (SNN).
Parameter name
Value
classes
2
n channels in
64
n
e__hannllss res
16
n channels channels_skip
32
h_hiiden
32
dilations
[2. 8,
2,
4, 8, 2, 8]
threshold
1.0
learning window
0.3
kemee_size bias
2
true
To
2
2
weight_scaling (init)
0.5
Table Parameters used for the HeySnips simulations (ANN).
Parameter name
Value
n_classes
n_channels___ in
64
e__chaneel__ree _es
16
e_channels_skee
32
hidden
32
dilations
[1, 8,2.4.8.
1, 3
2, 4, 1, 1, 1,2,4
kernel_size
bias
true
Table 6: Parameters used for the HeySnips simulations (SNN).
Parameter name
Value
classes
2
n_ channels in
64
n channels res
16
n a.channelssskii
n midiiden dilations
32
[2, 4, 8, 16, 2, 4, 8, 16]
threshold
1.0
learning window woirdoggwwindowww
0.3
kernel size bias
2
true
Tp
2
weight-ssaling (init)
0.5
Table 7: Parameters used for the SpeechCommands simulations (ANN).
Parameter name
Value
n classes
35
n_channels_in in
64
n_channels_ree
16
n. channels skip
32
hidden
32
dilations
[1, 4, 8, 1, 1, 1, 2, 4, 8,
1,
kernel_siz
bias
true
14

---

Under review as a conference paper at ICLR 2022
Table 8: Parameters used for the SpeechCommands simulations (SNN).
Parameter name
Value
n_ c_caasses
35
n channels in
64
c_hannels_rres
32
n__chanels__sip
64
hidden
128
dilations
12, 4, 8, 16, 2, 4, 8, 16, 2, 4, 8, 16]
threshold
1.0
Saariig__wndowe
0.3
kernel_size
bias
true
Tv
2
weight_scalle scalins (init)
0.5
15


Reviews:
[{"review_id": "pIauqpFLveK", "reviewer": "Reviewer_nmri", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper demonstrates that the time constant of the leaky-integrate-and-fire (LIF) can be effectively used to implement time delays, as required for temporal convolution. This can reduce memory requirements of temporal spiking algorithms in general, compared to buffering spikes or using synaptic transmission delays.\n2. Simple and standard neuron and synapse models allow the proposed SNN to be compatible with most neuromorphic processors. Therefore, it can have applications in low power AI systems.\n3. The results presented show that the proposed SNN performs competitively against the other existing SNN models in keyword spotting tasks and is only slightly less accurate than the ANN models.\n\nWeaknesses:\n1. The paper provides limited technical novelty. The network architecture, peak detection, training loss, and learning algorithm are not new. The ANN architecture was used for keyword spotting by Coucke et al. 2018.  Also, the paper lacks the justification behind selecting the network architecture, training scheme etc. with quantitative or qualitative reasoning.\n2. The paper proposes that the time constants be hand tuned based on the task. This requires at least these two comparisons to justify, which are missing from the paper: 1) a graph showing the effect of varying the time constant on accuracy for different tasks, i.e. to show quantitatively the relation between tasks and time constants and 2) comparison of the hand tuned time constant versus learning the time constant, similar to SRNN by Yin et al. 2020.\n3. The paper does not quantify the improvement in memory usage from the alternative design, i.e. temporal convolution using spike buffers or using synaptic delays. The significance of the memory saved via the proposed method compared to the memory [available in standard neuromorphic processors is not known.\n4. Since keyword recognition is based on maximum neuron activation during presentation, it is not clear in deployment how this proposed method will handle the absence of any keywords, because there will always be a maximum activation and intuitively a thresholding is required to avoid any false positives.\n5. Since the prediction by the proposed SNN is a factor of the network state, its prediction performance is dependent on the order in which the test set is presented. The  paper should present accuracy results as the average accuracy over a large number of runs, where in each run, the test set is presented in a new random order.", "summary_of_the_paper": "The paper proposes a spiking neural network (SNN) that takes advantage of the time constant of the neuron spiking dynamics to implement temporal convolution, instead of using synaptic delays, in order to reduce memory requirements. Input temporal data, such as audio streams, are processed continuously without any buffering, which makes it suitable for always on audio recognition systems. Further, the spiking architecture makes it suitable for low power applications using neuromorphic processors. It adopts an existing artificial neural network (ANN) architecture for generating audio waveforms, converts it to the spiking domain, and trains the network using an established spiking backpropagation algorithm. The experimental results show improved audio keyword recognition performance compared to the existing spike based methods.", "summary_of_the_review": "While the presented results show that the proposed approach of temporal dilated spiking convolutional network can be trained to recognize audio keywords with competitive accuracy, the main contribution of the paper is a system that combines ideas from multiple existing works. Some key experiments pertaining to the claims are missing, such as the benefit of manual time constant selection or how the manual approach generalizes to new tasks, how it detects the no keyword case in deployment etc. The experiment section is missing data about memory requirement reduction than alternative approaches and throughput comparisons on a standard neuromorphic processor.\n\nThe paper can be strengthened by the addition of the following: \n1) Time constant optimization instead of hand selection and its effect on accuracy on different tasks. In the current setup, it should analyze the  effect of varying the constant in Equation 8.\n\n2)  Compare the memory requirement of the same WaveSense equivalent implemented using synaptic delays vs. WaveSense using time constants, to show the reduction in memory and computation requirements using synaptic time constant as mentioned in Section 2.1.\n\n3) Experimental results to show robustness regarding Weakness 5.\n \nIf detailed experiments and analysis are presented on the above issues in the revised version, with justification of the related claims made, I will be inclined to reconsider my score to a marginal accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ipUEEcyujM", "reviewer": "Reviewer_u5kv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n1. Simulating temporal convolution in SNN by varying synaptic time constant is a neat idea. As already pointed out in the paper, it's expensive to use spike delays for neuromorphic computations. The concept of using different time constants to approximate the dynamics of the delays can potentially be more efficient to compute on the neuromorphic processor.\n\n2. The motivation of using peak loss function for fast detection is interesting and may be useful for many other streaming applications which require instantaneous detection. It will be interesting to see how the improvement in detection speed using the peak loss function compares against conventional loss functions that use the last step output. \n\n3. The proposed SNN outperformed other SNN methods on keyword spotting. This shows that SNN using temporal convolutions is more suitable for this task than other SNNs that use recurrent or fully-connected layers.\n\n\nCons\n\n1. The paper lacks evidence to back the primary motivation of the proposed approach. The work is motivated by the need for low-power keyword spotting systems. However, the authors do not show any energy/computation cost comparison between the proposed SNN method and existing approaches. Without this comparison, there's no quantifiable advantage for the proposed SNN which invalidates the method's value. Energy efficiency is not a default property of SNN, and an SNN needs to be deployed appropriately on a neuromorphic processor to realize this advantage. The deployment is not trivial, and the efficiency highly relies on the dynamics and resource cost of the network. The authors need to either estimate the computation cost of the SNN by counting the number of operations, or deploy the SNN on a neuromorphic processor for energy benchmarking.\n\n2. In addition, other low-power variants of deep networks, like binary neural networks, exist for keyword spotting (for example, [Liu, Bo, et al. 2019], [Peter, David, et al. 2020]). The authors need to discuss the advantages of using SNN over these approaches and compare the SNN with these low-power approaches on energy cost and accuracy.\n\n3. The paper lacks ablation studies to show that the network's high performance comes from the proposed spiking temporal convolution mechanism. Since spiking neurons have an inherent information integration in the temporal dimension, the high accuracy is not enough to prove the temporal convolution is correctly functioning. When compared with the other SNN approaches, the result for the proposed method might be better due to the more complex network structure. Thus, the authors need to provide additional ablation studies on SNN without temporal convolution to prove the proposed mechanism works. Moreover, the author can also compare the synaptic time constant with the delay approach to show how much the approximation influences representation capacity.\n\n4. The structure and writing of the paper need considerable improvements to reach the standards of the conference. The current structure of the introduction might be hard to follow for the readers who are unfamiliar with the work. The reviewer recommends the authors to divide the section into an introduction section and a related works section. The method section is also hard to follow with network design, implementation details, and experiment data mixed together. The description of the spiking temporal convolution lacks important details for readers to understand the method. The method section should be clear enough so that the readers can reproduce the method using just the write-up.\n\n5. Figures in the paper lack sufficient details for understanding the proposed method better. Figure 1A is very similar to Figure 3 in [Oord, Aaron van den, et al. 2016], with no additional information on how SNN works with this network design. With the figure and descriptions in the text, it's very difficult to understand how the SNN computation is unrolled, and whether or not spiking neurons in each layer are updated simultaneously. Figure 2 is similar to Figure 4 in [Oord, Aaron van den, et al. 2016]. Although specific layers in [Oord, Aaron van den, et al. 2016] have been replaced by spiking layers, the paper didn't argue why these layers are needed in the block.\n\nOord, Aaron van den, et al. \"Wavenet: A generative model for raw audio.\" 2016.\n\nCoucke, Alice, et al. \"Efficient keyword spotting using dilated convolutions and gating.\" 2018.\n\nLiu, Bo, et al. \"An ultra-low power always-on keyword spotting accelerator using quantized convolutional neural network and voltage-domain analog switching network-based approximate computing.\" 2019.\n\nPeter, David, et al. \"Resource-efficient DNNs for Keyword Spotting using Neural Architecture Search and Quantization.\" 2020.\n\n", "summary_of_the_paper": "This paper proposes a Spiking Neural Network (SNN) with temporal convolutions for low-power keyword spotting. The proposed SNN architecture comes from the WaveNet ([Oord, Aaron van den, et al. 2016]) with modifications to adapt to the SNN dynamics. The idea for using such an architecture for keyword spotting originates from [Coucke, Alice, et al. 2018]. By training the SNN using Backprop Through Time (BPTT), the proposed method achieved better performance than other SNN-based methods and reached near SoA performance for the keyword spotting datasets.\n\nOord, Aaron van den, et al. \"Wavenet: A generative model for raw audio.\" 2016.\n\nCoucke, Alice, et al. \"Efficient keyword spotting using dilated convolutions and gating.\" 2018.\n", "summary_of_the_review": "Overall, the reviewer recommends not accepting the paper. The authors gave a neat idea on how to implement temporal convolution with spiking neurons efficiently. However, the overall quality of the paper does not reach the standards of the conference. First, additional experiments are needed to validate the effectiveness of the proposed method and demonstrate the advantage of the method compared to existing approaches. Second, the structure and writing of the paper need considerable improvements to make it more readable. More detailed problems of the paper are listed in the cons in the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e0rzWXtmqUt", "reviewer": "Reviewer_kAcR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is hard to follow due to its tinpot writing and organization. \n\nFor example, there should be some parentheses to surround the cite or reference. \n\nBesides, it’s necessary to give a formal introduction for the notations in the spiking neural model, such as $\\tau_s$. It’s not reasonable to put the formulation about the spiking neural model to appendix. There are two large blanks wasteful in Page 6.\n\nFig. 2 does not corresponds to the context in Subsection 2.2. \n\nIn addition, using the WaveNet architecture, the authors should list all the trainable parameters and introduce their roles played in this model. \n\nThe points above prevent the understanding of this work.", "summary_of_the_paper": "This paper presents the WaveSense model consists of spiking neurons and the WaveNet architecture for ultra-low power local signal processing. The authors conducted experiments on several real-world data sets for evaluating the efficacy of the proposed models.", "summary_of_the_review": "This paper needs to strictly rewritten and organized for proofreading, and thus, I will take the score of “rejection”.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YlWzSMllSkY", "reviewer": "Reviewer_SLVL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This study addresses a valid problem; because although neuromorphic systems have the obvious advantage of low-power usage, regarding the performance, existing neuromorphic solutions are not at par with traditional methods.\n\n2. Authors clearly explain how they transform the original WaveNet architecture to their SNN architecture, WaveSense, to realize an efficient neuromorphic system. I think that the experimentation configurations are also detailed sufficiently for reproducibility of the results. Authors will open source their code as well.\n\n3. Performance evaluations are comprehensive by testing on three different public datasets, and by comparing against both existing SNN and ANN methods. The results adequately support the performance claims of the proposed model.\n\n4. Discussion section is quite informative.\n\n\nWeaknesses:\n\n1. I think the main weakness of this paper is the limited evaluations regarding the efficiency. I understand that neuromorphic hardware is by nature low-power compared to classical digital processors, and thus we may accept that they are always efficient. However, authors do not compare the efficiency of their method with respect to previous SNN models, except the model size and parameter count in Table 1. As efficiency is the selling point, it is desirable to see the comparison against the SNN models given in Table 2 as well. Also, it would be helpful if authors can elaborate more on if neuron and parameter counts are directly comparable across different neuromorphic systems.\n\n2. Authors often use the term “spatio-temporal”, as in “... we focus on audio tasks as spatio-temporal tasks…”. I don't quite understand the use of this term in the context of this paper because there is no spatial domain with a single channel audio. Authors should clarify this issue.\n\n3. Authors obtain 0.95 FAPH with a 0.8% FRR, while (Coucke et al., 2018) obtains FRR 0.12% for a fixed FAPH of 0.5, on the HeySnips dataset. Authors say that their results were slightly worse. However, it is not easy to compare these numbers since authors do not fix their FAPH score. Authors need to explain how they reach that conclusion based on these numbers. A plot of FAPH versus FRR could be helpful.\n", "summary_of_the_paper": "This paper proposes spiking neural networks (SNNs) as an efficient neuromorphic alternative to dilated temporal convolutions, based on the WaveNet architecture for keyword spotting. The main idea is to model the delay periods in dilated convolutions of WaveNet as synaptic time constants in SNNs for efficient implementation with neuromorphic hardware. For this task, authors introduce an effective transformation of the original WaveNet architecture to a qualitatively equivalent SNN architecture which they named as WaveSense. Then they show by experimental evaluation that WaveSense surpasses the SOTA SNN performances and come close to the SOTA performance of CNN and LSTM based methods, in keyword spotting.", "summary_of_the_review": "Although the proposed method looks promising and comparative evaluations show performance improvements compared to SNNs nearing the ANN performances, the efficiency aspect compared to SNNs have not been justified well and there are some other critical flaws as pointed by the other reviewers. Authors did not address these issues. Therefore my decision is reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I-0oahotiH0", "reviewer": "Reviewer_4EoH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The results are interesting and promising, the benchmark is quite weak though.\n\nWeaknesses:\n1. The translation of WaveNet into SNN-based network itself appears correct. However, I indicate that the SNN considered is absolutely not an SNN given the use of multiple spikes per timestep. Fundamentally, spiking neurons in an SNN should communicate via one-bit/timestep data given the event-based data processing in nature. This is the common constraint on spike data among all SNN models. Otherwise, the network is equivalent to DNN unfolded in the time-domain with integer activations. In this regard, strictly speaking, the network considered in the present work is not SNN. Therefore, the comparison of this work with other SNNs is not fair at all. For a correct comparison, the authors should limit the number of spikes per timestep for a neuron should be limited to one and re-evaluate the performance.\n\n2. This work hardly includes important fundamental aspects of the proposed network. The authors’ translation of WaveNet into SNN-based network is understandable; yet, it is not clear where the breakthrough lies, and how it works in compared with DNN-based WaveNet. Also, ablation study is missing. \n\n3. Benchmark is weak. The authors used only relatively new datasets and compared their results with only few of previous works. There should be a number of previous works on google speech commands that should be compared.", "summary_of_the_paper": "The authors propose a way to translate WaveNet into SNN-based network referred to as WaveSense. The performance of spiking wavenet on three different datasets was compared with a few previous results. Several technical aspects of the proposed method were reported including data preprocessing and translation into spiking neurons and so forth.", "summary_of_the_review": "Based on my concerns listed above, I recommend for reject.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose in this manuscript to use spiking neural networks (SNNs) as an efficient alternative to dilated temporal convolutions. They propose to utilize the membrane time constant of neurons instead of synaptic delays for memory efficiency. Training such networks with BPTT achieves better performance than other SNN-based methods and achieve close to SOTA compared to ANN solutions for keyword spotting.

Pros:
- The manuscript addresses an interesting problem.
- Performance is good

Cons:
- Limited evaluations regarding efficiency, although this is a main point of the paper.
- The technical novelty is limited.
- One reviewer noted that the model is not actually an SNN, due to the use of multiple spikes per time step.
- Benchmarking is weak. Little comparison with previous work.
- Structure and writing of the paper needs improvement.

The authors did not reply to any of these critical points. In summary, although the idea seems interesting, the manuscript is not ready for publication.
Decision: Reject