Decoding EEG With Spiking Neural Networks on Neuromorphic Hardware
Authors: Neelesh Kumar
Guangzhi Tang
Raymond Yoo
Konstantinos P. Michmizos
Venue: Accepted by TMLR ()

Abstract: Decoding motor activity accurately and reliably from electroencephalography (EEG) signals is essential for several portable brain-computer interface (BCI) applications ranging from neural prosthetics to the control of industrial and mobile robots. Spiking neural networks (SNNs) is an emerging brain-inspired architecture that is well-suited for decoding EEG signals due to their built-in ability to integrate information at multiple timescales, leading to energy-efficient solutions for portable BCI. In practice, however, current SNN solutions suffer from i) an inefficient spike encoding of the EEG signals; ii) non-specialized network architectures that cannot capture EEG priors of spatiotemporal dependencies; and iii) the limited generalizability of the local learning rules commonly used to train the networks. These untapped challenges result in a performance gap between the current SNN approaches and the state-of-the-art deep neural network (DNN) methods. Moreover, the black-box nature of most current SNN solutions masks their correspondence with the underlying neurophysiology, further hindering their reliability for real-world applications. Here, we propose an SNN architecture with an input encoding and network design that exploits the priors of spatial and temporal dependencies in the EEG signal. To extract spatiotemporal features, the network comprised of spatial convolutional, temporal convolutional, and recurrent layers. The network weights and the neuron membrane parameters were trained jointly using gradient descent and our method was validated in classifying movement on two datasets: i) an in-house dataset comprising of complex components of movement, namely reaction time and directions, and ii) the publicly available eegmmidb dataset for motor imagery and movement. We deployed our SNN on Intel's Loihi neuromorphic processor, and show that our method consumed 95\% less energy per inference than the state-of-the-art DNN methods on NVIDIA Jeston TX2, while achieving similar levels of classification performance. Finally, we interpreted the SNN using a network perturbation study to identify the spectral bands and brain activity that correlated with the SNN outputs. The results were in agreement with the current neurophysiological knowledge implicating the activation patterns in the low-frequency oscillations over the motor cortex for hand movement and imagery tasks. Overall, our approach demonstrates the effectiveness of SNNs in accurately and reliably decoding EEG while availing the computational advantages offered by neuromorphic computing, and paves the way for employing neuromorphic methods in portable BCI systems.

Published in Transactions on Machine Learning Research (06/2022)
Decoding EEG With Spiking Neural Networks on Neuromorphic Hardware
Neelesh Kumar Deportment of Computer Science
neelesh.kumarrrrutgee
Rutgers University
Guangzhi Tang
wwangzhi.langg@ruugrrr
Department of Computer Science Rutgers University
Raymond Yoo
mymonnd.yoo&ruutgrrs..
Department of Computer Science Rutgers University
Konstantinos P. Michmizos
michmissos-ccs.rtggrrs.oo
Department of Computer Science Rutgers University
Reviewed on OpenReview: https: openreveew. net/ forum? 2PBJPGX3Ba
Abstract
Decoding motor activity accurately and reliably from electroenephalography (EEG) sig- nals essential for several portable brain-computer interface (BCI) applications ranging from neural prosthetics to the control of industrial and mobile robots. Spiking neural net- works (SNNs) is an emerging brain-inspired architecture that is well-suited for decoding EEG signals due to their built-in ability integrate information leading to energy-efficienl solutions for portable BCI. In practice, however, current SNN solutions suffer from i) an inefficient spike encoding of the EEG signals; ii) non-specialized network architectures that cannot capture EEG priors of spatiotemporal dependencies; and iii) the limited generalizability of the local learning rules commonly used to train the net- works. These untapped challenges result a performance gap between the current SNN approaches and the state-of-the-art deep neural network (DNN) methods. Moreover, the black-box nature of most current SNN solutions masks their correspondence with the under- lying neurophysiology, further hindering their reliability for real-world applications. Here, we propose an SNN architecture with an input encoding and network design that exploits the priors of spatial and temporal dependencies in the EEG signal. To extract spatiotem- poral features, the network comprised of spatial convolutional temporal convolutional, and recurrent layers. The network weights and the neuron membrane parameters were trained jointly using gradient descent and our method was validated in classifying movement on two datasets:
multiple timescales,
in-house dataset comprising of complex components of movement, namely reaction time and directions, and ii) the publicly available eegmmidb dataset for motor im- agery and movement. We deployed our SNN on Intel's Loihi neuromorphic processor, and show that our method consumed 95% less energy per inference than the state-of-the-arl DNN methods on NVIDIA Jeston TX2, while achieving similar levels classification per- formance. Finally, we interpreted the SNN using network perturbation study to identify the spectral bands and brain activity that correlated with the SNN outputs. The results in agreement with the current neurophysiological knowledge implicating the activation

---

Published in Transactions on Machine Learning Research (06/2022)
patterns in the low-frequency oscillations over the motor cortex for hand movement and imagery tasks. Overall, our approach demonstrates the effectiveness of SNNs in accurately and reliably decoding EEG while availing the computational advantages offered by neuro- morphic computing, and paves the way for employing neuromorphic methods in portable BCT systems.
1 Introduction
The accuracy and reliability in decoding electroencephalograplyy (EEG) signals are the two main factors that brain-computer interface (BCI) relics on for its diverse set of applications, ranging from neurorchabilitation
and neural prostheses (Baud et al., 2018 Chaudhary et al. 2016 Lebedev & Nicolelis, 2017). to the control
of industrial (Buerkle et al. 2021 Garcia et al. 2013) and mobile robots (Bi et al. 2013) For their use to be practical in the real-world, BCI systems often need to be portable and run on limited energy resources (Zhang let al. 2019 Kartsch et al.) 2019). Deep neural networks (DNNs) have been quite effective in decoding EEG for various tasks (LeCun et al.] 2015) due to their ability to learn features from raw representations with strong gencralization (Zhang et al. 2016). However, the high energy cost associated with inference using DNN (Dong et. al.) 2017) hinders their use in portable BCI.
Energy-efficiency is long- promised advantage offered by spiking neural networks (SNN), emerging brain- inspired architocture where neurons compute asynchronously through discrete events, the spikes. The hard- ware realization of the SNNs on neuromorphic chips (Davies et al. 2018) has spurred their applicability as energy-effiienn solutions for many learning tasks (Tang et al. 2019
Hwu et al. 2017). Moreover, mounting evidence suggests that SNNs may be more effective than DNNs for the classification of signals that have non-stationary characteristics (Kugele et al., [220 Deng et al.)
2020 Reid et al. 2014).. This has been
attributed to the ability of the spiking neurons in representing information at multiple timescales through
their dynamics (Deng et al.) 2020 Yin et al.] 2020 Bellec et al.
2018
Tang et al. 2021 However, current
SNN-based EEG classification methods (Antelis et al. 2020 Luo et al. 2020 Kasabov 2014) are unable to
match the performance of the stale-of-the-arl DNN methods for EEG classification (Kumar & Michmizos
2022 Schirrmeister et al. 2017 which renders SNNs ineffective for most practical applications.
The reported performance gap of SNNs is mainly due to three reasons. First, the common methods for encoding EEG into spikes (Schrauwen & Van Campenhout] 2003) are heuristics-basee and require heavy task-independent tuning of encoding-related parameters. Consequently, an inappropriately tuned encoding of the spikes can significantly affect the classification performance. Second, several current approaches employ a reservoir of randomly connected recurrent neurons as their network topology (Antelis et al. 2020 Kasabov 2014). While effective train-inspired tuning methods for reservoirs have recently been introduced (Ivanov & Michmizos 2021). : they can only partially consider the important inductive priors associated with EEG in their design, such as the inherent spatial and temporal dependencies in the input signal (Burle el al. 2015) Last, local correlation- based learning rules (Markram et al. [997) are very popular in training SNNs but they lack the expressiveness of error-driven training algorithms such backpropagation, and restrict network topologies to shallow architectures (Shrestha et al. 2019). While learning rules such as spatiotemporal backpropagation (Wu et al. 2018) are now starting to gain traction in training SNNs for several tasks (Tang et al.) 2021 2020 Wu et al. 2018). they inherit the disadvantages found in their conventional counterparts, including interpretability. For example, the black-box nature of the SNN trained using backpropagation masks the correspondence between the learned features and the underlying neurophysiology, resulting in only weak guarantocs for their reliability in analyzing EEG signals (Montavon et al., 2018). Therefore, the development of an accurate and reliable SNN solution to EEG classification requires a complete rethinking of the SNN design that can tackle the issues of input encoding, network topology, training, and interpretability.
In this work, we propose an SNN that extracts task-discriminative spatiotemporal EEG features by incor- porating the priors of ial and temporal dependencies in the brain activity directly in its input encoding and network design (Figure []. The encoding parameters, network weights, and the neuron membrane pa- rameters were jointly optimized using spatiotemporal backpropagation (Wu et al.) 2018). We deployed the trained SNN on Intel's Loihi neuromorphic processor to demonstrate the energy efficiency for the inference. We validated our method an in-house, IRB-approved, dataset for classifying complex movement compo-
2

---

Published in Transactions on Machine Learning Research (06/2022)
Kapoho Bay
Intel's Loihi
yow Hot wot You whe thes whu MAt eit why
Slow Fast
Reaction Time
toe apo with Appt
Flatten
Left Right
Down
EEG recorded using 120 Channel System
Spiking Convolutional
Recurrent Layer
Fully Connected
Up
EEG with Convolutional Encoding
Layers
Layers
Directions
EEG Acquisition
Loihi-run Spiking Neural Network
Classification
Figure 1: Experimental setup and study workflow: 12 subjects performed a goal-oriented motion task while high demsity EEG data were simultaneously acquired with movement kinematics. The proposed SNN received topography-preeeevingg EEG inputs, and was deployed on Intel's Loihi neuromorphic chip to classify complex components of hand movements.
nents, namely reaction time (RT) and directions, and the publicly available eegmmidb dataset for classifying motor imagery and execution tasks. Our method on Loihi consumed 95% less energy per inference than the tate--of-the-ar DNN methods on NVIDIA Jetson TX2, while achieving similar levels of classification perfor- mance. To interpret the trained SNN, we used a network perturbation study and identified the low- fwwequequccc oscillations as well as the localized patterns of brain activity, alignment with the current ncurophysio- logical knowledge for motor tasks (Hamel-Thibaull et al.) 2018 Batula et al. 2017). Overall, our results demonstrate the effectiveness of SNN solutions in classifying temporal signals and discovering task-relevant features in the brain signals in an energy-efficient way.
2 Related Works
Several studies have employed EEG to decode several behavior tasks. The earliest approaches relied on statistical methods that compute hand-crafted features such as spatial features (common spatial pattern (CSP) (Ang al., 2008)), temporal features (fractal dynamics (Gupta et al. 2018)), or spectral features (wavelet transforms, power spectral density (Hazarika el al., 1997; Shaker 2006 Kim et al. 2018) from preprocessed EEG signals. A classifier such as support vector machine (SVM) is then employed to segregate these fcatures. While statistical EEG decoding methods are highly explainable, they require extensive preprocessing steps as well as domain-specific hand-crafted features, which results in high computational costs during inference and impedes their use in real-time systems. Tn addition, the reliance on a limited number of features affects their generalizability (Samek et al. 2012 Lotte & Guan 2011) and as such, they do not classify well tasks that are important for decoding natural human behavior, such as hand movement directions [Robinson et al. 2012).
The recent DNN methods eliminate the need for domain-specific features by learning features directly from representations (LeCun et al. 2015) and also exhibit strong generalization (Zhang al. 2016). The current DNN methods for EEG classification can be broadly categorized based on imput representation and network design. Most cxisting approaches provide raw EEG representations with little or no preprocessing as inputs to the network. This type of input representation makes them suitable for network architectures such as 1D and 2D convolutional neural networks (ID-CNN (Acharya al. 20188) 2D-CNNs (Schirrmeister et al. 2017 Cui al. 2020 Lawhern al. 2018 Gao et al. 2019) ), or recurrent architectures such LSTM (Zheng & Chen 2021p. While such approaches are truly end-to-end, the limited inductive priors in their input representation and nelwork design limit their generalizability, and increase the amount of training data needed. Some recent approaches represent EEG inputs as 2D images either in the form of spectral
"Code available http:://gthhab.cccm/mmmmbeeaal/
3

---

Published in Transactions on Machine Learning Research (06/2022)
maps (Bashivan et al. 2015 leracitano et al. 2019 Jiao et al. 2018 topological maps stacked in time to
form 3D representations (umaar& & Michmizos] 2022 Zhao et al. 2019). Given the 3D input representations.
these approaches employ convolationall-ecurren architectures such as CNN-LSTM (Bashivan et al., 2015)
or 3D convolutional ncural networks (3D-CNN) (Kumar & Michmiios
2022 Zhao et al. 2019). Although
DNN approaches like the ones based on 3D input representations are eflective in classilying EEG signals
for several tasks, their high energy consumption during inference (Dong et al. 2017) limits their use in applications that require portable BCIs.
SNNs provide highly energy-efficient solutions for many learning tasks (Davies et al. 2018 Tang et al. 2019 The current SNN-based approaches to EEG classification extract features from preprocessed EEG signals and feed it to an SNN-based multilayer perceptron (MLP) (Antelis et al., 2020) or CNN (Yan et al. 2022) for classification. These approaches suffer from the same problems found in traditional statistical methods, namely high computational costs during inference and limited generalizability. More recent approaches attempt to Iearn directly from the raw EEG signals by first encoding them into spikes using temporal or rale-based encoding (Schrauwen & Van Campenhout, 2003) and then leeding the spikes to reservoir-style network architecture such NeuCube for classification (Kasabov 2014). Local learning rules (Markram ct al. 1997) and brain-inspired tuning methods [Ivanov & Michmiizss 2021 are then be used to train these reservoirs in an unsupervised manner, and biologically inspired supervised learning rule is used to train
the readout layer for classification (Kasabov & Capecci 2015 Tan et al. 2021 Alzhrani et al. 2021 y. These approaches have the appeal of being biologically relevant but they do not explore the full scale of learning rules present in the brain and therefore can perform well only in simple classification tasks.
With respect to interpretability, brain-inspired solutions such as NeuCube lend themselves well to neurophyysoolgical inter- pretation, such as the identification of clusters of neurons that are implicated in a given cognitive task (Doborjeh et al. 2018 Kasabov & Capecci 2015). Such interpretations are, however, lacking for hierarchical SNN architectures that are trained us- ing backpropagation. While significant progress has been made in developing interpretation tools for DNNs (Selvaraju et al. 2017), such techniques cannot be directly employed for SNNs because they do not account for the temporal nature of SNN backpropagation (Kim & Panda 2021). Overall, the use of houristics- based encoding, local learning rulcs, and shallow net- work architectures limit the effectiveness and interpretability of
A5
DEL
G
A12
AG
AMS
Alt
CEC
ON
C12
C1U
A13
MI
Ate
CtS
CIG
AMS
A2S
CH
CZ
AZ2
ACO
A24
ARG
A28
C2E
A21
Sr
BS
DE
P10
B
SID
SH
B12
the current approaches for complex EEG classification tasks.
R1C
SYC
bx2
D2S
3 Methods
BSC
Azc
DOP
DRT
We propose an SNN that incorporates the inductive priors of EEG spatial and temporal dependencies in its input encod- ing and network design. The encoding parameters, network weights, and neuron parameters were jointly trained using the spatiotemporal backpropagation learning algorithm (Wu el al. 2018). Then, we deployed the trained model on Loihi for enorgy-efficient inference. Finally, we interpreted the network using the network perturbation technique (Schirrmeister et al. 2017) to determine the EEG sensors and frequencies important for classification decisions.
Figure 2: Input representations preserv- ing the brain's topography applied on BioSemi's 128 sensor layout. Same colored sensors appeared in the same row in the input matrix. To ensure that the num- ber of sensors per row are same, periph- cral channels that were in close proximity to the picked channels were dropped. The dropped sensors are indicated in black.
3.1 Neuron Model
The building block of the SNN is the current-based leaky- integrate-and-frn (LIF) neurons (Lin al. 2018).. The states of the Rth layer neurons at time were updated in two stages:
4

---

Published in Transactions on Machine Learning Research (06/2022)
First, the input spikes were integrated into synaptic current as
()))) = D, (lt-1)(b) + psp((
(1)
where e is the synaptic current, De is a non-negative vector of current decay factors for cach neuron in the layer, psp is the function that generates post-synaptic potential defined by the 4th layer, and is an array of binary variable (0 or 1) which indicates the spike events of the presynaptic neurons.
Then, the synaptic current was integrated into membrane voltage and transformed to spikes as
(()(() =D,- (l-11)(A) + (1)())
(2)
o(0(00 & ((0(k) 0, if U, ((((0)) > V%.
where v is the membranc voltage, D is the non-negative vector of voltage docay factors for cach ncuron in the layer and Va, is the voltage threshold which was either static or adaptive. To lessen the dependence on the initialization of hyperparameters and make them task dependent (Rathi Roy] 2020), we set De and D, as trainable parameters which were optimized using gradient descent. In addition, the adaptive firing thresholds were updated to increase after every spike and decay otherwise using
V (f) = ViA (t - 1) + thamp: if o, 11) (t)+ =1
(t) = max 1) - thaecay. thbase), otherwise,
where thamp is the amount by which voltage threshold increases th is the decay factor for voltage threshold and thoss is the minimum voltage threshold. Neurons with adaptive firing thresholds and decays are known to improve SNN performance for image classification tasks (Yin et al., 2020; Rathi & Roy 2020 as they allow for multiscale temporal integration of information (Bellec et al.) 2018). In our network, we set the thresholds for all the layers except the recurrent layer to be static. The recurrent layers used neurons with adaptive thresholds.
3.2 Spiking Neural Network for EEG Classification
The EEG data exhibit high spatial and temporal dependencies that are often correlated with the physical or mental activity being performed (Berka et al. 2007 Burle et al., 2015 Cohen & Kohn, 2011). Here, we propose an SNN that incorporated these dependencies in its architectural priors, enabling the extraction of disiiiiscreennative spatiotemporal features for EEG classification.
The SNN exploited the spatial dependencies in the EEG data in three ways: through input representation, spike encoding, and network activity. We mapped the EEG signals from the spatially distributed channels onto a 2D matrix, leading to a were then fed to a convolutional spike encoding layer which converted them to spikes while preserving the spatial dependencics. As opposed to popular heuristics- 2019), this direct approach made the encoding trainable and hence removed the need for manual tuning of encoding parameters. The encoded spikes were then passed through several layers of convolution in the SNN to extract spatial fcatures (Equation The post-synaptic potential function for the convolutional layer (pspH) for the neuron at the location
imput representation (Fig, 2 The mapped EEG signals
encoding algorithms for EEG (Potro et al.,
(i,j) with the convolution channel f in the ttth layer is computed as
N-1 Σe
pSpH (00 (1)0)
(1)R - (i- - -n) m))(()) He, (m, n) + bH..))
(4)
m=0 n=0
where (-(k) Ho. and (k) Ho, are the kernel weights from convolution channel e in layer k-1 to the channel f in layer k. The convolution kernels are of the size M x N. An average pooling layer between the convolutional layers downsampled the outputs for lessening the computational load and providing some translational invariance.
The SNN captured the multiscale temporal dependencics in the EEG data through the dynamics of the spiking neurons, temporal convolution layer, and recurrent layer. The dynamics of the spiking neurons enabled the network to learn weak short-term temporal dependence as they linearly integrated their past activities into their current activity (Equation 2). The temporal convolution layer and the recurrent layer in the SNN captured complex non-linear temporal relationships.
5

---

Published in Transactions on Machine Learning Research (06/2022)
The temporal convolution layer performed non-linear in- A. tegration of information from short window of previ- ous timesteps, allowing for learning of short-range tem- Inpur poral representations. Specifically, neurons at time in- tegrate imputs from their presynaplic neurons over a win- dow of previous timesteps from to t win into their post-synaptic potential (pspTc) as defined by Equation win
Hakd
Ha.x
Hake1
Convolution
pspto(o(((61)) (1)--1)))))))))))))))))))))))))
(k)(dd)) (tt-dt((k-1) + byC (())d)) TC
HaR-
Hz.x
Hakel
di=U
where ((X))) (k)) TC DyC are the weights and biases for each time dt in the window.
Hogd
H2ux
HaKI
The recurrent layer captured complex non-linear tem- poral relationships over larger timescales. Specifically, spikes of neuron from the previous timestep were fed to it input in the current timestep which then integrated into their post-synaptic potential (pspR) as per Equation
Temporal Convolution
TC
TC
R
Rat
Recurrent
ps (1(k ()--))-= = (00-))))))1) (*)) + w (t-110)))
(t-1)(8) + b
(6)
where (((), (8) are the feedforward and (((k) b are
(k)
B.
the recurrent weights and biases.
Activation
Elemeniiwwse operation
Finally, the spatiotemporal features extracted by the SNN were fed to decoder module composed of spiking and non-spiking fully-connected layer which transformed them into a vector of class probabilities at each timestep according to Equation 7
V11
0
O
psp
+
psp
(o (K)K))))) (K(--) +b(k)
X
Vector transfer
Copy
(t) Wout (oot (1)(K) + bowt
Figure 3: A. The proposed SNN architecture. Topography-preserve
where W and (k) denote the weights and biases of the spiking fully-connected layer K receiving inputs from layer K-1, and Wout and bot are the weights and biases of the non-spiking fully-connected layer. A weighted sum of the probabilities at each timestep (()) yielded the final class probabilitics Pelass (Equation
inputs were fed to con- volutional layers for extracting spatial features. Temporal feat were extracted using temporal convolutional and recurrent layers. B. Transfor- mation of neuron states by spiking neuron unit.
Pelass Σ Wil (1)
(8)
where (((0) the decoding weight for each timestep. The decoding weights on the timesteps allowed the network to learn the relative importance of each timestep towards classification (Appendix C).
3.3 Network Training
The SNN parameters and the neuron decays (D.: D.,) were trained minimize the average cross-cntropy loss L over the training samples. The gradients of the parameters with respect to L were computed using the spatiotemporal backpropagation algorithm (Wu et al. 2018). To handle the non-differentiabilly of the

---

Published in Transactions on Machine Learning Research (06/2022)
threshold function, we used the rectangular function with respect to the membrane voltage v
Wi et al. 2018 to approximate the gradient of spike
if 2 Val < a2
=(0)
(9)
0
otherwise
where is the pseudo-gradient, is the amplifier, is the threshold window for passing the gradient.
The gradient of the loss with respect to the neuron states- current, voltage, and output spikes is given by:
Vicua I. z(v(()())) Votaa L do -o (1((k) Vice
Votexa vi de Vcc-ioo))
(10)
Vo w(k'" Veceoon
Using this, compute the gradient of the loss with respect to the network parameters. Specifically, the gradients for the spatial convolution and linear layers are
V
w L
(0k-1)) V Vocuaa, ba
=
V
eene L,
(11)
t=1
t=1
and the gradients of the temporal convolution layer are
V
(e)ca L
Σ
(t-
(t-dt)(k-1)
V Larta
L=
V Lo
(12)
W
brc
t=wii++
t=win=1
The gradients for the recurrent layer are
( Σ (t-))()) VecooL V
va
V cl L.
(13)
Finally the gradients for the docays are
V L Σ (11)()) (1-1))()..evii L
VD Σ (((11)1()). (10))))-(1-0000 (t-1)()) Veono
(14)
3.4 Neuromorphic Realization
We deployed trained SNN Intel's Loihi neuromorphic processor (Davies al. 2018) for energy-officient inference. To maximize the inference speed, introduced an interaction framework between Loihi and the host machine that minimized the communication overhead during each inference. The convolutional spike encoding layer was implemented on the host machine, with the generated spikes sent to Loihi in spike-time buffers to avoid redundant communications at every timestep. The SNN decoding module was implemented on Loihi. To do so, we utilized the low-frequency x86 cores that is embedded on Loihi for interfacing with the on-chip networks during runtime. The sequence of operations were as follows: First, the x86 chip computed the weighted sum of output spikes (K)(K) as per Equation 15
(K)
Ew0..000 (0)())
(15)
SuM
Then, the computed sum was sent to the host machine to be converted to final class probabilities through the non-spiking fully-comnected decoding layer as follows,
Palass
aind (K)
b out
(16)

---

Published in Transactions on Machine Learning Research (06/2022)
Table 1: List of hyperparameterss
Hyperparammeters
Reaction Time
Directions
EEGMMIDB
EEGMMIDB w/Loihi
Segmentation intervals
-0.5-0s
-0.5-1,5s
O-1s
0-1s
Initial voltage decay (D,)
0.1
0.1
0.1
0.1 (Fixced
Initial current decay (D.)
0.1
0.1
0.1
0.1 (Fixed)
Initial voltage threshold (Vi.)
0.1
0.1
0.1
0.2 (Fixed)
Feedforward timesteps
125
500
160
160
Convolutional layer architecture
61C1-128C2-266C3
61C1-128C2-266C
61C1-128C2-266C3
4C1-8C2-128C3
Timesteps for temporal convolationa((vinn)
3
3
3
Pseudograd amplifier (a1)
0.3
0.3
0.3
0.3
Pseudograd window (#2)
0.3
0.3
0.3
0.3
Weight learning
1c-4
1o-4
1o-4
lc44
Decays learning
1e-3
1e-3
le-4
Batch size
64
64
64
where the Palass computed here is mathematically equivalent to the definition in Equation
We usod layer-wise rescaling to map the full-precision weights of the trained SNN to the bits integer weights on Loihi (Tang et al. 2020). Each convolution operation defined in Equation had a separate group of physical connections Loihi, with all groups sharing the same weights. The temporal convolution operation was realized through spike delays which Loihi supports in the form of buffered presynaptic spikes on the neuron cores. Due to hardware limitations, the SNN realized on Loihi did not have neurons with adaptive thresholds and decays.
3.5 Network Perturbation
To interpret the features that the SNN used in its decision-making performed network perturbation study for correlating changes (perturbations) in the spectral bands of the EEG inputs with the network outputs (Schirrmeister et al.] 2017). Doing so allowed us to determine the causal effect of changing the band-power on the network output. To perturb the spectral bands of the FEG trials, we first converted the trials into the frequency domain using Fourier transformation. Then, we added a small Gaussian noise (with mean and variance 1) to the amplitudes while keeping the phases unchanged. Nexi, we transformed the signals back into the time domain using inverse Fourier transform, which gave us the perturbed EEG trials. Then, we computed the network output for each layer for the unperturbed and perturbed trials. Finally, we measured the correlation in the change in the network outputs with the change in the spectral amplitudes.
4 Experiments and Results
To demonstrate the effectiveness of our method, we applied to motor decoding tasks and validated on two datasets: i) an in-house dataset consisting of complex components of arm movements performed on a plane, and ii) the public cegmmidb dataset (Schalk et al.) 2004) for motor imagery and movement. Then, we performed network perturbation study to demonstrate the neurophysiological interpretability of our method. Lastly, we deployed the trained networks on Loihi to highlight the energy efficiency of our method.
4.1 Data Preprocessing
We preprocessed the EEG data to get rid of the contaminations. The preprocessing was minimal to allow the SNN to learn the representations by itself. To remove the low and high-frequency artifacts and drifts, applied zero-phase FIR bandpass filter of H..zzz00HHz (for the in-house dataset) and 0. Hzz-80lz (for cegmmidb). We then applied independent component analysis (ICA) (Radüntz et al.) 2015) to the data from each subject to get rid of ocular artifacts. To do so, we applied the FastICA (Hyvärinen & Ojal 2000) algorithm to construct 25 ICA components. The components corresponding to ocular artifacts were then manually identified and removed. Subsequently, we sogmented the data into trials containing the events of interest. The time windows used for segmentation for the different classification tasks are provided in Table For the in-house dataset, applied baseline correction from the beginning of the data until the time
8

---

Published in Transactions on Machine Learning Research (06/2022)
Table 2: Two-class classification of Reaction Time evaluated using leave-one-subjeeclout (%).
Subject
3D-CNN
2D-CNN
CNN-TC
CNN-LSTM
EEGNet
SNN (Ours)
1
80.00
67.30
82.81
67.85
72.22
87.03
2
79.68
65.85
77.78
70.58
78.57
80.35
3
81.03
69.64
84.37
54.68
71.15
84.61
4
82.25
81.96
76.47
70.58
73.17
75.61
5
79.68
73.21
73.21
69.41
78.68
80.32
6
79.51
58.53
75.00
70.58
73.43
81.53
7
74.51
77.58
79.31
65.95
74.11
75.29
8
80.00
79.31
82.75
67.21
79.68
92.30
9
73.91
76.78
84.37
64.06
79.31
82.75
10
85.33
72.41
84.12
74.60
82.81
89.06
11
81.57
82.35
80.95
60.93
70.21
76.59
12
80.26
80.32
80.85
73.43
80.95
77.81
Average
79.81 3.07
73.77 7.34
80.17 3.82
67.49 = 5.55
76.19 4.24
81.93 + 5.45
point zero get rid of unrelated temporal drifts. Lastly, normalized the trials after segmentation using z-score normalization with statisties computed across all the trials of training data. More details the preprocessing steps are provided in Appendix B.
4.2 Datasets, Baselines and Evaluation
We performed an IRB-approved experiment in which 12 subjects performed goal-directed movements plane in response to visual stimuli using BioNik's InMotion Arm Rehabilitation Robot. The experiment protocol is described in Appendix A. High density EEG data were simultaneously acquired using a 128- sensors Biosemi ActiveOne EEG system at 1024Hz., which was later downsampled to 250Hz. The resulting dataset consisted of i) Reaction time (RT), measured as the time difference between the onset of stimulus and the start of the movement. We discretized the RT into two classes- fast and slow, by choosing suitable thresholds determined from the histogram of RTs for each subject separately, based on the distribution of their RT across the experiment. The RT dataset consisted of 50 concurrent EEG trials average per class for each subject; ii) Directions: We separated the data into 4 classes representing the 4 orthogonal directions left, right, up, and down movements. This dataset consisted of 52 concurrent EEG trials, per direction, for cach subject.
The publicly available eegmmidb datasel comprised of EEG recordings from 109 subjects acquired using a 64-channel system during motor movement and imagery tasks (Schalk et al.) 2004). Subjects were required to move imagine moving their fists feet in response visual cucs. We trained the SNN to classify whether the movement was performed using the left hand, or the right hand, the feet, for both real and imagined movements. Each class had 2524 EEG trials on average.
We compared our mothod against the following statc-of-the-art DNN mothods:
A 3D-CNN comprising of 3D convolutional layers extract spatiotemporal features from EEG inputs with topographical representation (Kunar Michmizos 2022
A 2D-CNN comprising of several layers of 2D convolutions extract spatiotemporal features from stacked EEG inputs (Kumar & Michmizos).
A CNN with separate temporal and spatial convolutions (CNN-TC) (Schirrmeister al. 2017
4. CNN-LSTM where the CNN learns the spatial features and the LSTM learns the temporal fea- tures (Zhang et al. 2018).
compact convolutional network, EEGNet, consisting of temporal convolution, depthwise convolu- tion, and separable convolution followed by pointwise convolution (Lawhern et. al. 2018).

---

Published in Transactions on Machine Learning Research (06/2022)
Table 3: Four-class classification of Directions evaluated using leave-one-suhjeett-ot (%)
Subject
3D-CNK
2D-CNN
CNN-TC
CNN-LSTM
E.GNNet.
SNN (Ours)
82.85
76.19
82.85
53.33
80.00
83.81
83.81
74.28
76.19
70.47
80.95
87.71
72.38
63.81
69.52
56.19
72.38
69.52
84.76
80.00
82.85
54.28
81.90
82.85
77.14
67.61
74.28
46.66
68.57
80.95
72.38
59.04
65.71
49.52
64.76
73.33
81.20
73.15
74.49
51.67
77.18
83.22
8
87.61
71.42
82.85
70.47
81.90
83.81
83.81
61.90
64.76
42.85
70.47
68.57
10
85.71
59.04
75.28
30.47
74.28
83.81
11
95.23
86.66
82.87
42.85
92.38
86.67
77.14
67.61
73.33
57.14
69.52
75.23
Average
82.00 + 6.24
70.05 + 8.53
75.41 + 6.54
52.16 + 10.79
76.19 + 7.68
79.95 + 6.28
Given that the baselines employed different EEG systems requiring network architecture changes, imple- mented our own versions of the basclines. The hyperparameters for the basclincs and the proposed method were optimized using a held-out set consisting of data from 1 subject for the in-house dataset, and 9 subjects for the eegmmidb dataset. We used the leave-k-out technique for evaluation, where data. from all but k subjects were used for training. The evaluation was done on the data from the left-out subjects. This evaluation technique allowed us to evaluate the ability of the SNN to generalize to new subjects that were trained different
not included in the training. To ensure that all the subjects were covered for evaluation, networks with different folds of k subjects left out from training. We then computed the average of the classification accuracies obtained from all the folds. For the in-house dataset, & 1, L.e. there were 11 training subjects and 1 test subject. For the eegmmidb dataset, k = 10, which meant that there were 90 training subjects and 10 test subjects. The list of hyperparamcters for training is provided in Table 1
4.3 Classification of Complex Movement Components
When evaluated using the leave-k-out technique (k: = 1), our SNN achieved comparable performance as the DNN methods on both the RT and directions classification tasks, achieving average accuracy of 81.93% + 5.45% for RT (Table 2 and 79.95% + 6.28% for directions (Table 3). To further highlight the importance of the diffcrent components in the SNN architecture, we performed an ablation study where we a) removed the recurent and temporal convolutional layers in the network ('No recurrent & TC*) while keeping all the other components intact b) removed the recurrent layer in the network ('No recurrent") while keeping all the other components intact; and e) trained the network without adaptation of decays or voltage thresholds ('No adaptation") while keeping all the other components intact. When compared against our SNN (Table D. the networks with 'No recurrent & TC', "No recurrent', ; and "No adaplation" suffered from drop in accuracios, thereby highlighting the importance of these components towards classification.
4.4 Classification of limb for Motor Execution and Imagery
For further validation, we evaluated the SNN on the publicly available eegmmidb dataset which comprises of EEG recordings from 109 subjects acquired using 64-channel system during motor movement and imagery tasks (Schalk et al. 2004). Subjects visual cucs. We trained the SNN to classify whether the movement was performed using the left hand, or the right hand, or the feet. When evaluated using the leave-k-out technique (k =k=100 10), the SNN achieved similar levels of performance as the shaate-oftthearte DNN methods (Table 5). Overall, our method successfully bridged the performance gap between existing SNN methods and the DNN methods, thereby suggesting SNN as a possible alternative DNN for EEG classification. Additional classification results are presented in Appendix E.
required to move imagine moving their fists feet in response to
10

---

Published in Transactions on Machine Learning Research (06/2022)
Table 4: Ablation: Classification of Directions (%)
Subject
No recurrent
TC
No recurrent
No adaptation
SNN (Ours)
75.23
78.09
81.90
83.81
2
78.09
80.95
84.76
87.71
3
54.28
65.70
63.81
69.52
4
78.09
80.00
80.95
82.85
5
60.09
64.76
79.04
80.95
6
59.04
67.61
69.52
73.33
7
68.45
77.85
80.53
83.22
8
74.28
79.19
81.90
83.81
9
55.23
65.71
72.38
68.57
10
60.95
75.23
76.19
83.81
11
70.47
85.71
85.71
86.67
12
72.38
74.28
76.19
75.23
Average
67.21 8.82
74.59 + 7.01
77.74 + 6.48
79.95 6.28
Table 5: Classification of Motor Movement/Tmagery evaluated using leave- 1have-10-uubjccts-ont (%)
Task
Real Imagined
3D-CNN
2D-CNN
CNN-TC
CNN-LSTM
EEGNet
SNN (Ours)
Hand (T.) va. Hand (R)
Real
82.68 + 3:22
78.51 + 3.45
78.37 + 3.92
75.48 + 3.11
74.64 + 4.84
80.27 3.87
Hand (L) vs. Feet
Real
77.95 1: 3.99
75.53 + 4.67
74.80 1::00::0 4.81
4.81
70.72 + 4.65
70.17 2.94
77.69 +: 4.00
Hand (R) vs. Feet
Real
74.63 2.43
72.55 1::5-=00 4.83
71.21
70.23 1.78
3.51
Hand (L) vs. Hand (R)
Imagined
81.02 1012::2.00 2.72
79.84 3.75
78.86 10.6=300 3.63
74.22 11.2+3.11
76.02
80.65 3383
Hand (L) vs. Foot
Imagined
75.60 3.55
74.62 1 3.94
73.77 1 2.69
70,95
14229
70.81 1211
74.33 3.90
Hond (R) vs. Foet
Imagined
73.48 1: 1.66
73.07 + 1.89
71.70
69.67 4.13
1
72.31 1.79
4.5 Network Interpretation using a Perturbation Study
Neuroimaging studies have shown that motor actions are associated with spectral power modulations at multiple frequency levels (Jasper & Penfield| 1949: Jeon et al.) 2011] Pfurtscheller et al. 20066. Hence, EEG band power features useful in the decoding tasks. To determine if the SNN used these features in its dccision-making, we performed network perturbation study to correlate the changes in the spectral bands with the network outputs (see Methods). The network perturbation study was performed on the eegmmidb dataset since more domain knowledge is available for the corresponding classification tasks, allowing us to validate our findings.
We averaged the correlation between spectral perturbation and network oulput (see Methods) for left and right motor cortex electrodes, and each frequency range (Delta: 1-4 Hz, Theta: 5-8 Hz, Alpha: 9-13 Hz, Beta: 14-30 Hz) over all the trials of representative validation subject, resulting in spectral map (Fig. Left). The averaged correlation values were then rescaled to be in the range [-1, 1] for facilitating comparisons across different frequency bands and across different subject groups. The raw absolute maximum correlation values are presented in Appendix D. For obtaining the topographical maps, we repeated the above procedure for all the electrodes. The computed topographical maps (Fig. Right) showed localized regions that the network possibly exploited its decision-making.
All spectral bands showed some degree of correlation for the movement and imagery tasks, with the delta band being the most prominent (Fig. 4 This is in alignment with recent studies implicating the role of low-frequency oscillations in predicting handedness of the movement (Hamel-Thibault et al. 2018) and in movement preparation (Caspers et al. 1984 Birbaumer et al.) 1990) and control (Tarkka & Hallett 1990; Yilmaz et. al., 2015). In addition, the lopography of the delta band correlations suggest localized patterns in both the movement and imagery tasks. Specifically, for the movement task, an amplitude decrease in the right sensorimotor area led an increase in the prediction of the Hand (I) class and vico-versa for the Hand (R) class. This localized pattern was previously observed experimentally in Thibault et al. 20118: Hoshi & Tanji (2006); Beurze et al. 2009) for hand selection in reaching tasks, and also for DNNs
11

---

Published in Transactions on Machine Learning Research (06/2022)
in Schirrmeister et al. 2017]. The opposite effect observed in the case of imagery tasks, suggesting differences in the activation patterns of the two lasks, consistent with the observations in (Batula et al. 2017). These patterns were less distinct for the alpha band, potentially due to the weaker correlation of the network's output with perturbation in the alpha band. Nevertheless, we see hint of similar patterns in the lower motor corlex for alpha band. However, as noted in Schirrmeister al. (2017), the perturbation results reflect the behavior of the trained model, and any interpretation about the data itself must be carefully made.
Movement
Hand (L)
Hand (R)
Beta
Alpha
Alpha
Theta
Delta
Hand (L)
Hand (R)
Hand (L)
Hand (R)
Delta
Left Motor Cortex
Right Motor Cortex
Correlation
Imagery
Hand (L)
Hand (R)
Beta
Alpha
Alpha
Theta
Delta
Hand (L)
Hand (R)
Hand (L)
Hand (R)
Delta
Left Motor Cortex
Right Motor Cortex
Correlation
Figure Neurophysiological network interpretation. The spectral maps oblained using the network per- turbation study identified all frequencies as being correlated with the movement tasks, with delta being the most prominent (a. Left, b. Left). The topographical maps of the correlations reveal contralateral activation pattern for the movement task (a. Right), and opposite pattern for the imagery task (b. Right). The maps are averaged over the trials from all the test subjects in one fold. The maps for the remaining folds are provided in Appendix D.Correlation values are scaled to be in the range [-1,1].
4.6 Energy-efficient Inference Loihi
For energy-efficient inference, we deployed the SNN trained for Hand (L) vs. Hand (R) (Real) task on Loihi. Due hardware limitations and allow for high inference speed, the SNN on Loihi had reduced number of convolutional kernels (Table []] and did not have neurons with adaptive threshold and decays. This resulted in marginal decrease in the performance of the SNN Loihi when compared against SNN full precision machine (Table 6). We computed the inference speed and energy consumption for EEG inference of the SNN on Loihi, and compared them against the SNN and DNN methods on the embedded AI chip NVIDIA Jetson TX2 (operating in energy-efficient mode Q). We computed the energy cost per inference as the ratio of the dynamic power and the number of inferences per second. Our SNN on Loihi consumed 95% less energy per inference than the 3D-CNN on the low-power processor for DNNs- Jetson TX2 (Table 6.. It worth noting that the encoding and decoding computations that took place outside of the Loihi constitute only 2.9% of the overall computations, and hence do not contribute to the energy costs much.
12

---

Published in Transactions on Machine Learning Research (06/2022)
Table 6: Performance across network architectures and hardware for Hand (T.) vs. Hand (R) (Real)
Network (Device)
SNN (Loihi)
SNN (TX2)
2D-CNN (TX2)
3D-CNN (TX2)
Accuracy (%)
78.04 + 3.25
79.40 + 4.03
78.51 + 3.45
82.68 + 3.22
Static Power (W)
1.639
1.606
1.672
1.691
Dynamic Power (W)
0.0367
4.871
4.040
4.481
Speed (Inference/s)
10.2
1.38
73.2
59.5
Energy /Inffeeeceee)
3.60
3529.71
55.20
75.31
5 Discussion and Conclusion
In this work, we demonstrated that through the synergy of a) an appropriate input encoding, b) network topology that reflected data priors, and c) a powerful learning algorithm, SNNs can reach the high perfor- mance of the DNNs in modeling intrinsically variable EEG signals, while being orders magnitude more energy efficient than their mainstream counterparts. To that effect, our SNN extracted task-discriminative spatiotemporal EEG features for motor imagery and movement classification and achieved similar levels of performance as the state-of-the-art DNN methods. The energy-efficient inference on Loihi further demon- strated the practical advantages of the emerging neuromorphic technology. Moreover, the correspondence of the learned features with the underlying neurophysiology increased the reliability of our method, allowing it to be placed in real-world portable BCI systems.
This work tackles an important gap in SNN approaches, that has hindered them from becoming a main- stream technology and replace or complement DNNs. While SNNs are arguably better suited than DNNs for spatiolemporal lasks (Kugele el al. 2020: Deng el. al. 2020 Reid et al. 2014). there has remained significant performance gap between SNN and DNN for EEG classification (Antelis et al. 2020 Luo et al., 2020 Kasabov 2014). We showed here that appropriately constructed SNN that incorporates the high spatiotemporal dependencies in its input encoding and network design through specialized layers (convolu- tional, temporal convolutional, and recurrent), and uses gradient-based learning rules as opposed to local learning rules could achieve similar levels of performance as the DNN methods. This is evident through the high leavo-one-out accuracy achieved by our method, thus highlighting its practical effectiveness in decoding inherently noisy EEG data.
Further practical gains of our method is demonstrated through neuromorphic realization, where our method consumed 95% less energy per inference than the DNN methods when deployed Loihi while oblaining comparable performance levels. The significance of this result lies in applications such as EEG- based portable BCI systems, which run on limited onboard power sources (Zhang et al., 2019 Kartsch et al., 2019). The use of DNN methods running on conventional microprocessors can quickly deplete the energy resources on such devices. On the other hand, our proposed method can arguably decrease the high energy costs by a large factor for such applications. In fact, further improvements in energy efficiency can be achieved by employing memristive neuromorphic processors (Ankit et al.) 2017
The decisions taken by the SNNs must be supported by the domain knowledge for the method to be reli- able enough for use in real-world applications. It is common knowledge that an unreliable SNN may use incorrect features to make its decisions, and achieve high accuracy on the limited validation set while ex- hibiting unintended consequences when deployed in the real-world (Sturm et al., 2016). That is why the interpretability of our SNN was central in this work. We interpreted the SNN using a perturbation study that identified the low-frequency oscillations and localized patterns of activations, both of which align well with the current neurophysiolloical knowledge for motor tasks (Hamel- Thibault et al. 2018 Batula et all, 2017). The correspondence of the learned features with the underlying neurophysiology increases the relia- bility of our method to be placed in critical real-time systems, including the ones for enhancing functional motor recovery (Chaudhary el al. 2016 Donoghue 2018
This paper describes our efforts in reducing the performance gap between SNNs and DNNs for EEG classi- fication. To this end, we suggested novel network architectures and learning rules that reflect the inductive biases in F.G.G Further performance improvements may be achieved by exploiting the advances in the training
13

---

Published in Transactions on Machine Learning Research (06/2022)
of DNN. Some of these include the development of SNN-equivalent of normalization layers such as Batch- Norm for stabilizing the training (Toffe & Szegedy 2015); attention-modules for capturing arbitrary lengths of spatial and temporal dependencies (Vaswani et al. 2017); and learning rules that more closely approxi- mate ANN backpropagation gradients. Efforts in these directions are already being made with architectures
that incorporate normalization layers (Zheng et al., 2020), temporal atlention modules (Yao et al.) 2021), and learning rules that do not rely on surrogate gradients (Wu al., 2021p. The enhanced performance, combination with the interpretability, can make our method directly deployable to any real-world task that involves decoding brain behavior from the inherently noisy EEG signals. Overall, our results demonstrate the effectiveness of SNN solutions in classifying EEG signals by discovering task-relevant features in the recorded brain activity, that can be harvested in an energy-efficient way.
Acknowledgements
This work supported through the Grant K12HD093427 from the National Center for Medical Rehabilitation Research, NTH/NICHD; and Intel's Neuromorphic Rescarch Community Grant Award.
References
U Rajendra Acharya, Shu Lih Oh, Yuki Hagiwara, Jen Hong Tan, and Hojjat Adeli. Deep convolutional neural network for the automated detection and diagnosis of seizure using eeg signals. Computers in Biology and Medicine, 100:270-278, 2018.
Wael Alzhrani, Maryam Doborjeh, Zohreh Doborjeh, and Nikola Kasabov. Emotion recognition and un- derstanding using eeg data in brain-inspired spiking neural network architecture. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1 9. IEEE, 2021.
Kai Keng Ang, Zheng Yang Chin, Haihong Zhang, and Cuntai Guan. Filter bank common spatial pattern (fbcsp) in brain-come crain-computer interface. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 2390-200000 2397. IEFE, 2008.
Aayush Ankit, Abhromil Sengupta, Priyadarshini Panda, and Kaushik Roy. Respare: A reconfigurable and energy-efficient architecture with memristive crossbars for deep spiking neural networks. In Proceedings of the 54th Annual Design Automation Conference 2017, pp. 1-6, 2017.
Javier M Antelis, Luis E Falcón, et al. Spiking neural networks applied to the classification of motor tasks in eeg signals. Neural Networks, 122:130-143, 2020.
Pouya Bashivan, Irina Rish, Mohammed Yeasin, and Noel Codella. Learning representations from eeg with deep reuurent-connvouutinnal neural networks. arXiv preprint arXiv:1511.0644 2015.
Alyssa M Batula, Jesse A Mark, Youngmoo E Kim, and Hasan Ayaz. Comparison of brain activation during motor imagery and motor movement using fnirs. Computational Intelligence and Neuroscience, 2017, 2017.
Maxime O Baud, Jonathan K Kleen, Emily A Mirro, Jason C Andrechak, David Stng-Stephens, Edward F Chang, and Vikram R Rao. Multi-day rhythms modulate scizure risk in epilepsy. Nature Communications, 9(1):1-10, 2018
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short- term memory and learning-to-learn in networks of spiking neurons. arXie preprint arXiv:1803.0954, 2018.
Chris Berka, Daniel J Levendowski, Michelle N Lumicao, Alan Yau, Gene Davis, Vladimir T Zivkovic, Richard E Olmstoad, Patrice D Tremoulet, and Patrick L, Craven. Eeg correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, Space, and Environmental medicine, 78(5):B231-BB)) 2007.
14

---

Published in Transactions on Machine Learning Research (06/2022)
Sabine Maria Beurze, Floris Pieter De Lange, Ivan Toni, and W Picter Medendorp. Spatial and effector processing in the human parietofrontal network for reaches and saccades. Journal of Newrophysiology, 101 (6):3053 3062, 2009.
Luzheng Bi, Xin-An Fan, and Yili Liu. Eeg-based brain-controlled mobile robots: A survey. IEFE Transac- tions on Huchine Systems, 43(2):161 176, 2013.
Niels Birbaumer, Thomas Elbert, Anthony G Canavan, and Brigitte Rockstroh. Slow potentials of the cerebral cortex and behavior. Physiological Reviews, 70(1):1 41, 1990.
Achim Bucrkle, William Eaton, Niels Lohse, Thomas Bamber, and Pedro Ferreira. Eog based arm movement intention recognition towards enhanced safety symbiotic human-robot collaboration. Robotics and Computer-Integnated Manufacturing, 70: 70:102137, 2021.
Boris Burle, Laure Spieser, Clémence Roger, Laurence Casini, Thierry Hasbroucq, and Franck Vidal. Spatial and temporal resolutions of eeg: Is it really black and white? a scalp current density view. International Journal of Psychophasiologyy 97(3):210-22) 220, 2015.
Heinz Caspers, Erwin-Josef Speckmann, and A Lehmenküüller. Electrogenesis of slow potentials of the brain. In Relf-Regulation of the Brain and Behavior, pp. 26-41. Springer, 1984.
Ujwal Chaudhary, Niels Birbaumer, and Ander nication and rehabilitation. Nature Reviews Neurology, 12(9):513, 2016.
Murguialday, Brain computer interfaces for commu-
Marlene R Cohen and Adam Kohn. Measuring and interpreting neuronal correlations. Nature Newroscience, 14(7):811, 2011.
Heng Cui, Aiping Liu, Xu Zhang, Xiang Chen, Kongqiao Wang, and Xun Chen. Eeg-based emotion recog- nition using an end-to-end asymmerrii convolutional neural network. Knowledge-Bor Basedd Based Systems, 205:106243, 2020.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82 99, 2018
Lci Deng, Yujic Wu, Xing Hu, Ling Liang, Yufci Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xic. Rethinking the performance comparison between smns and anns. Neural Networks, 121:294 307, 2020.
Zohrch Gholami Doborjeh, Nikola Kasabov, Maryam Gholami Doborjeh, and Alexander Sumich. Modelling peri-percepturl ual brain processes in a deep learning spiking neural network architecture. Scientific Reports, 8(1):1-13, 2018
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5840 5848, 2017.
John P Donoghue. Brain computer interfaces: Why not better? In Neuromodulation, pp. 341 356. Elsevier, 2018.
Zhongke Gao, Xinmin Wang, Yuxuan Yang, Chaoxu Mu, Qing Cai, Weidong Dang, and Siyang Zuo. Eeg- based peatio-tempone convolutional neural network for driver fatigue evaluation. IEEE Transactions Newral Networks and Learning Systems, 30(9):2755 2763, 2019.
Angel Perez Garcia, Ingrid Schjolberg, and Serge Gale. Eeg control of an industrial robot manipulator. In 2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom), pp. 39 44. IEEE, 2013.
Gaurav Gupta, Sérgio Poquito, and Paul Bogdan, Ro-thinking cog-basod non-invasive brain interfaces: Modeling and analysis. In 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS), pp. 275 286. TEFE, 2018.
15

---

Published in Transactions on Machine Learning Research (06/2022)
Audrey Hamel-Thibault, François Thénault, Kevin Whittingstall, and Pierre-Michel Bernier. Delta-band oscillations in motor regions predict hand selection for reaching. Cerebral Corlex, 28(2):574-558 -544, 2018.
Neep Hazarika, Jean Zhu Chen, Ah Chung Tsoi, and Alex Sergejew. Classification of eeg signals using the wavelet transform. Signal Processing, (9(1):1177, 1997.
Eiji Hoshi and Jun Tanji. Differential involvement of neurons in the dorsal and ventral premotor cortex during processing of visual signals for action planning. Journal of Newrophysiology, 95(6):3596 17596-6666, 2006.
Tiflany Hwu, Jacob Isbell, Nicolas Oros, and Jeffrey Krichmar. A self-driving robot using deep convolutional neural networks on neuromorphic hardware. In 2017 International Joint Conference on Neurol Networks (IJCNN), pp. 635-641. IEEE, 2017.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Newral Networks, 13(45)):411445 2000.
Cosimo leracitano, Nadia Mammone, Alessia Bramanti, Amir Hussain, and Francesco Morabito. A convo- lutional neural network approach for classification of dementia stages based on 2d-spectral representation of eeg recordings. Newrocompulingg 323:96 107, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint ar Xrxi:: 1rx:::152200331 031667 2015.
Vladimir Ivanov and Konstantinos Michmizos. Increasing liquid state machine performance with edge-of- chaos dynamics organized by astrocyto-modulatee plasticity. Advances in Newral Information Processing Systems, 34, 2021.
Herbert Jasper and Wilder Penfield. Electrocorticograms in man: effect of voluntary movement upon the electrical activity of the precentral gyrus. Archiv für Psychiatrie und Nervenkrankheiten, (83(1-2):163-174 1949.
Yongwoong Jeon, Chang S Nam, Young-Joo Kim, and Min Cheol Whang. Event-related (dc) synchronization (erd/ers) during motor imagery tasks: Implications for crain-compuler interfaces. International Journal of Industrial Ergonomics, 41 (15):2884438) 2011.
Zhicheng Jiao, Xinbo Gao, Ying Wang, Jie Li, and Haojun Xu. Deep convolutional neural networks for mental load classification based on eeg data. Pattern Recognition, 76:582-595, 2018
Victor Kartsch, Giuseppe Tagliavini, Marco Guermandi, Simone Benatti, Davide Rossi, and Luea Benini. Biowolf: A sub-10-mw 8-channel advanced brain computer interface platform with nine-core processor and ble connectivity. IEEE Transactions on Biomedical Circuits and Systems, 13(5):893-906, 2019.
Nikola Kasabov and Elisa Capecci. Spiking neural network methodology for modelling, classification and understanding of eeg spatio-temporal data measuring cognitive processes. Information Sciences, 294: (75,575, 2015.
Nikola K Kasabov. Neucube: spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data. Neural Networks, 52:62-76, 2014.
Chungsong Kim, Jinwei Sun, Dan Liu, Qisong Wang, and Sunggyun Paek. An effective feature extraction method by power spectral density of eeg signal for 2-class motor imagery-based bei. Medical & Biological Engineering & Computing, 56(9):1645 1658, 2018
Youngeun Kim and Priyadarshini Panda. Visual explanations from spiking neural networks using interspike intervals. arXie prepriat or orXiv:2103. 14441, 2021.
Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca Efficient processing of spatio- temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.
16

---

Published in Transactions on Machine Learning Research (06/2022)
Neelesh Kumar and Konstantinos P Michmizos. Deep learning of movement intent and reaction time for reg-informed adapiation of rehabilitation robots. In 2020 8th IEEE RAS/EMBA International Conference for Biomedical Robotics and Biomechatronics (BioRob), pp. 527 532. IEEE.
Neclesh Kumar and Konstantinos P. Michmizos. A neurophysiologicaly interpretable deep neural network predicts complex movement components from brain activity. Scientific Reports, 12(1). January 2022. doi: 10.1038/s41198--000 00006/254915000022200000---9 URL http://p///ooo.ooo//////////////
Vernon J Lawhern, Amelia J Solon, Nicholas R. Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. Eegnet: a compact convolutional neural network for eeg-based train-computer interfaces. Journal of Neural Engineering, 15(5)0056013, 2018.
Mikhail A Lebedev and Miguel AL Nicolelis. Brain-machine interfaces: From basic science neuroprostheses and neurorehabilitation. Physiological Reviews, 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553)):366 444, 2015.
Chit-Kwan Lin, Andreas Wild, Gautham N Chinya, Yonggiang Cao, Mike Davies, Daniel M Lavery, and Hong Wang. Programming spiking neural networks on intel's loihi. Computer, 51(3):52 61, 2018.
Fabien Lotte and Cuntai Guan. Regularizing common spatial patterns to improve BCT designs: unified thcory and new algorithms. IEEE Transactions on Biomedical Engineering, 58()):35--362, 2011.
Yuling Luo, Qiang Fu, Juntao Xie, Yunbai Qin, Guopei Wu, Junxiu Liu, Frank Jiang, Yi Cao, and Xuemei Ding. Eeg-based emotion classification using spiking neural networks. IEEE Access, 8:46007-46016, 2020.
Henry Markram, Joachim Lübke, Michael Frotscher, and Bert Sakmann. Regulation of synaptic efficacy by coincidence of postsynaptic aps and epsps. Science, 275(5297):213221) 1997.
Grégoire Montavon, Wojeiech Samek, and Glluss-Robers Müller. Methods for interpreting and understanding doep neural networks. Digital Signal Processing, 73:1-15, 2018.
Balint Petro, Nikola Kasabov, and Rita M Kiss. Selection and optimization of temporal spike encoding methods for spiking neural networks. IEEE Transactions Neural Networks and Learning Systems, 31 (2):358-377 370, 2019.
Gert Pfurtscheller, Clemens Brunner, Alois Schlögl, and FH Lopes Da Silva. Mu rhythm (de) synchronization and eog single-trial classification of different motor imagery tasks. Neurolmage, (1)1:1153-15)) 2006.
T Radüntz, J Scouten, O Hochmuth, and B Meffert. EEG artifact elimination by extraction of ICA- component features using image processing algorithms. Journal of Neuroscience Methods, 243:84-93, 2015.
Nitin Rathi and Kaushik Roy. Diel-smm: Direct input encoding with leakage and threshold optimization deep spiking neural networks. arXiv preprint 2r08v:2008. 03658, 2020.
David Reid, Abir Jaafar Hussain, and Hissam Tawfik. Financial time series prediction using spiking neural networks. PloS One, (8)3c10365,, 2014.
Neethu Robinson, A Prasad Vinod, Cuntai Guan, Kai Keng Ang, and Tee Keng Peng. A modified wavelet- common spatial pattern method for decoding hand movement directions in brain computer interfaces. In The 2012 International Joint Conference on Neural Networks (LJCNN), pp. 1-5. IEEE, 2012.
Wojeiech Samek, Carmen Vidaurre, Klaus-Robert Müller, and Motoaki Kawanabe. Stationary common spatial patterns for brain computer interfacing. Journal of Neural Engineering, 9(2):026013, 2012.
Gerwin Schalk, Dennis J McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan R Wolpaw. Bci2000: general-purpose brain-computer interface (bci) system. IEEE Transactions on Biomedical Engineering. 51())10034-104) 2004.
17

---

Published in Transactions on Machine Learning Research (06/2022)
Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermarn, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, 38(11):339155420, 2017.
Benjamin Schrauwen and Jan Van Campenhout. Bsa, fast and accurate spike train encoding scheme. In Proceedings of the International Joint Conference Neural Networks, 2003., volume 4, pp. 2825-2883 TEEE, 2003.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference Computer Vision, pp. 618-626, 2017.
Maan M Shaker. Eeg waves classifier using wavelet transform and fourier transform. Brain, 2(3), 2006.
Amar Shrestha, Haowen Fang, Qing Wu, and Qinru Qiu. Approximating back-propagation for biologically plausible local learning rule in spiking neural networks. In Proceedings of the International Conference Newromorphic Systems, pp. 1-8, 2019.
Trene Sturm, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert. Müller. Interpretable deep neural networks for single-trial eeg classification. Journal of Neuroscience Methods, 274:141 145, 2016.
Clarence Tan, Marko Sarlija, and Nikola Kasabov. Neurosense: Short-term emotion recognition and un-
derstanding based on spiking neural network modelling of spatio-temporal eeg patterns. Neurocomputing, 434:137-148, 2021.
Guangzhi Tang, Arpit Shah, and Konstantinos P Michmizos. Spiking neural network neuromorphic hard- ware for energy-efficient unidimensional slam. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4176-4181. IEEE, 2019.
Guangzhi Tang, Neclesh Kumar, and Konstantinos P Michmizos. Reinforcement co-learning of deep and spik- ing neural networks for energy-efficient mapless navigation with neuromorphic hardware. arXiv preprint arXiv:2003.00 201.00003.0 01157, 2020.
Guangzhi Tang, Neclesh Kumar, Raymond Yoo, and Konstantinos Michmizos. Deep reinforcement learning with population-coded spiking neural network for continuous control. In Conference on Robot Learning, pp. 1-10. PMLR, 2021.
IM Tarkka and Mark Hallett. Cortical topography of premotor and motor potentials preceding self-paced, voluntary movement of dominant and non-dominant hands. Eleetrenceehalogrpphy and Clinical Neuro- physiology, 75(1-2):36 1990.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Jibin WU, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions on Newral Networks and Learning Systems, 2021.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training pegohperformance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018
Zhanglu Yan, Jun Zhou, and Fai Wong. Eeg classification with spiking neural network: Smaller, better, more energy efficient. Smart Health, pp. 100261, 2022.
Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li. Temporal-wise attention spiking neural networks for event streams classification. arXio preprint arXiv:2107117111, 2021.
18

---

Published in Transactions on Machine Learning Research (06/2022)
Ozge Yilmaz, Niels Birbaumer, and Ander Ramos- Murguialday. Movement related slow cortical potentials in severely paralyzed chronic stroke patients. Frontiers in Human Neuroscience, 8:1033, 2015.
Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with multiple- timescale spiking recurrent neural networks. arXiv preprint arXiv:2005. 1633, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXi:::611.035300 2016.
Dalin Zhang, Lina Yao, Xiang Zhang, Sen Wang, Weitong Chen, Robert Boots, and Boualem Benatallah. Cascade and parallel convolutional recurrent neural networks on eeg- beg-based intention recognition for brain computer interface. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Yu Zhang, Xiong Zhang, Han Sun, Zhaowen Fan, and Xucfei Zhong. Portable brain-computer interface based novel convolutional neural network. Compulers in Biology and Medicinee 107:248 007:248-256, 2019.
Xinqiao Zhao, Hongmiao Zhang, Guilin Zhu, Fengxiang You, Shaolong Kuang, and Lining Sun. A multi- branch 3d convolutional neural network for eeg-based baged motor imagery classification. IEEE Transactions on Newral Systems and Rehabilitation Engincering, 27(10))2164-2177 2019.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. aixie preprint urXiv:2011.0528 2020.
Xiao Zheng and Wanzhong Chen. An attention-base bi-lstm method for visual object classification via cog. Biomedical Signal Processing and Comtrol, 63: 63:102174, 2021.
19


Reviews:
[{"review_id": "uuR2BSpBJe", "reviewer": "Reviewer_Gshi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The manuscript describes an interesting SNN architecture. The main issues to me are that the evaluation on public datasets/comparison to prior work is limited and I could not see a statement about code availability?\n\n1. In my view, to ensure that the research community can properly assess the decoding performance of spiking neural networks, there need to be direct comparisons to published results. The manuscript however only provides comparisons to reimplemented baselines “Given that the baselines employed different EEG systems requiring network architecture changes, we implemented our own versions of the baselines and optimized the hyperparameters using a separate held-out set.”... I would much favor to still see comparison to published results, on eegmmidb or also BCI Competition IV  datasets, e.g. BCI Competition IV 2a… This would greatly help assess if/how big of a performance gap there is, since the compared results will have been well-tuned.\n\n2. Why are the comparisons on eegmmidb only binary and not multiclass?\n\n3. Code availability is always a plus as it allows other researchers to apply the proposed model to other datasets.\n\n4. Regarding the energy-efficiency/energy comparisons, I am not so sure how to interpret them due to: “ The convolutional spike encoding layer was implemented on the host machine, with the generated spikes sent to Loihi in spike-time buffers to avoid redundant communications at every timestep. The SNN decoding module was implemented on Loihi.” Is this common practice that some part of the model is not taken into account when computing energy consumption? Isn’t this comparison unfair then, assuming e.g. 2d-CNN all the energy is taken into account, while for SNN some part is left out? How big is this part, the convolutional spike encoding layer? Some clarifications here would be helpful.\n\n5. Also “SNN on Loihi had reduced number of convolutional kernels (Table 1) and did not have neurons with adaptive threshold and decays.”.. Shouldn’t this then also be done for the SNN outside of Loihi? Also for “We used layer-wise rescaling to map the full-precision weights of the trained SNN to the 8 bits integer weights on Loihi (Tang et al., 2020).”\n\n6. Also a CNN that was designed to be compact, EEGNet, https://iopscience.iop.org/article/10.1088/1741-2552/aace8c/meta, should be added to the comparisons in my view.\n\n7. The ablation studies in table 4 are helpful, another column only removing recurrence, but keeping temporal convolution, would be informative.\n\n8. Which projection exactly do you use for the 2d-grid EEG projection? Would be nice to include this detail.\n\n9. Regarding the visualization, would be good to also include some caveats of what can be learned from such perturbation visualizations as mentioned inhttps://onlinelibrary.wiley.com/doi/full/10.1002/hbm.23730 and related to the issues mentioned in https://www.sciencedirect.com/science/article/pii/S1053811913010914 \n\nMinor things:\n“To handle the non-differentiability of the threshold function, we used the rectangular function to approximate the gradient of a spike with respect to the membrane voltage v:” Is this a novel idea of this work or has this been done before? Would be good to cite or clarify appropriately.\n\n “we augmented the SNN with a temporal convolution layer and a recurrent layer.” In this part I got a bit confused during the reading, what is now part of SNN, this way it first sounded like there will be some non-SNN-layers added to the SNN. Maybe change the wording a bit to make things very clear here.\n\n“ th_amp is the increase factor of voltage threshold” To me factor implies something is multiplied.. But I am also not a native speaker :)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript describes a novel spiking neural network architecture for EEG decoding. The SNN architecture is evaluated in terms of accuracy and energy usage on EEG movement-decoding tasks. The manuscript claims 95% reduction in energy usage for the SNN implemented on Loihi and comparable decoding accuracy compared to regular convolutional neural networks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Comparisons to published results\n2. Add EEGNet to comparisons\n3. Provide code\n4. Clarify energy efficiency comparison\n5. Ablation only removing recurrence\n6. Discuss visualization caveats\n7. Add projection details\n", "broader_impact_concerns": "No Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7lN6JCGSza", "reviewer": "Reviewer_WM3g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe rational for exploiting spatiotemporal patterns for EEG in a spiking neural network is reasonable. The related work seems applicable.  The efforts towards interpretation seems to be readily comprehensible. The energy efficiency results are very promising. \n\nWeaknesses: \nThe presentation of the network is not clear in how the input is fed to the network.  Everything is given as a function of the previous layer and many equations seem to overload the definition of the post-synaptic potential (psp). \n\nThe specific novel contributions are not clear. Not that novelty is not sufficient but to clarify what was previous work from other authors, especially for the spatiotemporal backpropogation learning rule. \n\nThe hyper-parameter selection in the context of leave-k-out for subjects is not discussed. Basic statistics of sample sizes etc. should be included. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes an approach for using spiking neural network for EEG decoding that will be much more efficient during inference. In particular, the results show that the predication performance (accuracy for discrete decoding) is comparable to current convolutional networks for EEG decoding but can run on specialized hardware for EEG at much higher efficiency. The paper describes an approach that is meant to exploit both the spatiotemporal patterns in EEG.  In covers some aspects of spiking neural network model and training. The contribution is mainly the architecture choices specific to EEG decoding. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Clarifications (answers are critical to recommend acceptance):\nIn Section 3.1, it is not clear if $\\mathbf{D}_c$ and $\\mathbf{D}_v$ have constraints (non-negativity perhaps). If so then how are these constraints enforced during the gradient descent. \n\nIn Equation 4  it is not clear how the neuron at location (i,j) mentioned before the equation relates to $A(i,j)$ in the equation, as after the equation $A$ is described as a convolutional kernel. Where is the input in equation 4? Is the input treated as the previous layer? \n\nThe $psp(\\cdot)$ appears on the left hand side in three different equations,  and it is not clear how to reconcile equations 4, 5, and 6. Are they perhaps supposed to use layer indexes of k from disjoint sets? \n\nEquation 7 may have some indexing errors as there doesn't seem to be a statement relating the parenthetical superscript K-1 in the first line with the K on the second line.\n\nIn Section 3.3., is there any particularly special about the gradients as given. Besides the first line of equation 10 that involves the $z$ function, these seem to be standard and it is not clear if there inclusion is necessary.  What is different from spatiotemporal back propagation of Wu et al. 2018?\n\nIn Section 3.4, I find the mix of hardware details a bit obscure. What is meant by \"low-frequency x86 chip\". Is that one of the processors on the Loihi system on chip? A full hardware description and how this affects the reported energy usage and computation is necessary to understand the import of this statement. \n\nIn section 4.1, details of the bandpass filter (order and type) should be given. Also how were the ICA components associated to ocular artifacts identified, manually? For time windows of multivariate data, the use of z-score normalization is ambiguous. Is this done per trial, across trails (based on only training data), at each time point in the window? It is also difficult because it is not clear how the time points in the window are used by the SNN or DNN. \n\nThe results in 4.3 and 4.4 are those shown in 4.2. The organization and presentation of the results is confusing. Basic information about the datasets (number of channels, number of training subjects, number of trials, validation set choice and usage, etc. should be reported in 4.2 before Tables 2 and 3. Additionally the class distribution and/or chance or naive prediction rate. \n\nIn 4.2, the baseline methods' hyper parameters are said to have used \"a separate held-out set\". It is not clear if that is the same as the proposed method and also how that is used in conjunction with the leave-k-out technique.  In Section 4.4, k=10 for the leave k out. In this case how is hyper-parameter selection via validation performed? \n\nAs the interpretability was done for a single subject, it is unclear how consistent this is. How was the subject selected? Can the interpretation maps for all validation subjects be shown? From a neurophysiological perspective it would be reasonable that they should be roughly the same.  (Also validation trials is mentioned in the Figure 4 caption, which is confusing given the hold-k subjects out method).  \n\nIn the last paragraph of Section 5, please clarify what aspects of the learning rules were novel and important in the context of EEG. \n\nMinor: \nIn Equation 3 please typeset max in roman font and use another symbol instead of asterisk for scalar multiplication to preclude confusion with convolution. \n\nIn Section 4.3 please place quotes or use different typesetting for \"No current and temporal conv\" and \"No adaptation\" in text.  \n \nMathematical equations and expressions should be treated as part of the normal syntax. Commas after equations if they are followed by dependent clauses and periods (full stops) at the conclusion of sentences. There is often no need to use a colon before them in the case \"We define something as:\".\n  \nBibliography style: Please capitalize acronyms consistent with normal usage. Journal names, conference venues, and book title should be capitalized according to title case. ", "broader_impact_concerns": "Nothing is particularly concerning assuming the IRB for human subjects is valid.  More broadly, efficient EEG decoding may enable some technology that is potentially beneficial for creating brain-computer interfaces. However, their could be deleterious impacts to such technology too. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7rWF0efwrh", "reviewer": "Reviewer_1m1u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: I sincerely thank the authors for their nice submission. The paper has a clear perspective, structure and narrative. Neuromorphic realization experiments is an important strength of this work. Overall it was a nice read. \n\nWeaknesses: There are some arising questions with regards to the experimental pipeline, interpretability results, and EEG data analyses that can be clarified in revisions. My comments/questions for the authors’ revisions are listed below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel spiking neural network architecture and training pipeline for decoding of motor movement related EEG data. Experiments are performed on both the EEGMMIDB dataset for motor imagery/execution classification, as well as an in-house dataset for classification of hand movement direction and reaction times. Authors deployed their discriminative SNN pipeline on Intel’s Loihi neuromorphic processor to realistically demonstrate energy-efficient inference. Finally a Fourier domain perturbation based network interpretability method is explored with demonstrations in accordance with neurophysiological evidence.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1) Details with regards to where static or adaptive thresholds were used is a bit unclear in the manuscript. Were they only static for the Loihi deployment phase? Another question for instance: following topography-preserving EEG mapping, a convolutional spike encoding layer is utilized. What was the firing threshold following this layer such that a sufficient amount of incoming spikes could be provided to the encoder block? Was it simply fixed to a constant value, and the learnable encoding layer alleviated any related issues?\n\n2) Could you please elaborate how essential was the use of a non-spiking fully-connected W_{out} layer at the end of the network? Did the authors explore alternative output decoding schemes, such as sum of spikes from o^{(t)(K)}?\n\n3) Can the authors provide more details to the ICA-based ocular artifact removal procedure, as it is currently superficially phrased in the manuscript? Did it involve manual IC selection? How many ICs were constructed initially by the algorithm? Did the authors use pooled ICA across subjects, or ICA applied on each subject’s data per experiment? \n\n4) Please elaborate the phrase: “we normalized the segmented trials using z-score normalization”. Does it actually mean that z-score normalization was applied to trials after segmenting? \n\n5) For the reaction time and direction experiments, was there any baseline correction applied to EEG segments?\n\n6) The paper lacks details on the in-house dataset specifications, and these can be included in depth. How was the experimental paradigm set? With regards to some relevant work, e.g., did it involve center-out reaching movements within a circle [1,2]?\n[1] “Classification of different reaching movements from the same limb using EEG”, Journal of Neural Engineering, 2017.\n[2] “Electroencephalographic identifiers of motor adaptation learning”, Journal of Neural Engineering, 2017.\n\n7) Since data analyses involve cross-subject EEG classification, it naturally brings up the question: did the authors use any kind of EEG transfer learning or stronger model regularization to impose representation invariance in EEG decoding for the baseline networks? For instance the CNN-TC architecture explicitly benefits from stronger dropout in this setting [Schirrmeister et al., 2017].\n\n8) Identifying relevance of low-frequency EEG components for reaction times would align well with previous similar evidence on movement-related cortical potentials (MRCPs). However Figure 4 analyses are performed on the EEGMMIDB motor imagery/execution data of a single subject. In this case one would perhaps expect results to show relevances in the alpha or beta bands for motor imagery/execution [Pfurtscheller et al.]. For a sanity check, it would be clarifying to also present how the other frequency band topographies look like (i.e., similar to the right side of Figure 4 where only delta-band results are shown)? Were there similar findings for some other subjects as well? Would model interpretations look similar for different models (e.g., CNN-TC) than the proposed SNN?\n\nMinor comments:\n- Typo in Table 1: “Psuedograd” -> “Pseudograd”.\n- Please add in captions of Tables 2 and 3 the information on these results being “k=1 fold” or “leave-one-subject-out”, and being averaged over 12 repetitions, and the number of classes per classification task.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After a careful and constructive review process with a good deal of discussion between the authors and the reviewers, 2/3 reviewers have recommended to accept. Final minor revisions to be made for the camera-ready version are:

1) All edits/editions should be converted to standard colour.
2) A link for the code should be added to the footnote on page 2 where it is currently just a place holder.
3) There was a small typo on page 6 that I detected:

"Finally, the spatiotemporal features extracted by the SNN were fed to a decoder module *comprising* of a spiking and a non-spiking fully-connected layer"

"comprising" should be "composed"
Decision: Accept (poster)