BioLCNet: Reward-modulated Locally Connected Spiking Neural Networks
Authors: Hafez Ghaemi
Erfan Mirzaei
Mahbod Nouri
Saeed Reza Kheradpisheh
Venue: ICLR 2022 Submitted (2022)
Keywords: Spiking Neural Networks
Locally Connected Networks
Spike-timing-dependent plasticity
Reinforcement Learning

Abstract: Recent studies have shown that convolutional neural networks (CNNs) are not the only feasible solution for image classification. Furthermore, weight sharing and backpropagation used in CNNs do not correspond to the mechanisms present in the primate visual system. To propose a more biologically plausible solution, we designed a locally connected spiking neural network (SNN) trained using spike-timing-dependent plasticity (STDP) and its reward-modulated variant (R-STDP) learning rules. The use of spiking neurons and local connections along with reinforcement learning (RL) led us to the nomenclature BioLCNet for our proposed architecture. Our network consists of a rate-coded input layer followed by a locally connected hidden layer and a decoding output layer. A spike population-based voting scheme is adopted for decoding in the output layer. We used the MNIST dataset to obtain image classification accuracy and to assess the robustness of our rewarding system to varying target responses.

Under review as a conference paper at ICLR 2022
BIOLCNET: REWARD-MODULATED LOCALLY CON- NECTED SPIKING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies have shown that convolutional neural networks (CNNs) are not the only feasible solution for image classification. Furthermore, weight sharing and backpropagation used in CNNs do not correspond to the mechanisms present in the biological visual system. To propose more biologically plausible so- lution, we designed a locally connected spiking neural network (SNN) trained using seaeee-tinnn-deppe
plasticity (STDP) and its ewward-modulated vari- ant (R-STDP) learning rules. The use of spiking neurons and local connections for
along with reinforcement learning (RI.) led us to the nomenclature our proposed architecture. Our network consists of rate-coded input layer fol- lowed by a locally connected hidden layer and a decoding output layer. A spike population-basee -based voting scheme is adopted for decoding in the output layer. We used the MNIST dataset to obtain image classification accuracy and to assess the robustness of our rewarding system to varying target responses.
INTRODUCTION
For many years, deep convolutional neural network (DCNN) has dominated the field of computer vision and object recognition Goodfellow al. (2016): LeCun al. (2015). Although novel meth- ods, such as visual transformers Carion et al. (2020) and very recent baseb models Tatsunami & Taki (202I) are threatening its reign. CNN is still the most popular architecture employed for solving visual tasks. However, CNNs lack biological plausibility. First of all, neuron activations in an artificial neural network (ANN) are static real-numbered values, that are modeled by differen- tiable, non-linear activation functions. This is in contrast to biological neurons that use discrete, and mostly sparse spike trains to transmit information between each other, and in addition to the rate of spikes (spatial encoding), they also use spike timing to encode information temporally Tavanaei et al. (2019). Therefore, a spiking neural network (SNN) is more akin to the neural networks in the brain. Spiking neural networks also require fewer labeled data and operations, which makes them compatible with energy-efficient neuromorphic hardware.
Secondly, the brain is incapable of error backpropagation, as done in traditional ANNs. One issue with error backpropagation in ANNs is the weight transport problem, i.e., the fact that weight con- nectivity in feedforward and feedback directions is symmetric Liao al. (2016); Baatunov et al. 2018 Additionally, error feedback propagation that does not affect neural activity is not compliant with the feedback mechanisms that biological neurons use for communication Lillicrap al. (2020)
Furthermore, although convolutional neural networks has shown great potential in solving any translation invariant
-nnvariionnnnvariantt task. its use of weight sharing biologically problematic. There no empirical support for explicit weight sharing in the brain Pogodin et al. (201)). However, local connections be- tween neurons is biologically plausible, since neurons in the biological visual system exploit them to have local visual receptive fields Gregor & LeCun (2010). To be compatible with this fact, we also used a locally-connected scheme without explicit weight sharing to design our network. Despite the biological nature of local connections, they mostly underperform convolution-based methods with weight sharing in the visual domain, especially on large-scale scale datasets Bartunov et al. (2018). This weaker performance may be mainly attributed to the smaller number of parameters and better generalization in CNNs. Fewer parameters in CNNs would also require less memory and computa- tional cost, and would lead faster training Poggio al. (2017). Studies are being done to bridge

---

Under review as a conference paper at ICLR 2022
the performance gap between convolutional and locally-connected networks Lillicrap et al. (2020); Bartunov et al. (2018).
Noting the above considerations, in this paper, we are proposing BioLCNet, reward-modulate locally-connected spiking neural network. Our network is trained using the unsupervised spike- timing-dependent plasticity and its semi-supervised variant goward-modulated STDP. The input im- ages are encoded proportional to the pixels intensity using Poisson rate-coding that converts intensity to average neuron firing rate in Hertz. In the output layer, there are neuronal groups for each class label, and decision making is based on aggregated number of spikes during the decision period. Our novel dynamic reward prediction error (RPE) mechanism exploits strongly supported empirical findings to improve classification performance. We test the classification capabilities of our network with different sets of hyperparameters on the MNIST dataset LeCun et al. (1999). We also conduct a classical conditioning experiment to prove the effectiveness of our decoding scheme and rewarding mechanisms.
RELATED WORK
Neuroscientists and deep learning researchers have long been searching for more biologically plau- sible deep learning approaches in terms of neuronal characteristics, learning rules, and connection types. Regarding neuronal characteristics, researchers have turned biological neuronal models and spiking neural networks. The vanishing performance gap between deep neural netwroks (DNNs)
and SNNs, and the compatibility of SNNs with neuromorphic hardware and online on-chip training
Schemmel et al.] (2010) has piqued the interest of researchers Mozafari et al. (2019). For compre-
hensive reviews on deep learning in spiking neural networks, see Tavanaei et al. (2019) Pfeiffer &
Pfeil (0018).
Spiking neurons are activated by discrete input spike trains. This differs from artificial neurons used in an ANN that have differentiable activation functions and can easily employ backpropagation and gradient-based optimization. There are works that use gradient-ban based methods with SNNs Kherad-
pisheh & Masquelier (2020); Wu et al. (2018);
[ Neftei et al. 2 Bellec al. (2020) and some of them have achieved great performances. On the other hand, many works in this area use derivations of the Hebbian learning rule where changes in connection weights depend on the activities of the pre and post-synaptic neurons Hebb] (1949). spikk-timing-deeenddeeee
plasticity (STDP) and its variants, apply asymmetric weight updates based on the temporal activities of neurons. Normal STDP re- quires an external read-out for classification Mozafari et al. (2018), and have been applied to image reconstruction and classification tasks by many researchers. Some have employed fully-connecteo architectures Beyeler et al.] (2013); Tavanaei & Maida (2015): Allred & Roy (201)). while others used convolutional layers for feature extraction Masquelier Thorpe|(2077) Panda & Roy (2016); Kheradpisheh et al. (2016) 2018). Reward-modulater STDP (R-STDP) uses reward (or punish- ment) signal to directly modulate the STDP weight change, and can be used to decode the output without external cue. Izhikevich (2007) solved the distal reward problem in reinforcement learn- ing by using a version of R-STDP with decaying eligibility traces that gives recent spiking activity more importance. Around the same time. Florian (2007) showed that R-STDP can be employed to solve a simple XOR task with both rate and temporal encoding of the output. Also, Caporale & Dan (2008) used R-STDP to generate specific spiking patterns the output of their spiking network. Historically, R-STDP was first adopted with temporal (rank-order) encoding for image classification Mozafari et al. (2018). They employed a convolutional architecture based on Masquelier & Thorpe (2007) and tmmett-ffrst-ppe decoding scheme. An extended architecture was later developed which had multiple hidden layers Mozafari al. (2019). The use of R-STDP with Poisson rate- coding has been mostly limited to fully-connected architectures for solving reinforcement learning robot navigation tasks Shim & L.i (2017); [Bing et al. (2019). To our knowledge, image recognition problems have not yet been addressed by combining R-STDP and rate-based encoding.
The most prevalent architectures used for image classification in deep learning with both DNNs and SNNs are based convolutional layers and weight sharing. However, there are arguments against the biological plausibility of these approaches Bartunov et al. (2018): Pogodin et al. (202
Locally connected (LC) networks an alternative to the convolutional ones. Illing al. (2019)
show that shallow networks with localized connectivity and receptive fields perform much better than fully-connected networks on the MNIST benchmark. However, Bartunov et al. (2018) showed
2

---

Under review as a conference paper at ICLR 2022
that the lower generalization of LC networks compared to CNNs results in their underperforming CNNs in most image classification tasks, and prevents their scalability to larger datasets such as ImageNet Deng et al. 2009). Very recently, Pogodin et al. (2021) proposed bio-inspired dynamic weight sharing and adding lateral connections to locally-connected layers to achieve the same reg- ularization goals of weight sharing and normal convolutional filters. The first work integrate locally-connece (LC) layer into an SNN [Saunders et al. (019)) used a network with no hidden layers where the rate-coded input is passed the output layer via local connections. They exploited recurrent inhibitory connections similar to the ones employed by Diehl & Cook (2015) to simu- late winner-take-all (WTA) inhibition mechanism in their output. Their learning rule is STDP, and therefore an external readout, in this case n-gram voting, is required for classification. Their network scheme was inspiring in designing our locally connected hidden layer.
recently,Pogoddi
3 THEORY
In this section, we will outline the theoretical foundations underlying our proposed method. Specif- ically, the dynamics of the spiking neuronal model, the learning rules used, and the connection type employed in our network will be described.
3.1 ADAPTIVE LIF NEURON MODEL
The famous leaky and integrate fire neuronal model is governed by the following differential equa- tion Gerstner et al. 2014).
du = (u(t) - ureat] + RR((),
(1)
dt
where u(t) denotes the neuron membrane potential and resistance, I(t) is any arbitrary input current, and Tm is the membrane time constant. Equation (I) dictates that the neuron potential exponentially decays to a constant value ürest over time. When pre-synaptic neuron fires (spikes), it generates current that reaches its post-synaptic neurons. In the simple leaky integrate and fire (LIF) model, a neuron fires when its potential surpasses a constant threshold uyhr- After firing, the neuron's 's potential resets to a constant treset and will not be affected by any input current for period of time known as the refractory period (Atref).
function of time, R is the membrane
A variant of the LIF model uses adaptive firing thresholds. In this model, thhr can change over time based on the neuron's 's rate of activity Diehl & Cook](2013) When a neuron fires, its tolerance to the input stimuli and consequently its firing threshold increases by constant amount, Io. otherwise the threshold decays exponentially with a time constant Ty to the default threshold Uthrr Equations ( to (4) explain the dynamics of the adaptive LIF model,
Usto (1) -Wehro + +9((),
where,
Todg/de /o. -g(1),
(3)
and
spike + g(t) = =g(t- - 1) + go.
(4)
3.2
STDP
Spike-tining-dependarr plasticity is a type of biological Hebbian learning rule that is also aligned with human intuition ("Neurons that fire together wire together." (Lowel & Singer 1992)). The normal STDP is characterized by two asymmetric update rules. The synaptic weights are updated based on the temporal activities of pre and post-synaptic neurons. When a pre-synaptic neuron fires shortly before its post-synaptic neuron, the causal connection between the first and the second neu- ron temporal activity acknowledged, and the connection weight is increased. On the other hand, if the post-synaptic neuron fires shortly after the pre-synaptic neuron, the causality is undermined and the synaptic strength will decrease Hebb (144).. These weight updates, called long-term po- tentiation (LTP) and long-temo depression (LTD), be performed with asymmetric learning rates to adapt the learning rule to the excitatory to inhibitory neuron ratio or the connection patterns of specific neural network. A popular variant of STDP that integrates reinforcement learning into
3

---

Under review as a conference paper at ICLR 2022
the learning mechanism of spiking neural networks is reward-modulated STDP (also known as R- STDP or MSTDP Florian (0077). ). In R-STDP, global reward or punishment signal, which can be function of time, is generated as the result of the network's 's activity task performance. Using a notation similar to Florian (2007), to mathematically formulate both STDP and R-STDP, we can define the spike train of pre-synaptic neuron the of Dirac functions the spikes of the post-synaptic neurons,
(1) Σ 8(1 i.)-
(5)
where of is the firing time of the 7th post-syanptic neuron. Now, we can define the variables sj and Pij to respectively track the influence of pre or post-synaptic spikes on weight updates. Now, the spike trace & for given spike from neuron i to can be defined as below,
Sty = P.t + ,1(),
(6)
where: (assuming the same,
dP, /dt -P / + Post pt({),
(7)
dP, - PT- mpre $ (t),
(8)
where assumed that P.j P, for all pre-synaptic connections related neuron j, and P.j P; for all post-synaptic connections related to neuron i.
The variables ти the time constants determining the time window in which spike affect the weight updates. Using larger time constants will cause spikes that are further apart to also trigger weight updates. The variables lpost and 7/ree determine the learning rate for LTP and LTD updates respectively. We denote the reward or punishment signal with r(t). The R-STDP update rules for positive and negative rewards can be written as,
dwij(t) dt where y is scaling factor. The update rule for normal STDP can also be written as,
(t)
(9)
dun,(t) (1) = (t))
(10)
dt
Based on Equation (}. we note that R-STDP updates only take effect when a non-zero modulation signal is received at time step l. However, STDP updates do not depend on the modulation signal, and are applied at every time step. In other words, STDP can be considered a special case of R- STDP where the reward function is equal in every time step. This causes STDP to respond to the most frequent patterns regardless of their desirability.
Weight-shared convolution
Local connection
Figure 1:
Visual comparison of convolutional and local connections for given filter; in convolu- tional connections, the weights are shared between all receptive fields. However, in local connec- tions, each receptive field has its own set of weights.
3.3 LOCAL CONNECTIONS
local connection in a neural network similar to a convolutional connection but with distinct filters for each receptive field. As seen in Fig. T in normal convolutional connections, there is one
4

---

Under review as a conference paper at ICLR 2022
filter for each channel that is convolved with all receptive fields it moves along the layer's input. This filter has one set of weights that are updated using the network's 's update rule. However, In local connection (LC), after taking each stride, the next receptive field. This type of connectivity between the input and the L.C layer resembles the physical structure of retinal Ganglion cells. Because there more filters in an I.C, the number of distinct synapses in local connection is greater than a convolutional connection, yet much lower than dense connection. Similar convolutional connection, assuming square filters, and equal horizontal and vertical strides, we can specify local connection by the number of channels (filters) (ch4e), the kernel size (le,, and the stride (s).
set of parameters characterize whole
filter for
ARCHITECTURE AND METHODS
BioLCNet consists of an input layer, a locally connected hidden layer, and a decoding layer. Each layer structure and its properties alongside the training and rewarding procedure will be delineated in this section. A graphical representation of our network is presented in Fig. 2 The simulation time T' is divided into three phases, adaptation period (Tadapt), decision period dec), and learning period (Tiearn). The details of each phase will be specified in the remainder of this section.
Figure 2: Graphical representation of the proposed network; locally connected filters will be applied to the rate-coded input image. Based on a winner-take-all inhibition mechanism, the most relevant features from each receptive field transmit their spikes to the decoding layer, which selects the most active neuronal group as the predicted label exploiting lateral inhibitory connections. The red lines indicate inhibitory connections.
4.1 ENCODING LAYER
The input of the network is image of dimensions (chin, hin win). For grayscale image dataset such as MNIST, chin equals to one. Each input channel is rate-coded using Poisson encoding scheme, i.e. the spiking neuron corresponding to the intensity of that pixel. By choosing the maximum firing rate fmaz. the spike trains average firing rates will be distributed in the interval [0, fmaa Hertz based on the pixel values.
each pixel has an average firing rate proportional
4.2 FEATURE EXTRACTION LAYER (LOCAL CONNECTIONS)
The encoded input at each simulation time step passes through local connections with chout distinct filters for each receptive field. Therefore, the output of this layer will have dimensions (chout> hauts Wout), where the output size depends on the size of the kernel and the stride. There are generally two approaches in the SNN literature for training feature extraction layer with rate-coded inputs using STDP to attain a rich feature representation and also prevent the weights from growing too large. One is allowing the weights to have negative values, which corresponds to having inhibitory neurons, as done the convolutional layers used by Lee et al. (0018). The other is to use a combination of recurrent inhibitory connections and adaptive thresholds done by Diehl & Cook (2015); Saunders cet al. (2018; 2019). In this work, we used the latter approach for our feature extraction LC layer. We adaptive LIF neurons and inhibitory connections between neurons that share the receptive
5

---

Under review as a conference paper at ICLR 2022
field. This is equivalent the winner-take-all inhibition mechanism which causes competition between neurons to select the most relevant features. The inhibitory connections are non non-plastic and they all have static negative weight Wink with a large absolute value.
In normal STDP, the LTP learning rate (Mpost) is usually chosen larger than the LTD rate (7pre) to suppress the random firing of neurons that triggens many LTD updates during the early stages of training. However, this may become problematic in the later stages, and the weights may grow too large. Therefore, in practice, different mechanisms, such weight clipping and normalization are used to prevent the weights running amok. In this work, we clipped the weights to stay in the range [0, 1]. We also employed the normalization technique used by Saunders al. (2019) and normalized the pre-synaptic weights of each neuron in the LC layer to have a constant mean of Crorm at the end of each time step.
4.3 DECODING LAYER AND REWARDING MECHANISMS
The final layer of our network is a fully connected layer for reward-basee decoding. The layer is divided into n. neuronal groups where n. the number of classes related the task. Consequently, the nout neurons in this layer are divided equally into neuronal groups. The predicted label for given test sample the class whose group has the most number of spikes aggregated over the deci- sion period (Tace). This decoding layer is trained using reinforcement learning and R-STDP during the learning period (Teerr)) based on the modulation signal generated by the rewarding mechanism. We designed two different rewarding mechanisms, static and dynamic reward prediction error (RPE). In the static mechanism, we use fixed reward or punishment signal for the whole learning period (Tlearn) based the prediction of the network for the jth training sample,
1: predicted label = target label otherwise
rs =
(11)
11
The second mechanism, dynamic RPE is based on the reward prediction error theory in reinforce- ment learning. According to this theory, the dopaminergic neurons in the brain release dopamine proportional to the difference between the actual reward and the expected reward (not solely based on the actual reward) Schultz et al. (1997); Sutton & Barto (2018). We formulate our dynamic RPE mechanism as below.
R R; Tr (r; EMAR)
(12)
where R₂ is the scalar R-STDP modulation signal used during the whole learning period (Tearn)) of the 1th training sample, Ti is the reward signal received based on the prediction, and EMA the exponential moving average of the modulation signals with a smoothing factor a.
4.4 TRAINING PROCEDURE
The network is trained in a layer-wise fashion. After initializing the weights uniformly between [0, 1]. we train the feature extraction LC layer in a completely unsupervised manner using STDP. Simulation time for training the feature extraction layer is Tlearn time steps. After this layer is trained, the weights are freezed, and we train the decoding FC layer in a semi-supervised manner using R-STDP and the selected rewarding mechanism. Training this layer requires all three simu- lation phases. The input image is first presented to the network for Tadopt time steps to let the LC layer neurons adapt to the input image and select its relevant features. During Taee time steps, the decoding layer accumulates the number of spikes received by each neuronal group to determine the predicted label. Afterwards, the modulation signal is generated and the decoding layer weights are updated using R-STDP for Tieern time steps.
When training the LCC layer, we observed that after a specific number of iterations (training samples), the weights of this layer converge and remain constant. Fig. [3a visualizes the filters learned after 2000 iterations for 100 filters of size 15 with a stride of applied to the input images. This fast convergence is evidence showing the strength of STDP learning. Considering these observations, and to save computation time, we limit the number of training sample the LC layer to 2000 for all of the hyperparameter configurations. Given input image (Fig. 3b), can plot the activation map of the LC layer (Fig. Bc). This map shows the post-synaptic neurons corresponding to the relevant features activate, and suppress the other neurons in accordance with the WTA inhibition mechanism.

---

Under review as a conference paper at ICLR 2022
The network is implemented using PyTorch Paszke et al. (2019), and mostly top of the BindsNet framework Hazan et al. (2018) to make our code more efficient. We reimplemented the local con- nection topology to make it compatible with multi-channe inputs and possible deep extension of our network.
53
1
125
(a)
(h)
(c)
Figure 3: Input and I.C layer visualizations. (a) LC layer learned filters; the red lines separate filters corresponding to each receptive field. (b) A sample input image. (c) The L.C layer activation map
corresponding to the sample input image shown.
Table I: BioLCNet paaaaeeepsssmseter::
best-performing value for (hyper-)paramettrs subject to grid search are in bold.
Parameter
Value
Mekro
-52 (m V)
Uyradt Ureadt
-65 (miV)
0.05 (m V
Tg
10 (ms)
Myef
(ms)
20 (ma)
(Hz)
hin, Win
22
Nout
[100, 500, 1000]
chie
[25, 50, 100. 250]
k
[11, 13, 15, 17]
[2, 3, 4]
T Tadapts T decs Tiearn
256 (m.s
(Mpre, Apost] SP
(0.0001, 0.01)
Uprres Upost) R-ST DP
(0.1, 0.1)
Trye
(statiic), 0.075, 0.125, 0.175, 0.25]
0.9
Winh
-TO0
Cnorm
0.25
EXPERIMENTS AND DISCUSSION
5.1 IMAGE CLASSIFICATION
To evaluate our network's classification performance, we trained our model on the MNIST bench- mark. Some of the hyperparameters were fixed and others were subject to grid search. The full list of hyperparameters are given in Table
Considering the hyperparameters mentioned in Table | we report in Table the classification accu- racy the whole MNIST test set (10000 samples) for four hyperparameter configurations chosen based on the highest test accuracy obtained after conducting a grid search. The number of neurons and synapses for each model are also reported in this table. The final models were all trained using 10000 training samples from the MNIST training set. Using more training samples did not improve the classification performance
be observed from Fig. 4 The mean and standard deviations
7

---

Under review as a conference paper at ICLR 2022
Table MNIST test dataset accuracies obtained by four different sets of hyper-parameters: the test accuracies are averaged over ten independent runs
Parameters [K, 8. Mrpe. Rout]
Ragnapees
Test accuracy
SVM test accuracy
[13, 3, 0.025, 100]
1700
430400
61.30 4:3.14
87.5++1.32
[15, 4, 0.175, 1000]
1884
4900000
75.00 42.68
83.31 1.74
[15, 4, 0.125, 1000]
1884
490000
76.40 #2.43
83.3
[15, 4, (static), 100]
984
130000
68.8 #2287
8.73
Table 3: MNIST test dataset accuracies obtained by different SNN approaches
Paper
Encoding
Architecture
Bio-plausibility criteria
Acc.
BioLCNet (proposed. RL)
rate-based
Locally Dennec
STDP, RL, LC
76.40
BioeC (proposed, SVM)
rate-based
Locally connected
STDP, LC
87.5
Beyeler et al. (2013)
rate-based
Dense
STDP
91.60
Diehi Cook 2015
rate-based
Dense
STDP
95.00
Tlaaaaci & Marda (2015)
rate-based
Dense
STDP
75.93
Allred & Roy (2016)
rate-based
Dense
STDP
86.59
Kheradpishch al.
rank-order
Convolutional
STDP
98.40
Saunders ct (2018)
rate-based
Convolutional
STDP
84.23
Lee et al. (2018)
rate-based
Convolutional
STDP
91.1
Mozatan (2019
rank-order
Convolutional
STDP, RL
97.2
Saunders et al. (2019)
rate-based
Locally connected
STDP, LC
95.07
reported are estimated from ten independent runs. In addition to the RL-based models, another clas- sification approach was employed. In this approach, for each training sample, we create feature vector containing the number of spikes aggregated over Tlearn time steps for every filter in the LC layer. We use these feature vectors train support vector machine (SVM) classifier. The SVM results are also obtained by training on 10000 training samples, and testing on the whole MNIST test set. The SVM test results for two different hyperparameter configurations are reported Table P and are compared to the RL-based results. The best performance of SVM and RL-based classifica- tion are 87.50, and 76.40 respectively. Table3] compares the MNIST test performance obtained by different SNN approaches along with the bio-plausibility criteria to which they adhere.
20N11 24E
Figure 4: Smoothed running accuracy over the training set for four sets of hyperparameters using R-STDP classifier
Overall, the supervised SVM has achieved better performance than the R-STDP method. Two important observations can be made from Table 2 First, the classification accuracy has positive correlation with the filter size, and the number of neurons in the decoding layer. Secondly, the dynamic RPE mechanism improved the classification performance compared to the default static rewarding mechanism. dynamic RPE plays similar role to the adaptive learning method em- ployed by Mozafari et al. (2018). yet with more biological roots and empirical support.
5.2 CLASSICAL CONDITIONING
In order to show the effectiveness of our rewarding mechanism, we perform a classical (Pavlovian) conditioning experiment. This type of conditioning pairs up neutral stimulus with automatic
8

---

Under review as a conference paper at ICLR 2022
Mma
18
3.4
WW
3.5
Telh1
300
200
300
Figure 5: Classical conditioning experiment; in this experiment, we tested the adaptability of the network to varying target responses. 'Te plot shows the rate of receiving reward and punishment averaged over 20 runs, and the decoding layer weight maps at iterations 0, 200, 300, 400, and 600. The right side of the weight maps correspond to the task target response neurons, and the left side corresponds to the task 2 target response neurons. The weights adapt to the varying target response during the experiment.
conditioned response by the agent. In this experiment, we present the network with images be- longing one class of the MNIST dataset the neutral stimuli. We used the pre-trained feature extraction layer of network with 25 filters of size 13 and stride 3, following by a decoding layer with 20 neurons for two-class prediction task. In the first half of the experiment (task 1), the target response is class 1, and the network receives a constant reward of it predicts this class regardless of the input. A punishment signal of -1 is received if the agent predicts class O. We mon- itor the rate of the reward and punishment received during the experiment. After the convergence in about 50 iterations, Fig. fig.5ssoww shows that the agent has become completely conditioned the rewarding response, After 200 iterations, we swap the rewarding and punishing classes, and continue running the network. In task 2, the network should predict the input images as class O. The RL agent (the network) adapts to the change notably fast, and completely changes its behavior after about 100 iterations. The heat maps in Fig. 5 visualize the weights of the output layer through the training.
The reward adaptability of an RL agent is critical because in many real-world problems the envi- ronment is stationarr. Integration of reward adaptation into spiking neural networks, as done in this work, can pave the path for models that simulate human behaviour with the same spike-based based computation as done in the human brain.
6 CONCLUSIONS AND FUTURE WORK
In this work, examined the capabilities of neural network with three-fold biological plausi- bility; spiking neurons, local visual receptive fields, and reward-mode modulated learning rule. The R-STDP learning rule has been only used for sequential decision making or temporal-codee tasks. As the first work to employ R-STDP in locally connected SNNs, we did not expect to achieve state-of-the-aa performance. However, we hope that using the novel dynamic RPE rewarding mech- anism alongside the emerging local connection scheme will make the future prospects of biological learning rules and architectures in solving real-word world problems, more promising.
visual
In the future, by bringing ideas such as dynamic weight sharing and lateral connections Pogodin al. (2021) to spiking neural networks, we may be able to obtain richer feature representations using locally connected SNNs. We also exploit the recent advances in SNN minibatch processing Saunders et al. (2020) and neuromorphic hardware (Schemmel et al. (2010) to extend our network with deeper architectures and solve more complex tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jason M Allred and Kaushik Roy. Unsupervised incremental stdp learning using forced firing of dormant or idle neurons. In 2016 International Joint Conference on Neural Networks (IJCNN, pp. 2492-2499. IEEE, 2016.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timo- thy Lillicrap. Assessing the scalability of biologicallymootivtee deep learning algorithms and architectures. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Cesa-Bianchi, and R. Gameett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL htwph: htt.tpps//ppocceedi.ng www.aiiinn.nnnn.neeee....... 7bc3ddcccb22ddoo 7b2 /dwaa//ddaaallla. 24dpe/////e22aaaaeeeeeeee2222220
Paperr-appp.... pddf
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neu- rons. Nature communications, 11()::1-1,, 2020.
Michael Beyeler, Nikil Dutt, and Jeffrey Krichmar. Categorization and decision-making in a neurobiologically plausible spiking network using stdp-like learning rule. Neural Networks, 48: 109-124, 2013.
Zhenshan Bing, Zhuangyi Jiang, Long Cheng, Caixia Cai, Kai Huang, and Alois Knoll. End to end learning of a multi-layered snn based on r-stdp for a target tracking snake-like robot. 2019 International Conference Robotics and Automation (ICRA), pp. 9645-9651. IEEE, 2019.
Natalia Caporale and Yang Dan. Spike timing- plasticity: a hebbian learning rule. Annu. Rev. Neurosci., 31:25-46, 2008.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255 2009. doi: 10.1 1091C//R..20000 200...
5206848.
Peter U Diehl and Matthew Cook. Unsupervised learning of digit recognition using spike-timing dependent plasticity. Frontiers in computational neuroscience, 9:99, 2015.
Räzvan V Florian. Reinforcement learning through modulation of spike-timing- synaptic plasticity. Neural computation, 19(6):168-114) 1502, 2007.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons networks and models of cognition. Cambridge University Press, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
7/www. www.deeeplearningboo..oo org
Karo Gregor and Yann LeCun. Emergence of complex-like cells a temporal product network with local receptive fields, 2010.
Hananel Hazan, Daniel J Saunders, Hassaan Khan, Devdhar Patel, Darpan T Sanghavi, Hava Siegelmann, and Robert Kozma. Bindsnet: A machine carning--rienttd spiking neural networks library in python. Frontiers in neuroinformatiic, 12:89, 2018
Donald Olding Hebb. The organisation of behaviour: a neuropsychological theory. Science Editions New York, 1949.
Bernd Illing, Wulfram Gerstner, and Johanni Brea. Biologically plausible deep learning- but how far can we go with shallow networks? Neural Networks, 118:90-101. 2019.
Eugene Izhikevich. Solving the distal reward problem through linkage of stdp and dopamine signaling. Cerebral cortex, 2450022443--455, 2007.
10

---

Under review as a conference paper at ICLR 2022
Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems, 30(06):2050027, 2020.
Saeed Reza Kheradpisheh, Mohammad Ganjtabesh, and Timothée Masquelier. Bio-inspired unsu- pervised learning of visual features leads to robust invariant object recognition. Neurocomputing, 205:382-392, 2016.
Saced Reza Kheradpisheh, Mohammad Ganjtabesh, Simon J Thorpe, and Timothée Masquelier. Stdp-based spiking deep convolutional neural networks for object recognition. Neural Networks, 99:56-67, 2018.
LeCun, Cortes, and Burges. The mnist dataset of handwritten digits (images). NYU: New York, NY. USA, 1999.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Chankyu Lee, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Deep spiking convolutional neural network trained with unsupervised wrike-timing-depente plasticity. IEEE Transactions on Cognitive and Developmental Systems, ((33):344394, 2018.
Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in backpropaga- tion? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Back- propagation and the brain. Nature Reviews Neuroscience, 21(6))335-346, 2020.
Siegrid Lowel and Wolf Singer. Selection of intrinsic horizontal connections in the visual cortex by correlated neuronal activity. Science, (5:555))2209-222 1992.
Timothée Masquelier and Simon J Thorpe. Unsupervised learning of visual features through spike timing dependent plasticity. PLoS computational biology, 3(2):e31, 2007.
Milad Mozafari, Saeed Reza Kheradpisheh, Timothée Masquelier, Abbas Nowzari Nowzari-Dalini, and Mo- hammad Ganjtabesh. First-spike- visual categorization using reward-modulated stdp. IEEE fransactions on neural networks and learning systems, (9(1)))677--619) 2018.
Milad Mozafari, Mohammad Ganjtabesh, Abbas Nowzari-Dalin Simon Thorpe, and Timothée Masquelier. Bio-inspired digit recognition using reward-modulate spike-timing plas- ticity in deep convolutional networks. Puttern recognition, 94:87-95, 2019.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrggate gradient learning in spiking neural networks: Bringing the power of gradient- baased optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-633 2019.
Priyadarshini Panda and Kaushik Roy. Unsupervised regenerative learning of hierarchical features in spiking deep networks for object recognition. In 2016 International Joint Conference on Neural Networks (IJCN),, pp- 299-306. IEEE, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 6024-8035. Curran Associates, Inc., 2019. URL http: http::::/ppppp.aeee www.ppppppp.....p.ppp........... wwtp:iipppppppppppp.pppapppppp 901 10155--ytorch--c an mperat ve sty e -er-eeiinn--eeeeeracece------ -eeperroooonnneeeeeeeeeeeeeeeeee learning iibrar.. pdf
Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: opportunities and chal-
lenges. Frontiers in neuroscience, 12:774, 2018.

---

Under review as a conference paper at ICLR 2022
Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao. Why and when can deep-but but not shallow-networks avoid the curse of dimensionality: a review. Inter- national Journal of Automation and Computing, 14(5):503-51) 519, 2017.
Roman Pogodin, Yash Mehta, Timothy P. Lillicrap, and Peter E. Latham. Towards biologically plausible convolutional networks, 2021.
Daniel Saunders, Hava T Siegelmann, Robert Kozma, et al. Stdp learning of image patches with convolutional spiking neural networks. In 2018 international joint conference on neural networks (IJCNN, pp. 1-7. IEEE, 2018.
Daniel Saunders, Devdhar Patel, Hananel Hazan, Hava T Siegelmann, and Robert Kozma. Locally connected spiking neural networks for unsupervised feature learning. Neural Networks, 119: 332-340, 2019.
Daniel Saunders, Cooper Sigrist, Kenneth Chaney, Robert Kozma, and Hava T Siegelmann. Mini- batch processing for speed-up and scalability of spiking neural network simulation. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Johannes Schemmel, Daniel Brüderle, Andreas Grübl, Matthias Hock, Karlheinz Meier, and Se- bastian Millner. A wafer-scale neuromorphic hardware system for large-scale neural modeling. In 2010 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1947-1950. IEEE, 2010.
Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 15155306):159-15555 1997.
Myung Seok Shim and Peng Li. Biologically inspired reinforcement learning for mobile robot collision avoidance. In 2017 International Joint Conference on Neural Networks (IJCN), pp- 3098-3105. IEEF, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuki Tatsunami and Masato Taki. Raftmlp: Do mip-based models dream of winning over computer vision? arXiv preprint DeXiv:2108.04388 2021.
Amirhossein Tavanaei and Anthony S Maida. A minimal spiking neural network to rapidly train and classify handwritten digits in binary and 10-digit tasks. International journal of advanced research in artificial intelligence, 4(7):1-8, 2015.
Amirhossein Tavanaci, Masoud Ghodrati, Saced Reza Kheradpisheh, Timothée Masquelier, and Authony Maida. Deep learning in spiking neural networks. Neural Networks, 111:47-63, 2019.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
12


Reviews:
[{"review_id": "qa2JJrYDc0R", "reviewer": "Reviewer_b4gF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- The paper is adequately written, and the research is motivated and\n  presented clearly.\n- To the best of my knowledge, the proposed architecture is a novel\n  combination of existing approaches, and the reported results\n  convincingly show that it achieves reasonable performance on the\n  benchmark task.\n- The problem of devising flexible architectures that can perform\n  local (Hebbian) learning is one of significant theoretical and\n  practical interest, especially with an eye to potential neuromorphic\n  applications.\n\n## Weaknesses\n- The paper seems rather incremental, in the sense that its main\n  innovations seem to be tweaks or variations of existing\n  approaches. The Introduction section ends with this sentence: *\"The\n  proposed network is the first to employ reinforcement learning with\n  Poisson rate-coded inputs for image recognition and the first\n  locally-connected SNN with a hidden layer.\"* Even if taken at face\n  value, being the first paper to \"employ reinforcement learning with\n  poisson rate-coded inputs for image recognition\" does not seem such\n  an impressive primate, if it means that there already exist papers\n  where reinforcement learning with poisson rate-coded inputs is\n  applied to a different domain, or where rate-coded inputs for image\n  recognition are used with a different learning scheme. Moreover, a\n  cursory search brought up https://doi.org/10.3389/fncom.2021.543872,\n  where reinforcement learning and Poisson rate-coded inputs are\n  applied to an MNIST classification benchmark. I haven't searched\n  more, but this should suffice to show that the claim may not even be\n  true (I'd like for the authors to comment on this). As for being the\n  first locally-connected SNN with a hidden layer (again assuming that\n  the claim is true), I'm similarly unsure how significant an\n  achievement that is, if there already exist convolutional and\n  fully-connected spiking networks with hidden layers, or\n  locally-connected networks without hidden layers.\n- Some of the arguments of the paper seem oddly fixated on the\n  importance of developing spiking networks that are biologically\n  plausible. For instance, on page 1: *\"although convolution has shown\n  great potential in solving any translation-invariant task, its use\n  of weight sharing is biologically problematic\"* . I agree that\n  biological plausibility is important if the goal of the study is to\n  learn something about biological vision, but this doesn't seem to be\n  the goal of the present paper, which is instead more aligned with\n  the interests of the computer vision field. In other words, several\n  passages in the paper present biological plausibility as something\n  desirable per se, but this is not a self-evident truth. Relatedly, the paper seems very\n  quick to dismiss or ignore alternative (and very successful) methods\n  for training SNNs, such as surrogate gradient-based ones (see for\n  instance https://doi.org/10.1109/MSP.2019.2931595 and\n  https://doi.org/10.1038/s41467-020-17236-y), because of their lack\n  of biological plausibility (see for instance on page 2: *\"the\n  majority of works in this area use derivations of the Hebbian\n  learning rule where changes in connection weights depend on the\n  activities of the pre and post-synaptic neurons\"* - is there a\n  reference for this statement?).\n\n## Minor points\n- The introduction contains a few surprising and unsubstantiated\n  statements that are likely to elicit surprise and confusion in the\n  reader. For instance, *\"in many learning problems, we do not have\n  direct access to the explicit label of the data. Consequently, we\n  may need to abandon gradient-based methods, and utilize\n  reinforcement and reward-modulated learning rules\"*: this seems\n  completely gratuitous, and it is not clear how going gradient-free\n  is a necessary implication of not having access to labels, as the\n  wealth of existing gradient-based unsupervised learning methods\n  shows.\n- When discussing biological plausibility, I do not understand the\n  insistence on the primate brain specifically. What is special about\n  the primate brain that is relevant to the arguments made in this\n  paper, and that does not apply, say, more generally to mammal or\n  vertebrate brains?\n- On page 3, please clarify in what sense *\"Spike-timing-dependent\n  plasticity is a type of biological Hebbian learning rule that is\n  also aligned with human intuition\"*. Is this supposed to imply that\n  other types of biological hebbian learning rules are not \"aligned\n  with human intuition\"? What does \"aligned with human intuition\"\n  mean?\n\n## Typos/clarity\n- It would help clarity if the final paragraph of the discussion\n  started with something like \"In the future...\", to signpost that\n  further developments are now being discussed.\n- Throughout the whole paper, please use the appropriate latex\n  citation commands - I believe that in most cases you will want to\n  use \\citep instead of \\cite or \\citet. In the current state of the\n  paper, many sentences are made needlessly hard to parse because the\n  name of the authors of some reference is just inserted into the\n  sentence without any parenthesis.\n- page 1: \"although convolution has shown\" → \"although convolutional\n  networks..\"\n- page 2: \"online on-cheap\" training → \"on-chip\"\n- page 2: \"this differs with artificial neurons\" → \"differs from\"\n- page 3: \"adapt the learning rule with the excitatory to\n  inhibitory...\" → \"adapt the learning rule to\"\n- page 4, above eq 5: I believe \"post-synaptic\" neuron should be\n  \"pre-synaptic\".\n- page 4, below eq 5: \"pre-syanptic\"\n- page 4, below eq 10: \"can be considered a special CASE of R-STDP\"\n\n", "summary_of_the_paper": "This paper presents a spiking neural network (SNN) architecture for\nimage classification, designed to be more biologically plausible than\ncomparable existing architectures. In particular, it eschews\nconvolutional connectivity in favour of local connectivity, and uses\nSTDP (vanilla and reward-modulated) instead of gradient-based\nlearning. The paper describes in detail the proposed architecture\n(providing introductory material on topics such as spiking artificial\nneurons and STDP), and then compares its performance on an MNIST\nbenchmark to other SNN architectures. Finally, the paper presents the\nperformance of the architecture on a conditioning task, which is used\nto highlight the capacity of the network to track a dynamically\nchanging reward.\n", "summary_of_the_review": "This paper presents a novel combination of existing approaches. It is\nin my opinion incremental in nature, but reasonably consistent in\nachieving its goals. In its current state I can't recommend it for\nacceptance, but I may revise my score if the confusing biological\nplausibility angle is softened and if better connections are made with\nrelated work, especially gradient based methods which at the moment\nseem unjustly penalized.\n\n\n---\nUpdate after discussion: I am updating my recommendation to \"marginally above the acceptance threshold\" to reflect the effort made by the authors to engage at least with some of my concerns. See my message downthread for more details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iBhvOdLkInX", "reviewer": "Reviewer_Urvj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\nThe manuscript proposed a training scheme mixed between STDP and R-STDP and proposed reward functions for training the 3-layer SNN with R-STDP. I think it is unique. However, unique does not mean good or better. \n\n**Weaknesses:**\n1. The authors proposed a dedicated 3-layer SNN network for image classification. The validation is only limited to MNIST but with worse performance than other wildly used SNN approaches. From my point of view, it is too limited and it is not convincing enough in the effectiveness of the proposed method.\n2. The authors claimed, “The proposed network is the first to employ RL … for image recognition”. Even it is true, why the community should care? From my point of view, the motivation for utilizing RL for classification is not very clear, especially since the performance is worse than other commonly used SNNs. \n3. It is not clear to me why the authors mixed STDP and R-STDP for training the network? I think it would make sense to show the performance with pure STDP and pure R-STDP training. \n4. The last sentence of the first paragraph of the introduction, “Spiking neural networks also require fewer labeled data….”, does not make sense to me. \n     * Why SNNs require fewer labeled data?\n     * Fewer labeled data and operations can also be compatible with regular GPU, right?\n5. I appreciate the authors tried to bring the bio-plausible concepts to the SNN domain. However, not all bio-plausible concepts revealed by biology/neuroscience research can be directly leveraged and meaningful for the SNN research as SNN neuron models themselves are just bio-inspired and they are super simple compared to the neurons in the nervous system. I do not think combining a few bio-plausible components together without clear insights could be a meaningful research project.  \n", "summary_of_the_paper": "The authors proposed an SNN model, dubbed BioLCNet, for image classification. The proposed network consists of three layers: 1) input layer, accepting spike trains as input;  2) locally connected (LC) hidden layer; 3) decoding fully connected (FC) output layer. The LC layer is for feature extraction and is trained first using STDP in an unsupervised manner. The decoding output FC layer is trained second with R-STDP in a semi-supervised manner. The proposed SNN is validated with the MNIST dataset to show its effectiveness. ", "summary_of_the_review": "The main concern on my end is the proposed approach lacks clear motivation why the developed SNN network and corresponding training approach are meaningful. As an SNN researcher, I did not really see the significant value. The validation is only with MNIST, and I think it is too limited. \n\nBut, my rating is not firmed at the moment. Based on the authors' feedback, if I realize I indeed miss something important, I am flexible to change my recommendation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N5qHlNdlir0", "reviewer": "Reviewer_BWum", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Major comments \n\n**Strengths.** The paper aims at combining multiple biologically plausible features of deep networks: spikes, local connections and RL-based error signals. This is a reasonable direction, and it’s been tried by multiple papers as shown in Tab. 3. \n\n**Weaknesses.**\n\nHowever, I struggle to see the novelty in the author’s approach: spikes and local connections alone have been tried many times (Tab.3 and also [1]). Training the output layer (rather than the whole network) with an RL-based rule is somewhat new, but I find this approach unreasonable for the following reasons:\n1. The last layer is usually trained with SGD + cross-entropy to assess the quality of representations built by previous layers. So the performance of R-STDP in any case would be limited by the representations it gets from earlier layers, which are arguably more important for training networks. (This paper tries to do that too with SVM, however.) \n2. There’s no reason for this approach to scale beyond MNIST, as the hardest part of training is done by a simple STDP rule. Maybe some layer-wise R-STDP can be a valid approach (akin to [2]), or a backprop-like RL error [3].\n\nAs a side point, I couldn’t run the code in colab (with PyTorch 1.8 and Bindsnet installed). Running `image_classification_experiment.py` gives `record() got an unexpected keyword argument 'n_labels'`. Disabling recording makes it go away, but then there’s a shape mismatch. And if you make the running time 256 instead of 256*3 to fix it, the accuracy doesn't improve at all.\n \nThe main result of the paper -- MNIST accuracy (Tab. 3) -- is very weak. It’s pretty straightforward to achieve 95%+ test accuracy with spiking networks, local connections and unsupervised pre-training (using SGD for predictions) [1] (Tab. 2 there). Therefore, even ignoring the potential weakness of the R-STDP in the final layer and concentrating on the STDP + SVM result (87.5%), it is clear that the network does not learn useful representations. There are multiple potential reasons for that:\n1. The STDP in the first layer is at fault, which would be a bit surprising given the clarity of filters in Fig.3A. As a sanity check, you can train an SVM on the hidden layer without any pre-training, and see if it improves the results. \n2. The decoding scheme is ill-fitted for SVM. I’d suggest using SGD with cross-entropy like in [1] and probably many papers in Tab. 3 of your paper. If you see a large improvement, then R-STDP needs some rethinking to properly make use of the pre-trained layer.\n3. Local connections make it harder. Some works in Tab. 2 of [1] successfully use LC layers, however. I would test performance with the same architecture, but using convolutions. Another thing I noticed is really large filters -- 15x15 filters for a 28x28 image are not too far from a fully connected layer.   \n4. When the winner-take-all in the LC layer makes a mistake by activating the wrong “digit” (and the filter weights do look like digits in Fig.3A), the readout layer can’t fix it. \n\nFinding the root of poor performance would improve the paper, but the overall approach (hidden layer STDP + last layer R-STDP) is still unlikely to scale to harder problems and deeper networks.\n\n**Recommendation**\n\nDue to limited novelty and unsatisfying results, I would recommend rejecting the paper. \n\n### Minor comments\n\n> It is noteworthy that in many learning problems, we do not have direct access to the explicit label of the data. Consequently, we may need to abandon gradient-based methods, and utilize reinforcement and reward-modulated learning rules\n\nGradient-based doesn’t mean it uses labels. See VAEs, self-supervised learning, etc. that all use backprop. \n\n> The proposed network is … the first locally-connected SNN with a hidden layer\n\nThat’s not true. See [1] and references therein.\n\nVarious problems:\n1. Eq. 3 has extra underscores. It has to be dg/dt.\n\n1. P_ij^+- in Eqs. 7-8 only need one index, j for Eq. 7 and i for Eq. 8.\n\n1. Eq. 12 is confusing. Where does the reward come from at each trial? Is one of the r_i taken from Eq. 11?\n\n1. Explaining the network model in Sec. 4.2 with equations would greatly improve clarity.\n\n[1] https://www.sciencedirect.com/science/article/pii/S0893608019301741\n\n[2] https://www.frontiersin.org/articles/10.3389/fnins.2018.00608/full\n\n[3] https://proceedings.neurips.cc/paper/2020/hash/1abb1e1ea5f481b589da52303b091cbb-Abstract.html\n", "summary_of_the_paper": "The paper discusses learning in biological networks, and proposes a model that combines spiking networks, local connections (convolutions without weight sharing) and reward-modulated spike-timing dependent plasticity. All three components are chosen to be more realistic compared to artificial neural networks. The model is tested on MNIST and on a classical conditioning task. ", "summary_of_the_review": "The paper presents a somewhat novel combination of biologically plausible features of deep nets. However, the overall approach is too simple to scale to hard tasks (STDP in hiddden layers + R-STDP in the last layer), and it performs poorly even on MNIST. The paper should be rejected. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a locally connected spiking neural network model trained to do classification of MNIST using spike-timing-dependent plasticity (STDP) and reward-modulated STDP. The authors show that this model can learn to classify MNIST images (though not at a very high accuracy) and that it can engage in classical conditioning. The reviews were initially all in the reject range. The common theme in the reviews was concerns about the weak and limited nature of the results. After a good amount of author response and reviewer replies to the authors, one reviewer increased their score to a borderline accept, but the other reviewers did not change their scores, producing scores of 3,3, and 6. Given these scores, and the reviewers' remaining concerns, a reject decision was reached.
Decision: Reject