Heterogeneous Neuronal and Synaptic Dynamics for Spike-Efficient Unsupervised Learning: Theory and Design Principles
Authors: Biswadeep Chakraborty
Saibal Mukhopadhyay
Venue: ICLR 2023 poster (2023)
Keywords: theory
spiking neural network
LIF
STDP
heterogeneity
memory capacity
spike efficiency
bayesian optimization

Abstract: This paper shows that the heterogeneity in neuronal and synaptic dynamics reduces the spiking activity of a Recurrent Spiking Neural Network (RSNN) while improving prediction performance, enabling spike-efficient (unsupervised) learning.
We analytically show that the diversity in neurons' integration/relaxation dynamics improves an RSNN's ability to learn more distinct input patterns (higher memory capacity), leading to improved classification and prediction performance. We further prove that heterogeneous Spike-Timing-Dependent-Plasticity (STDP) dynamics of synapses reduce spiking activity but preserve memory capacity. The analytical results motivate Heterogeneous RSNN design using Bayesian optimization to determine heterogeneity in neurons and synapses to improve $\mathcal{E}$, defined as the ratio of spiking activity and memory capacity. The empirical results on time series classification and prediction tasks show that optimized HRSNN increases performance and reduces spiking activity compared to a homogeneous RSNN.

Published as a conference paper at ICLR 2023
HETEROGENEOUS NEURONAL AND SYNAPTIC DYNAM- ICS FOR
UNSUPERVISED LEARNING: THEORY AND DESIGN PRINCIPLES
Biswadeep Chakraborty & Saibal Mukhopadhyay Department of Electrical and Computer Engineering
Georgia Institute of Technology (biswadeep, smukhopahyay6 ) @egatchh.ccc... edu
ABSTRACT
This paper shows that the heterogeneity in neuronal and synaptic dynamics re- duces the spiking activity of a Recurrent Spiking Neural Network (RSNN) while improving prediction performance, enabling spike-efficient (unsupervised) learn- ing. We analytically show that the diversity in neurons' integration/relaxation dynamics improves an RSNN's ability to learn more distinct input patterns (higher memory capacity), leading to improved classification and prediction performance. We further prove that heterogeneous Spike-Timing Seeeiin--Deeeeeettt Dependeet--lastiictyy dynamics of synapses reduce spiking activity but preserve memory capacity. The analytical results motivate Heterogeneous RSNN design using Bayesian optimiza- tion to determine heterogeneity in neurons and synapses to improve E, defined as the ratio of spiking activity and memory capacity. The empirical results on time series classification and prediction tasks show that optimized HRSNN increases performance and reduces spiking activity compared homogeneous RSNN.
Plasticity (STDP)
INTRODUCTION
Spiking neural networks (SNNs) (Ponulak & Kasinski 2011 use unsupervised bio-inspired neu- rons and synaptic connections, trainable with either biological learning rules such as spike-timing dependent plasticity (STDP) (Gerstner & Kistler 2002) or supervised statistical learning algorithms such as surrogate gradient (Nefici et 20199) Empirical results on standard SNNs also show good
performance for various tasks, including spatiotemporal data classification, (Lee et al.) 2017 Khoei
et al. 2020), sequence-to-sequene
mapping (Zhang & Li, 2020), object detection (Chakraborty ct al. 2021] Kim et al. 2020), and universal function approximation (Gelenbe et al. 1999) lannella
Back 2001). An important motivation for the application of SNN in machine learning (ML) the
sparsity in the firing (activation) of the neurons, which reduces energy dissipation during inference
(Wu et al. 2019). Many prior works have empirically shown that SNN has lower firing activity than artificial neural networks and can improve energy efficiency (Kim et al.) 2022] [Sriniasann & Roy 2019). However, there are very few analytical studies on how reduce the spiking activity of an SNN while maintaining its learning performance. Understanding and optimizing the relations between spiking activity and performance will be key to designing energy-efficient SNNs for complex ML tasks.
In this paper, we derive analytical results and present design principles from optimizing the spiking activity of a recurrent SNN (RSNN) while maintaining prediction performance. Most SNN research in ML. considers simplified network model with a homogeneous population of neurons and synapses (homogeneous RSNN (MRSNN)) where all neurons have uniform integration/relaxation dynamics, and all synapses use the same long-term potentiation (LTP) and long-term depression (LTD) dynamics in STDP learning rules. On the contrary, neurobiological studies have shown that a brain has wide variety of neurons and synapses with varying firing and plasticity dynamics, respectively (Destexhe) & Marder 2004; Gouwens et al., 2019 Hansel et al. 1995] Prescott et al. 2008). We show that optimizing neuronal and synaptic heterogeneity will be key to simultaneously reducing spiking activity while improving performance.

---

Published as a conference paper at ICLR 2023
Table 1: Notations Table
Notation
Mearing
Notation
Meaning
Notation
Meuning
Notation
Meuning
Heterogeneous
HRSNN
Recurrent Spiking
S.(1)
Spike train from neuron
Spike indicator
Fumation
x(()
States RSNN
Neural Network
MRSNN
Recurient Spiking
Homogenenus
Synaptic Weight Update
9/b
Threshold voltage
Covariance Matrix of
r(f)
Neural Network
Time Constant
Membrane
Synaptic Time
Coestants
Memory Capacity
Model Input
Mernhrane
Poteritial
Scaling Functions
Model Ourput
with delay
E
Spike Efficiency
Vesst
Resting potential
Decay Factor
Constani
R in RSNN
Recurrett Layor
Sn. Sar
HRSNN (R), MRSNN (M)
Total Spike Count
Avg. Spike Count
Symupec weight
Heterogencity
Spake Ireqeecc
SR. SM
per neuron
W3+
neuron
Neuronal
4g. Ort
HRSNN (R).
HRSNN (R), MRSNN (M)
to neuron
Parameters
MRSNN (M)
We define the spike efficiency & of an RSNN the ratio of its memory capacity C and average spiking activity S. Given fixed number of neurons and synapses, higher C. implies network can learn more patterns and hence, perform better in classification or prediction tasks (Aceituno et al. 2020 Goldmann et al., 2020); lower spiking rate implies that network is less active, and hence, will consume less energy while making inferences (Sorbaro et al. 2020 Rathi et al., 2021). We analytically show that a Heterogeneous Recurrent SNN (HRSNN) model leads to more spike-efficient learning architecture by reducing spiking activity while improving C (i.e., performance) of the learning models. In particular, make the following contributions to the theoretical understanding of an HRSNN.
We prove that for a finite number of neurons, models with heterogeneity among the neuronal dynamics has higher memory capacity C.
We prove that heterogeneity in the synaptic dynamics reduces the spiking activity of neurons while maintaining C. Hence, a model with heterogeneous synaptic dynamics has lesser firing rate than model with homogeneous synaptic dynamics.
We connect the preceding results to prove that simultancously using heterogeneity in neurons and synapses, as in an HRSNN, improves the spike efficiency of a network.
We empirically characterize HRSNN considering the tasks of (a) classifying time series ( (Spoken Heidelberg Digits (SHD)) and (b) predicting the evolution of a dynamical system (a modified chaotic Lorenz system). The theoretical results are used to develop an HRSNN architecture where modified Bayesian Optimization (BO) is used to determine the optimal distribution of neuron and synaptic parameters maximize ε. HRSNN exhibits better performance (higher classification accuracy and lower NRMSE loss) with lesser average spike count Š than MRSNN.
Related Works Inspired by the biological observations, recent empirical studies showed potential for improving SNN performance with heterogencous neuron dynamics PaeessseeeeeeNeeee Nieves et al. 1 Chakraborty & Mukhopadhyay 2023). However, there is a lack of theoretical understanding of why heterogeneity improves SNN performance, which is critical for optimizing SNNs for complex tasks. She et al. (022) have analytically studied the universal sequence approximation capabilities of a feed- forward network of neurons with varying dynamics. However, they did not consider heterogeneity in plasticity dynamics, and the results are applicable only for a feed-forward SNN and do not extend to recurrent SNNs (RSNN). The recurrence is not only a fundamental component of biological brain (Soures & Kudithipudi 2019). but as machine learning (ML) model, RSNN also shows good performance in modeling spatiotemporal and nonlinear dynamics (Pyle& & Rosenbaum 2017| Gilra & Gerstner 2017). Hence, it is critical to understand whether heterogeneity can improve learning in an RSNN To the best of our knowledge, this is the first work that analytically studies the impact of heterogeneity in synaptic and neuronal dynamics in RSNN This work shows that only using neuronal heterogeneity improves performance and does not impact spiking activity. The number of spikes required for the computation increases exponentially with the number of neurons. Therefore, simultancously analyzing and optimizing neuronal and synaptic heterogeneity, as demonstrated in this work, is critical to design an energy-efficient recurrent SNN.
2

---

Published as a conference paper at ICLR 2023
2 PRELIMINARIES AND DEFINITIONS
We now define the key terms used in the paper. Table 1 summarizes the key notations used this paper. Figure I shows the general structure of the HRSNN model with heterogeneity in both the LIF neurons and the STDP dynamics. the rest of the paper: Firstly, the heterogeneous network hyperparameters are estimated before the training and inference. The hyperparameters are frozen after estimation and do not change during the model evaluation. Secondly, this paper introduces neuronal and synaptic dynamics heterogeneity by using a distribution of specific parameters. However, other parameters can also be chosen, which might lead to more interesting/better performance or characteristics. We assume mean-field model where the synaptic weights converge for the analytical proofs. In addition. it must be noted that LIF neurons have been shown to demonstrate different states [Brunel (2000). Hence, for the analytical study of the network, we use the mean-field theory to analyze the collective behavior of dynamical system comprising many interacting particles.
is to be noted here that there are a few assumptions we use for
Heterogeneous LIF Neurons We use the Leaky Integrate and Fire (LIF) neuron model all our simulations. In this model, the membrane potential of the i-th neuron (t) varies over time as:
LIF Neurons
STDP Dynamics
Homogeneous
Ti (t:(t) d Urest) T:())
-v=()))
(1)
where Tm the membrane time constant, Urest is the resting potential and I; is the input cur- rent. When the membrane potential reaches the threshold value um a spike is emitted, vy re-
Heterogeneous
M1M
sets the reset potential U, and then enters refractory period where the neuron cannot spike. Spikes emitted by the j th neuron at finite set of times {t,} can be formalized spike train
S.(() = Σ 8 (1-4)).
Figure 1: Concept of HRSNN with variable Neu- ronal and Synaptic Dynamics
Let the recurrent layer of an RSNN be R. We incorporate heterogeneity in the LIF neurons by using different membranc time constants Tm. and threshold voltages V for each LIF neuron in R. This gives distribution of time constants and threshold voltages of the LIF neurons in R.
Heterogeneous STDP:
The STDP rule for updating a synaptic weight (Aw) is defined by Pool & Mato 2011
Aw(At) { A, (,(w) l if
At 0 Ad - A. (w) if At <
s.l, (w) 17- (Wmax w) A (w) 17- (w Wmin)
(2)
where At= t fee is the time difference between the post-synaptic spike and the pre-synaptic one, with synaptic time-constam T,. In heterogeneous STDP, we use an ensemble of values from a distribution for t, and the scaling functions 1/+-
Heterogeneity:
We define heterogeneity measure of the variability of the hyperparameters in RSNN that gives rise to an ensemble of neuronal dynamics.
Entropy is used to measure population diversity. Assuming that the random variable for the hyperpa- rameters follows a multivariate Gaussian Distribution (X ~ (M, E)), then the differential entropy
of I on the multivariate Gaussian distribution, is H(x) In 2m) In |=|++ n 5 Now, if we take
any density function g(x) that satisfies [ ((x)x,,yddx = Eng and p = NO E), then H(1) H(p). (Proof in Suppl. Sec. A) The Gaussian distribution maximizes the entropy for given covariance. Hence, the log-determinant of the covariance matrix bounds entropy. Thus, for the rest of the paper, use the determinant of the covariance matrix measure the heterogeneity of the network.
3

---

Published as a conference paper at ICLR 2023
Memory Capacity: Given input signal x(t), the memory capacity C of trained RSNN model is defined as a measure for the ability of the model to store and recall previous inputs fed into the
network (Jagger 2001 Jaeger et al. (200]).
In this paper, we use C as measure of the performance of the model, which is based on the network's ability retrieve past information (for various delays) from the reservoir using the linear combinations of reservoir unit activations observed at the output. Intuitively, HRSNN can be interpreted as a set of coupled filters that extract features from the input signal. The final readout selects the right combination of those features for classification or prediction. First, the r-delay C measures the performance of the RC for the task of reconstructing the delayed version of model input (t) at delay
(i.e., sx( - (T) ) and is defined as the squared correlation coefficient between the desired output ( and the observed network output y/- (r((), given as:
7-time-step delayed input signal, x(((-)))) -))
THRNK Σ C(T)= Timax y=1
THng Cov² (x(t-T),
T), Vr Vr (+)))
C= lim
lim Timax
T N,
Var(x(t)) Var (y, (t))
where Cov(-) and Var(.) denote the covariance function and variance function, respectively. The y- (t) is the model output in this reconstruction task. measures the ability of RSNN to reconstruct precisely the past information of the model input. Thus, increasing C indicates the network is capable of learning a greater number of past input patterns, which in turn, helps in increasing the performance of the model. For the simulations, we use Tmax 100.
(E) of a trained RSNN model is defined as the ratio of the memory capacity to the average total spike count per neuron S.
Spike-Efficieney
Given an input signal x(t), the
E is analytical measure used to compare how C and hence the model's performance is improved with per unit spike activity in the model. Ideally, we want to design a system with high C using fewer spikes. Hence we define & as the ratio of the memory capacity using NR neurons C(NR) to the average number of spike activations per neuron (5) and is given as:
C(NR) M S. S S
f==[",(()) s;(()dtt ss ppost J (btdt
NR.
where Npnsa is the number of postsynaptic neurons, &, the inter-spike interval spike frequency for neuron i, and 7 is the total time. It is to be noted here that the total spike count S is obtained by counting the total number of spikes in all the neurons in the recurrent layer until the emission of the first spike at the readout layer.
3 HETEROGENEOUS RSNN: ANALYTICAL RESULTS
We present three main analytical findings. Firstly, neuronal dynamic heterogeneity increases memory capacity by capturing more principal components from the input space, leading better performance and improved C. Secondly, STDP dynamic heterogeneity decreases spike activation without affecting C, providing better orthogonalization among the recurrent network states and a more efficient repre- sentation of the input space, lowering higher-order correlation in spike trains. This makes the model more spike-efficient since the higher-order correlation progressively decreases the information avail- able through neural population (Montani et al. 2009 Abbott & Dayan 1999). Finally, incorporating heterogeneity in both neuron and STDP dynamics boosts the to spike activity ratio, i.c., E, which enhances performance while reducing spike counts.
The performance of an RSNN depends on its ability to retain the memory of previous inputs. To quantify the relationship between the recurrent layer dynamics and C, we note that extracting information from the recurrent layer is made using a combination of the neuronal states. Hence, more linearly independent neurons would offer more variable states and, thus, more extended memory.
Memory Capacity:
Lemma 3.1.1:
The state of the neuron can be written follows:
re(t)
NR Ng X (c,)) "h) (vw), x(( - k). where Vn. vnl V are, respectively, the left and right K=On=1
ΣΣ
eigenvectors W, w the input weights, and X X belongs the diagonal matrix containing
4

---

Published as a conference paper at ICLR 2023
the eigenvalues of W; a, = [a,, 01.1....] represents the coefficients that the previous inputs X; = (tt have on rt()).
Short Proof: (See Suppl. Sec. full proof) As discussed by Accituno et al. 2020), the state of the neuron can be represented as r(t) = Wr(t - 1) + w x(t), where w are the imput weights. We can simplify this using the coefficients of the previous inputs and plug this term into the covariance between two neurons. Hence, writing the input coefficients a as a function of the eigenvalues of W,
NR Σ 0=m
r(t) Σ W w x((k)) Σ (VA*V-1) w ((-kk) = r:(1)
Σ
wh)) (vn); xt(
Theorem 1:
If the memory capacity of the HRSNN and MRSNN networks are denoted by Ch and Cad respectively, then, Cи 2 CMs, where the heterogeneity in the neuronal parameters H varies inversely to the correlation among the neuronal states measured as Σxa Eme, Cov² (1) I' (t)) which in turn varies inversely with
(See Suppl. Sec. Bfor full proof) Aceituno et al. (2020) showed that the C increases when the variance along the projections of the input into the recurrent layer are uniformly distributed. We show that this can be achieved efficiently by using heterogeneity in the LIF dynamics. More formally, let us express the projection in terms of the state space of the recurrent layer. We show that
Intuitive Proof:
INa X (E)
the raw variance in the neuronal states I can be written as J
? where An (E) is
(E A, (E)
the nth eigenvalue of Σ. We further show that with higher H. the magnitude of the eigenvalues of W decreases and hence leads to a higher J. Now, we project the inputs into orthogonal directions of the
network state space and model the system as r(t) Σ at (t) where the vectors a, e R.V
7-1
correspond the linearly extractable effect of 2x(t onto r(1) and (1) is the nonlinear contribution of all the imputs onto the state of r(t). First, we show that C increases when the variance along the projections of the input into the recurrent layer is more uniform. Intuitively, the variances at directions a, must fit into the variances of the state space, and since the projections are orthogonal, the variances must be along orthogonal directions. Hence, we show that increasing the correlation among the neuronal states increases the variance of the eigenvalues, which would decrease our No.
memory bound C*. We show that heterogeneity is inversely proportional to Σ Cov (In(t), (1))
n=1
We see that increasing the correlations between neuronal states decreases the heterogeneity of the eigenvalues, which reduces C. We show that the variance in the neuronal states is bounded by the determinant of the covariance between the states; hence, covariance increases when the neurons become correlated. As H increases, neuronal correlation decreases. Aceituno et al. (2020) proved that the neuronal state correlation is inversely related to C. Hence, for HRSNN, with
CH C..
Spiking Efficiency We analytically prove that the average firing rate of HRSNN is lesser than the average firing rate of the MRSNN model by considering a subnetwork of the HRSNN network and modeling the pre-and post-synaptic spike trains using a nonlinear interactive Hawkes process with inhibition, as discussed by Duval et al. (222). The details of the model are discussed in Suppl. See
B
Lemma 3.2.1: If the neuronal firing rate of the HRSNN network with only heterogeneity LTP/LTD dynamics of STDP is represented as O R. and that of MRSNN represented as & M, then the HRSNN model promotes sparsity in the neural firing which can be represented as R < M.
Short Proof: (See Suppl. Sec. Boor full proof) We show that the average firing rate of the model with heterogeneous STDP (LTPALTDD) dynamics (averaged over the population of neurons) is lesser than the corresponding average neuronal activation rate for model with homogeneous STDP dynamics. We prove this by taking a sub-network of the HRSNN model. Now, we model the input spike trains of the pre-synaptic neurons using multivariate interactive, nonlinear Hawkes process with multiplicative inhibition. Let us consider a population of neurons of size N that is divided into population A (excitatory) and population B (inhibitory). We use particular instance of the model given in terms of family of counting processes (Z,,..,, Z,NA (population A) and (2,Na=1 ..... Z,²) (population
5

---

Published as a conference paper at ICLR 2023
B with coupled conditional stochastic intensities given respectively by A and A follows:
1 AAN
A
J jeA
Σ
h1(t-uddZ)
BB-A
Σ
J
ha(t-u)dZ?
N
N
jeB
B.N. 1
Σ jeB
L.
+ AA
- N
So hall ha(l-w)dZ? w)d JEA
Σ
(5)
N
where A, B are the populations of the excitatory and inhibitory neurons, respectively, X is the intensity of neuron i, Φ positive function denoting the firing rate, and (t=; is the synaptic kernel associated with the synapse between neurons and Hence, we show that the heterogeneous STDP dynamics increase the synaptic noise due to the heavy tail behavior of the system. This increased synaptic noise leads to a reduction in the number of spikes of the post-synaptic neuron. Intuitively, heterogeneous STDP leads to non uniforrfor scaling of correlated spike-traing leading to de- correlation. Hence, we can say that heterogeneous STDP models have learned a better-orthogonallzee subspace representation, leading better encoding of the input space with fewer spikes.
Theorem For a given number of neurons NR, the spike efficiency of the model & C(NNa) for HRSNN (Ex) is greater than MRSNN (Eas) i.e., &n EM
Short Proof: "See|Supl.. Sec. forr full proof) First, using Lemma 3.2.1, we show that the number of spikes decreases when we use heterogeneity in the LTP/LTD Dynamics. Hence, we compare the efficiencies of HRSNN with that of MRSNN as follows:
ENT (a(-))))))))) Var(a)Tr(())
ʃ fras
1 D
En CR(NR) SSM Es (Nr
(6)
NN Cov (ar")) (ar(a(
t CpMdh
Aye!
Since SR S SM and thso,the covariance increases when the neurons become correlated, and as neuronal correlation decreases, Hx increases (Theorem 1), we see that En/E 2 2 1 En 2 E.
Optimal Heterogeneity using Bayesian Optimization for Distributions To get optimal heterogene- ity in the neuron and STDP dynamics, we use a modified Bayesian Optimization (BO) technique. However, using BO for high-dimensional problems remains significant challenge. In our case, optimizing HRSNN model parameters for 5000 neurons requires the optimization of two parameters per neuron and four parameters per STDP synapse, where standard BO fails to converge to optimal solution. However, the parameters to be optimized are correlated and can be drawn from a probability distribution as shown by PPerez-Nivves parameter distributions instead of individual parameters for the LIF neurons and the STDP synapses, for which we modify the BO's surrogate model and acquisition function. This makes our modified BO highly scalable over all the variables (dimensions) used. The loss for the surrogate model's s update is calculated using the Wasserstein distance between the parameter distributions. We use the modified Matern function on the Wasserstein metric space as a kernel function for the BO problem. The detailed BO methods are discussed in Suppl. Sec. A BO uses a Gaussian process to model the distribution of an objective function and acquisition function to decide points to evaluate. For data points x X and the corresponding output y Y, an SNN with network structure V and neuron parameters W acts problem can be defined as: miny,w ExeX,yeY L (y, (x)) where V is the set of hyperparameters of the neurons in R and W the multi-variate distribution constituting the distributions of: (i) the membrane time constants T T of LIF neurons, (ii) the scaling function constants (A,,A.) and (iii) the decay time constants T., T_ for the STDP learning rule in SRR-
al. (202I). Thus, we design a modified BO to estimate
function (v,)(x)) that maps input data x y. The optimization
EXPERIMENTAL RESULTS
Model and Architecture We empirically verify our analytical results using HRSNN for classification and prediction tasks. Fig. [ shows the overall architecture of the prediction model. Using rate-encoding methodology. the time-series data is encoded to a series of spike trains. This high- dimensional spike train acts as the input HRSNN. The output spike trains from HRSNN act the

---

Published as a conference paper at ICLR 2023
Encoding Layer
Decoding
Reado et
Insue
InputSplle
Decaded Cutputt Signal
Fully actted Layers
Proditted
Splies
Figure 2: Block Diagram showing the methodology using HRSNN for prediction
BO optimizing
BO optimizing 5
BO optimizing E
0.8
0.8
100
1/5
0.6
100
400
1/5
0.6
400
1000
1000
2000
0.4
2000
0.4
5.0
7.5
10.0
12.5
15.0
5
10
15
Memory Capacity (C)
Memory Capacity (C)
(a)
(b)
Figure 3: Figure showing the results of ablation studies of BO for the following three cases: (a) for prediction problem - the radius indicates the normalized NRMSE loss with a smaller radius indicating lower (better) NRMSE (b) for classification problem the radius indicates the normalized accuracy with larger radius indicating higher (better) accuracy. The numbers represent the number of neurons used in each model, and the line joins the three corresponding models with the same model size. The detailed results are shown in Suppl. Sec. A
input to a decoder and readout layer that finally gives the prediction results. For the classification task, we use similar method. However, we do not use the decoding layer for the signal but directly feed the output spike signals from HRSNN into the fully connected layer. The complete details of
the models used and description of the different modules used in Fig. is discussed in Suppl. Sec. A
Datasets: Classification: We use the Spoken Heidelberg Digits (SHD) spiking dataset benchmark the HRSNN model with other standard spiking neural networks (Cramer et al. 2020).
Prediction: We multiscale Lorenz 96 system (Lorenz 1996) which is set of coupled nonlinear ODEs and an extension of Lorenz's original model for multiscale chaotic variability of weather and climate systems which we as testbed for the prediction capabilities of the HRSNN model Thornes et al. 2017). Further details on both datasets are provided in Suppl. Sec. A
Bayesian Optimization Ablation Studies: First, we perform an ablation study of BO for the following three cases: (i) Using Memory Capacity C as the objective function (ii) Using Average Spike Count Š as the objective function and (iii) Using & as the objective function. We optimize both LIF neuron parameter distribution and STDP dynamics distributions for each. We plot C, Š, the empirical spike efficiency E, and the observed RMSE of the model obtained from BO with different numbers of neurons. The results for classification and prediction problems are shown in Fig. (Ja) and (b). respectively. Ideally, we want to design networks with high C and low spike count, I.c., models in the upper right corner of the graph. The observed results show that BO using as the objective gives the best accuracy with the fewest spikes. Thus, we can say that this model has learned a oratooooothogo subspace representation, leading to better encoding of the input space with fewer spikes. Hence, for the remainder of this paper, we focus on this BO model, keeping the & as the objective function. This Bayesian Optimization process to search for the optimal hyperparameters of the model is performed before training and inference using the model and is generally equivalent to the network architecture search process used in deep learning. Once we have these optimal hyper-parameters, we freeze these hyperparameters, learn (unsupervised) the network parameters
7

---

Published as a conference paper at ICLR 2023
Memory Capacity
Avg. Spike Count
Spike Efficiency (e)
SAGE Values
0.6
SAGE Values
0.6
0.4
0.4
0.2
0.2
0.0
0.0
7+
n-
Tm
Urest
T-
T+
7+
n
Im Urest
Neuron
STDP
Parameters
(a)
(b)
Parameters
Figure 4: Bar chart showing the global importance of different heterogeneous parameters using HRSNN on the dataset. The experiments were repeated five times with different parameters from the same distribution (a) Classification (b) Prediction
(i.e., synaptic weights) of the HRSNN while using the frozen hyperparameters, and generate the final HRSNN model for inference. In other words, the hyperparameters. like the distribution of membrane time constants or the distribution of synaptic time constants for STDP, are fixed during the learning and inference. Further details of the Bayesian Optimization procedure, including the parameterized variables and the final searched distributions of the hyperparameters, are shown in SSppll. Sec. A where also discuss the convergence analysis of the three different BOs discussed above.
Heterogeneity Parameter Importance: We use SAGE (Shapley Additive Global importancE) (Covert al. 2020), game-theoretic approach understand black-box models calculate the significance of adding heterogeneity each parameter for improving C and S. SAGE summarizes the importance of each feature based on the predictive power it contributes and considers complex feature interactions using the principles of Shapley value, with higher SAGE value signifying more important feature. We tested the HRSNN model using SAGE on the Lorenz96 and the SHD datasets. The results are shown in Fig. We see that Tm has the greatest SAGE values for C, signifying that it has the greatest impact on improving C when heterogeneity is added. Conversely, we see that heterogeneous STDP parameters (viz., T., T/+) play a more critical role determining average neuronal spike activation. Hence, we confirm the notions proved in Sec. 3 that heterogeneity in neuronal dynamics improves the while heterogeneity in STDP dynamics improves the spike count. Thus, need to optimize the heterogeneity of both achieve maximum E.
We perform an ablation study to evaluate the performance of the HRSNN model and compare it to standard BP-based spiking models. We study the performances of both the SHD dataset for classification and the Lorenz system for prediction. The results are shown Table We compare the Normalized Root Mean Squared Error (NRMSE) loss (prediction), Accuracy (classification), Average Spike Count Š and the application level empirical spiking efficiency &
Results:
2]
NRMSE, S (prediction) and Accuracy
calculated as
$ (classification. We perform the experiments
using 5000 neurons in R. on both classification and prediction datasets. We see that the HRSNN model with heterogeneous LIF and heterogeneous STDP outperforms other HRSNN and MRSNN models in terms of NRMSE scores while keeping the Š much lower than HRSNN with heterogeneous LIF and homogencous STDP. From the experiments, we can conclude that the heterogeneous LIF neurons have the greatest contribution to improving the model's ' performance. In contrast, heterogeneity in STDP has the most significant impact on spike-efficient representation of the data. HRSNN with heterogeneous LIF and STDP leverages the best of both worlds by achieving the best RMSE with low spike activations, as seen from Table 2 Further detailed results on limited training data are added in Suppl. Sec. A We also compare the generalizability of the HRSNN vs. MRSNN models, where empirically show that the heterogencity in STDP dynamics helps improve the overall model's generalizability. In addition, we discuss how HRSNN reduces the effect of higher-order correlations, thereby giving rise more efficient representation of the state space.
8

---

Published as a conference paper at ICLR 2023
Table 2: Table showing the comparison of the Accuracy and NRMSE losses for the SHD Classification and Lorenz System Prediction tasks, respectively. We show the average spike rate, calculated as the ratio of the moving average of the number of spikes in a time interval T. For this experiment, we choose 7 4ms and rolling time span of 2mss, which is repeated until the first spike appears in the final layer. Following the works of Paul et al. (2022), we show that the normalized average spike rate is the total number of spikes generated by all neurons in an RSNN averaged over the time interval T. The results marked with denotes we implemented the souncsource code for the model and evaluated the given results.
SHD (Classification) Normalized Avg. Firing Rate (
Chaotic Lorenz System (Prediction) Normalized Avg. Firing Rate (3)
Method
Accuracy (A)
Efficiency C +
Efficiency
NRMSE
NRMSE
MRSNN
(Homogencous LIF.
73.58
-0.508
18.44 10
0.395
-0.768
0.787 10
Unsupervised
Homogeneous STDP)
RSNN
HRSNN
(Heterogeneous LIF Homogeneous STDP)
78.87
0.277
17.19 10-9
0.203
-0.143
1.302 10-3
HRSNN
(Homogeneous LIF
74.03
-1.292
22/17 10-3
0.372
-1.102
0.932 10-
Heterogenzous STDP)
HRSNN
(Heterogeneous LIF.
80.49
-1.154
24.35 10
0.195
-1,018
1.725 10"3
Heterogeneous STDP)
MRSNN-BP
(Homogeneous LIF,
81.42
0.554
16.9 10
0.182
0.857
1.16 10
RSNN with BP
BP)
HRSNN-
(Heterogeneous LIF, BP) Adaptive SRNN
83.54
1.292
15:42 10
0.178
1.233
1.09 10
84.46
0.831
17.21" x 10-3
0.174'
0.941
1.19* x 10-3
Yin et al. 2020
5 CONCLUSION
This paper analytically and empirically proved that heterogeneity in neuronal (LIF) and synaptic (STDP) dynamics leads to an unsupervised RSNN with more memory capacity, reduced spiking count, and hence, better spiking efficiency. We show that HRSNN can achieve similar performance as MRSNN but with sparse spiking leading to the improved energy efficiency of the network. In conclusion, this work establishes important mathematical properties of an RSNN for neuromorphic machine learning applications like time series classification and prediction. However, is interesting to note that the mathematical results from this paper also conform to the recent neurobiological research that suggests that the brain has large variability between the types of neurons and learning methods. For example, intrinsic biophysical properties of neurons, like densities and properties of ionic channels, vary significantly between neurons where the variance in synaptic learning rules invokes reliable and efficient signal processing in several animals (Marder & Taylor 2011 Douglass et al. 1993). Experiments in different brain regions and diverse neuronal types have revealed a wide range of STDP forms with varying neuronal dynamics that vary in plasticity direction, temporal dependence, and the involvement of signaling pathways (Sjostrom et al. 2008 Korte & Schmitz 2016). Thus, heterogeneity is essential in encoding and decoding stimuli biological systems. In conclusion, this work establishes connections between the mathematical properties of an RSNN for neuromorphic machine learning applications like time series classification and prediction with neurobiological observations. There are some key limitations to the analyses in this paper. First, the properties discussed are derived independently. An important extension will be to consider all factors simultaneously. Second, we assumed an idealized spiking network where the memory capacity is used to measure its performance, and the spike count measures the energy. Also, we mainly focused on the properties of RSNN trained using STDP. An interesting connection between synchronization and heterogeneous STDP remains a topic that needs to be studied further - whether we can optimally engineer the synchronization properties improve the model's 's performance. Finally, the empirical evaluations were presented for the prediction task on a single dataset. More experimental evaluations, including other tasks and datasets, will strengthen the empirical validations.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported by the Army Research Office and was accomplished under Grant Number W91 ENF--19--00447 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government.
REFERENCES
Larry F Abbott and Peter Dayan. The effect of correlated variability on the accuracy a population code. Neural computation, 11(1):91-001, 1999.
Pau Vilimelis Aceituno, Gang Yan, and Yang-Yu Yu Liu. Tailoring echo state networks for optimal learning. iscience, 23(9):001440, 2020.
Robert M Blumenthal and Ronald K Getoor. Some theorems on stable processes. Transactions of the American Mathematical Sociery, 95(2):263-27) 1960.
Martin Boerlin and Sophie Denève. Spike-based population coding and working memory. PLoS computational biology, 7(2)ee0001880, 2011.
Martin Boerlin, Christian K Machens, and Sophie Denève. Predictive coding of dynamical variables in balanced spiking networks. PLoS computational biology, 9(11)< 1003258, 2013.
Ralph Bourdoukan, David Barrett, Sophie Deneve, and Christian K Machens. Learning optimal
spike-based representations. Advances in neural information processing systems, 25, 2012.
Nicolas Brunel. Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons. Journal of computational neuroscience, 8()):183-208, 2000.
György Buzsáki and Kenji Mizuseki. The log-dynamic brain: how skewed distributions affect network operations. Nature Reviews Neuroscience, 15(4):264-278, 2014.
Biswadeep Chakraborty and Saibal Mukhopadhyay. Characterization of generalizability of spike time dependent plasticity trained spiking neural networks. arXiv preprinl 1rXiv:2105. 2021.
Biswadeep Chakraborty and Saibal Mukhopadhyay. Heterogeneous recurrent spiking neural net- work for spatio-temporal classification. Frontiers in Neuroscience, 17, 2023. ISSN 1662-453X. doi: 10.3388/fnins.22223.9999 994517.. URL sttpp::/www.i http://ww..rotttiees...ooooo
10. 3189338//f wtt.9//fnns....00 094511999999
Biswadeep Chakraborty, Xueyuan She, and Saibal Mukhopadhyay. A fully spiking hybrid neural network for energy-efficient object detection. arXiv preprint 1rXiv:2104.10719 2021.
Ashesh Chattopadhyay, Pedram Hassanzadeh, and Devika Subramanian. Data-driven predictions of a multiscale lorenz 96 chaotic system using machine- learning methods: reservoir computing, artificial neural network, and long short-term memory network. Nonlinear Processes in Geophysics, 27(3):373-38) 2020.
Julien Chevallier, María José Cáceres, Marie Doumic, and Patricia Syynaud-Bouree Microscopic approach a time elapsed neural model. Mathematical Models and Methods in Applied Sciences, 25(14):2669-21) 2719, 2015.
ES Chornoboy, LP Schramm, and AF Karr. Maximum likelihood identification of neural point process systems. Biological cybernetics, 59(4-5):265-275, 1988.
Lee Cossell, Maria Florencia lacaruso, Dylan R Muir, Rachael Houlton, Elie N Sader, Ho Ko, Sonja B Hofer, and Thomas D Mrsic-Flogel. Functional organization of excitatory synaptic strength in primary visual cortex. Nature, 518(7539):399-444 403, 2015.
Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33:17212- 17223, 2020.
10

---

Published as a conference paper at ICLR 2023
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Sophie Denève and Christian K Machens. Efficient codes and balanced networks. Nature neuro- science, 19(3):35--38) 2016.
Alain Destexhe and Eve Marder. Plasticity in single neuron and circuit computations. Nature, 431 (7010):789-795, 2004.
John K Douglass, Lon Wilkens, Eleni Pantazelou, and Frank Moss. Noise enhancement of information transfer in crayfish mechanoreceptors by stochastic resonance. Nature, 365(6444))337-3340 1993.
Céline Duval, Eric Luçon, and Christophe Pouzat. Interacting hawkes processes with multiplicative inhibition. Stochastic Processes and their Applications, 148:180-226, 2022.
Igor Farkaš and Peter Gergel'. Maximizing memory capacity of echo state networks with orthog- onalized reservoirs 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2437-2442. IEEE, 2017.
Igor Farkaš, Radomír Bosák, and Peter Gergel'. Computational analysis of memory capacity in echo state networks. Neural Networks, 83:109-120, 2016.
Jean Feydy, Thibault Séjourné, François-Xavis Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. The 22nd International Conference Artificial Intelligence and Statistics, pp. 2681 PMLR, 2019.
Peter Frazier. A tutorial bayesian optimization. arXiv preprint 1807.0288 2018.
Antonio Galves and Eva Löcherbach. Modeling networks of spiking neurons as interacting processes with memory of variable length. Journal de la Société Française de Statistique, 157(1):17-32, 2016.
Erol Gelenbe, Zhi-Hong Mao, and Yan-Da Li. Function approximation with spiked random networks. IEEE Transactions Neural Networks, 10(1):3-9, 1999.
Felipe Gerhard, Moritz Deger, and Wilson Truccolo. On the stability and dynamics of stochastic spiking neuron models: Nonlinear hawkes process and point process glms. PLoS computational biology, (3(2)=e0053300 2017.
Wulfram Gerstner and Werner M Kistler. Mathematical formulations of hebbian learning. Biological cybernetics, 4155):4044411 2002.
Aditya Gilra and Wulfram Gerstner. Predicting non-linear dynamics by stable local learning in recurrent spiking neural network. Elife, 6:e28295, 2017.
Mirko Goldmann, Felix Köster, Kathy Lüdge, and Serhiy Yanchuk. Deep time-delay reservoir computing: Dynamics and memory capacity. Chaos: An Interdisciplinary Journal of Nonlinear Science, 30(9):093124, 2020.
Nathan W Gouwens, Staci A Sorensen, Jim Berg, Changkyu Lee, Tim Jarsky, Jonathan Ting, Susan M Sunkin, David Feng, Costas Anastassiou, Eliza Barkan, et al. Classification of electrophysio- logical and morphological neuron types in the mouse visual cortex. Nature neuroscience, 22(7): 1182-1195, 2019.
David Hansel, Germán Mato, and Claude Meunier. Synchrony in excitatory neural networks. Neural computation, 7(2):307-337, 1995.
Niels Richard Hansen, Patricia Reynaud-Bon Bourre, and Vincent Rivoirard. Lasso and probabilistic inequalities for multivariate point processes. 2015.
Hatsuo Hayashi and Jun Igarashi. Ltd windows of the stdp learning rule and synaptic connections hav- ing a large transmission delay enable robust sequence learning amid background noise. Cognitive neurodynamics, 3(2):119-130. 2009.
11

---

Published as a conference paper at ICLR 2023
Nicolangelo lannella and Andrew D. Back. A spiking neural network architecture for nonlinear
function approximation. Neural Networks, 14(6):933 909,999,990 2001. ISSN 0893-6080. doi: https:
/ttooooo/||..0000000090000000000 sci
URL https:/ /www/ www.sciencedirect.t.m com/
Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks-with an erratum note, Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34):13, 2001.
Herbert Jaeger. Short term memory in echo state networks. gmd-report 152. GMD-Germer
National Research Institute for Computer Science (2002), htttp://www. faculty, jacobs-university. pdf. Citeseer, 2002.
Herbert Jaeger et al. Short term memory in echo state networks, volume 5. GMDD-Forschungszentuu Informationstechhiik Bremen, Germany, 2001.
Mina A Khoei, Amirreza Yousefzadeh, Arash Pourtaherian, Orlando Moreira, and Jonathan Tapson. Sparet: Sparse asynchronous neural network execution for energy efficient inference. In 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp- 256-260 IEEE, 2020.
Daehyun Kim, Biswadeep Chakraborty, Xueyuan She, Edward Lee, Beomseok Kang, and Saibal Mukhopadhyay. Moneta: A processing- poncessingii-mmmoore convolutional spiking neural network with online learning. Frontiers in Neuroscience, 16, 2022.
hardware platform for the hybrid
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11270-11277 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini 1412,9940,.6980, 2014.
Martin Korte and Dietmar Schmitz. Cellular and system biology of memory: timing, molecules, and beyond. Physiological reviews, 96(2):647-633,000
Lukasz Kušmierz, Shun Ogawa, and Taro Toyoizumi. Edge of chaos and avalanches in neural networks with heavy-tailed synaptic weight distribution. Physicul Review Letters, 125(2):02810) 2020.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 1rX00:1711.00165, 2017.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 1 11:309, 2017.
Eva Löcherbach. Spiking neurons: interacting hawkes processes, mean field limits and oscillations.
ESAIM: Proceedings and Surveys, 60:90-103, 2017.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2):130-14) 1963.
Edward N Lorenz. Predictability: A problem partly solved. In Proc. Seminar on predictability, volume 1, 1996.
Cheng Ly. Firing rate dynamics in recurrent spiking neural networks with intrinsic and network heterogeneity. Journal of computational neuroscience, 39(3)311-337, 2015.
Rosario N Mantegna and H Eugene Stanley. Scaling behaviour in the dynamics of an economic index. Nature, 376(6535):46-49, 1995.
Eve Marder and Adam L Taylor. Multiple models to capture the variability in biological neurons and networks. Nature neuroscience, 1332)1133-138, 2011.
12

---

Published as a conference paper at ICLR 2023
Cyrille Mascart. Efficient simulation of point processes with applications neurosciences. PhD
thesis, Université Côte d' Azur, Nice, France, 2021.
Sam McKenzie, Roman Huszár, Daniel F English, Kanghwan Kim, Fletcher Christensen, Euisik Yoon, and György Buzsáki. Preexisting hippocampal network dynamics constrain optogenetically induced place fields. Neuron, 109(6): 1040-1054, 2021.
Fernando Montani, Robin AA Ince, Riccardo Senatore, Ehsan Arabzadeh, Mathew Diamond, and Stefano Panzeri. The impact of high-order interactions on the rate of synchronous discharge and information transmission in somatosensory cortex. Philosophical Transactions of the Royal Sociery A: Mathematical,, Physical and Engineering Sciences, 367((101):3297
2009.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks. IEEE Signal Processing Magazine, 36:61-63, 2019.
Erkki Oja. Simplified neuron model principal component analyzer. Journal of mathematical biology, 15(3):267-2735 1982.
Erkki Oja. Neural networks, principal components, and subspaces. International journal of neural systems, 1(01):61-6,, 1989.
Ankita Paul, Stefan Wagner, and Anup Das. Learning in feedback-driven recurrent spiking neural networks using full-force training. In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1-10. IEEE, 2022.
Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. Nature communications, 12(1):1-9, 2021.
Balint Petro, Nikola Kasabov, and Rita M Kiss. Selection and optimization of temporal spike encoding methods for spiking neural networks. IEEE transactions on neural networks and learning systems, 31(2):358-370 2019.
Peter Pfaffelhuber, Stefan Rotter, and Jakob Stiefel. Mean-field limits for non-linear hawkes processes with excitation and inhibition. Stochastic Processes and their Applications, 153:57-78, 2022.
Filip Ponulak and Andrzej Kasinski. Introduction to spiking neural networks: Information processing, learning and applications. Acta newrobiologiae experimentalis, 71(4) 4333, 2011.
R Rossi Pool and Germán Mato. Spike-timing-deepedennt plasticity and reliability optimization: the role of neuron dynamics. Neural computation, 23(7): 2011.
Steven A Prescott, Yves De Koninck, and Terrence J Sejnowski. Biophysical basis for three distinct dynamical mechanisms of action potential initiation. PLoS computational biology, (10))e1000119 2008.
Ryan Pyle and Robert Rosenbaum. Spatiotemporal dynamics and reliable computations in recurrent spiking neural networks. Physical review letters, 1 (18(1)001100) 2017.
Nitin Rathi, Amogh Agrawal, Chankyu Lee, Adarsh Kumar Kosta, and Kaushik Roy. Exploring spike-based learning for neuromorphic computing: Prospects and perspectives. 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 902-907. IEEE, 2021.
Patricia Reynaud- -eynauu-Bouuee, Vincent Rivoirard, Franck Grammont, and Christine Tuleau-Malot Goodness-of- fit tests and nonparametric adaptive estimation for spike train analysis. The Journal of Mathematical Neuroscience, 4: 4:1-41, 2014.
Xueyuan She, Saurabh Dash, and Saibal Mukhopadhyay. Sequence approximation using feedfor- ward spiking neural network for spatiotemporal learning: Theory and optimization methods. In Proceedings of 10th International Conference on Learning Representations, ICLR (forthcoming). 2022. URL httlltttt:///ppeeeeee htt.://wwww..t.eeoooooooooaaoap.
XC
Michael F Shlesinger, BJ West, and Joseph Klafter. Lévy dynamics of enhanced diffusion: Application to turbulence. Physical Review Letters, 58(11) (3(11):110 1000, 1987.
13

---

Published as a conference paper at ICLR 2023
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise deep neural networks. arXiv preprint arXiv:1901.060S3. 2019.
Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization neural networks, Advances in Neural Information Processing Systems, 33, 2020.
P Plssper Sjostrom, Ede A Rancz, Amd Roth, and Michael Hausser. Dendritic excitability and synaptic plasticity. Physiological reviews, 88(2):769-88) 2008.
Martino Sorbaro, Qian Liu, Massimo Bortone, and Sadique Sheik. Optimizing the energy consump- tion of spiking neural networks for neuromorphic applications. Frontiers in neuroscience, 14:662,
2020.
Nicholas Soures and Dhireesha Kudithipudi. Spiking reservoir networks: Brain-inspired recurrent algorithms that use random, fixed synaptic strengths. IEEE Signal Processing Magazine, 36(6): 78-87, 2019.
Gopalakrishnan Srinivasan and Kaushik Roy. Restocnet: Residual stochastic binary convolutional spiking neural network for memory-efficient neuromorphic computing. Frontiers in neuroscience, 13:189, 2019.
Benjamin Staude, Stefan Rotter, and Sonja Grün. Cubie:
cumulant based inference of higher-order correlations massively parallel spike trains. Journal of computational neuroscience, 29(1): 327-350, 2010.
Tobias Thomes, Peter Düben, and Tim Palmer. On the use of scale-dependent precision in earth system modelling. Quarterly Journal of the Royal Meteorological Society, 143(703):897-908 2017.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019.
Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with multiple- timescale spiking recurrent neural networks. In International Conference on Neuromorphic Systems 2020, pp. 1-8, 2020.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances Neural Information Processing Systems, 13:12022-12033, 2020.
Yan Zhou, Yaochu Jin, and Jinliang Ding. Surrogate-assisted evolutionary search of spiking neural architectures in liquid state machines. Neurocomputing, 406:12-23, 2020.
14

---

Published as a conference paper at ICLR 2023
Supplementary Section
Table of Contents
Supplementary Section A
15
A.
Experimental Sctup
15
A.2
Maximum entropy distribution
17
A.3
Optimal Hyperparameter Selection using Bayesian Optimization
18
A.4
Comparing Bayesian Optimization Objective Functions
20
A.5
Comparing the Generalizability
20
A.6
Results on Limited Training Data
21
Further Evaluations
22
B
Supplementary Section B
24
B.
Approximations and assumptions
24
B.2
Mean Field Reduction Model of HRSNN
24
B.3
Analytical Results
26
4
Memory Capacity
27
B.S
Spiking Efficiency
C
Supplementary Section C
35
Higher Order Correlation
35
A SUPPLEMENTARY SECTION A
A.1 EXPERIMENTAL SETUP
A.1.I HRSNN AND MRSNN
Table 3: Table showing the description of the models described in the paper
LIF Neurons STDP parameters
HRSNN MRSNN
Heterogeneous
Heterogeneous
Homogeneous
Homogeneous
The models used in this paper were the Heterogeneous Recurrent SNN (HRSNN) and the Homoge- neous Recurrent SNN (MRSNN), Both models use STDP as the learning method. For MRSNN, we use STDP with uniform parameters for all the synapses. However, for HRSNN, we use distribution for each parameter to get a rich class diverse LTP/LTD dynamics. But, at the core, all the training done using STDP.
A.1.2 LIF NEURON NUMERICAL IMPLEMENTATION
To implement the LIF model, we discretize time into multiples of a small-time step At so that spikes can only happen at multiples At. (Cramer et al. 2020 Perez-Nieves al. 2021) Thus, we can approximately solve Eq. as
ny (1++At) n; [[+1]] 3 (v:[t) vn) + uo+ to (1-3)
St(t] s.t. 3 exp (-t//Tm) (7)
It is to be noted here that this approximation for numerically solving the LIF neurons. Hence, although we use continuous notations for the remainder the paper, to be noted that we use the discrete form discussed here for numerical solutions.
15

---

Published as a conference paper at ICLR 2023
A.1.3 SPIKE CODING
Encoding: For the RSNN process our time series, the signal must be represented spikes. We use temporal encoding technique for representing signals in this paper. The spikes are only generated whenever the signal changes in value. The implementation of the temporal encoding used in this research is based on the Step-Forwart (SF) algorithm (Petro et al., 2019). The percentage of neurons to input the spikes to (ax) is also chosen to provide good recurrent layer dynamics.
To represent the recurrent state, we use exponentially decreasing rate decoding strategy by taking the sum of all the spikes s over the last r timesteps into account follows:
Decoding:
Σ =r T=
xxx(0)
(1)
Vie
where X denotes the model representation. The parameters and y are balanced to optimize the memory size of the stored data (e.g. 7 50) ) and its containment of information, which includes adjusting the pace which the temporal data is presented and processed. The state of the recurrent layer will be only based on the output of excitatory neurons. Thus, it is crucial for the discount y not to be too small, the sliding window unusable. Recent spikes hardly affect the recurrent layer state when setting too high in combination with a large window size. This causes the decoder to react too late recent information provided by the recurrent layer and complicates the learning process of the readout layer.
possibly flattens older values in the window
making part of
A.1.4 READOUT
After the initialization of the recurrent layer, the readout is the only component of the LSM with trainable parameters. It consists The readout does not have to be any deeper, as the output of the recurrent layer is already high- dimensional representation of the processed input. x and y present the continuous signals of the time series T and model representation X.
single fully connected layer for regression classification.
x² x²(1+ k)= = yf (1) (1. (1)
The mean squared error (MSE) is used as the loss function to train the readout, and the network was trained using the stochastic optimizer Adam (Kingma & Ba 2014.
C C 1 - E 1 )
A.1.5 DATASETS
Lorenz96: Loreezz 1ur objective is more clearly demonstrated using the canonical chaotic system we will use as a test bed for the prediction capabilities of the HRSNN model. We use a multiscale Lorenz 96 system which is set of coupled nonlinear ODEs and extension of Lorenz's original model (Tornes et al. 2017), (Chattopadhyay et al., 2020).
dXx di
he 2, Y,,,
Xx-1 (X1(XK11 - ( F-
b
dY,,k cbYf=1. (YJ+2,4- Y - -Y + he Xx+ 2
he
dt
dZi,j. =edZi-1JA (Z1-1j, Zz225,)) ge d
he Yg
(8)
dt
This set of coupled nonlinear ordinary differential equations (ODEs) is three-tier extension of Lorenz's 's original model (Lorenz) 1963) and has been proposed by Thornes et al Thornes et al.) (2017) as fitting prototype for multiscale chaotic variability of the weather and climate system and useful test bed for novel methods. these equations, F = 20 is a large-scale forcing that makes the system highly chaotic, and b= = d g 10 and h = are tuned to produce appropriate spatiotemporal variability. For this paper, we focus on predicting Y axes, which have relatively moderate amplitudes compared X, Z and demonstrate high-frequency variability and intermittency, which makes the
16

---

Published as a conference paper at ICLR 2023
10
Loren796
-10
20
40
60
80
100
Time
Figure 5: Figure showing a snippet for the Y dimension of the Lorenz96 time series used for the prediction problem
prediction problem difficult. It to be noted here that the Lorenz 96 is complex, diffieult dataset for climate modeling. A snippet of the time series is shown in Fig. 5
We the Spoken Heidelberg Digits spiking dataset benchmark the HRSNN model with other standard spiking neural networks (Cramer et al. 2020). It was created based on the Heidelberg Digits (HD) audio dataset which comprises 20 classes of spoken digits from zero to nine in English and German, spoken by 12 individuals. For training and evaluation, the dataset (10420 samples) split into training set (8156 samples) and test (2264 samples). To apply our RSNNs, we converted all audio samples into 250- by-700 binary matrices. For this, all samples fit within the second window, shorter samples were padded with zeros, and longer samples were cut by removing the tail. Spikes were then binned in time bins, both of sizes 10ms and 4ms; for the RSNNs, the presence or non-presence of any spikes the time bin is noted as a single binary event.
SHD dataset:
A.1.6 HYPERPARAMETERS
The hyperparameters used in this paper are summarized in Table
Table 4: Table showing the hyperparameters used in the experiments and their values
Parameter
Value
Description
EM X
80%
Excitatory/mhiiitory ratio
2
Leak Exponent
r
50
Sliding window size
--1
0.02
Sliding Window Leak
SHD Parameters
Lorenz-63 Parameters
Parameter
Value
Description
Parameter
Value
Description
17
Time delay
28.0
p. parameter
0.2
parameter
10.0
d-parameter
0.1
parameter
3
8/3
3-parameter
10
parameter
20
[1.0,1.0.100]
Initial Condition
Time delta between
To
1.2
Initial Condition
0.03
two discrete timesteps
Time delta between
1.0
two discrete timesteps
A.2
MAXIMUM ENTROPY DISTRIBUTION
This subsection proves that the maximum entropy distribution with a fixed covariance matrix is Gaussian.
17

---

Published as a conference paper at ICLR 2023
Lemma:
Let q()) be any density satisfying f (())=;,,dr== Lij. - Σ). Then h(q) h(p)
Proof.
0 (L(()))= = q())) log g(r) dr
p(r)
= -1((1) f ((r) logg(r)))r
-h(1) - f log
= -h(()) + h(p)
since and yield the same moments for the quadratic form encoded by log p(r).
A.3 OPTIMAL HYPERPARAMETRA SELECTION USING BAYESIAN OPTIMIZATION
Most recent rescarch in Bayesian Optimization (BO) applications is limited to low-dimensional problems, as BO fails catastrophically when generalizing to sigh-dimensional problems (Frazier 2018). However, in this paper, we aim to use BO to optimize the neuronal and synaptic parameters of a heterogeneous RSNN model. This BO problem thus entails a huge number of hyperparameters to be optimized; hence, using standard BO algorithms remains a significant challenge. Hence, to overcome this issue, we used novel BO algorithm based on the assumption that our hyperparameters to be optimized are not completely random and uncorrelated but can be thought of as being drawn from probability distribution as shown by Perez et al Peree----Neee for the individual parameters themselves, we use modified BO estimate parameter distributions for the LIF neurons and the STDP dynamics. After learning the optimal distributions, we simply sample from the distribution to get the distribution of hyperparameters used in the model. To learn the probability distribution of the data, we modify BO's surrogate model and acquisition function to treat the parameter distributions instead of individual variables. This makes our modified BO highly scalable over all the variables (dimensions) used. The loss for the surrogate model's update is calculated using the Wasserstein distance between the parameter distributions.
et al. (202T). Thus, instead of searching
BO uses Gaussian process to model the distribution of an objective function and acquisition function to decide on points to evaluate. For data points in a target dataset æ X and the corresponding label y Y, an SNN with network structure V and neuron parameters W acts as function (v,w(2) that maps input data x to predicted label ÿ. The optimization problem in this work is defined as
min Σ L C fv.(()) V.W DeX.yEY
(9)
where is the set of hyperparameters of the neurons in R (Details of hyperparameters given in the Supplementary) and W is the multi-variate distribution constituting the distributions of (i) the membrane time constants Tm-E, mm-J of the LIF neurons, (ii) the scaling function constants AA,,A_) and (iii) the decay time constants T_, r_ for the STDP learning rule in SRR-
Again, BO needs a prior distribution of the objective function f(i) on the given data Disk (Zris, I (i)=)) }- In the Gaussian Process (GP)-based BO, we assume that the prior distribution of I' (Ti.k) follows the multivariate Gaussian distribution, which follows a GP with mean ÄD and covariance ED... Thus, we estimate DD,, using the modified Matern kernel function. We use the loss function as d(z,,²) x²), which the Wasserstein distance between the multivariate distributions of the different parameters. That is, given two distributions of hyperparameters I 11,2, the distance between these two distributions (given as d(x), X22) is used as the loss function the Matern kernel for the modified BO. We want to learn the optimal distribution of hyperparameters x', which maximizes the performance. It is to be noted here that for higher-dimensioe distance as regularized version of the Wasserstein distance to approximate the Wasserstein distance (Feydy et al. 2019).
metric spaces, we use the Sinkhorn
Dir are the points evaluated by the objective function. The GP will estimate the mean T...... and variance t' for the rest unevaluated data D The acquisition function used in this work is the expected improvement (EI) of the prediction fitness as:
EI (it=) = (ji,... - (Thest )) (T)) + TD.... ol ())
(10)
where ((-) and o()) denote the probability distribution function and the cumulative distribution function of the prior distributions, respectively. f (#hea) max f (T1+t) is the maximum value that
18

---

Published as a conference paper at ICLR 2023
Table 5: The list of parameter settings for the Bayesian Optimization-aasee hyperparameter search
Parameter
Initial Value
Range
(0,50)
(0.10)
2.5
(0,10)
(0,3)
(0,10)
(0.1)
(0,100)
À (SHD)
(0.2)
X (Lorenz)
(0,4)
Pir
(0,0.1)
Tu-E: Mu.EB.nn- Tn-F (SHD)
50nn
100ms)
(Lorenz)
100ms
(0mss, 300ms)
Aes- Ages Apt. Are. Air
30
(0,60)
Table 6: Table showing the average final distributions of the hyperparameters
Parameter
Distribution
p 18.235
T
Normal
8 1.522
STDP Parameter
22.382
T_
Normal
ē 1.768
I 0.516
Normal
0.0055
A= = 0.448
7/-
Normal
0 0.0057
á
LIF
Tm
Gamma
1/3 0.248
Parameter
Tim
.(i)
Gamma
1/3 0.313
5.14
has been evaluated by the original function ff in all evaluated data Disk and 2 AA ( ) BO
0Dam
will choose the data v; = arginax { [E7 (Teen) 3 E Tien) as the next point to evaluated using the original objective function.
A.3.1 OPTIMIZED HYPERPARAMETERS
The list the hyperparameters optimized using the Bayesian Optimization technique shown in Table 5 We also show the range of the hyperparameters used and the initial values. In addition to this, Table [6] enjitt the final optimized distributions of the STDP and the LIF parameters obtained using BO.
Memory Capacity (c)
10
10
Avg. Spike Count
6 10a
Spike Efficiency
6 10
5 10a
4 x 10
10
10"
100
0
20
40
20 Evaluations
Evallations
Evaluations
(a)
(c)
Figure 6: Figure Showing the convergence behaviors of the three types of BO described in the paper
(a) BO optimizing the memory capacity (b) BO optimizing the average spike count Š and (c) BO optimizing the spike efficiency 3
19

---

Published as a conference paper at ICLR 2023
Table 7: Table showing the performance of the Bayesian Optimization on the SHD Classification dataset for the three different cases where BO 1 optimizes C, BO 2 optimizes S and BO 3 optimizes
3
NLR
BFT Memsory Capacity
BOT Average Spikke Count
BO
Memory
B03
RF2 Average Spikk
BO Accurncy
Memory Capacitt
BO
B03
Spike
BO 3
Accuracy
Capacity
Accurncy
5.31
120.14
113.00
Dv.41
622
0.58
G25:23 5.74
1.2
1208.80
05.07
200
5,05
0.30
2
135.12
07.85
+ 0.16
5,38 0.53
1255. 10= 122.56
67.21
5.26+ 0.34
18 1:26
131 45
09.51 1
300
0,72 0.4
1441:14 148 83
68:41
+ 4.87
6.42 0.57
1325.25 + 128 47
67.06+ 5.11
6.54+ 0.30
1301
140.37
70.1 1.0
1:21
230.315
70.11
0.51
1-25,00
125.
20.25
7:01
1511.79
71.54
8.50
170s 24
287.94
709 0.38
1005.29
00.31 n 5.03
8,03 0.38
16218 a 250.40
13.05
1000
0..22
2251 37
319.75
72.50
a 3.38
9.00
0.44
2015.24 a 147.44
70.88 4.91
12:25 0.39
2102.09 279 2G
T0.32 3.44
2000
9.51
2566 21 848.08
76.30
= 3.29
9.89 0.86
2011.99 s 151.18
72.53 4.86
13,98
0.31
24100.08 = 901 57
17.25=
3000
14.07 0.55
2825.47
366.87
8.38
10.49
2623.41 177.54
74.53
4.983
0.42
2708-62
315.31
T8.21
3551
07 200.19
78.06
3.25
11.67
30145.28
73. 15. 4.85
15.87 0.38
3218.
328
79.36
5000
00.57
4186. 49 385.09
78.92233.31 3.31
11.08 a 0.48
8299,62 241.
75.87 4.81
10.03 0.41
3613.1
331 18
3.15
Table 8: Table showing the performance of the Bayesian Optimization on the Lorenz System
Prediction dataset for the three different cases where BO optimizes C, BO optimizes S and BO 3 optimizes 3
BOT Memory Cupaciiy
Average Spike
EO I
BO 1 RMSE
Memory
B02
BO Average Spike
BO RMSE
BO3 Memory
B03 Average Spike
BO.3 RMSE
Count
Capacity
Count
Capacity
Count
11.36 3,56
0.52
1101
0.08L 11.126
3.45
1207.52
11.39
1428.00
01587
3.80 1:38
a 118.57
0.030 I1.127
1922AT SRE.
0C11 0:015
300
5:35 1.37
1409.322
141.79
a 0.027
4.40 1.55
1257.264 a: 1257 26
0.568 a 0.054
5.15 0:34
1335,81
168.02
u.dd1 a 0.0287
400
6.37
0.41
1628.75 228 87
0.450 0.033
5.87 0.48
1304.35
126. 16
0.467 + 0.04
6.05-10 6:35
1415.32 215.40
0:182 0.0347
500
7.25
0.45
1601
277.97
0,399 0.00
0.20 0.446
1306.35
137.01
0.421 a 0,143
6.87 0.33
1500 290.29 a 219.58
0:11 x 0,0377
10:32
1868.14
300.17
0.336
1569.25
138.76
1229:27 275-79
0ZRP 0:0317
2105.005
0.5
1851.35
2014.12
0.001
2508.35
0.266
8.9 0.63
210
1700.5
0.33% 0.005
1387
2230.00
(121 0.0422
4000
15.15=0.55
2051.56
357 06
0.212 0.063
0.55 0.55
2115::1
217.73
0,325 10.07
4185= 0:41
2510.25 = 280.8
0.227 = 0.0619
3000
15.45 0.51
3186.41 353.51
0.2033 0.001
0.00
0.35
2651..00
251.63
0.302 e 0.071
15.12= 0:42
2506.21
e 307.14
01196 0. 0.0055
A.3.2 CONVERGENCE ANALYSIS
We compare the convergence analysis of the three Bayesian Optimization techniques and the results are shown in Fig. 6 Each of the experiments was repeated five times and the mean and variance of the observations are shown in the Figure. It is to be noted here that since we define the BO as minimization principle, we minimize _ S and E
A.4 COMPARING BAYESIAN OPTIMIZATION ORJECTIVE FUNCTIONS
We show the results of Bayesian Optimization results for the three cases we are considering in this paper for both the classification and prediction problems. The results for the classification problem are shown in T We tabulate the memory capacity, the average spike count and the observed accuracy for the three BO cases. Similarly, the results for the prediction problem are shown in Table B In that case, we tabulate the memory capacity, the average spike count and the observed NRMSE for the three BO cases. We rerun each of the experiments 5 times and report the mean and standard deviation of the results obtained.
A.5 COMPARING THE GENERALIZABILITT
We observed that increasing the neuronal heterogeneity increases the memory capacity of the network.
However, this increment in the memory capacity might lead to model which overfits the training data. However, the heterogeneous STDP model with varying synaptic dynamics gives rise to a heavy-tailed Feller process. Recent works Siissekli et al. (2020). Chakraborty & Mukhopadhyay (2021) show that the Hausdorff dimension of the trajectories of the sample paths of the learning algorithm can control the generalization error. This is intimately linked to the tail behavior of the driving process. The authors showed that heavier-tailed processes achieve better generalization. Thus, the tail index of the process can be used as a notion of capacity metric that estimates the generalization error, which does not necessarily grow with the number of parameters. The authors discuss that the stochastic process for the synaptic weights behaves like a Lévy motion around a local point. Because of this locally regular behavior, the Hausdorff dimension can be bounded by the Blumenthal- Getoor (BG) index (Blumenthal & Getoor 1960). which depends on the tail behavior of the Lévy process. Thus, can the BG index bound for the Hausdorff dimension of the trajectories from the STDP
20

---

Published as a conference paper at ICLR 2023
Table 9: Table showing the Ablation Study for the comparison of the Generalizability of heterogeneous networks
Training Accuracy (A)
Testing Accuracy (R)
Generalization Error
BG Index
(IA-BI)
Hom
Hom STDP
1.522
87.33
73.58
13.75
Hom LIF
Het STDP
1.438
85.31
74.03
11.28
Het LIF
1835
95.29
78.87
16.42
Hom STDP
Hct LIF
1.711
94,32
80.49
13.83
Het STDP
Table 10: Table showing results with limited training data
Train Accuracy
Test Accuracy (B)
Generalization
Train Accuracy
Test
Generalization
Percentange Training Data
Error IA-BI
Accuracy (B)
Error
(A)
(A)
IA-BI
Heterogencous LIF,
Homogeneous LIF,
Heterogeneous STDP
Homogeneous STDP
100
94.32
80.49
13.83
87.33
73,58
13.75
90
94.89
78.34
16.55
87.83
67.84
19.99
80
95.47
76.72
18.75
88.86
65.86
23
70
96.15
74.92
21.23
89.95
62.19
27.76
60
96.92
70.34
26.58
91.58
61.25
30.33
50
97.69
69.44
28.25
94.38
59.51
34.87
40
98.21
63.76
34.45
96.85
55.93
40.92
30
98.43
54.01
44.42
98.43
45.86
52.57
20
99.43
43.87
55.56
99.49
42.68
56.81
10
100
31.43
68.57
100
30.18 18
69.82
5
100
15.32
84.68
100
14.38
85.62
Heterogeneous LIF, Homogeneous STDP
Homogeneous LIF, Heterogeneous STDP
100
97.29
78.87
18.42
86.31
74.03
12.28
90
97.41
77.48
19.93
86.94
68.59
18.35
80
97.65
76.32
21.33
87.75
67.58
20.17
70
97.95
74.03
23.92
88.17
65.25
22.92
60
98.03
71.16
26.87
89.52
63.11
26.41
50
98.43
68.48
29.95
90.48
60.86
29.62
40
98.79
61.93
36.86
93.15
57.31
35.84
30
99.56
51.68
47.88
96.34
48.41
47.93
20
100
44.52
55.48
98.43
43.59
54.84
10
100
30.68
69.32
99.56
32.57
66.99
5
100
14.15
85.85
100
18.48
81.52
learning process. Now, as the Hausdorff dimension a measure of the generalization error and also controlled by the tail behavior of the process, heavier tails imply less generalization error. In this paper, we empirically study the generalization ability the HRSNN network using the BG index as metric. We did the experiments the ablation study models for the classification task on the SHD dataset, and the results are reported in Table From the table, we see that the heterogeneity in STDP improves the generalization error the most, while the heterogeneity in the LIF neurons increases the training and testing accuracies.
A.6 RESULTS ON LIMITED TRAINING DATA
We have trained the models with limited training data. We observe that the HRSNN model with heterogeneous LIF and STDP dynamics not only has better testing accuracy but also shows better generalization behavior when compared to other homogeneous RSNN or the other ablation heteroge- neous models (with heterogeneity in only one of them). Also, we see that the HRSNN model with heterogeneous STDP shows distinctly better generalization ability than the generalization ability of
21

---

Published as a conference paper at ICLR 2023
SHD
SSC
CIPARIODYS
Iraining Accuracy
Iesting
acternlizatio
Training Accuracy
Iesting Accuracy
Generaliation
Training Accurasy
Iestag
Generaliation
Accuracy
Eeroe
Error
Accuracy (D))
Levor
(d)
(1))
IA- 1
LA)
(27)
14 M1
1,0
IA 1-1
Hos LIF Hom STDP
W6.22
1.36
72,89 1.85
1108 1.04
74x0 1.72
47.3 1334
26.75 1.92
$2.01
18
60.35 2.01
Hom LIF Het STOP
$576 00209 1.20
1:27
70.01 + 1.40
11.85 1:25
70.70 1.58
52.06
1:73
23.90
1.20
83 60.98+110 1.52
67 67.05= 2.97
16-42 1.24
Hit Hom STDF
78,00 a 1.42
1o00 1.33
84.20 1.85
MM.1E 1.04
29.15 1.12
96.81 1.79
08.97 4.05
15.00 1.22
Ha LIF Har STOP
94.07 176
50.00 +
1400+ 172
80:41
1.00
10:28 1.35
27.13
84.19
70.51
16,85 + 1.38
HRSNN with heterogeneous LIF neurons. On the other hand, the latter showcases significantly higher training and testing accuracy compared to the former model. This can be interpreted as follows: since heterogeneous LIF dynamics increase the memory capacity, it leads to overfitting of the data. Heterogeneous STDP dynamics help in obtaining more generalizable solutions from this. Each has its own downsides; however, using HRSNN with both heterogeneous LIF and STDP dynamics shows better performance and generalization abilities, as seen from Table I0
A.7 FURTHER EVALUATIONS
we argued that as the heterogeneity in the neuronal parameters increases, the covariance decreases; hence the neurons become less correlated. In this section, we give empirical results to support the theory. We tested the model on more complex datasets (i) The Spiking Heidelberg Digits (SHD) dataset (ii) the Spiking Speech Command (SSC) dataset are both audio-based classification datasets for which input spikes and output labels are provided providedCCramer Cramer et al. (2020) and (iii) CIFARI0 DVS dataset Li et al. (017).
In Section B,
Impact of Heterogeneity on Covariance: We plot the covariance matrices for different levels of heterogeneity J (Eq, (46) for a small network with neurons. The covariance matrix is calculated by taking the average neuronal states before the appearance of the first spike in the final layer. We see that as the heterogeneity the neuronal parameters increases, the correlation between the neurons decreases. The results are shown in Fig. 7
Impact of Heterogeneity on Principal Components: From the covariance plots, we see that increasing J. reduces the correlation between neurons. We also plot the probability density functions of the eigenvalues of the covariance matrix of the neurons with increasing heterogeneity in the neuronal parameters. We see that with higher heterogeneity in the neuronal parameters J, the distribution of the eigenvalues of the covariance becomes flatter. This signifies that the covariance matrix has a lower variance for higher J. A flatter distribution also indicates that larger number of principal components are active. This supports our hypothesis that heterogeneity in the neuronal parameters increases the number of principal components and helps increase the model's memory capacity. The result is shown in this Fig. g
Impact Heterogeneity in STDP on Firing Rate: We plot the mean firing rate of the neurons for the four types of HRSNNs and MRSNN with homogeneous LIF and STDP dynamics. We plot the results for a smaller network with 100 neurons and a Poisson input process. The MRSNN model shows much higher firing rate, especially at higher frequency, demonstrating that MRSNN requires significantly more spikes than the HRSNN model. The result is shown in Fig. 9
We note here that in this paper, we use (homogeneous or heterogeneous) STDP to learn the synaptic conductance connecting various neurons in the SNN. Therefore, we do not control the synaptic coupling strength as independent variables and hence, cannot perform control experiments with various extents of coupling strength. An interesting future extension of the results will be quantifying the coupling strength for HRSNN with heterogeneity in LIF and STDP dynamics. We can leverage McKenzie et al MeKzzie et al. (2021). where the authors proposed statistical tools to estimate synaptic coupling dynamics from spike-spike correlations.
Coupling Strength:
22

---

Published as a conference paper at ICLR 2023
1.0
10
10
20
20
30
30
40
40
20
20
20
40
Heterogeneity H Increases
Figure 7: Figure showing as the heterogeneity in the neuronal parameters increases, the covariance between the neurons decreases
N=0.63
1.4
1t0.45
1.2
1/00337
0.8
Inernasing "
0.6
0.4
0.2
0.0
0.5
Covoriance Eigenvalues
Figure 8: With higher heterogeneity in the neuronal parameters H. the distribution of the cigenvalues of the covariance becomes flatter. This signifies that the covariance matrix has lower variance for higher H. A flatter distribution also signifies greater number of principal components are active, which supports our hypothesis that heterogeneity the neuronal parameters increases number of principal components and helps in increasing the memory capacity of the model.
Homogeneous LIF neurons
Heterogeneous LIF neurons
20
2.5
Spiic Count.
Spice Count
Homogeneous
STDP
20
10
15
Firing
Fiiing
20
Spike Count
Spica Count
10-
Heterogeneous
STDP
10
15
20
Firing
Firing Ratz (Hz)
Figure 9: Figure showing the histograms of the firing rates for the four kinds of heterogeneous RSNN with homogeneous /heterogeneous neurons and synapses
23

---

Published as a conference paper at ICLR 2023
B SUPPLEMENTARY SECTION B
B.1 APPROXIMATIONS AND ASSUMPTIONS
We make several approximations and assumptions for this section's theoretical analysis of the heterogeneous RSNN networks. Firstly, must noted that in this paper, the analytical relations are derived by taking the heterogeneity individually. ie., when consider heterogeneity in the neuronal parameters, we consider homogeneous STDP dynamics and vice-versa. In addition to this, we assume diffusion approximation. That is, if a neuron receives Poissonian uncorrelated input spike trains and the contribution of a single synaptic connection is small compared to the distance hetween reset and threshold << (Ve- Vg), the random input can be approximated by Gaussian white noise with mean and noise intensity o². This approximation does not hold if the network features highly correlated activity receives strong external input common to many neurons. Also, we assume a fast/slow synaptic regime in which the synaptic time constant is much shorter/longer than the membrane time constant Tm- In this work, we consider a mean-fiele approximation of the HRSNN network with heterogeneity in the parameters of the LIF neurons and the STDP dynamics independently.
B.2 MEAN-FIELD REDUCTION MODEL OF HRSNN
In this section, we model the HRSNN network using heterogeneity in only the LIF neuron parameters. Following the works of Ly et Ly (2015), we can write the equations for the excitatory neurons indexed by j {1,2...., Ne) are:
Tm dvj
dl -2 Die (t) (1-- - I0e (t) (u3 - EE) + a E7,(t)
(11)
e/ (t') 2 0 refractory period ) = 2/ (t' + +rees) =0
(12)
dnj
Tn dt = - + TT)
(13)
Iee (t)= Qj Nee jel presyn cells)
Yee
Σ
Gt,(t)
(14)
(1) Pei,s Ke( presyn Icells)
Ves Σ
Gx (1)
(15)
dG, -G2+ Aj
Td
(16)
dt
dA, -A, + T 5
Tr dt
(17)
where the inhibitory and excitatory reversal potentials are ε₁, and Eg, respectively, with & < 0 < Eg. (L) are uncorrelated white noise processes, Pay is the proportion of neuron type y (randomly chosen) that provides presynaptic input to neuron type x (I, y E (e, The second line in the equations describes the refractory period spike time t*. When the neuron's voltage crosses threshold 0,, the neuron goes into refractory period for Tref where the voltage is undefined, after which we set the neuron's 's voltage to In the last equation, & denotes the spike times of the j th excitatory neuron. Now, for the mean- mean-field analysis, we use ęjz to model the synaptic heterogeneity between the pre-and post-synaptic neurons by modulating the synaptic conductance for both the excitatory and inhibitory neurons.
We note here the numerical assumptions for the nean-field analysis:
1. finite size effects are negligible (N e/i 1
2. the firing rate of presynaptic neurons is governed by Poisson process
3. the population firing rate averaged over q and Tm is a good approximation the average presynaptic input rate and
4. single p.d.f. function is sufficient to describe the population behavior) (finite N)
24

---

Published as a conference paper at ICLR 2023
Similarly, for the inhibitory neurons indexed by e (1,2...... ,,,,, the equations
Tm duk
dt UR (t) (vx- E1) gei (e)(t) (vx Eσ) + m
(18)
2k (t*) 1( refractory period) ; (t + Tref) =0
(19)
dyp
Tn dt
-l] + v Ttt((t)
(20)
(i) qi Tie
Σ
GL((1)
(21)
Pie Ne wel presyn cclls)
ga(1)
Via
Σ (R.(1)
(22)
PuNs we [presyn cells]
dGa -G + Aa
Ta
(23)
dt
dA A6 + T L
Tr dl
(24)
Please refer to the paper by Ly et al. Ly 2015) for details regarding the equations. Since the recurrent coupled stochastic network is difficult describe theoretically, where an equation determines the probability of a neuron being in a particular state. The variables in the populations are determined using distribution functions. The two forms of heterogeneity introduce a large number of dimensions. For simplicity, one can track a family of probability density functions for each (ąj. pair for each neuron. The subsequent equations are good approximation the HRSNN network with the following assumptions: (i) finite size effects are negligible (Nefi 1) (ii) the firing rate of presynaptic neurons is governed by Poisson process (iii) the population firing rate averaged over q and Tm a good approximation to the average presynaptic input rate (iv) a single p.d.f. function is sufficient to describe the population behavior, and the heterogeneity is driven by (m. J). the probability density function p is defined by:
population density methods,
(gn; J) For each pair of values (q),
WE. UI. w),t) dugdwgdujdw = P ((E
wi())
E S2)
(25)
where wx denotes the other states variables of the corresponding neuron type X E (E, consisting of conductance, colored noise: wx 7x). The evolution of the p.d.f.'s is governed by continuity equation and boundary conditions:
0p
.J
(26)
Ot
(Jugy JOE Jag Jnn: Jups Jan Jni
(27)
Jog
[UE Nleig (vg - Et) qeeell (vg - Eσ) dENE]p
(28)
Tm
Joy
[ur + Tigr (vr - Ei) + Nall (wr Ex) + orppp
(29)
Jg =
[gx ax]p
(30)
ax
ux((t) f.aee p d'...) do'x
Jax =
(31)
T
1
02p
Jnx =
nxp+
+
On
(32)
Tn
On
[S] Tm
Ux
(t) Ux(()
Jox dddTT
(33)
Jwx |dwx= =
(34)
25

---

Published as a conference paper at ICLR 2023
8
Efficiency E 1000M/SS)
14
MRSNN
Memory Capacity
12
U
x
u
0.7
100
500
1000
2000
3000
4000
5000
Neuronal Heterogeneity H(Pa) Σp(x)log.ptxx))
Number Neurons
(a)
(b)
Figure 10: (a)Figure showing the variation of memory capacity with neuronal heterogeneity (b)
Figure showing the variation of efficiency & with number of neurons NR
The definitions of gxy in the LIF neuron equations defined above result in a total conductance of 9XY on average.
We describe an insightful analytic reduction that captures how the range of excitatory firing rates changes in different regimes. We focus on only the excitatory neurons, which have fewer variables if the inhibitory population is ignored or assumed to be known.
Let us denote the approximate excitatory firing rate(s) ug as r. The deterministic firing rate of the equation
Tm dug
UE 991 (ug Eτ) qgæ (vg E3) + Tī
(35)
is given by
0,
if L S 0
(+g(dg+0r)
ro Tmi WB)
1+0 9 +r)
if ( 9
(36)
Tm
13---11 t
1+9(gE+9i)
We define: ЯE := YeeGE, ف1 == YesTı, ME == OETE- Finally, the given state variables are integrated against their marginal density get:
r(g. 0) E
ro
(1B S1. (E) dw
(37)
roTre
1 roTref
There a slight abuse of notation because the auxiliary variables ax effect the conductances but are not written in the previous equation; the emphasis is how (gg.9) 91, πH)) directly effects Since the external noise is applied indiscriminately. TE is independent of the other variables and the marginal density factors into:
(VElle))) TEVI
أ (gE, 91, TE) = (gE.
(38)
However, (ge, 91) still not analytically tractable, leading us rely on Monte Carlo simulations to numerically estimate (9E.ği).
must be noted here that this reduction model for the HRSNN network with many simplifying assumptions. It is not complete mean-field derivation of the HRSNN model with heterogeneous LIF neurons, and heterogeneous STDP dynamics a fascinating research question but beyond the scope of this paper.
B.3 ANALYTICAL RESULTS
Analytical Results of Memory Capacity Neuroscience networks of spiking neurons are increasingly used understand mechanisms underlying phenomena observed in electrophysiological recordings.
26

---

Published as a conference paper at ICLR 2023
There are two complementary strategies for studying such recurrent network of spiking neurons (a) numerical simulations and (b) analytical methods using mean field models. With numerical simulations, we simulate any network model without any approximation. However, this method typically works in high dimensional parameter space and is, thus, hard to interpret. Also, it is generally hard to characterize parameter regions where specific behaviors are found using numerical simulations. On the other hand, with analytical calculations, we obtain deeper insights into mechanisms underlying specific behaviors and can obtain critical parameters that control specific behaviors. So, now, we analytically study the variance of the estimated memory capacity with the change in the heterogeneity of neuronal parameters. We plot the change in the estimated memory capacity C, calculated using Eq. 3 neuronal parameters for the HRSNN model. The result plotted in Fig. aoaa). We use a HRSNN model with NR= = 1000 and sequences of 4,000 random inputs chosen from -11; ;; 1]. We see that, as predicted, the memory capacity of the model increases linearly with the increase in hetcrogeneity within the limits of the application, proved in Theorem 1. The error bars in Fig. T0 represent the standard deviation of the observations.
We plot this with respect to the neuronal heterogeneity H. measured using the entropy of the
Analytical Study of Spike Efficiency We calculate the average firing rate of the heterogeneous spiking neural network for the prediction task during inference, and the results are shown in Fig[TO])) Using heterogeneity in the STDP parameters reduces the average number of spiking activations while keeping the memory capacity almost equal. This result shows that Heterogeneous STDP leads to sparse activation of neurons, as proved in Theorem 2.
We compare the analytical results obtained with some of the standard recurrent LIF network models in the literature. Brunel et al Brunel (2000) analytically study the dynamics of sparsely connected a network of sparsely connected excitatory and inhibitory integrate-ane cnerrate-and-ffe and-fire neurons. The authors showed the existence of a diverse set of states, including synchronous states in which neurons fire regularly; asynchronous states with stationary global activity and very irregular individual cell activity; and states in which the global activity oscillates but individual cells fire irregularly, typically at rates lower than the global oscillation frequency. In this paper, we use heterogeneity in the LIF neurons. This leads to a diverse set of states for the ncurons, which consequently helps orthogonalize the state space dynamics to increase the information stored in the memory of the network.
Comparison with Neuroscience Works:
Deneve et al. Denève & Machens (016)) discussed the inefficiency of irregular Poisson rate encoding in the brain. The authors argue that the Poisson point process, which we use to model the spike firing rate, is extremely inefficient it exponentially increases the number of spikes required to convey information. The authors further discuss that a continuum exists between loosely balanced and tightly balanced spike-coding networks in neuroscience. Though loosely balanced networks are inefficient, they are cheap in terms of the number of connections per neuron and structure (Boerlin et al. 2013) Boerlin & Denève, 2011 [ourdoukan et al. 2012). On the other hand, tightly-balanced spike-coding networks are highly efficient but extremely structured, dense connections that STDP rules must constantly maintain. For the HRSNN model, since we are engineering an artificial spiking neural network model, our network is highly structured and constantly updated using the heterogeneous STDP rules. Thus, we might say that the HRSNN model is tightly-coupled network that helps an efficient transfer of information. This hypothesis is supported by the results shown in Table 2 where the HRSNN model shows a higher performance using a lesser number of spikes.
B.4 MEMORY CAPACITY
Let (t) E U (where -00 < < +oo and U c R is compact interval) be single-channel stationary input signal. Assume that we have an RSNN, specified by its internal weight matrix W. its input weight vector wt and the unit output functions f, rom The network receives sr()) at its input unit. For a given delay r and an output unit y, with connection weight vector w we consider the determination coefficient
d [ (x(t (x(1- () (1))
=d -d(z(t- - w
out ( x((1)
r(t) )
Cov2 (z(t -). (t))
) (1))
27

---

Published as a conference paper at ICLR 2023
where Cov denotes covariance and o² variance. The r-delay Memory capacity of the network is D0
defined by C+ max d [w,"]) (x(1 - -), (t)). The Memory capacity of the network is C = C
The determination coefficient of two signals is the squared correlation coefficient. It ranges between O and 1 and represents the fraction of variance explainable in one signal by the other. Thus, the Memory capacity measures how much variance of the delayed input signal can be recovered from optimally trained output units, summed over all delays. Note that the output units do not interfere; arbitrarily, many output units yr can be attached to the same network.
The performance of the heterogeneous network model derives from its ability to retain the memory of previous inputs. To quantify the relationship between the recurrent layer dynamics and the memory capacity, we note that the extraction of information from the recurrent layer is made through linear combination of the neurons' states. Hence, more linearly independent neurons would offer more variable states and, thus, more extended memory.
For reservoir computing (RC), Jaeger et al. Jaeger (2022) shows that C is bounded by the reservoir network size of the linear RC with the identity activation function and the independent and identically distributed (i.i.d.) model input. Memory capacity ( C) is used to quantify the memory of RSNN. Such memory capacity measures the ability of RC to reconstruct precisely the past information of the model input. Also, the network's s structural properties can greatly impact the C. of the linear RC. Now, the question arises what is the need to maximize the memory capacity of the network? The C normally serves as global index to quantify the memory property of the network. To comprehensively examine the memory property deeply, the local measurement of its memory property is indispensable. Thus, maximizing the C acts as an estimator for better prediction results of the trained network.
Since the first-order approximation of the model is linear, the heterogeneity between state variables depends on all the eigenvalues of the adjacency matrix, with larger mean eigenvalue meaning higher heterogeneity. Hence we can use the eigenvalues {}i} of the weight matrix W to quantify approximately how fast the input decays in the recurrent layer. In other words, the eigenvalues of W should be related to the memory capacity of the heterogeneous neural network model. Indeed, we find that the average eigenvalue modulus: (IAl) = L/NR, r. NR [Aa] strongly correlates with H and therefore with C as well. Note that, instead of and H. ([>l) is much easier to compute and is solely determined by the recurrent layer network.
The memory capacity reflects the precision with which previous inputs can be recovered. The nonlinearity of the recurrent layer and other far-in-the-past inputs induce noise that complicates recovery. Thus, similar to the analysis done by Aceituno et al |ceeito et al. (2020) for Echo state networks. the variance of the linear part of the recurrent layer is placed to maximize the recoverable information. Thus, the inputs are projected into orthogonal directions of the recurrent layer state space to not add noise to each other. The variance spread across the different dimensions should be evenly distributed within those orthogonal directions, quantified by the neurons' covariance.
We start by noticing that the linear nature of the projection vector Wout implies that we treating the system as
20
r(t) Σ (axx(t-)++(())
(39)
where the vectors a, e R. correspond to the linearly extractable effect of zx(t onto r(t) and E(t) is the nonlinear contribution of all the inputs onto the state of r(t).
Previous works have shown that linear recurrent layers have more extended memory, but nonlinearity is needed to perform interesting computations. Here we show that for a fixed ratio of the nonlinearity, greater heterogeneity leads to a lesser neuronal correlation, leading to higher memory capacity.
To maintain this trade-off between linear and non-linear behavior, we will assume that linear and non-linear strengths distribution is fixed. This can be achieved if we impose that the probabilities of the neuron states do not change, meaning that the mean, variance, and other moments of the neuron outputs are unchanged; hence, the strength of the non-linear effects is unchanged. A first constraint can also be obtained from the maintained strength of the linear side of Eq 39
Var Σ 7-
a a (1-
(40)
28

---

Published as a conference paper at ICLR 2023
where e cis constant.
Lemma 3.1.1: The state of the neuron can be written as follows:
NR Σ (M wh)) ( k)
(1(1)
(41)
12
where Vn; Vi, V are, respectively, the left and right eigenvectors of W, and X X belongs the diagonal matrix containing the eigenvalues W: a; [ar.0, O;,1,-...] represents the coefficients that the previous inputs x, (tt have (t).
Proof: We build on the work Aceituno et al Aceituno et al. 2020) where they showed that higher heterogeneity among the neuronal states implies higher memory capacity. Here aim to show that as the number of neurons Ng in the recurrent layer decreases, heterogeneity increases the spectral radius. More formally, the spectral radius [na] is directly proportional H. NR decreases. We express the state of neuron (t) as
ri(t) Σ (W" w x(t x() Σ (axxo(t-a (a., Xs) kw0
(42)
k-0
where the vector a; = [0 ag,1....] represents the coefficients that the previous inputs x; [=<()),<(l+ .). have r1). We then plug this into the covariance between neu- rons,
1 T-T
Cov (ri,r,) = lim T-000
Σ
(a, x,y
(a).
T q-t
T
T
= aa)
lim T
Σ
Σ
(x N4,)
T
= aa;;a;;
lim
(xg: Xx)
T-00
T 7-0
= aa;,
x E x (1)
= az)
(43)
Now we write a as a function of the eigenvalues of W. Using the eigenvalue decomposition of the weight matrix W, rewrite the state of the neuron follows:
Nu Σ K-O m=1
(t())
At (u win) (Un);
(44)
where v., v V are, respectively, the left and right eigenvectors of W, and X À belongs to the diagonal matrix containing the eigenvalues of W; a; = [a [..., represents the coefficients that the previous inputs x, [=(t), x(t ... have rt).
Theorem 1: If the memory capacity of the HRSNN and MRSNN networks are denoted by Ch and Ca respectively, then, Cи 2 CM. where the heterogeneity in the neuronal parameters H varies inversely to the correlation among the neuronal states measured as Σ,A Σ Cov² (I,, xm (t)) which in turn varies inversely C.
As shown by Aceituno et al Aceituno et al. (000)), the memory capacity increases when the variance along the projections of the input into the recurrent layer state has higher heterogeneity. This can be expressed in terms of the state space of the recurrent layer. Now, we aim project the inputs into orthogonal directions of the network state space. Thus, we model the system as
Proof:
r(1)= Σ a, x e(1)
(45)
where the vectors a, R correspond to the linearly extractable effect of æ(t - -7) onto r(t) and a()) is the nonlinear contribution of all the inputs onto the state of r(t).
29

---

Published as a conference paper at ICLR 2023
Since goal is to have variance as homogeneous as possible along with the directions of ay need a variance that as homogeneous along with orthogonal directions, where the vectors a, E R- correspond to the linearly extractable effect of the input variable zz(t) onto the states of the neurons (r(1)). Since the eigenvectors of Σ preserve orthogonality across the covariance matrix Σ, the new variances are given by the eigenvalues of the covariance matrix, A, (E). Thus, work the distribution of the eigenvalues of the covariance matrix. Specifically, we want to show that increasing the heterogeneity in the neuronal membrane time constants decreases the correlation between the neuron states, which decreases the variance of the neuronal states of the eigenvalues, which would increase the memory capacity C. We quantify the heterogeneity using the mean with respect the square root of the raw variance of the eigenvalues of the covariance matrix given by
J Ena (2)
(46)
EE An (E))²
where Xn (E) is the nahh eigenvalue of E. To get intuition of how this metric reflects the hetero- geneity in the neuronal parameters, consider the case of two eigenvalues \1, when \ A2-very homogeneous then J 3. but when Å₁ 0, A2 = 0- heterogeneity is more and hence, J 1. The membrane time constant is given by the product of the membrane resistance Rm and membrane capacitance Cm, such that Tm Cm Rm is the inverse of the permeability; the higher the permeability, the lower the resistance, and vice versa. Thus, the lower the time constant, the faster or more rapidly membrane will respond stimulus. The effects of the time constant on propagation velocity will become clear below. Hence, variability in the membrane time constants will lead to variability in the propagation velocity of action potentials.
Now,
No. Var Var(r.(()))
Σ Am 2)) = (trl (
(47)
==1
which is constant by the assumption that the probability distributions of the neuron activities are fixed. Hence we can focus on the value of ENN X ()) which is true since
Nn = tr [>3]
E'en () > >n) e.. (x)
(E)e,
Σ tr)
(48)
e (E) and An E) are, resp. the rath eigenvector and eigenvalue of Σ. Hence, we can compute this by decomposing the square of the covariance matrix as follows:
where e,
NR NR.
No No E Cov? (In(1), (1))
Enm
Emn
Σ
(49)
m=11
mmm1
where Dij the factor matrices obtained using Cholesky decomposition of Σ. Thus, EN A increases as the neurons become more correlated; hence heterogeneity decreases.
Σ
Thus, from Eqs.
49 we can write the heterogeneity as inversely proportional to EAN Cov2 (I), Fm (t)) We see that increasing the correlations between neuronal states de- creases the heterogeneity of the eigenvalues, which would reduce the memory capacity of the model. We show that the determinant of the covariance between neuronal parameters bounds the heterogene- ity, Thus, as H increases covariance decreases neurons become less correlated. Accituno et aleAcciuuno et al. (2020) proved that the neuronal state correlation is inversely related to the memory capacity of the network. Hence, we claim that as H. increases, the memory capacity C also increases. Hence, for HRSNN, with H > O, CH 2 C.M.
46
B.5 SPIKING EFFICIENCY
In this section, we model the spiking activity using point process called the multivariate Point process model. A point process is a collection random points on some underlying mathematical space, such as the real line, the Cartesian plane, more abstract spaces.
The notion of using point process models, especially the interactive Hawkes processes, to model the spiking dynamics of LIF network dynamics has been studied in the literature previously (Locherbach)
30

---

Published as a conference paper at ICLR 2023
Galves & Löcherbach 2016 Mascart 2021: Pfaffelhuber al. 2022). We leverage these results to prove that heterogeneity in the synaptic dynamics can help reduce the spike count, as already discussed in the paper. We highlighted the key assumptions used in deriving the results in the Suppl. Sec. C. We apologize if there is still confusion, and we will add more in-depth discussion in the final manuscript as discussed below. In their paper, Locherbach al [6chherbacch (2017) survey some aspects of the study of Hawkes processes in high dimensions to model biological neural systems and study their long-term behavior. Galves of point processes used as stochastic models for interacting neurons in discrete and continuous time. Similarly, Hawkes processes have met a recent interest in the mathematical neuroscience literature for their ability to model the dependence of neuron's ' activity in the network's 's history (Mascart) Pfaffelhuber et al. 2022 Galves& & Löcherbach 2016; Gerhard et al., 2017: Zhou et al. 2020 Duval et al. (022). Other works have also used a nonlinear interactive Hawkes process to model spiking ncural networks with excitatory and inhibitory neurons (Chevallier et al.) 2015 Chornoboy al. 1988; Hansen et al. 2015 Ercynaud-Boures et al. 2014)) Drawing from these works, we use microscopic model describing a large network of interacting neurons that can generate oscillations in macroscopic frame. In the model, the activity of each neuron is represented by point process indicating the successive times at which the neuron emits a spike, where each realization of this point process is the spike train. We take the spiking intensity of neuron as the probability of emitting a spike during the next instant, depending on the history of the neuron and the activity of other neurons in the network. The neurons interact through their synapses. This means that a spike of a pre-synaptic neuron leads to an increase of the membrane potential of the post-synaptic neuron if the synapse is excitatory or a decrease if the synapse is inhibitory, possibly after some delay, like the process of synaptic integration. The neuron fires a spike when the membrane potential reaches a certain upper threshold. Thus, excitatory inputs from the neurons in the network increase the firing intensity, and inhibitory inputs decrease it. Hawkes processes provide good models of this synaptic integration phenomenon by the structure of their intensity processes. This paper uses a general class of mean-fied field interacting Hawkes processes, modeling the reciprocal interactions between population of excitatory neurons and a population of inhibitory neurons.
2017
Galves & Löcherbach (2016) provided an overview
2021:
Let us consider subsection of the HRSNN network as shown in Fig. 11 denoted by N₂. We use the multivariate Point process model to create a probabilistic model that relates the inner structure of the sub-network and its spiking activity. In this model, each neuron has background spiking intensity i caused by neurons outside the network. We know that when a neuron spikes, it impacts its spiking activity and the spiking activity of its output neurons. The impact of neuron j on neuron i is modeled by real function (t). This impact can be excitatory or inhibitory depending on whether the pre-synaptic neuron is excitatory or inhibitory, as shown in Fig. II While the spikes from excitatory neurons try to excite another spike, spikes originating from inhibitory neurons try to inhibit the spiking of the cascading neuron.
A Hawkes process is a point process in which each point is commonly associated with event occurrences in time, where every event time impacts the probability that other events will take place subsequently. These processes are characterized by the conditional intensity function, seen as an instantaneous measure of the probability of event occurrences. A Hawkes process is a point process in which each point is commonly associated with event occurrences. In this past-dependent model, every event time impacts the probability that other events take place subsequently. These processes characterized by the conditional intensity function, seen as an instantaneous measure of the probability of event occurrences. Although the self-exciting Hawkes process remains widely studied, there has been growing interest in modeling the opposite effect, known as inhibition, in which the apparition of certain events lowers the probability of observing an event. In practice, this amounts to considering negative kernel functions. To maintain the positivity of the intensity function, a non-linear operator is added to the expression, which in turn entails the loss of the cluster representation. This model is known as the non-lineat Hawkes process, where the existence of such processes was proved via construction using bi-dimensional marked Poisson processes. The general Hawkes framework can be written as:
X/- $; ( Σ S S(nnax(t--u)d22 (t- jESi
(50)
where X is the intensity of neuron i, Q a positive function, Zz.e is the counting process associated with neuron j, ht) is the synaptic kernel associated with the synapse between neurons j and i.
31

---

Published as a conference paper at ICLR 2023
Excitatory Pre-synaptic
Inhibitory
Pre-synaptic
Post-Synaptic
Excitatory
Pre-synaptic
Figure 11: Figure showing the excitatory and inhibitory pre-synaptic neurons with excitatory and inhibitory spikes respectively incident on the post-synaptic neuron. We use this model to model the nonlinear interacting Hawkes process with inhibition.
To simplify the notation, we rewrite Eq. 50 as
(t) = 2
Σ Sous Neet (0,1)
s)dtt))
(51)
where (t- s) measures the influence of neuron & on neuron i and how this influence vanishes with the time. More precisely, hus - s) describes how a spike of neuron k lying back t t- s time units in the past influences the present spiking rate time L.
The goal of using heterogeneity in the STDP dynamics is to get better orthogonalization among the recurrent network states to lower higher-order correlations in spike trains. Studies have shown that the correlation of higher order progressively decreases the information available through neural population (Montani al. 2009 Abbott & Dayan 1999). Since we are trying engineer spike- efficient model, we leverage the heterogeneity in the STDP dynamics to reduce the higher-order correlations. The hypothesis is that using heterogeneity STDP helps us orthogonalize the recurrent layer that can help us achieve an efficient representation of the input spike patterns with fewer spikes. This may be interpreted as the recurrent layer acting as an orthogonal bases function where inputs are projected onto these bases. Thus, having orthogonal bases can efficiently map inputs without much loss. While heterogeneous LIF neurons help us increase the number of principal components, thereby enabling to store a greater subclass of features, heterogeneous STDP helps us efficiently encode this orthogonalization of the recurrent layer, resulting in fewer spikes compared to homogeneous RSNN. Thus, in effect, heterogeneous STDP parameters can learn the output more precisely, which is projected back into the recurrent network. One of the primary reasons why heterogeneous STDP helps project the input to orthogonal activations of the recurrent network can be attributed to the distribution of LTD dynamics, as this increases the competition and helps distribute the input projection to multiple principal components. We discuss that the heterogeneous LTP/LTD dynamics in STDP lead to fewer spikes in the transmission of information.
Lemma 3.2.1: the neuronal firing rate of the HRSNN network with only heterogeneity in LTP/LTD dynamics of STDP is represented as Φn and that of MRSNN represented as + then the HRSNN model promotes sparsity in the neural firing which can be represented as t n < 4M.-
Proof: In this lemma, we show that the average firing rate of the model with heterogeneous STDP (LTP/LTD) dynamics (averaged over the population of neurons) lesser than the corresponding average neuronal activation rate for a model with homogeneous STDP dynamics. We prove this by taking a sub-network of the HRSNN model as illustrated by Fig. II Now, we model the input spike trains of the pre-synaptic neurons using a multivariate interactive, nonlinear Hawkes process with multiplicative inhibition (Duval et al. 2022).
We consider population of neurons of size N that is divided into population A (excitatory) with size NA == aN and a population B (inhibitory) with size NB = (1 a.N. A particular instance of the model is then given in terms of a family of counting processes (z! 1.l....,7,, .7,* (population A) and (Z,N+11 Zz") (population B with coupled conditional stochastic intensities given respectively by 2 and AB Consider on a filtered probability space (2, (FF) P) an independent family of i.i.d. Poisson measures ( ds, dæ), {1....,N), with intensity measure ds x dz on [0, oo) x [0, o0). Let (x,y) r F F(x,, y) and (x, G(x,y) y) two nonnegative functions defined on (0, o0)
32

---

Published as a conference paper at ICLR 2023
We assume that F and G satisfy
F(z,y) 4 (0)
< + + AA)B(y)
where A, 1 A- &B and 4 A- B are nonnegative functions, each of them globally Lipschitz with B- bounded (and with no loss of generality assume 0 A 1
Let us consider the family of càdlåg (Fi) point processes (Z;) 12...= given by
Zi = L So Lecy mo ds, dz), I,...,,
where the intensity X = I,..., N. is given
1 J.N = AA
Σ
[''
h1(t u)dz
PB+A
Σ
J. h2(t- - u)dzz
(52)
N
N
JeA
jeB
S. halt - sa(t-u)dZy
X B.N = 2B
Σ JEB
- ca(t-w)d23
+ +AABB
Σ JEA
(53)
where A&B are the populations of the excitatory and inhibitory neurons, respectively.
The dynamics given by Eq. 53 is of Hawkes type: each particle's intensity depends on the whole system's 's history, through memory kernels h; 1.....4 4 and firing rate functions O and d The multiplicative influence of inhibitory population B onto population A, is represented using the inhibition kernel & B-A which decreasing nonnegative function [0, +o0), with & (0) 0- i.e., activity of population A should decrease as activity of population B
and & A
rises. The model secondly incorporates retroaction from population A onto population B, which is supposed to be mostly additive, although possibly modulated by a nonlinear feedback kernel A
Now, without loss of generality we assume that A and 4g are linear i... Pa(x) HA+ (z(x)= μB x, 0, where MA pB O, and h, 2 0 for 1,....,..
Hence, Eq. 53 becomes
A₂ (HA + h1 w) du B-A ((1 a) So h2(1 u) du 2 = =B= (1 (1-a) a) Ji ha(t u) u du DA-B (a So ha(t - u) A1 du
(54)
For heterogencous neuron populations, there exists asymmetry of the weights. Based on balanced spiking neural networks with heterogeneous connection strengths, previous works have revealed that such heterogeneous networks possess heavy-tailed Lévy fluctuations (Shlesinger al. 1987
Mantegna & Stanley] 1995] Cossell et al. 2015). The heterogeneous heavy-tailed distributions of synaptic weights have been fitted lognormal distributions (Buzsáki & Mizuseki 2014 Kušmierz et al. 2020). We model the inputs to neuron e E as:
4(t) WEX TIEE Σ cijs;(t) + WEETmE Σ cij8s(t) - WEITHB Σ (t)8)() JeX
(55)
jeB
jel
En App + (t)
(56)
where μ denotes the mean inputs such that μg K KTmE (WEXTX -EETE WErTt);
Ap; "quenched' fluctuations (from neuron neuron) with variance (Ap²)
K-nn (W2 (r3x + + (r2+ Ar2)+ w (+7+ due to random connectivity. Finally, 77 denotes temporal fluctuations due to spiking activity. We assume that the pre-synaptic neurons fire as using the interactive Hawkes process described above.
Consider case where A >> 1, HB 0 and h 1[0.0]
{
x (Ma + a l/' hn (t w)d @B-A ((1 ((1 a) h2 -oett-uddy :)
(57)
X = (1 - a) l/ ha add a ht - u)d
33

---

Published as a conference paper at ICLR 2023
In normal case, the excitatory and inhibitory populations follow the following steps: (1) fx 0, 0,
AA is high and X," O is small (2) Feedback from A to B: Xe increases (3) Inhibition of B to A when X gets high, 4 B-B reduces X (4) ha has compact support: after a time 02, B no longer feels the influence of A intensity of B is back to μB * and A to its normal high activity Mr (State
This leads to oscillations which lead to spikes. However, heterogeneity in the synaptic dynamics increases the stochasticity of the pre-synaptic spike arrival. Thus, due the heterogeneity, & F--Λ promotes the system in the inhibition state (state 3) and inhibits the system's 's movement to system 4 and system 1, thereby creating spike. Hence, $ $ Similarly, for the inhibitory neurons, we can show that + R $ M.- Thus, we get & Φ M
This lemma might be interpreted as the heterogeneous STDP dynamics increasing the synaptic noise, which reduces the number of spikes of the post-synaptic neuron. heterogeneous STDP leads to non-uniform scaling of correlated spike trains leading to de-correlation. Hence, we can say that heterogeneous STDP models have learned a wetter-orthogonaaized subspace representation, leading to better encoding of the input space with fewer spikes.
is to be mentioned here that the synaptic noise might be thought of as analogous to the stochasticity in the gradient descent algorithm. As recently proved by Simsekli et al. (Simsekli et al. 2020 2019)
stochasticity plays an important role in the generalization ability of the model. We might interpret the synaptic noise in the heterogeneous STDP to play similar role and helps in better generalizability of the HRSNN model. This hypothesis is empirically proven in Supplementary Section A. However, a detailed theoretical analysis would be very interesting direction for future work.
Theorem For a given number of neurons NR, the spike efficiency of the model & C(Na) for HRSNN (É)) is greater than MRSNN (Eys) i.e.. ER EM
Proof: To study the effect of the spike time when the weight wi changes, we look into the expected value of the time difference in the post-synaptic spikes, which is given as:
E tpsss ] E [Eposs Epost
(E [pposs ] tpos Pr[s]
(58)
where Pr[3s] is the probability of occurrence of the post-synaptic spike. Thus, the expected input to the neuron at time (E [i(()), which comprises of its excitatory and inhibitory components
E ((),, can be expressed as:
B[Ai(t)] [t) AB((())
for t < &ppst
(59)
where E (() Pe Jo² Mw, (w, 1) t)dw
[[(((t)] = So² Mwx, (w,,w,,)de t)dw
(60)
where Per P: are the rates of incoming spikes and thx. (w, t). Mww (w, t) the probabilities of the weights associated to time t. Now, considering the case for RSNNs with homogeneous STDP (M) and with heterogeneous STDP (I), the difference the variances of the two populations is given as:
A Var VM] - A [ar[ = A S [E] - [=] -E(-)(((())()()))))
(61)
Since t < Epese STDP potentiates both inhibitory and excitatory synapses, AE (E(((t)
0, AE [t) > O. The term Bii [1) = 0 by the symmetry of the weights, and it is main- tained at zero by the symmetry of the STDP. But for heterogeneous neuron populations, as described above, there exists an asymmetry of the weights. Based on balanced spiking neural networks with heterogeneous connection strengths, previous works have revealed that such heterogeneous networks possess heavy-tailed, Lévy fluctuations (Shlesinger et al. 1987; Mantegna & Stanley 1995 Cossell al. 20155. This implies [t) 0= A Var VR] A Varl v(t)ar] We calculate the number of post-synaptic spikes triggered when the stimulus is present. Now, representing the spike rate of the HRSNN and the MRSNN as R & PM resp.,
So ('on())dss S' (1) SR= NR FIsI
T
S NR.
S
(62)
FISI
34

---

Published as a conference paper at ICLR 2023
Thus, spikes decrease when we use heterogeneity in the LTP/LTD Dynamics. Hence, compare the efficiencies of the HRSNN with that of MRSNN as follows:
Σm Cov (a0)))))))))))(
td
ER_ MR(NRR SM
Var(a"
leef
(63)
m SR x Mar (NR)
DNR Cov x 1d
00
Var(a))Far(1))
tref
Since SR & SM and also, covariance increases when the neurons become correlated, and as neuronal correlation decreases, H. increases (Theorem 1), we see that En 2 1 = En 2 EMs
Ex
C SUPPLEMENTARY SECTION C
C.I HIGHER ORDER CORRELATION
In this paper, we took inspiration from results in reservoir computing, which show that we can maximize memory capacity using orthogonalization among reservoir states in the case of reservoir computers (Farkaš & Gergel" 2017 Farkaš et al., 2016). The goal of using heterogeneous STDP dynamics is to get better orthogonalized recurrent network states to achieve more efficient information transfer with lower higher-order correlations in spike trains. Recent studies (Montani et al., 2009) Abbott & Dayan 1999) have shown that the correlation of higher order progressively decreases the information available through the neural population. The decrease in information becomes larger as the interaction order grows. Since are trying to engineer spike-efficient model, we leverage the heterogeneity in neuronal parameters to reduce the higher-order correlations. The hypothesis is that orthogonal recurrent layer can help efficiently represent the input spike patterns with fewer spikes. This may be interpreted as the recurrent layer acting as an orthogonal bases function where the inputs are projected onto these bases. Thus, having orthogonal bases can efficiently map the inputs without much loss. The heterogeneous STDP helps us efficiently achieve this orthogonalization of the recurrent layer, resulting in a lesser voltage variance across the neuron population. This leads to fewer spikes (since the mean is constant) compared to homogeneous RSNN. Thus, in effect, heterogeneous STDP parameters can learn the output more precisely, which is projected back into the recurrent network. Hence, using heterogeneous STDP parameters leads to better orthogonalization among the neuronal states and hence, a higher C.
In this paper, we show that using a distribution of LTPALD dynamics in the STDP parameters helps us in mappings the input onto the orthogonal activations of the recurrent network to capture the principal components of the input signal. The LTD dynamics play an important role in determining the orthogonality of neuronal activations. LTD windows of the STDP rules enable robust sequence learning amid background noise in cooperation with a large signal transmission delay between neurons and a theta rhythm (Hayashi & Igarashi] 2009). The LTD window in the range of positive spike-timing plays important role in preventing noise influences with sequence learning. Oja (Oja 1982 1989) showed that the LIF neuron's 's time constant is very fast compared to the time constant of learning in which the weights wys change. The learning is assumed to take place according to the STDP type conjunction of the inputs & and the integrated effect of the inputs, 14,, with an
additional forgetting term attributed the LTD dynamics: de avjξ, - f (1y, E,, wyx) In the case of homogencous STDP, f()) is a constant; hence, the model can only efficiently learn the first principal component of the input. However, quite interesting functions emerge when considering STDP to have a distribution. This also helps us determine the next principal components other than the first one. Hence the diversity in the different LTD dynamics increases the competition and helps that not all inputs are mapped the first principle component. Thus, the diversity the LTD dynamics helps in projecting the input to orthogonal activations of the recurrent network.
dwe
Now, for homogeneous RSNNs, several higher-order correlations, which according to our hypothesis, arise because of the poor orthogonalization among the network states. This results in the redundancies of spikes for encoding the same information. In this paper, we use heterogeneous STDP dynamics to learn an efficient orthogonal representation of the state space, which result in the network learning the same patterns but using fewer spikes. (theorem: 3) We also show that heterogeneity in the neuronal parameters decreases the neuronal correlation (theorem And fig 2a). Thus, since heterogeneity results in better orthogonalization among the neuronal states, it results in fewer higher- order cor- relations. Moreover, recent studies have shown that the correlation of higher order progressively
35

---

Published as a conference paper at ICLR 2023
Table 11: Table Showing the estimated highest order of correlation for HRSNN vs. MRSNN using CuBIC
Estimated Highest-
p-value
order of correlation
C L)
HRSNN
0.423
MRSNN
5
0,358
6
decreases the information available through the neural population, and the decrease in information becomes larger as the interaction order grows. Since we are trying to engineer an efficient model, we aim to reduce the higher-order correlations using heterogeneity in neuronal parameters (as shown in Theorem 1). In addition to this, to verify this, we used CuBIC (Staude et al., 2010), a cumulant- based inference of higher-order correlations in massively parallel spike trains. The details of the experimental methodology are given in Supplementary Section C. The outcome of CuBIC is a lower bound & on the order of correlation in the spiking activity of large groups of simultaneously recorded neurons. CuBIC can provide statistical evidence for large correlated groups without the discouraging requirements on sample size that direct tests for higher-order correlations have to meet. This is achieved by exploiting constraining relations among correlations of different orders. However, must be noted that CuBIC is not designed to estimate the order of correlation directly; the inferred lower bound might not always correspond to the maximal order of correlation present in given data set.
36


Reviews:
[{"review_id": "sY6YETFzkH", "reviewer": "Reviewer_Rdb2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes an endeavor to enhance the memory capacity $\\mathcal{C}$ with heterogeneity in neuronal dynamics, which refers to different membrane time constants. The authors also show heterogeneous STDP, which means using a probabilistic synaptic time constant, helps to suppress the firing rate while the memory capacity is maintained. The results are both analytical and empirical.", "summary_of_the_review": "Many contents are confusing, and need to be clarified.", "strength_and_weaknesses": "**Pros**\n\nAlthough using a heterogeneous time constant is not novel in this area, the elaboration upon its effect on the memory capacity of SNNs is clearly novel to me.\n\n**Cons**\n\n1. There are some things that need to be clarified or properly placed notations.\n2. The statement in Lemma 3.2.1, that is, heterogeneous dynamics reduce the variance of membrane voltages and thus reduce the rate of spikes, seems to omit the discussion of the mean of membrane voltages.\n\n**Questions**\n\n1. The notations are a bit confusing even when the authors have made a table of notations. Here is a non-exhaustive lists\n   -  ${\\bf{w}}^{\\rm{in}}$ first appears in the statement of Lemma 3.1.1, but is first defined in short proof.\n   -  In Eq.2, the authors choose $[\\cdot]$ to express discretized variable and distinguish it from continuous one $(\\cdot)$. However, in Eq.4 and many following equations with integer $\\tau$, they still use $(\\cdot)$.\n   - When using $\\Delta$, only the meanings of $\\Delta w$ and $\\Delta t$ are clearly given. Notations like $\\Delta\\text{Var}$ in Lemma 3.2.1 are not defined and can be confusing.\n2. The statement in Lemma 3.2.1 that a lower variance of membrane voltage distribution leads to a lower firing rate is not sufficient. A zero variance (constant) voltage but higher than the threshold leads to persistent firing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "kcLrbtnmPm", "reviewer": "Reviewer_LwLn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission studies discrete-time recurrent spiking neural networks of LIF neurons with STDP both analytically and numerically.  The main analytical finding is that heterogeneity of neuron parameters increases the memory capacity of the recurrent spiking neural network. This finding is based on the fact that with increasing heterogeneity of neuron parameters (threshold and membrane time constant), the covariance is decreasing ints covariance. A lower covariance results in a higher capacity. The numerical analysis confirms that. Moreover, a SAGE-attribution analysis reveals that the main contribution to the improved performance comes from heterogeneous membrane time constants.", "summary_of_the_review": "The submission studies the role of the heterogeneity of membrane time constants and threshold heterogeneity on the capacity of recurrent spiking neural networks.\nThe two main weaknesses of the submission are a lack of biological plausibility and a lack of discussion of the scope and rigor of the \"analytical\" theory (which seems to be a slight extension of  (Aceituno 2020), i.e (including discussion of the implicit assumptions and when it would break down) and details (and code) that would make the findings reproducible.\n\nOverall, I cannot recommend the manuscript in the current form for publication.", "strength_and_weaknesses": "Strength:\n* It is a rare case of an analytical study of the computational capabilities of a recurrent spiking neural network.\n\nWeaknesses:\n* The main finding is that heterogeneity of membrane time constants increases the capacity, however, in biological neurons, the membrane time constant is not a free parameter but is tightly controlled by the resistance across the membrane and the capacitance of the membrane. It is thus not biologically plausible that the brain has the freedom to choose a heterogeneous distribution of membrane time constants. A more plausible candidate for that would be the heterogeneity of synaptic time constants.\n* A direct comparison of analytical results and numerical simulations is missing. While figure 3 shows a similar trend as expected from theory, it would be desirable to actually directly compare them. \n* table 2: it would be better to measure the firing rates of the neurons in 1/s or 1/tau_m and not as 'spike count'.\n* It would be nice to relate this work to other work with recurrent networks of LIF neurons, e.g. Brunel 2000, Jahnke/Memmesheimer/Timme 2009, Deneve/Machens 2016. \n* The theoretical part seems to rely on some kind of mean-field assumption that allows to express the covariance of the activity in terms of linearized dynamics and a stationarity assumption. As LIF networks can have a very diverse set of states (e.g. oscillations, synchronization, splay-states, asynchronous irregular states, etc. see e.g. Brunel 2000), it is not clear how this assumption is justified here. Also, no direct numerical tests of the analytical results are being presented, which makes it hard to evaluate them. \n* Details the Bayesian Optimization are missing, the numerical experiments are not fully reproducible in the current version.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kTd6D24S4J5", "reviewer": "Reviewer_rFWy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed the spike-efficiency measurement for spiking neural networks and showed the relationship between spike-efficiency and the heterogeneity. They also performed some empirical experiments to demonstrate the significance of neuronal and synaptic heterogeneity in reducing spiking activity. ", "summary_of_the_review": "Overall, this is a paper that proposes an interesting question of measuring the performance of RSNN. However, the discussion remains a bit superficial.  ", "strength_and_weaknesses": "Strength:\n1. The manuscript is clearly written with sufficient details. \n2. The idea of quantifying the capacity and heterogeneity is interesting. \n\nWeakness:\nAlthough this topic appears to be interesting, there seem to lack sufficient evidence to support the claim. Specifically, I have the following concerns. \n1. The capacity can be intuitively understood as a measurement of how much variance in the past are still maintained in the current state. However, what's the relationship between this capacity and the model performance or generalization ability? As is known for the deep learning setup, overfit on the training dataset is a typical phenomenon. Thus increasing only the capacity is not enough for such cases. \n\n2. The definition of spike-efficiency is a bit random especially the factor 1000. Is there any reasoning on why we should define it as a ratio? Since both the memory capacity and spiking rate are unit-free variables, their relationship can be composed in multiple ways. Thus it would make more sense if this definition is derived under a certain criterion rather than simply claimed. \n\n3. Intuitively one may expect that if we improve the heterogeneity of parameters, it would lead to more complex models thus higher ability in memorizing certain modes. For the results in Table 2, soly increasing heterogeneity of LIF does not necessarily improve the efficiency or not. How should we understand such difference with STDP? Specially for Lemma 3.2.1, the $\\Delta Var$ is not defined. Also, it seems that the proof here is a directive corralory from the heavy-tail claim, I am not 100% sure about the reliability here. \n\n4. The experiments are on fairly simple cases. To validate the capacity and the assumption, experiments on some more complex static/dynamic dataset may be included. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j4VY44AWcl", "reviewer": "Reviewer_MSRY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the role of neural and synaptic heterogeneity in improving the computational capacity and/or energy efficiency of spiking recurrent neural nets. It also provides an optimization procedure for determining the optimal degree of heterogeneity for a problem.", "summary_of_the_review": "Overall, a solid contribution with clear problem framing, theoretical and numerical support and robust numerical results. ", "strength_and_weaknesses": "Strengths:\n+ biological motivation\n+ what in hardware may be traditionally deemed as a bug becomes a computational feature of the system\n+ rigorous analysis of implications\n+ numerics are solid\n\nWeakness:\n- capacity measure somewhat restricting as a measure of computational power\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster