Title: Evolving Connectivity for Recurrent Spiking Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: neuromorphic computing

spiking neural networks

evolutionary algorithms

inference-only approach

hardware-friendly

robotic locomotion tasks
Abstract: Recurrent spiking neural networks (RSNNs) hold great potential for advancing artificial general intelligence, as they draw inspiration from the biological nervous system and show promise in modeling complex dynamics.
However, the widely-used surrogate gradient-based training methods for RSNNs are inherently inaccurate and unfriendly to neuromorphic hardware.
To address these limitations, we propose the evolving connectivity (EC) framework, an inference-only method for training RSNNs.
The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions.
Our EC framework circumvents the need for gradients and features hardware-friendly characteristics, including sparse boolean connections and high scalability.
We evaluate EC on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs, even solving the complex 17-DoF humanoid task.
Additionally, the EC framework demonstrates a two to three fold speedup in efficiency compared to directly evolving parameters.
By providing a performant and hardware-friendly alternative, the EC framework lays the groundwork for further energy-efficient applications of RSNNs and advances the development of neuromorphic devices.
Our code is publicly available at https://github.com/imoneoi/EvolvingConnectivity.

Evolving Connectivity for Recurrent Spiking Neural Networks

Abstract

Recurrent spiking neural networks (RSNNs) hold great potential for advancing artificial general intelligence, as they draw inspiration from the biological nervous system and show promise in modeling complex dynamics. However, the widely-used surrogate gradient-based training methods for RSNNs are inherently inaccurate and unfriendly to neuromorphic hardware. To address these limitations, we propose the evolving connectivity (EC) framework, an inference-only method for training RSNNs. The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions. Our EC framework circumvents the need for gradients and features hardware-friendly characteristics, including sparse boolean connections and high scalability. We evaluate EC on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs, even solving the complex 17-DoF humanoid task. Additionally, the EC framework demonstrates a two to three fold speedup in efficiency compared to directly evolving parameters. By providing a performant and hardware-friendly alternative, the EC framework lays the groundwork for further energy-efficient applications of RSNNs and advances the development of neuromorphic devices. Our code is publicly available at https://github.com/imoneoi/EvolvingConnectivity .

1 Introduction

Inspired by the remarkable information processing power of the human brain, neuromorphic computing aims to design and implement computational systems that mimic the architecture and functionality of biological neural networks, paving the way for a new era of intelligent machines. Specifically, the highly recurrently connected and spike-emitting brain network inspired recurrent spiking neural networks (RSNNs), which employ discrete, spike-based signals for transmitting information through recurrent connections in an event-driven manner. RSNNs can serve as realistic models of the brain incorporating the latest anatomical and neuro-physiological data [Billeh et al., 2020], leading to the discovery of general principles of computing, e.g., noise robustness [Chen et al., 2022]. Furthermore, RSNNs can exhibit self-organized critical properties [Poil et al., 2012] and serve as a reservoir for complex temporal dynamics [Jaeger, 2001, Maass et al., 2002], making them suitable for sequential tasks like robotics [Lele et al., 2020], path-finding [Rueckert et al., 2016], and others.

Despite the importance of RSNNs, developing end-to-end training algorithms remains a challenge. Inspired by the success of deep learning, a line of research [Wu et al., 2018, Shrestha and Orchard, 2018, Bauer et al., 2022] uses error-backpropagation [Rumelhart et al., 1985] with carefully chosen surrogate gradients to address the non-differentiability problem and achieve performance comparable to deep learning. However, incorporating surrogate gradients into RSNNs introduces two concerns. Algorithmically, the surrogate gradient leads to inherent inaccuracy in the descent direction [Li et al., 2021] and sensitivity to function scale selection [Zenke and Vogels, 2021]. At the implementation level, gradient-based training is incompatible with prominent neuromorphic devices [Pei et al., 2019, Davies et al., 2018, Mayr et al., 2019] due to the requirement of accessing the full network state over every timestep [Werbos, 1990]. Consequently, this raises a critical research question: can we design a training method for RSNNs that bypasses the need for gradients without compromising performance?

In response to this challenge, we observe that connection probability distribution information from brain connection maps across various species exhibit similarities [Haber et al., 2023] and can be used to construct large-scale neural computing models [Billeh et al., 2020, Schmidt et al., 2018]. Existing studies demonstrate that networks with binary connections alone can achieve high performance, rivaling that of weighted networks [Gaier and Ha, 2019, Frankle and Carbin, 2018, Malach et al., 2020]. Moreover, computing boolean connections relies primarily on integer arithmetic rather than resource-intensive floating-point operations, enabling simpler on-chip implementation and enhanced energy efficiency. Considering these features, we reformulate the architecture of RSNNs, where connections are sampled independently from parametric Bernoulli distributions, and adopt Natural Evolution Strategies (NES) [Wierstra et al., 2014] to optimize this parametric probability distribution, providing a scalable, inference-only, and effective algorithm for tuning parameters.

In this paper, we introduce the evolving connectivity (EC) framework for training RSNNs. The EC framework reformulates RSNNs as boolean connections with homogeneous weights and employs NES to evolve the connection probabilities of the network. To evaluate effectiveness and efficiency, we conduct extensive experiments on locomotion tasks, a set of widely-used sequential decisionmaking tasks [Brockman et al., 2016, Freeman et al., 2021]. Experimental results show that our proposed EC method achieves performance comparable to deep recurrent neural networks, surpassing the asymptotic performance of surrogate gradients. Despite using a GPU, which is not specifically optimized for our framework, the EC method yields a speed improvement of 2 ∼ 3 × compared to directly evolving parameters using Evolution Strategies [Salimans et al., 2017] and exhibits better efficiency than surrogate gradients. Additionally, our EC method can effectively train deep recurrent neural networks (RNNs), demonstrating versatility for different architectures and potential for quantized deep neural networks.

Our main contributions are summarized as follows:

Novel framework : We propose a novel inference-only training framework for RSNNs by reformulating weight-tuning as connection probability searching through evolution-based algorithms.

High performance : Our method can solve the complex 17-DoF humanoid locomotion task with RSNN, achieving performance on par with recurrent neural networks and outperforming gradient-trained RSNNs.

Hardware-friendly : By producing RSNNs with sparse 1-bit boolean connections, as well as enabling inference-only training and high scalability, our method is highly compatible with neuromorphic devices. This compatibility holds promising potential for further energy-efficient applications of RSNNs.

2 Related Works

2.1 Training Recurrent Spiking Neural Networks

The study of training algorithms for RSNNs endeavors to unravel the mechanism that enables the brain to learn. For decades, biological plasticity rules, especially spike-timing-dependent plasticity (STDP) [Bi and ming Poo, 1998], have been considered as a foundational of RSNNs training [Diehl and Cook, 2015, Kheradpisheh et al., 2018, Mozafari et al., 2019]. Recently, with the success of deep learning, surrogate gradient-based approaches [Wu et al., 2018, Shrestha and Orchard, 2018, Bauer et al., 2022] have been predominately utilized in training RSNNs. Surrogate gradients share conceptual similarities with the straight-through estimator [Bengio et al., 2013] used in deep learning.

Both techniques employ continuous surrogate functions to compute gradients of non-differentiable functions during the backward pass, while preserving the original non-differentiable functions in the forward pass. Slayer [Shrestha and Orchard, 2018] adopted an exponential decaying surrogate function with a spike response model (SRM), while STBP [Wu et al., 2018] incorporated BPTT in multi-layer SNN training and proposed several surrogate function choices. Chen et al. [2022] trained an anatomical and neurophysiological data constrained RSNN and demonstrated its robustness and versatility. However, gradient-based approaches face the new problem of being difficult to implement on neuromorphic devices. To address the limitation, E-prop [Bellec et al., 2020] utilize eligibility trace to compute truncated gradient, becoming an exemplary training algorithm on neuromorphic devices like Loihi2 [Davies et al., 2018] and SpiNNaker2 [Mayr et al., 2019], but have a strict restriction on the temporal relationship of SNN models and still lags behind surrogate gradient methods in performance. Our approach tackles the gradient-effectiveness dilemma from the beginning, by introducing the inference-only training framework.

2.2 Weight-agnostic Neural Networks

Training a neural network typically involves assigning appropriate values to the network's weights. However, the weight-agnostic neural network (WANN) [Gaier and Ha, 2019] has demonstrated that a network's topology can be highly informative, similar to that of a weighted neural network. Additionally, research on the lottery ticket hypothesis (LTH) [Frankle and Carbin, 2018, Zhou et al., 2019, Ramanujan et al., 2020] suggests that an over-parameterized network contains an effective subnetwork, even when using its initial parameter weights. Several theoretical studies have also shown that a sufficiently large, randomly-weighted deep neural network contains a subnetwork capable of approximating any function [Malach et al., 2020, Fischer et al., 2022]. Our approach, furthering the existing research on connection-encoded networks, proposes a framework that utilizes connection probability to parameterize the network.

2.3 Deep Neuroevolution

Deep neuroevolution utilizes evolutionary algorithms for training deep neural networks. Natural Evolution Strategies (NES) [Wierstra et al., 2014] have laid the foundation for gradient estimation in this domain. A well-known variant, Evolution Strategies (ES) [Salimans et al., 2017], has been employed for training deep neural networks in sequential decision-making tasks. ES optimizes a single set of continuous network weights by applying Gaussian perturbations and updating the weights using the NES gradient estimator. Following the success of ES, numerous evolutionary algorithms have been proposed for training deep neural networks. For example, Such et al. [2018] showed that deep neural networks could be effectively trained using simple genetic algorithms that mutate weights. Furthermore, Conti et al. [2018] integrated exploration and novelty-seeking into ES to overcome local minima. While the majority of prior research has primarily focused on continuous parameters, our proposed framework presents a novel approach by concentrating on the search for connection probability distributions. This shift in perspective offers new possibilities for evolving recurrent spiking neural networks in a hardware-friendly manner.

3 Preliminaries: Recurrent Spiking Neural Networks

RSNN is a class of spiking neural networks which incorporate feedback connections. In this paper, we adopt a typical RSNN architecture from the reservoir network [Jaeger, 2001, Maass et al., 2002] for sequential tasks. It is worth noting that, despite that we adopt a specific RSNN model as an example, our framework can be broadly applied to search for connectivity distributions in any type of RSNN, as it does not depend on network-specific assumptions and only relies on evaluating the network with a set of parameters.

According to Dale's law, our network consists of an excitatory neuron group and an inhibitory neuron group. Each neuron is modeled as a leaky integrate and fire (LIF) neuron. A neuron will fire a spike when its membrane potential u exceeds the threshold, and the membrane potential will hard-reset to 0 . More specifically, our model defines the dynamics of membrane potential u and synaptic current c as follows:

Population

Global Distribution

B(p)

RSNN

RSNNSimulation

Environment

Current-based

Indhno

0

Input

Integrator

1

0

0

1

excitatory

Action

Observation

0

1

0

1

1

Action

0

1

>

1

1

1

0

Observation

Rk

0

1

0

0

inhibitory

1-bit

LIF

Physics

Firing Calculation

Simulation

where g = { Exc,Inh } denotes the excitatory and inhibitory group, respectively, and R denotes the resistance.

The current input was modeled using exponential synapses to retain information for a short duration, commonly adopted in robotic tasks [Tang et al., 2020, Naya et al., 2021].

where t s ( g j ) j denotes the spike time of neuron j in neuron group g j , δ denotes Dirac delta function. I g was set to define the connection strength of excitatory and inhibitory synapse respectively, where I Exc > 0 and I Inh < 0 . Weight W ( g i g j ) was defined as a matrix with non-negative elements, connecting group g j to group g i . Besides, the external input signal I ext is extracted using linear projection of observation x .

Discretizing the LIF differential equations (Eq. 1 and 2) with ∆ t as a time-step, we could obtain the following difference equation for our RSNN model:

where d c = e -∆ t τsyn and d v = e -∆ t τm are two constant parameters.

The output vector o is extracted using linear projection of neuron firing rates in a short time period τ , i.e.,

where W ( g ) out denotes the output weight, and k is a function averaging the time period τ .

4 Framework

In this section, we present our proposed Evolving Connectivity (EC) framework, which is depicted in Fig. 1. Our approach consists of three main steps: (1) reformulating the neural network architecture from weight-based parameterization to connection probability distribution, (2) employing the Natural Evolution Strategies (NES) method to optimize the reformulated parameter space, and (3) deterministically extracting the final parameters from the distribution.

Reformulation. A sparse connected weight matrix can be described into a weight matrix w and a connection mask θ . Traditionally, we can use Erd˝ os-Rényi random matrix to describe the connection

mask, in which connections are independently drawn from a Bernoulli distribution, i.e.,

Inspired by the success of subnetworks in deep neural networks [Ramanujan et al., 2020], we can reformulate this in a connection framework. In this view, we aim to find a connection probability matrix, ρ = ( ρ ij ) , where each element represents the connection probability between two neurons, and set all weights w ij to be unit size, i.e.,

Optimization. We optimize ρ to maximize the expected performance metric function R ( · ) across individual network samples drawn from the distribution, which can be expressed as the following objective function:

To optimize this objective, we employ Natural Evolution Strategies (NES) [Wierstra et al., 2014]. NES provides an unbiased Monte Carlo gradient estimation of the objective J ( ρ ) , by evaluating the performance metric R k on multiple individual samples θ k drawn from B ( ρ ) . Specifically, NES estimates the gradient of J ( ρ ) with respect to the parameters of the distribution ρ by computing the expectation over samples from B ( ρ ) :

Thus, we obtained an inference-only estimation of the gradient by simply sampling and evaluating the metric. Then gradient descent can be carried over the estimated gradients, following the NES approach [Wierstra et al., 2014]. Furthermore, as in Williams [1992], we scale the step size proportionally to the variance, i.e. α = η · Var[ B ( ρ )] , and obtain the update rule:

In addition, all elements of ρ clipped in the interval [ ϵ, 1 -ϵ ] , where ϵ → 0 + , to guarantee a minimal level of exploration within the search space. In our experiment, we set ϵ = 0 . 001 and initialize ρ = 0 . 5 . Additionally, we adopted a similar fitness shaping trick, center rank transform, as discussed in Salimans et al. [2017], which sort the return of a population, then linearly rescale the sorting indexes into a fixed interval [ -0 . 5 , 0 . 5] .

Once a sufficient number of updates have been performed, the final parameter θ can be deterministically obtained as the parameter with maximum probability, which is subsequently used for deployment. In the context of a Bernoulli distribution, this process is equivalent to thresholding ρ at 0 . 5 to produce a boolean matrix containing only { 0 , 1 } values.

5 Properties of EC Framework

In this section, we further discuss the important properties of our proposed EC framework in implementation, thanks to leveraging the connection probability distribution.

Inference only. The most significant challenge in neuromorphic devices is the absence of an effective hardware-friendly learning algorithm [Li et al., 2023]. Most neuromorphic chips, such as Loihi [Davies et al., 2018], Tianjic [Pei et al., 2019], TrueNorth [Akopyan et al., 2015], SpiNNaker2 [Mayr et al., 2019], lack support for directly calculating error backpropagation, which is crucial for surrogate gradient methods. The inference-only EC framework enables an alternative method for training on these inference chips.

1-bit connections. EC employs 1-bit sparse connections throughout training and deployment, replacing the traditional floating-point weight matrix. Therefore, the 1-bit connections permit the use of more economical integer arithmetic instead of costly floating-point cores. This approach not only accelerates computations on devices like GPUs, but also holds a promise of driving the creation of novel 1-bit connection neuromorphic computing hardware.

Scalability. The inference-only property of the EC framework implies no data dependence between evaluations, which allows for the distribution of sampled parameters θ k across independent workers and the collection of their reported performance measures, making EC highly scalable. Moreover, the random seed for sampling the population can be transmitted across nodes instead of the connections θ k to minimize communication overhead, facilitating scalar-only communication.

6 Experiments

6.1 Experimental Setups

Tasks. We focus on three robotic locomotion tasks in our experiments, Humanoid, Walker2d, and Hopper, as they are commonly used for sequential decision-making problems in the reinforcement learning domain [Brockman et al., 2016, Freeman et al., 2021]. As illustrated in Fig.

2, these tasks involve controlling robots with varying degrees of freedom (DoF), to perform certain actions that maximize the return within a fixed-length episode T . In the EC framework, the performance metric function can be defined as the expected return over episodes R ( θ ) = E τ ∼ π θ [ ∑ T t =0 r t ] , and a single evaluation R i corresponds to the return of one episode using network parameter θ k .

RSNN architecture. In our study, we employed a RSNN consisting of an input layer, a hidden recurrent layer with LIF neurons, and an output layer, as delineated in the Preliminaries section. To achieve an equilibrium within the excitatory inhibitory balance, input and hidden layers were evenly divided into excitatory and inhibitory group.

Baselines. To thoroughly evaluate the effectiveness of our framework and its advantages over prior methods, we compare our EC framework with deep RNNs, as well as RSNN trained with Surrogate Gradients (SG) and Evolution Strategies (ES). Detailed network architecture and precision is illustrated in Table 1.

For deep RNNs, we employ widely-used recurrent deep neural networks, specifically long-short term memory (LSTM) [Hochreiter and Schmidhuber, 1997] and gated recurrent unit (GRU) [Cho et al., 2014], trained using Evolution Strategies (ES) [Salimans et al., 2017] as our baselines.

For RSNN, we employ the same structured but excitatory/inhibitory not separated RSNN trained with ES and Surrogate Gradient (SG). ES is directly applied to search the weight matrix W of RSNNs, optimizing the performance metric in a gradient-free manner. In contrast, the Surrogate Gradient is combined with Proximal Policy Optimization (PPO) [Schulman et al., 2017], a prominent reinforcement learning method, to optimize the weight W for sequential decision-making tasks where returns cannot be differentiated.

Implementation Details. Our EC framework and all baselines are implemented using the JAX library [Bradbury et al., 2018] and just-in-time compiled with the Brax physics simulator [Freeman et al., 2021] for efficient GPU execution. The population is vectorized automatically, and 1-bit firing

0

200

400

600

800

1000

0

2000

4000

6000

8000

10000

12000

14000

16000

Humanoid (17-DoF)

EC-RSNN (Ours)

ES-RSNN

SG-RSNN

ES-GRU

ES-LSTM

0

200

400

600

800

1000

0

1000

2000

3000

4000

5000

6000

Walker2d (6-DoF)

0

200

400

600

800

1000

0

500

1000

1500

2000

2500

3000

Hopper (3-DoF)

# Generations

Return

calculations take advantage of INT8 arithmetic for performance optimization. As a result, the training process achieves over 180 , 000 frames per second on a single NVIDIA TITAN RTX GPU. Each experiment's result is averaged over 3 independent seeds, with the standard deviation displayed as a shaded area. For detailed information on hyperparameters and hardware specifications, please refer to Appendix G.

6.2 Performance Evaluation

Firstly, we compare RSNN trained with our EC framework (EC-RSNN) to deep RNNs. To ensure a fair comparison, we utilized the evolution-based training approach ES [Salimans et al., 2017] for deep RNNs, including ES-GRU and ES-LSTM. Typically, SNNs face challenges in surpassing the performance of their DNN counterparts. However, as shown in the experimental results in Fig. 3, our EC-RSNN achieves competitive performance and training efficiency compared to deep RNNs, even surpassing ES-GRU and ES-LSTM on the Walker2d and Hopper tasks. In the complex 17-DoF Humanoid task, our EC-RSNN could also outperform ES-GRU. Moreover, EC-RSNN demonstrates comparable performance to GRU and LSTM trained with PPO, as detailed in Appendix E. It is worth noting that, while GRU and LSTM use densely connected, full precision (32-bit floating-point) weights and hard-coded gating mechanisms, EC-RSNN is constrained on 1-bit sparse weights and adheres to Dale's law for weight signs.

Then, as for the same structured RSNNs, our EC-RSNN outperforms ES-RSNN, especially significantly surpassing ES-RSNN on complex Humanoid and Walker2d tasks. We highlight two aspects that potentially account for the superior performance of our EC framework. On the one hand, schema theory [Whitley, 1994] suggests that selecting over a population of binary strings could combine partial binary string patterns (schemata) to implicitly work on many solutions without evaluating them explicitly, leading to massive parallelism. EC leverages the discrete { 0 , 1 } -string space, which may enable more efficient optimization due to implicit parallelism. On the other hand, ES uses fixed-scale noise to perturb a set of deterministic parameters, which may often fall in wide areas in the landscape of the objective function and fail to proceed on sharp maxima [Lehman et al., 2018]. In contrast, EC operates over a probabilistic distribution, providing the flexibility to adjust variance on sensitive parameters and achieving more fine-grained optimization.

Finally, gradient updates cannot be directly converted into evolution generations, thus we only compare the final converged return for surrogate gradient trained RSNNs. Considering that the surrogate gradient approach is sensitive to the selection of the surrogate function and its parameters [Zenke and Vogels, 2021], we choose three commonly-used surrogate functions with different β and γ parameters to thoroughly validate the performance: SuperSpike [Zenke and Ganguli, 2018], the piecewise linear function proposed by Esser et al. [2016], and the derivative of the Sigmoid function [Zenke and Ganguli, 2018]. As shown in Fig. 4, the performance of our EC-RSNNs is consistently higher than SG-RSNNs across all surrogate functions and hyper-parameters on the complex 17-DoF Humanoid task. We adopt the best parameter set of the piecewise linear surrogate function, with γ = 0 . 8 , β = 2 as the baseline in Fig. 3.

0

50000

100000

150000

200000

250000

0

2000

4000

6000

8000

10000

12000

14000

SuperSpike:

h

(

x

) =

γ

1

(

β

|

x

|

+1)

2

EC-RSNN (Ours)

γ

= 0

.

8

, β

= 5

γ

= 0

.

8

, β

= 10

γ

= 0

.

8

, β

= 20

0

50000

100000

150000

200000

250000

Esser et al.:

h

(

x

) =

γ

,

1

-

β

|

x

|

)

γ

= 0

.

8

, β

= 2

γ

= 0

.

8

, β

= 5

γ

= 0

.

8

, β

= 10

0

50000

100000

150000

200000

250000

σ

′

:

h

(

x

) =

γ

βe

βx

(

e

βx

+1)

2

γ

= 0

.

8

, β

= 1

γ

= 0

.

8

, β

= 2

γ

= 0

.

8

, β

= 5

Humanoid (17-DoF)

# Gradient steps

Return

One possible reason to explain the performance of SG-RSNN is that due to RSNNs having both explicit and implicit recurrence, the choice of surrogate function may determine whether gradients vanish or explode [Zenke and Vogels, 2021]. Moreover, the gradient approximation of SG is inaccurate and may deviate from the steepest descent direction [Li et al., 2021]. In contrast, our EC framework leverages unbiased Monte-Carlo gradient estimation, which approaches the optimal direction given that the population is sufficiently large [Zhang et al., 2017].

6.3 Efficiency Comparison

In this section, we compare the computational efficiency of our proposed EC framework with other baselines, as shown in Fig. 5. To ensure a fair comparison, we evaluate the EC and baseline methods using the same wall-clock computation time and implement them on identical hardware.

As expected, EC-RSNN exhibits slower training than deep RNNs due to the higher complexity of RSNNs, which necessitates simulating multiple timesteps to compute firing and integrate differential equations. Additionally, the deep RNN models employed in this study, specifically LSTM and GRU, are well-established and highly optimized for GPU implementation. Nevertheless, within the same computation time, the performance achieved by EC-RSNN remains competitive with deep RNNs.

Furthermore, we execute both ES and EC for 1,000 generations using the same population size and RSNN architecture. Experimental results in Fig. 5 have shown that our 1-bit EC-RSNN achieves a speedup of approximately 2 ∼ 3 × over 32-bit floating-point ES-RSNN. This finding emphasizes the value of 1-bit connections within the proposed framework. Utilizing integer arithmetic typically results in higher throughput than floating-point arithmetic across most accelerators, and incorporating smaller data types reduces both memory requirements and memory access time. It is also worth mentioning that the 1-bit connections are implemented using INT8 on GPU, which is not fully optimized. By implementing the framework on hardware supporting smaller data types, connection size could be further reduced by up to 8 × , enabling more significant acceleration and cost reduction.

Additionally, our EC-RSNN demonstrates faster convergence compared to SG-RSNN. This can be attributed to the fact that EC-RSNN is an inference-only method with 1-bit connections, requiring

0

10

20

30

40

0

2000

4000

6000

8000

10000

12000

14000

Humanoid (17-DoF)

EC-RSNN (Ours)

ES-RSNN

SG-RSNN

ES-GRU

ES-LSTM

0

10

20

30

40

0

1000

2000

3000

4000

5000

6000

Walker2d (6-DoF)

0

10

20

30

40

0

500

1000

1500

2000

2500

3000

Hopper-v5 (3-DoF)

Wall-clock time (hours)

Return

only a single forward pass using integer arithmetic, while SG-RSNN demands both a forward and a backward pass employing computationally-intensive floating-point operations.

Finally, we calculated the estimated energy consumption of the EC-RSNN on both GPU and neuromorphic devices, aiming to assess the power efficiency of the EC to train RSNN. In our study, all experiments were consistently conducted on the same NVIDIA Titan RTX GPU, operating at a stable 100% GPU power (280W). Consequently, the energy consumption of the EC training process on GPU is approximated to be directly proportional to the computation wall time, resulting in a 9 MJ. For neuromorphic device, we have conducted computations pertaining to the estimated energy consumption of EC-RSNN when implemented on the Loihi chip [Davies et al., 2018]. As detailed in Appendix D, the estimated energy consumption of EC is 28 kJ, which indicate a noticeable reduction in power consumption by more than two orders of magnitude.

6.4 General 1-bit Framework

The task of discretizing a neural network into a 1-bit representation presents a considerable challenge, particularly when employing continuous training techniques such as ES and SGD. To illustrate this difficulty, we attempted to discretize both ES and SG for training 1-bit connection RSNNs utilizing the straightthrough estimator [Bengio et al., 2013]. Fig. 7 (a) showed that while both ES-RSNN (1bit) and SG-RSNN (1bit) exhibited learning progress, they were surpassed by EC-RSNN (1bit). Fig. 7 (b) and (c) suggest that ES and SG demonstrated superior performance on continuous FP32 weights as opposed to discrete 1-bit representations. These findings imply that continuous optimization methods excel when applied to continuous parameters; conversely, for 1-bit discrete connections, EC is recommended

0

200

400

600

800

1000

0

2000

4000

6000

8000

10000

12000

14000

Humanoid (17-DoF), RNN

EC-RNN (1bit)

ES-RNN (FP32)

PPO-RNN (FP32)

# Generations

Return

due to its specific design for discrete 1-bit optimization and the provision of unbiased gradients. A comprehensive description of the discretization for ES and SG methods is provided in Appendix C.

Moreover, Despite EC have demonstrated the efficacy on RSNNs, it serves as a versatile 1-bit training framework to non-spiking networks as well. To substantiate EC's potential in training deep RNN, we conducted a series of experiments employing a standard RNN model trained with EC, while using ES and PPO as baselines. The RNN has 256 tanh units in the hidden layer. For 1-bit EC, the weight magnitudes are 0-1 connection matrix, and the weight signs are separated to excitatory or inhibitory. The results in Fig. 6 demonstrate that EC can effectively train 1-bit deep recurrent neural networks and has the potential for different architectures and quantized neural networks.

7 Conclusion

In this study, we present the innovative Evolving Connectivity (EC) framework, a unique inferenceonly approach for training RSNN with 1-bit connections. The key attributes of the EC framework,

0

200

400

600

800

1000

# Generations

0

2000

4000

6000

8000

10000

12000

14000

(a)

EC-RSNN (1bit)

ES-RSNN (1bit)

SG-RSNN (1bit)

0

200

400

600

800

1000

# Generations

(b)

ES-RSNN (1bit)

ES-RSNN (FP32)

0

50000

100000

150000

200000

250000

# Gradient steps

(c)

SG-RSNN (1bit)

SG-RSNN (FP32)

Humanoid (17-DoF), RSNN

Return

such as its inference-only nature and scalability, render it particularly suitable for training RSNNs on neuromorphic devices. The use of 1-bit connections significantly reduces memory requirements and computational cost, facilitating faster training on GPUs and paving the way for additional cost reductions in neuromorphic chip production.

We performed extensive experiments on a variety of intricate locomotion tasks, showcasing the competitive performance of our proposed EC framework in comparison to deep RNNs like LSTM and GRU. Moreover, the EC framework outperforms other RSNN training methods such as Evolution Strategies and Surrogate Gradient, both in terms of performance and computational efficiency.

8 Limitations

Evolutionary algorithms demand the storage of N distinct parameter sets for population evaluation, leading to a space complexity of O( N | θ | ) . In contrast, gradient-based approaches utilize a single parameter set but necessitate the storage of intermediate results at each timestep, resulting in a space complexity of O( NHS + | θ | ) , where H corresponds to the number of BPTT timesteps, and S represents the size of the intermediate results. This situation gives rise to a trade-off: evolutionary methods offer greater memory efficiency for tasks featuring long time horizons, whereas gradient-based techniques with larger parameter sizes and shorter time horizons require less memory. Recognizing this trade-off is crucial in practical applications. Although our proposed EC framework, as an evolutionary algorithm, retains the same space complexity and trade-off, it can achieve a constant term reduction by storing 1-bit connections.

9 Discussions

Neuromorphic hardware. One primary bottleneck in the development of neuromorphic hardware is the lack of effective on-chip learning algorithms to build applications compared to deep learning approaches. This paper proposes a novel EC framework that circumvents the requirement for gradients and demonstrates efficacy on locomotion tasks, providing a potential solution for solving the on-chip learning challenge. Typically, the demand for computing devices falls into two categories, including cloud and edge. In the cloud, our proposed EC framework supports large-scale learning, while at the edge, it enables energy-efficient applications. Therefore, our framework offers a potential approach to further building neuromorphic applications.

Another practical challenge lies in the trade-off between numeric precision and cost. We suggest that it is possible to drastically reduce connections from floating point to 1-bit, providing a novel design principle for the next generation of neuromorphic hardware. Therefore, our method holds the potential to significantly decrease the manufacturing and energy costs of neuromorphic hardware.

Neuroscience. Our EC framework introduces a novel method for neuroscience research. First, as opposed to toy tasks or simulated signals, which are often studied in previous neuromodeling work, we employed RSNN in a complex, real-world like locomotion task. Additionally, controlling signals in neuroscience like 'GO' and 'NOGO' signals in the basal ganglia can be easily integrated into the task by concatenating them to the environment observation vector, enabling the creation of novel neuroscientific tasks. Our work lays the foundation for further investigation of decision-making processes and motor control.

Moreover, our framework provides a novel type of data to analyze: neuron-to-neuron level connection probability. Connection probability is one of the most fundamental properties in brain-wide connectomes. However, obtaining neuron-to-neuron connection probability from the whole mammalian brain is experimentally implausible due to limitations in nowadays neuroconnectomic technology. Our work provides in silico connection data for further analysis, including covariances, motifs, clustered engrams, and dimensional properties.

Finally, our framework is capable of incorporating neuroanatomical and neurophysiological data to construct a novel neurosimulation model, since our framework is able to train arbitrary models in principle. Neuroscientists have discovered that connection probability between two neurons is determined by various factors, including spatial distance, receptive field, and neuron types. Our framework can leverage these findings and conduct in silico experiments with data-driven models.

References

Akopyan, J. Sawada, A. S. Cassidy, R. Alvarez-Icaza, J. V. Arthur, P. Merolla, N. Imam, Y. Nakamura, P. Datta, G.-J. Nam, B. Taba, M. P. Beakes, B. Brezzo, J. B. Kuang, R. Manohar, W. P. Risk, B. L. Jackson, and D. S. Modha. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 34:1537-1557, 2015.

C. Bauer, G. Lenz, S. Haghighatshoar, and S. Sheik. Exodus: Stable and efficient training of spiking neural networks. arXiv preprint arXiv:2205.10242 , 2022.

Bellec, F. Scherr, A. Subramoney, E. Hajek, D. Salaj, R. Legenstein, and W. Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications , 11(1): 3625, 2020.

Bengio, N. Léonard, and A. C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR , abs/1308.3432, 2013. URL http://arxiv.org/ abs/1308.3432 .

Bi and M. ming Poo. Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. The Journal of Neuroscience , 18:10464 - 10472, 1998.

N. Billeh, B. Cai, S. L. Gratiy, K. Dai, R. Iyer, N. W. Gouwens, R. Abbasi-Asl, X. Jia, J. H. Siegle, S. R. Olsen, et al. Systematic integration of structural and functional data into multi-scale models of mouse primary visual cortex. Neuron , 106(3):388-403, 2020.

Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.

Chen, F. Scherr, and W. Maass. A data-based large-scale model for primary visual cortex enables brain-like robust and versatile visual processing. Science Advances , 8, 2022.

Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014.

Conti, V. Madhavan, F. P. Such, J. Lehman, K. O. Stanley, and J. Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents, 2018.

Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro , 38 (1):82-99, 2018.

U. Diehl and M. Cook. Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Frontiers in computational neuroscience , 9:99, 2015.

Ding, J. Zhang, Z. Yu, and T. Huang. Accelerating training of deep spiking neural networks with parameter initialization, 2022. URL https://openreview.net/forum?id=T8BnDXDTcFZ .

K. Esser, P. Merolla, J. V . Arthur, A. S. Cassidy, R. Appuswamy, A. Andreopoulos, D. J. Berg, J. L. McKinstry, T. Melano, D. Barch, C. di Nolfo, P. Datta, A. Amir, B. Taba, M. Flickner, and D. S. Modha. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences , 113:11441 - 11446, 2016.

Fischer, A. Gadhikar, and R. Burkholz. Lottery tickets with nonzero biases, 2022.

Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github.com/google/ brax .

Gaier and D. Ha. Weight agnostic neural networks. Advances in neural information processing systems , 32, 2019.

Haber, A. Wanner, R. W. Friedrich, and E. Schneidman. The structure and function of neural connectomes are shaped by a small number of design principles. bioRxiv , pages 2023-03, 2023.

Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9:1735-80, 12 1997. doi: 10.1162/neco.1997.9.8.1735.

Jaeger. The'echo state'approach to analysing and training recurrent neural networks. 2001.

R. Kheradpisheh, M. Ganjtabesh, S. J. Thorpe, and T. Masquelier. Stdp-based spiking deep convolutional neural networks for object recognition. Neural Networks , 99:56-67, 2018.

Lehman, J. Chen, J. Clune, and K. O. Stanley. Es is more than just a traditional finite-difference approximator, 2018.

S. Lele, Y. Fang, J. Ting, and A. Raychowdhury. Learning to walk: Spike based reinforcement learning for hexapod robot central pattern generation. 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) , pages 208-212, 2020.

Li, L. Deng, H. Tang, G. Pan, Y. Tian, K. Roy, and W. Maass. Brain Inspired Computing: A Systematic Survey and Future Trends. 1 2023. doi: 10.36227/techrxiv. 21837027.v1. URL https://www.techrxiv.org/articles/preprint/Brain_Inspired_ Computing_A_Systematic_Survey_and_Future_Trends/21837027 .

Li, Y. Guo, S. Zhang, S. Deng, Y. Hai, and S. Gu. Differentiable spike: Rethinking gradientdescent for training spiking neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 23426-23439. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/ paper_files/paper/2021/file/c4ca4238a0b923820dcc509a6f75849b-Paper.pdf .

Maass, T. Natschläger, and H. Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation , 14:2531-2560, 2002.

Malach, G. Yehudai, S. Shalev-Schwartz, and O. Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6682-6691. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/ malach20a.html .

Mayr, S. Hoeppner, and S. B. Furber. Spinnaker 2: A 10 million core processor system for brain simulation and machine learning. ArXiv , abs/1911.02385, 2019.

Mozafari, M. Ganjtabesh, A. Nowzari-Dalini, S. J. Thorpe, and T. Masquelier. Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks. Pattern recognition , 94:87-95, 2019.

Naya, K. Kutsuzawa, D. Owaki, and M. Hayashibe. Spiking neural network discovers energyefficient hexapod motion in deep reinforcement learning. IEEE Access , 9:150345-150354, 2021.

Pei, L. Deng, S. Song, M. Zhao, Y. Zhang, S. Wu, G. Wang, Z. Zou, Z. Wu, W. He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature , 572(7767):106-111, 2019.

S.-S. Poil, R. Hardstone, H. D. Mansvelder, and K. Linkenkaer-Hansen. Critical-state dynamics of avalanches and oscillations jointly emerge from balanced excitation/inhibition in neuronal networks. The Journal of Neuroscience , 32:9817 - 9823, 2012.

Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What's hidden in a randomly weighted neural network?, 2020.

Rueckert, D. Kappel, D. Tanneberg, D. Pecevski, and J. Peters. Recurrent spiking networks solve planning tasks. Scientific reports , 6(1):1-10, 2016.

E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.

Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017.

Schmidt, R. Bakker, K. Shen, G. Bezgin, M. Diesmann, and S. J. van Albada. A multi-scale layer-resolved spiking network model of resting-state dynamics in macaque visual cortical areas. PLOS Computational Biology , 14(10):e1006359, 2018.

Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.

B. Shrestha and G. Orchard. Slayer: Spike layer error reassignment in time. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/ 82f2b308c3b01637c607ce05f52a2fed-Paper.pdf .

P. Such, V. Madhavan, E. Conti, J. Lehman, K. O. Stanley, and J. Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning, 2018.

Tang, N. Kumar, and K. P. Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 6090-6097, 2020. doi: 10.1109/IROS45743.2020.9340948.

J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78(10):1550-1560, 1990.

Whitley. A genetic algorithm tutorial. Statistics and computing , 4:65-85, 1994.

Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research , 15(27):949-980, 2014. URL http://jmlr. org/papers/v15/wierstra14a.html .

J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning , pages 5-32, 1992.

Wu, L. Deng, G. Li, J. Zhu, and L. Shi. Spatio-temporal backpropagation for training highperformance spiking neural networks. Frontiers in neuroscience , 12:331, 2018.

Zenke and S. Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation , 30(6):1514-1541, 2018.

Zenke and T. P. Vogels. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks. Neural Computation , 33(4):899-925, 03 2021. ISSN 0899-7667. doi: 10.1162/neco_a_01367. URL https://doi.org/10.1162/neco_a_ 01367 .

Zhang, J. Clune, and K. O. Stanley. On the relationship between the openai evolution strategy and stochastic gradient descent, 2017.

Zhou, J. Lan, R. Liu, and J. Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. CoRR , abs/1905.01067, 2019. URL http://arxiv.org/abs/1905.01067 .

B Comparing EC and NES

X = 0

X = 1

Y = 0

Y = 1

N = 12

N = 31

N = 14

N = 43

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

X

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

Y

Samples

Param

µ

Param

σ

For readers familiar with evolutionary algorithms, we present the similarities and main differences between our Evolving Connectivity (EC) method and commonly used evolutionary algorithms, namely Natural Evolution Strategies (NES, Wierstra et al. [2014]) and Evolution Strategies (ES, Salimans et al. [2017]). This comparison will facilitate better understanding and implementation of these algorithms. EC, ES, and NES share the same unbiased Monte-Carlo gradient estimator, gradient update approach, and fitness shaping trick, as described in Section 4.

The primary distinction between these algorithms lies in their search spaces, as illustrated in Figure S1. Both ES and NES employ real-valued parameters, akin to conventional deep learning techniques, and sample the search space using parameterized normal distributions. In contrast, EC utilizes a 1-bit discrete search space parameterized by Bernoulli distributions. This 1-bit formulation reduces the parameter size, endowing EC with unique advantages such as faster computation, improved memory efficiency, and reduced energy and cost requirements.

Additionally, there is a minor difference in how EC enhances exploration. As described in Section 4, EC clips the elements of connection probability matrix to boost exploration.

C Optimizing 1-bit Networks Using ES and SG

ES and SG are continuous optimization techniques designed for continuous parameters. To apply these methods to the training of 1-bit connection RSNNs proposed in this paper, we utilize discretization methods as described by Zhou et al. [2019]. Specifically, we employ ES and SG to optimize a continuous parameter θ , which is then discretized into 1-bit weights W by applying a threshold at 0 as W = H ( θ ) , where H denotes the Heaviside step function. For SG, the straight-through estimator is utilized as a standard approach [Bengio et al., 2013]. It is crucial to acknowledge that employing continuous optimization methods in conjunction with discretization may result in biased gradient estimation, while EC naturally provides unbiased gradient estimation for discrete 1-bit connections.

D On Chip Power Consumption Estimation

We estimate the power consumption of EC by considering the number of spiking neuron operations and the energy per operation provided by Loihi [Davies et al., 2018], as a representative neuromorphic

device. The data employed for this estimation is presented in Table 2. Initially, we computed the estimated energy consumption for a single network inference as follows:

Subsequently, we determined the total energy consumption during the training process:

It is important to note that these calculations are approximations and serve as an initial assessment. In future research, we plan to conduct experiments using neuromorphic chips to empirically validate the energy efficiency of RSNNs.

E Performance Comparison with PPO

In order to assess the efficacy of the EC-RSNN approach in relation to contemporary deep reinforcement learning (RL) techniques, we conducted additional training with GRU and LSTM models using Proximal Policy Optimization (PPO) [Schulman et al., 2017] on the most complex Humanoid task. The results of these experiments are presented in Figure S3. The data clearly indicates that EC-RSNN surpasses the performance of PPO-LSTM, illustrating a level of performance that is on par with state-of-the-art deep RL methods.

F Full SG Results

We present a comprehensive analysis of the Surrogate Gradient (SG) results, considering various damping factors ( γ ) and the parameter ( β ). The outcomes are illustrated in Figure S2.

G Detailed Experiment Settings

G.1 Hardware

All experiments were conducted on a single GPU server with the following specifications. Additionally, all efficiency experiments, including those that measure wall-clock time, were executed on a single GPU within this server.

8x NVIDIA Titan RTX GPU (24GB VRAM)

2x Intel(R) Xeon(R) Silver 4110 CPU

252GB Memory

G.2 Hyperparameters

In this section, we provide the hyperparameters for our framework, Evolving Connectivity (EC), and the baselines, including Evolution Strategies (ES) and Surrogate Gradient (SG). All methods utilize the same hyperparameter set for all three locomotion tasks. We also list the settings of neural networks below.

0

50000

100000

150000

200000

250000

2500

5000

7500

10000

12500

15000

17500

20000

PPO-GRU (FP32)

PPO-LSTM (FP32)

EC-RSNN (1bit)

# Gradient steps

Return

0

2000

4000

6000

8000

10000

12000

14000

SuperSpike:

h

(

x

) =

γ

1

(

β

|

x

|

+1)

2

EC-RSNN (Ours)

γ

= 1

, β

= 5

γ

= 1

, β

= 10

γ

= 1

, β

= 20

Esser et al.:

h

(

x

) =

γ

,

1

-

β

|

x

|

)

γ

= 1

, β

= 2

γ

= 1

, β

= 5

γ

= 1

, β

= 10

σ

′

:

h

(

x

) =

γ

βe

βx

(

e

βx

+1)

2

γ

= 1

, β

= 1

γ

= 1

, β

= 2

γ

= 1

, β

= 5

0

50000

100000

150000

200000

250000

0

2000

4000

6000

8000

10000

12000

14000

EC-RSNN (Ours)

γ

= 0

.

8

, β

= 5

γ

= 0

.

8

, β

= 10

γ

= 0

.

8

, β

= 20

0

50000

100000

150000

200000

250000

γ

= 0

.

8

, β

= 2

γ

= 0

.

8

, β

= 5

γ

= 0

.

8

, β

= 10

0

50000

100000

150000

200000

250000

γ

= 0

.

8

, β

= 1

γ

= 0

.

8

, β

= 2

γ

= 0

.

8

, β

= 5

Humanoid (17-DoF)

Return

# Gradient steps

H Source Code and Licensing

The source code associated with this paper is licensed under the Apache 2.0 License and is publicly available at https://github.com/imoneoi/EvolvingConnectivity . In our research, we incorporated locomotion tasks and the Brax physics simulator from Freeman et al. [2021], as well as the JAX framework [Bradbury et al., 2018]. Both of these resources are also released under the Apache 2.0 License.

Peer reviews:
comments: SUMMARY:
This study presents an application of a previously developed approach, NES, to train RSNNs formulated based on connection probabilities. However, there are concerns regarding the ethical aspect of this work, specifically the reproducibility and proper citation of related works (see Weaknesses). Due to these reasons, it is challenging to recommend this manuscript for acceptance in its current form.

STRENGTHS:
n/a

WEAKNESSES:
o	The lack of code availability at the time of review poses a significant barrier to verifying the claims made by the authors. 

o	Additionally, it appears that an existing work by Stockl et al., 2021 from Maass group, which applies NES to learn RSNNs formulated in terms of probability skeleton inspired from Billeh et al., is not cited in this manuscript. This omission warrants clarification as it may influence the perceived contribution of this work.

o	Beyond the ethical issues outlined above, this paper has several limitations that need to be addressed. Firstly, the results presented are purely empirical and there is a lack of theoretical contributions. Secondly, the manuscript lacks an analysis of how varying the sample size impacts the results, which leaves unanswered questions regarding the robustness and efficiency of the proposed methodology. Lastly, it is unclear whether the authors have performed a thorough tuning of the hyperparameters, which could significantly impact the performance of the benchmarks.

QUESTIONS:
n/a

LIMITATIONS:
See weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

comments: SUMMARY:
Facing on the inaccurate and unfriendly limitation of current surrogate gradient-based learning methods for recurrent spiking neural networks (RSNN), this study develops the evolving connectivity (EC) framework for inference-only training. The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions. The performance of the proposed EC is evaluated on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs.

STRENGTHS:
1.	The motivation is reasonable and the application scene is interesting.
2.	The framework of EC considering the weight reparameterization and connection evolution is reasonable.

WEAKNESSES:
1. It seems that the weight-based parameterization method and the NES method used in EC is not novel, it seems to use the method proposed in other papers.
2. The energy consumption should be computed to show the efficiency of the proposed framework.
3. The experiments seems not enough to verify the effectiveness of the proposed model.

QUESTIONS:
1.	The novelty of the weight-based parameterization method and the NES methods in EC framework should be explained further.
2.	The energy consumption should be computed to show the efficiency of the proposed framework.
3.	How about the performance comparison with Transformer models?

LIMITATIONS:
See the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes an evolutionary training algorithm to optimize the binary weight matrix of a recurrent spiking neural network. The optimization algorithm is derived using natural evolutionary strategies assuming that the weights follow a Bernoulli distribution with parameter $\\rho$. The resulting weight update is remarkably simple, and despite that, it seems to work very well.\n\nThe algorithm is tested on the classical locomotion tasks of simulated robotics, and the performance is reported in terms of task return and time to solution in wall clock time. On all the task RSNN model trained in this way performs better or similar to the LSTM or GRU baselines trained with ES, and it is always better than the spiking networks trained with surrogate gradient. By leveraging the discrete values of the weight matrix, the implementation is also faster than a floating point evolutionary strategy.\n\nSTRENGTHS:\nAs said in the paper: inference-only training solutions can be very useful for hardware where the training algorithm cannot be implemented easily. This approach is likely to have important for training or fine-tuning architectures when deployed on unconventional devices.\n\n I would also highlight an important achievement that the authors did not mention enough in my opinion: it appears that this training algorithm is better than all existing training algorithms for spiking networks which is remarkable for something that is practicable and competitive with deep learning solutions to such problems. It could in fact provide opportunities for quantized and spiking architectures which are otherwise bounded in their performance by the approximate gradients that are usually computed with surrogate gradient and straight through. This is solved here since the gradients are unbiased (not mentioned by the authors as far as I see). This was probably why surrogate gradient baselines do not quite reach the performance of LSTM and GRU baselines as seen here as in previous papers, but the algorithm provides a solution to this problem.\n\nWEAKNESSES:\n1) There are a few data points that would be useful to fully evaluate the quality of the training algorithm and separate the benefits of the spiking architecture from the benefits of the algorithm.\na. Is it possible to train an LSTM or a GRU with EC? What is its return after training, is that also better than ES for the spiking neurons?\nb. What is the return obtained with LSTM / GRU and PPO? This is both important to verify that PPO is well-implemented and claim or not that the EC-RSNN is competitive with the regular deep learning approach. (If the performance is low, then maybe there is a problem with the surrogate implementation too).\n\n2) I think it would be valuable to clarify that most surrogate gradient techniques were typically tested mainly without recurrent connections (Slayer, super-spike for instance). I find it admirable that the authors tested so many baselines already, but also unfortunate that they might have missed important details that were reported the be crucial in the presence of recurrent connections: depending on the weight initialization it was shown that the pseudo derivate should be multiplied by a dampening factor < 1 to avoid the accumulation of approximation errors, with Glorot initialization you might be fine without this (see Celotti and Rouat 2022). But what initialization did you use and did you test this?\n\n3) I realized only later that most of the concrete hyperparameters are detailed in the appendix: How many layers, units per layer, batch size, epsilon. Please state clearly in the main text that all this important is available in the appendix when it's relevant. I find it in general even better to put directly these numbers in the main text when possible. (For instance, write $\\epsilon = 0.001$ instead of $\\epsilon \\rightarrow 0$). Same remark when talking about the hardware. Instead of writing \"GPGPU\" or \"identitical hardware\" which I find vague since neuromorphic hardware is often mentioned in the main text. When possible, I would encourage writing just NVIDIA GPU and pointing out the appendix where the hardware reference is provided. On a related note, I would also encourage the authors to prefer a specific reference to the Figure number and panel at line 277 when commenting on their results.\n\n4) In the context of this conference which is broadly addressed to the machine learning community, I think it is important that the surrogate gradient is nothing more than a variant of the straight-through gradient estimator from Bengio 2013 which is much better known outside of the neuromorphic community and was discovered before.\n\n5) I find the first sentence line 20 a bit strange, and I find it weird to reference Pei et al. at this point. AGI is also never used elsewhere in the text (for good reasons since you have more concrete and good results to discuss), so why defining this acronym here ?\n\nQUESTIONS:\nI realized that points 1a.b and 2 above would be best addressed with extra simulations. Given the time and effort that it requires, I do not consider those simulations necessary, but I would encourage the authors to address those questions verbally during the rebuttal and consider the feasibility and the cost/benefits of these potential simulations at some point.\n\nFor points 3, 4 and 5. I would encourage the authors to do minor edits.\n\nMore general question: Do the authors believe that this approach can replace gradient descent learning in some other context ? Or is it that ES and EC and bounded to make only sense in this type of toy reinforcement leanring task where the network size is small and the batch size can be super large?\n\nLIMITATIONS:\nI see no potential negative impact. The limitations are clearly stated in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new evolutionary algorithm, evolving connectivity (EC) to train recurrent spiking neural networks (RSNN). The algorithm alleviates the gradient estimation problem of surrogate gradient-based (SG) training methods which are often hard to implement in neuromorphic hardware. Further, by focusing the algorithm only on evolving network connectivity instead of the weight magnitude, EC can reduce all network weights down to 1bit {0,1} alternatives making it further hardware friendly. The paper also highlights the superior accuracy and performance of their training algorithm for a few robotics tasks involving sequential decision making against state-of-the-art Surrogate Gradient and Evolution Strategies based algorithms.\n\nSTRENGTHS:\nThe authors present a strong argument for developing RSNNs which focus more on the connectivity probability between layers instead of selecting specific weight magnitudes. This enables the network to implicitly fit many samples without overfitting to any specific subset of data. \n\nThe paper explains the EC algorithm clearly and highlights justifications for its superiority in performance and accuracy compared to other algorithms (ES/SG) for targeted robotics benchmarks. \n\nRemoving the gradient estimation requirement and reducing the weight matrix to 1-bit are significantly useful optimizations that make the algorithm hardware friendly.\n\nWEAKNESSES:\nThe paper doesn’t clearly describe the RSNN architecture that is adopted for the experiments discussed and how it compares in terms of number of neurons/connections to the Deep RNNs that are used as baseline for performance/accuracy. This makes the comparisons to the ES-RNN results difficult to understand for me. \n\nFurther, comparing the RSNN based experiments only (ES/EC/SG), it appears that only EC algorithm is 1-bit owing to the focus on connectivity only. But this makes me wonder if 1-bit networks applying ES/SG algorithms could compete against EC approach in terms of performance. I understand that by the authors’ assertion that connectivity is more critical than assigning wide ranging weights to a fixed set of connections. However, for experiment’s sake I am curious to understand that if ES/SG algorithms also had the HW friendly assumption of only creating a binary SNN then would their accuracy/performance improve or deteriorate?\n\nQUESTIONS:\n1.\tDescribe the RSNN architecture and how it compares to the RNNs used in Experiments in terms of number of layers/params/model size/precision? \n2.\tExperiment with 1-bit RSNN utilizing ES/SG algorithms for training to understand the impact on accuracy and performance?\n3.\tThe combination of ES and RSNN performs notably well on all the benchmarks tested, but it would be good to understand what part of it comes from the network architecture (try another architecture), what part comes from bit precision and lastly what is contributed by the training algorithm? This kind of study will help strengthen the credibility of EC algorithm for other benchmarks and network architectures.\n\nLIMITATIONS:\nThe authors provide a useful discussion on the memory footprint of the different training algorithms discussed. Please address the concerns related to network architecture and precision impacting accuracy and performance as discussed in weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}