Evolving Connectivity for Recurrent Spiking Neural Networks
Authors: Guan Wang
Yuhao Sun
Sijie Cheng
Sen Song
Venue: NeurIPS 2023 poster (2023)
Keywords: neuromorphic computing
spiking neural networks
evolutionary algorithms
inference-only approach
hardware-friendly
robotic locomotion tasks

Abstract: Recurrent spiking neural networks (RSNNs) hold great potential for advancing artificial general intelligence, as they draw inspiration from the biological nervous system and show promise in modeling complex dynamics.
However, the widely-used surrogate gradient-based training methods for RSNNs are inherently inaccurate and unfriendly to neuromorphic hardware.
To address these limitations, we propose the evolving connectivity (EC) framework, an inference-only method for training RSNNs.
The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions.
Our EC framework circumvents the need for gradients and features hardware-friendly characteristics, including sparse boolean connections and high scalability.
We evaluate EC on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs, even solving the complex 17-DoF humanoid task.
Additionally, the EC framework demonstrates a two to three fold speedup in efficiency compared to directly evolving parameters.
By providing a performant and hardware-friendly alternative, the EC framework lays the groundwork for further energy-efficient applications of RSNNs and advances the development of neuromorphic devices.
Our code is publicly available at https://github.com/imoneoi/EvolvingConnectivity.

Evolving Connectivity for Recurrent Spiking Neural Networks
Guan 2 Yuhao 3 Cheng" 1 Sen Song2.3t
Deptartment of Computer Science and Technology, Tsinghua University
2 Laboratory of Brain and Intelligence, Tsinghua University & DDepartment of Biomedical Engineering, Tsinghua University 'Intiittee for Al Industry Research (AIR), Tsinghua University imonenext@gmail. com, {syh18, songsen@tsinghua. edu.
tsinghua. edu cn,
Abstract
Recurrent spiking neural networks (RSNNs) hold great potential for advancing artificial general intelligence, as they draw inspiration from the biological ner- vous system and show promise in modeling complex dynamics. However, the widely-used surrogate gradient- based training methods for RSNNs are inherently inaccurate and unfriendly neuromorphic hardware. To address these limitations, we propose the evolving connectivity (EC) framework, an inference-only method for training RSNNs. The EC framework reformulates weight-tuning as search into parameterized connection probability distributions, and employs Natural Evo- lution Strategies (NES) for optimizing these distributions. Our EC framework circumvents the need for gradients and features hardware-friendly characteristics, including sparse boolean connections and high scalability. We evaluate EC on a series of standard robotic locomotion tasks, where it achieves comparable per- formance with deep neural networks and outperforms gradient-traine RSNNs, even solving the complex 17-DoF humanoid task. Additionally, the EC framework demonstrates a two to three fold speedup efficiency compared to directly evolving parameters. By providing performant and hardware-friends alternative, the EC framework lays the groundwork for further energy-efficien applications of RSNNs and advances the development of neuromorphic devices. Our code is publicly available at htttt::////// //gt thub. cowm//iooeoo//iiloiiiiooooooo
Introduction
Inspired by the remarkable information processing power of the human brain, neuromorphic comput- ing aims to design and implement computational systems that mimic the architecture and functionality of biological neural networks, paving the way for new of intelligent machines. Specifically, the highly recurrently connected and spike-emitting brain network inspired recurrent spiking neural networks (RSNNs), which employ discrete, baske- signals for transmitting information through recurrent connections an event-driven manner. RSNNS can serve as realistic models of the brain incorporating the latest anatomical and neuro-physiological data [Billeh et al. 2020], leading the discovery of general principles of computing, c.g., noise robustness [Chen et al. 2022]. Furthermore, RSNNs can exhibit self-organized critical properties (Poil et 2012] and serve as reservoir for complex temporal dynamics Jaeger 2001 Maass et 2002). making them suitable for sequential
tasks like roboties [Lele et al. 2020 aath-finding [Rueckert et al. 2016). and others.
'Equal contribution CCoresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Despite the importance of RSNNs, developing end-to-end training algorithms remains a challenge. Inspired by the success of deep learning, a line of research [Wu et al.
2018 Shrestha and Orchard 1985 with carefully chosen [Rumelhart et al.
2018 Bauer et al. 2022 uses eroorbbacproppggatee
surrogate gradients to address the non-differentiabiity problem and achieve performance comparable to deep learning. However, incorporating surrogate gradients into RSNNs introduces two concerns. Algorithmically, the surrogate gradient leads to inherent inaccuracy in the descent direction [Li et al. 2021] and sensitivity to function scale selection Zenke and Vogels 2021]. At the implementation level, gradient- bbaed training is incompatible with prominent neuromorphic devices [Pei al. 2019 Davies et al. 2018 Mayr al. 2019] due to the requirement of accessing the full network state over every timestep Werboss| 1990]. Consequently, this raises a critical research question: can we design a training method for RSMNs that bypasses the nee gradients without compromising performance?
In response to this challenge, we observe that connection probability distribution information from brain connection maps across various species exhibit similarities [Haber et al. [2023 and can be used to construct large-scale neural computing models [Billeh et al.) [020] [Schmidt et al. 2018). Existing studies demonstrate that networks with binary connections alone can achieve high performance. rivaling that of weighted networks [Gaier and Hap 2019, Frankle and Carbin 2018 Malach et al.) 2020]. Moreover, computing boolean connections relies primarily on integer arithmetic rather than resource-intensive floating-pont operations, enabling simpler on-chip implementation and enhanced energy efficiency. Considering these features, we reformulate the architecture of RSNNs, where connections are sampled independently from parametric Bernoulli distributions, and adopt Natural Evolution Strategies (NES) [Wierstra al. 2014] to optimize this parametric probability distribution, providing a scalable, inference-only, and effective algorithm for tuning parameters.
In this paper, we introduce the evolving connectivity (EC) framework for training RSNNs. The EC framework reformulates RSNNs as boolean connections with homogeneous weights and employs NES to evolve the connection probabilities of the network. To evaluate effectiveness and efficiency, we conduct extensive experiments on locomotion tasks, a set of widely-used sequential decision- making tasks [Brockman et al. 2016 Freeman et al.] [02T]. Experimental results show that our proposed EC method achieves performance comparable to deep recurrent neural networks, surpassing the asymptotic performance of surrogate gradients. Despite using GPU, which is not specifically optimized for our framework, the EC method yields a speed improvement of ~ 3x compared
to directly evolving parameters using Evolution Strategies (Salimans et al. 2017] and exhibits better efficiency than surrogate gradients. Additionally, our EC method can effectively train deep recurrent neural networks (RNNs), demonstrating versatility for different architectures and potential for quantized deep neural networks.
Our main contributions are summarized as follows:
Novel framework: We propose a novel inference-only training framework for RSNNs by reformu-
lating weight-tuning as connection probability searching through evolution- algorithms.
High performance: Our method can solve the complex 17-DoF humanoid locomotion task with RSNN, achieving performance on par with recurrent neural networks and outperforming gradient-trained RSNNs.
Hardware-friendly: By producing RSNNs with sparse 1-bit boolean connections, as well as enabling inference-only training and high scalability, our method is highly compatible with neuromorphic devices. This compatibility holds promising potential for further energy-efficient applications of RSNNs.
2
Related Works
2.1 Training Recurrent Spiking Neural Networks
The study of training algorithms for RSNNs endeavors to unravel the mechanism that enables the brain to learn. For decades, biological plasticity rules, especially ppikk-timinng-ee -iming-dependent plasticity (STDP) Bi and ming Poo| 1998). have been considered as a foundational of RSNNs training (Diehl) and Cook 2015 Kheradpisheh et al. 2018 Mozafari et al., 2019]. Recently, with the success of deep learning, surrogate gradient-based approaches [Wu et al. 2018 Shrestha and Orchard 2018 [Bauer et al., 2022) have been predominately utilized training RSNNs. Surrogate gradients share conceptual similarities with the straighttthrough estimator [Bengio al.) 2013 used in deep learning.

---

Both techniques employ continuous surrogate functions to compute gradients of con-differentiable functions during the backward pass, while preserving the original con-differentiable functions in the forward pass. Slayer [Shrestha and Orchard 2018) adopted an exponential decaying surrogate function with spike response model (SRM), while STBP [Wu al. 2018) incorporated BPTT in multi-layer SNN training and proposed several surrogate function choices. [Chen et al. [2022] trained an anatomical and neurophysiological data constrained RSNN and demonstrated its robustness and versatility. However, gradient-based approaches face the new problem of being difficult to implement on neuromorphic devices. To address the limitation, E-prop [Bellec et al. 2020] utilize eligibility trace to compute truncated gradient, becoming exemplary training algorithm
neuromorphic devices like Loihi2 [Davies et al., 2018) and SpiNNaker2 [Mayr et al., 2019]. but have a strict restriction on the temporal relationship of SNN models and still lags behind surrogate gradient methods in performance. Our approach tackles the wradient-effectiveen dilemma from the beginning, by introducing the inference-only training framework.
2.2 Weight-agnostic Neural Networks
Training a neural network typically involves assigning appropriate values to the network's 's weights. However, the weight-agnostic neural network (WANN) (Gaier and Ha 2019] has demonstrated that a network's 's topology can be highly informative, similar to that of a weighted neural network. Additionally, research on the lottery ticket hypothesis (LTTH) [Frankle and Carbin 2018 Zhou et al.) 2019 Ramanujan et al. 2020] suggests that an over-parameterizze network contains effective subnetwork, even when using its initial parameter weights. Several theoretical studies have also shown that a sufficiently large, ranihel deep neural network contains subnetwork capable of approximating any function [Malach et al., 2020 Fischer et al.) 2022]. Our approach, furthering the existing research coanection-encodeee networks, proposes framework that utilizes connection probability to parameterize the network.
2.3 Deep Neuroevolution
Deep neuroevolution utilizes evolutionary algorithms for training deep neural networks. Natural Evolution Strategies (NES) [Wierstra et al. 2014) have laid the foundation for gradient estimation in this domain. A well-known variant, Evolution Strategies (ES) (Salimans et al., (017], has been employed for training deep neural networks in sequential decision- makinng tasks. ES optimizes a single set of continuous network weights by applying Gaussian perturbations and updating the weights using the NES gradient estimator. Following the success of ES, numerous evolutionary algorithms have been proposed for training deep neural networks. For example, (Such et al.) [2018] showed that deep neural networks could be effectively trained using simple genetic algorithms that mutate weights. Furthermore, Conti et al. (2018) integrated exploration and novelty-seeking into ES to overcome local minima. While the majority of prior research has primarily focused on continuous parameters, our proposed framework presents a novel approach by concentrating on the search for connection probability distributions. This shift in perspective offers possibilities for evolving recurrent spiking neural networks in a hardware-frient manner.
3 Preliminaries: Recurrent Spiking Neural Networks
RSNN is a class of spiking neural networks which incorporate feedback connections. In this paper, we adopt typical RSNN architecture from the reservoir network [Jaeger] 2001 Maass et al.) 2002 for sequential tasks. It is worth noting that, despite that we adopt a specific RSNN model as an example, our framework can be broadly applied search for connectivity distributions in any type of RSNN, as it does not depend on network-speciffc assumptions and only relies on evaluating the network with a set of parameters.
According to Dale's law, our network consists of an excitatory neuron group and an inhibitory neuron group. Each neuron is modeled as leaky integrate and fire (LIF) neuron. A neuron will fire spike when its membrane potential u exceeds the threshold, and the membrane potential will hard-reset to 0. More specifically, our model defines the dynamics of membrane potential u and synaptic current as follows:
du(s)
Fm
(9) + Rc(s)
(1)
dt
3

---

Population
Global Distribution
RSNN
Bia)
Enviranment
Cument Oumene-based
CARME 9000
B2
Irput
eeckatory
Cluerrsta
Otevrrazio
R₂
1-hE
Physics
Firing
Simulction
Figure 1: Architecture of evolving connectivity (EC). The connectivity O₂ of the population is sampled from the global distribution B(p) and then evaluated in parallel. The RSNN consists of excitatory and inhibitory neurons, simulated using 1-bit firing calculations and the LIF model.
where y = (Exe Inh} denotes the excitatory and inhibitory group, respectively, and R denotes the resistance.
The current input was modeled using exponential synapses retain information for short duration, commonly adopted in robotic tasks (Tang et al. 20200 Naya et al. 2021].
dela)
cla)
91 Σ
w (9:9))
ö(t 15105) + Lezt
dt
Tayn
where 1 (2(08) denotes the spike time of neuron j in neuron group 9j. 8 denotes Dirac delta function. 1g set define the connection strength of excitatory and inhibitory synapse respectively, where IEac > 0 and Inh O. Weight W(s.ss) was defined matrix with non-negative elements, connecting group 9j to group 9,. Besides, the external input signal Lent is extracted using linear projection of observation
Discretizing the LIF differential equations (Eq. Tand with At as a time-step, we could obtain the following difference equation for our RSNN model:
(t-g) (tclt-1.0)) +
Σ Lg, (t- (1.9)
(3)
cxtt
95
y (t.g) d.u (t-1) + (tg)
(4)
s ((.9) = ((1.9) >
(5)
u (1.0) ((1.0) ((.9)
where de Taym and d, are constant parameters.
The output vector is extracted using linear projection of neuron firing rates a short time period т, ie.,
Σ ((()) Σ w(e) out
((t-r.))
where w(s) out denotes the output weight, and le is function averaging the time period т.
Framework
In this section, we present our proposed Evolving Connectivity (EC) framework, which is depicted
in Fig. [ Our approach consists of three main steps: (I) reformulating the neural network architec- ture from weight-based parameterization to connection probability distribution, (2) employing the Natural Evolution Strategies (NES) method to optimize the reformulated parameter space, and (3) deterministically extracting the final parameters from the distribution.
Reformulation. sparse connected weight matrix can be described into a weight matrix and connection mask 0. Traditionally, we use Erdös-Rén Rényi random matrix describe the connection

---

mask, in which connections are independently drawn from Bernoulli distribution, i.e.,
W Wij Oij, where Ozj- B(p)
(8)
Inspired by the success of subnetworks in deep neural networks [Ramanujan et al. 2020 we can reformulate this in connection framework. In this view, we aim to find a connection probability matrix, p (pi)), where each element represents the connection probability between two neurons, and set all weights Wij to be unit size, i.e.,
W s) Ozg, where P,y B(pi)).
Optimization. We optimize p maximize the expected performance metric function R(.) across individual network samples drawn from the distribution, which can be expressed as the following objective function:
= arg max (pp arg Bo-xEo_BBp [(()) [[(O]]
(10)
To optimize this objective, we employ Natural Evolution Strategies (NES) Wierstra al. 2014].
NES provides an unbiased Monte Carlo gradient estimation of the objective J [p), by evaluating the performance metric Rg on multiple individual samples 0₂ drawn from (p). Specifically, NES
estimates the gradient of (p) with respect to the parameters of the distribution p by computing the expectation over samples from Bp)):
V p = Eo B(p) V p log R(O)))
(11)
0 P R())]
Ee
(12)
p(1 -) p)
N
0g P Rs
(13)
N
p(1 p)
Thus, obtained inference-only estimation of the gradient by simply sampling and evaluat- ing the metric. Then gradient descent can be carried over the estimated gradients, following the
NES approach [Wierstra et al., 2014). Furthermore, as in Williams [1992]. we scale the step size proportionally to the variance, l.e. a = 17 Var[B(p)), and obtain the update rule:
N
P==P--1 a P
(0 Rs
(14)
In addition, all elements of p clipped in the interval [e, 1 e]. where 0+, to guarantee minimal level of exploration within the search space. In our experiment, we set e = 0.001 and initialize p==00 0.5. Additionally, we adopted a similar fitness shaping trick, center rank transform, as discussed in Salimans et al. [2017). which sort the return of a population, then linearly rescale the sorting indexes into fixed interval [-0.5.0.5].
Once a sufficient number of updates have been performed, the final parameter 0 can deter- ministically obtained as the parameter with maximum probability, which is subsequently used for deployment. the context of a Bernoulli distribution, this process equivalent thresholding p at 0.5 to produce boolean matrix containing only {0, 1) values.
5 Properties of EC Framework
In this section, we further discuss the important properties of our proposed EC framework in implementation, thanks to leveraging the connection probability distribution.
The most significant challenge in neuromorphic devices is the absence of an effective fordware-friendle learning algorithm Li et al. 2023]. Most neuromorphic chips. such as Loihi [Davies et al. 2018 Tianjic [Pei et al., 2019]. TrueNorth (Akopyan al., 2015), SpiN- Naker2 [Mayr et al. 2019), lack support for directly calculating error backpropagation, which is crucial for surrogate gradient methods. The inference-only EC framework enables an alternative
Inference only.
method for training on these inference chips.
5

---

1-bit connections. EC employs 1-bit sparse connections throughout training and deployment, replacing the traditional dooatingpoiit weight matrix. Therefore, the 1-bit connections permit the use of more economical integer arithmetic instead of costly floating-point cores. This approach not only accelerates computations on devices like GPUs, but also holds a promise of driving the creation of novel 1-bit connection neuromorphic computing hardware.
Scalability. The inference-only property of the EC framework implies no data dependence between evaluations, which allows for the distribution of sampled parameters 0g across independent workers and the collection of their reported performance measures, making EC highly scalable. Moreover, the random seed for sampling the population can be transmitted across nodes instead of the connections 0₂ to minimize communication overhead, facilitating scalar-only communication.
6 Experiments
6.1 Experimental Setups
Tasks. We focus on three robotic locomotion tasks in our experiments, Humanoid, Walker2d, and Hopper, they are commonly used for se- quential decision-makinn
problems in the re- inforcement learning domain [Brockman al.) 2016 Freeman et al. [202T]. As illustrrted in Fig. these tasks involve controlling robots with varying degrees of freedom (DoF), to perform certain actions that maximize the return within a fixed-length episode T. In the EC framework, the perfor-
Figure 2: Locomotion tasks illustrated from left
right: Humanoid (17-DoF), Walker2d (6-DoF), and Hopper (3-DoF).
mance metric function can be defined the expected return episodes R(0) E E E rel.
and a single evaluation R3 corresponds to the return of one episode using network parameter 0g.
RSNN architecture. In our study, we employed a RSNN consisting of an input layer, hidden recurrent layer with LIF neurons, and an output layer, as delineated in the Preliminaries section. To achieve an equilibrium within the excitatory inhibitory balance, input and hidden layers were evenly divided into excitatory and inhibitory group.
Baselines. To thoroughly evaluate the effectiveness of our framework and its advantages over prior methods, we compare EC framework with deep RNNs, well as RSNN trained with Surrogate Gradients (SG) and Evolution Strategies (ES). Detailed network architecture and precision illustrated in [a
For deep RNNs, we employ widely-used recurrent deep neural networks, specifically long-short term memory (LSTM) [Hochreiter and Schmidhuber 1997] and gated recurrent unit (GRU) (Cho et al. 2014), trained using Evolution Strategies (ES) Saliiaan al. 2017] as our baselines.
For RSNN, we employ the same structured but excitatory/inhibioory not separated RSNN trained with ES and Surrogate Gradient (SG). ES is directly applied search the weight matrix W of RSNNs, optimizing the performance metric in gradient-free manner. In contrast, the Surrogate Gradient combined with Proximal Policy Optimization (PPO) [Schulman al. 2017], a prominent reinforcement learning method, to optimize the weight W for sequential decision decision-making tasks where returns cannot be differentiated.
Table 1: Comparison of model architectures.
Model
Hidden size
# Params
Precision
Size (MB)
RSNN
256
193K
1-bit(EC)) / FP32(Others)
24KB(EC) / 768KB(Others)
GRU
256
386K
FP32
1544KB
LSTM
128
191K
FP32
764KB
Implementation Details. Our EC framework and all baselines are implemented using the JAX
library [Bradbury al., 2018) and just-in-time compiled with the Brax physics simulator [Freeman al. 2021 I] for efficient GPU execution. The population is vectorized automatically, and 1-bit firing

---

Humanold (17-DoF)
Walker2d (6-DoP)
Hopper (3-DoP)
2000
330
Rot-or
RC-RENN (Ours)
FSS TISNN
300
SC- RENN
230
ES-CRU
530
E5-L81M
200
340
4.00
660
12.0
400 660
030
10.00
Cinneration
Figure 3: Performance evaluation on locomotion tasks. SG-RSNN is plotted based on the final return, gradient- based updates not directly comparable to generations in evolution-based methods. The proposed Evolving Connectivity (EC) framework effectively solves the 17-DoF Humanoid locomotion task, demonstrating competitive performance with deep RNNs and outperforming RSNNS trained using both Surrogate Gradient and Evolution Strategies across all tasks.
calculations take advantage of INT8 arithmetic for performance optimization. As a result, the training process achieves over 180, 000 frames per second on single NVIDIA TITAN RTX GPU. Each experiment's result is averaged over 3 independent seeds, with the standard deviation displayed as a shaded area. For detailed information hyperparameters and hardware specifications, please refer to Appendix G
6.2 Performance Evaluation
Firstly, compare RSNN trained with our EC framework (EC-RSNN) deep RNNs. To ensure a fair comparison, we utilized the evolution-based training approach ES [Salimans et al. 2017] for deep RNNs, including ES-GRU and ES-LSTM. Typically, SNNs face challenges in surpassing the performance of their DNN counterparts. However, as shown in the experimental results in Fig. 3 our EC-RSNN achieves competitive performance and training efficiency compared to deep RNNs. even surpassing ES-GRU and ES-LSTM on the Walker2d and Hopper tasks. In the complex 17-DoF Humanoid task, our EC-RSNN could also outperform ES-GRU. Moreover, EC-RSNN demonstrates comparable performance to GRU and LSTM trained with PPO, as detailed in Appendix E It is worth noting that, while GRU and LSTM use densely connected, full precision (32-bit floating-poins weights and hard-coded gating mechanisms, EC-RSNN is constrained on 1-bit sparse weights and adheres to Dale's ' law for weight signs.
Then, as for the same structured RSNNs, our EC-RSNN outperforms ES-RSNN, especially signifi- cantly surpassing ES-RSNN on complex Humanoid and Walker2d tasks. We highlight two aspects that potentially account for the superior performance of our EC framework. On the one hand, schema theory [Whitley] [994] suggests that selecting over population of binary strings could combine partial binary string patterns (schemata) to implicitly work on many solutions without evaluating them explicitly, leading to massive parallelism. EC leverages the discrete {0, }]-string space, which may enable more efficient optimization due to implicit parallelism. On the other hand, ES uses fixed-scale noise to perturb a set of deterministic parameters, which may often fall in wide areas in the landscape of the objective function and fail to proceed on sharp maxima [Lehman et al., 20188). In contrast, EC operates over a probabilistic distribution, providing the flexibility to adjust variance on sensitive parameters and achieving more fine- fine-grained optimization.
Finally, gradient updates cannot be directly converted into evolution generations, thus we only com- pare the final converged return for surrogate gradient trained RSNNs. Considering that the surrogate gradient approach is sensitive to the selection of the surrogate function and its parameters [Znke and Vogels, 2.2T]. we choose three commonly-
surrogate functions with different ,3 and y parameters to thoroughly validate the performance: SuperSpike [Zenke and Ganguli 2018), the piecewise linear function proposed by Esser et al. [2016], and the derivative of the Sigmoid function [Zenke and Ganguli 2018). As shown Fig. the performance of our EC-RSNNs consistently higher than SG-RSNNs across all surrogate functions and byper-parameters on the complex 17-DoF Humanoid
task. We adopt the best parameter set of the piecewise linear surrogate function, with y = 0.8, as the bascline in Fig. 3
7

---

Humanoid (17-DoF)
SuperSpiieer Mar 11)
Eesor et al: A(tr) = (0, 1- -14)
d' M(s)
Retura
2940
PC TUNNN (Chure)
0.8.
0.8,1
Cradiinnt steps
Figure 4: Surrogate Gradient on the Humanoid task, with y 0.8 and several surrogate functions, parameters. For 1.0 results, please refer to Appendix P. Performance is sensitive to the chosen surrogate function and its parameters. In this context, EC surpass all considered parameters of SG.
One possible reason explain the performance of SG-RSNN is that due RSNNs having both explicit and implicit recurrence, the choice of surrogate function may determine whether gradients vanish explode [Zenke and Vogels 202T]. Moreover, the gradient approximation of SG is inaccurate and may deviate from the steepest descent direction (Li et al. [202]]. In contrast, our EC framework leverages unbiased Monte-Carke gradient estimation. which approaches the optimal direction given that the population is sufficiently large [Zhang et al., 2017).
6.3 Efficiency Comparison
In this section, we compare the computational efficiency of our proposed EC framework with other To ensure a fair comparison, we evaluate the EC and baseline methods using the same wall-clock computation time and implement them on identical hardware.
baselines, as shown in Fig. 3
As expected, EC-RSNN exhibits slower training than deep RNNs due to the higher complexity of RSNNs, which necessitates simulating multiple timesteps compute firing and integrate differential equations. Additionally, the deep RNN models employed in this study, specifically LSTM and GRU, are well-established and highly optimized for GPU implementation. Nevertheless, within the same computation time, the performance achieved by EC-RSNN remains competitive with deep RNNs.
Furthermore, execute both ES and EC for 1,000 generations using the same population size and RSNN architecture. Experimental results in Fig. [5] have shown that our b- EC-RSNN achieves speedup of approximately 2 ~ 3 over 32-bit floating-poin ES-RSNN. This finding emphasizes the value of 1-bit connections within the proposed framework. Utilizing integer arithmetic typically results in higher throughput than floating-point arithmetic across most accelerators, and incorporating smaller data types reduces both memory requirements and memory access time. is also worth mentioning that the 1-bit connections are implemented using INT8 on GPU, which is not fully optimized. By implementing the framework on hardware supporting smaller data types, connection size could be further reduced by up 8x, enabling more significant acceleration and cost reduction.
Additionally, our EC-RSNN demonstrates faster convergence compared to SG-RSNN. This can be attributed to the fact that EC-RSNN is an inference-one only method with 1-bit connections, requiring
Humanoid (17-DoF)
Walkor (6-DoF)
Hopper-v5 (3-DoF)
Bettora
EC-RENN (Ours)
SC- RENN
1800
ES--GRU
2.0
ES-LSTM
Wall-clock Eima (bouu))
Figure 5: Efficiency comparison in locomotion tasks. Evolution-based approaches (EC, ES) are executed for 1,000 generations, while surrogate gradient (SG) runs for 250,000 gradient steps to attain a comparable run- run-time EC, providing a fair comparison. When training RSNNs, EC attains a 2 ~ 3x speedup over ES and demonstrates faster convergence than SG.
8

---

only single forward pass using integer arithmetic. while SG-RSNN demands both a forward and a backward pass employing computationally-nnn -intensive floating-poin operations.
Finally, we calculated the estimated energy consumption of the EC-RSNN on both GPU and neu- romorphic devices, aiming to assess the power efficiency of the ECC to train RSNN. In our study, all experiments were consistently conducted on the same NVIDIA Titan RTX GPU, operating at a stable 100% GPU power (280W). Consequently, the energy consumption of the EC training process on GPU is approximated to be directly proportional to the computation wall time, resulting in a 9 MJ. For neuromorphic device, we have conducted computations pertaining to the estimated energy consumption of EC-RSNN when implemented on the Loihi chip [Davies et al. 2018]. As detailed in Appendix D the estimated energy consumption of EC is 28 kJ, which indicate a noticeable reduction in power consumption by more than two orders of magnitude.
6.4 General 1-bit Framework
Humanoid (17-DoF), RNN
The task of discretizing a neural network into 1-bit represen- tation presents a considerable challenge, particularly when em- ploying continuous training techniques such as ES and SGD. To illustrate this difficulty, we attempted discretize both ES and SG for training 1-bit connection RSNNs utilizing the straight- through estimator (Bengio et al.) 2013]. Fig. 7 (a) showed that while both ES-RSNN (Ibit) and SG-RSNN (Ibit) exhibited learn- ing progress, they were surpassed by EC-RSNN (Ibit). Fig- 7 (b) and (c) suggest that ES and SG demonstrated superior per- formance on continuous FP32 weights as opposed to discrete 1-bit representations. These findings imply that continuous opti- mization methods excel when applied to continuous parameters; conversely, for 1-bit discrete connections, EC is recommended due to its specific design for discrete 1-bit optimization and the provision of unbiased gradients. A comprchensive description of the discretization for ES and SG methods is provided in Appendix C
1360
2000
EC-RNN (1bis)
GES (PP32)
PPO-RNN
200
Generations
Figure 6: Performance compari- son of vanilla RNN trained with EC, ES, PPO. EC surpasses base- lines despite using 1-bit connec- tions.
Moreover, Despite EC have demonstrated the efficacy on RSNNs, it serves as a versatile 1-bit training framework to non-spiking networks as well. To substantiate EC's potential in training deep RNN, we conducted series of experiments employing a standard RNN model trained with EC, while using ES and PPO baselines. The RNN has 256 tanh units in the hidden layer. For 1-bit EC, the weight magnitudes are 0-1 connection matrix, and the weight signs are separated to excitatory or inhibitory. The results in Fig. 6 demonstrate that EC can effectively train 1 1-bit deep recurrent neural networks and has the potential for different architectures and quantized neural networks.
7 Conclusion
In this study, we present the innovative Evolving Connectivity (EC) framework, a unique inference- only approach for training RSNN with 1-bit connections. The key attributes of the EC framework,
Humanoid (17-DoF), RSNN
(a)
(b)
4000
EC-ISNN (1bic) ES RSNN (Thit)
ES-RSNN (TTIL) TS-RISNN (FPOR)
SG-RSNY (1big)
SG-R5SN (1hir)
SG-RENN (PT332)
800
400
1000
100
300000
206000
1 Generatione
Cenerattion
Graditar stops
Figure 7: Optimization of the 1-bit RSNN proposed in this study using ES and SG. (a) In the discrete 1-bit RSNN settings, EC significantly outperforms ES and SG. (b) (c) Comparison of FP32 and 1-bit precision for ES and SG respectively, indicating that these continuous optimization methods achieve optimal results with FP32 continuous weights.

---

such as its inference-only nature and scalability, render it particularly suitable for training RSNNs on neuromorphic devices. The use of 1-bit connections significantly reduces memory requirements and computational cost, facilitating faster training on GPUs and paving the way for additional cost reductions in neuromorphic chip production.
We performed extensive experiments on a variety of intricate locomotion tasks, showcasing the competitive performance of our proposed EC framework in comparison to deep RNNs like LSTM and GRU. Moreover, the EC framework outperforms other RSNN training methods such as Evolution Strategies and Surrogate Gradient, both in terms of performance and computational efficiency.
8 Limitations
Evolutionary algorithms demand the storage of N distinct parameter sets for population evaluation, leading space complexity of ( In contrast, gradient-based approaches utilize a single parameter set but necessitate the storage of intermediate results at each timestep. resulting in a space complexity of ONNSS + 10)., where H corresponds the number of BPTT timesteps, and S repre- sents the size of the intermediate results. This situation gives rise to a trade-off: evolutionary methods offer greater memory efficiency for tasks featuring long time horizons, whereas gradient-based tech- niques with larger parameter sizes and shorter time horizons require less memory. Recognizing this trade-offis is crucial in practical applications. Although our proposed EC framework, as an evolutionary algorithm, retains the same space complexity and trade-off, it can achieve constant term reduction by storing 1 bitt connections.
9 Discussions
Neuromorphic hardware. One primary bottleneck in the development of neuromorphic hardware the lack of effective on-chip learning algorithms to build applications compared to deep learning approaches. This paper proposes a novel EC framework that circumvents the requirement for gradients and demonstrates efficacy on locomotion tasks, providing a potential solution for solving the on-chip learning challenge. Typically, the demand for computing devices falls into two categories, including cloud and edge. In the cloud, our proposed EC framework supports large-scale learning, while the edge, it enables energy energy-efficient applications. Therefore, our framework offers a potential approach to further building neuromorphic applications.
Another practical challenge lies in the trade-off between numeric precision and cost. We suggest that it is possible to drastically reduce connections from floating point to 1-bit, providing novel design principle for the next generation of neuromorphic hardware. Therefore, our method holds the potential to significantly decrease the manufacturing and energy costs of neuromorphie hardware.
Neuroscience. Our EC framework introduces a novel method for neuroscience research. First, as opposed to toy tasks or simulated signals, which are often studied in previous neuromodeling work, we employed RSNN in a complex, real-world like locomotion task. Additionally, controlling signals in neuroscience like 'GO' and 'NOGO' signals in the basal ganglia can be easily integrated into the task by concatenating them to the environment observation vector, enabling the creation of novel neuroscientific tasks. Our work lays the foundation for further investigation of decision-makie processes and motor control.
Moreover, our framework provides a novel type of data to analyze: neuron-to-neuron level connection probability. Connection probability is one of the most fundamental properties in brain-wide connec- tomes. However, obtaining nuronnto-neurrn connection probability from the whole mammalian brain is experimentally implausible due to limitations in nowadays neuroconnectomic technology. Our work provides in silico connection data for further analysis, including covariances, motifs, clustered engrams, and dimensional properties.
Finally, our framework is capable of incorporating neuroanatomical and neurophysiological data to construct novel neurosimulation model, since our framework is able to train arbitrary models in principle. Neuroscientists have discovered that connection probability between two neurons is determined by various factors, including spatial distance, receptive field, and neuron types. Our framework can leverage these findings and conduct in silico experiments with data-driven models.
10

---

References
F. Akopyan, J. Sawada, A. S. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y. Naka- mura, P. Datta, G.-J. Nam, B. Taba, M. P. Beakes, B. Brezzo, J. B. Kuang, R. Manohar, W. P. Risk, B. L. Jackson, and D. S. Modha. Truenorth: Design and tool flow of a 65 mw million neuron programmable neurosynaptic chip. IEEE Transactions on Computer-Aides Design of Integrated Circuits and Systems, 34:1537-1557, 2015.
F. C. Bauer, G. Lenz, S. Haghighatshoar, and S. Sheik. Exodus: Stable and efficient training of spiking neural networks. arXiv preprint arXiv:2205. 10242, 2022.
G. Bellec, F. Scherr, Subramoney, Hajek, Salaj, R. Legenstein, and W. Maass. A solution to the learning dilemma for recurtent networks of spiking neurons. Nature communications, 11(I): 3625, 2020.
Y. Bengio, N. Léonard, and A. C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1 3tss/1108.3332 2013. URL http://axpx/ org/ abs/ 3432
G. Bi and M. ming Poo. Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. The Journal of Neuroscience, 18: 1814444 10472, 1998.
Y. N. Billeh, B. Cai, S. L. Gratiy, K. Dai, R. Iyer, N. W. Gouwens, R. Abbasi-Asl, X. Jia, J. H. Siegle, S. R. Olsen, et al. Systematic integration of structural and functional data into multi-scale models of mouse primary visual cortex. Neuron, 106(3):388-40) 2020.
J. Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milng and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URLhttp:: http::ttttt:hta com/ googloooleeaaaa
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and Zaremba. Openai gym, 2016.
G. Chen, F. Scherr, and W. Maass. A data-based large-scale model for primary visual cortex enables brain-like robust and versatile visual processing. Science Advances, 8, 2022.
K. Cho, B. van Merrienboer, C. Gulcehre, D). Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation, 2014.
E. Conti, V. Madhavan, F. P. Such, J. Lehman, K. O. Stanley, and J. Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents, 2018.
M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al. Loihi: neuromorphic manycore processor with on-chip learning. leee Micro, 38 (1):82-99, 2018.
U. Dichl and M. Cook. Unsupervised learning of digit recognition using spike-timing -eminn-e plasticity. Frontiers in computational neuroscience, 9:99, 2015.
J. Ding, J. Zhang. Z. Yu, and T. Huang. Accelerating training of deep spiking neural networks with parameter initialization, 2022. URL
S. K. Esser, P. Merolla, J. V. Arthur, A. S. Cassidy, R. Appuswamy, A. Andreopoulos, J. Berg, J. L. McKinstry, T. Melano, D. Barch, C. di Nolfo, P. Datta, A. Amir, B. Taba, M. Flickner, and D. S. Modha. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences, 11:: 1441 11446, 2016.
Fischer, A. Gadhikar, and Burkholz. Lottery tickets with nonzero biases, 2022.
Frankle and Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.0080 2018

---

C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax differentiable physics engine for large scale rigid body simulation, 2021. | com/ google/ brax
A. Gaier and D. Ha. Weight agnostic neural networks. Advances in neural information processing systems, 32, 2019.
A. Haber, A. Wanner, R. W. Friedrich, and E. Schneidman. The structure and function of neural connectomes are shaped by small number of design principles. bioRxiv, pages 2023-03, 2023,
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9:1735-80, 12 1997. doi: 10.1162/neco..1/9
H. Jaeger. Thhoecho state"approace to analysing and training recurrent neural networks. 2001.
S. R. Kheradpisheh, M. Ganjtabesh, S. J. Thorpe, and T. Masquelier. Stdp-based spiking deep convolutional neural networks for object recognition. Neural Networks, 99:56 67.56-77-55 2018.
J. Lehman, J. Chen, J. Clune, and K. O. Stanley. Es more than just a traditional finite-difference approximator, 2018.
A. S. Lele, Y. Fang, J. Ting, and A. Raychowdhury. Learning to walk: Spike based reinforcement learning for hexapod robot central pattern generation. 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pages 208-212, 2020.
G. Li, L. Deng, H. Tang, G. Pan, Y. Tian, K. Roy, and W. Maass. Brain Inspired Com- puting: Systematic Survey and Future Trends. 1 2023. doi: 10.36227/tecrrxiv 21837027.v1. URL htLphtts://wwwt techrxiv org/ wwtpccceepppppprttee.. Barain_Inspirdd S.saeesaatysyssssaaa... Satuurveyaan_Futttee http://aaaa..uuurreeeeeeeee////e
Y. Li, Y. Guo, Zhang, S. Deng, Y. Hai, and S. Gu. Differentiable spike: Rethinking gradient- descent for training spiking neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems. volume 34, pages 23426 23439. Curran Associates, Inc., 2021. http///ttttp:ppprreeeiing..uuu wwwrppss..........ee..........e.
pdf
W. Maass, T. Natschläger, and H. Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14:2531-2560, 2002.
E. Malach, G. Yehudai. S. Shalev-Schwarte and Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6682-6686 1691. PMLR, 13-18 Jul 2020. URL https: /ttpoe///proceedings mlr p.t..prees//v1/// malach20a.htm1
C. Mayr, S. Hoeppner, and S. B. Furber. Spinnaker 2: A 10 million core processor system for brain simulation and machine learning. ArXiv, abs/191 1.02385, 2019.
M. Mozafari, M. Ganjtabesh, A. Cowzari-Dalini S. J. Thorpe, and T. Masquelier. Bio-inspired -iiiiii-mmeeennn plasticity in deep convolutional
digit recognition using swward-modulated networks. Pattern recognition, 94:87-95, 2019.
K. Naya, K. Kutsuzawa, D. Owaki, and M. Hayashibe. Spiking neural network discovers energy- efficient hexapod motion in deep reinforcement learning. IEEE Access, 1:55055-150034, 2021.
Pei, L. Deng, S. Song, M. Zhao, Y. Zhang, S. Wu, G. Wang, Z. Zou, Z. Wu. W. He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106-111 2019.
S.-S. Poil, R. Hardstone, H. D. Mansvelder, and K. Linkenkaer-Hansee Critical-state dynamics of avalanches and oscillations jointly emerge from balanced excitation/inhibiiionn neuronal networks. The Journal of Neuroscience, 32:9817 - 9823, 2012.
12

---

V. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What's hidden in randomly weighted neural network?, 2020.
E. Rueckert, D. Kappel, D. Tanneberg, D. Pecevski, and J. Peters. Recurrent spiking networks solve planning tasks. Scientificc reports, 601):1-10, 2016.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies scalable alternative to reinforcement learning, 2017.
M. Schmidt, R. Bakker, K. Shen, G. Bezgin. M. Diesmann, and S. J. van Albada. A multi-scale layer-resolved spiking network model of resting-staie satting-ttate dynamics in macaque visual cortical areas. PLOS Computational Biology, (4100)e100635) 2018.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
S. B. Shrestha and G. Orchard. Slayer: Spike layer error reassignment in time. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httpp::/pproceediingg www.iingnn.nenui..eeeeem www.nnurruucmmm wwtpuripssccpppppppeeee wwtpccppppppeeeeeeeee httppppr.iieeeepppp///////////// 22250000ab0s57667/eefff2222eeee2
pdf
F. P. Such, V. Madhavan, E. Conti, J. Lehman, K. O. Stanley, and J. Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning, 2018
G. Tang, N. Kumar, and K. P. Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 6090 0097, 2020. doi: 10.1109/RR0455433. 2020.939999099
J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 18(10):150-11560 1990.
D. Whitley, genetic algorithm tutorial. Statistics and computing, 4:65-85, 1994.
D. Wierstra, Schaul, T. Glasmachers, Y. Sun, Peters, and Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15(27):949-98)) 2014. URI.htp:://jml org/ htge/paapprs//////reesttaaaaa html
R. J. Williams. Simple statistical
algorithms for connectionist reinforcement leamning. Reinforcement learning. pages 5-32, 1992.
Y. Wu, L. Deng, G. Li, J. Zhu, and L. Shi. Spatio-temporal backpropagation for training high- performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
F. Zenke and S. Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 30(6):151411541 2018.
Zenke and T. P. Vogels. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks. Neural Computation, 33(4):899-925, 03 2021. ISSN 0899-7667. doi: 10. 1621/e/coooaaaoc a_03367 URL https:///oi. org/ /00 1162/neco._ 01367
X. Zhang. J. Clune, and K. O. Stanley. On the relationship between the openai evolution strategy and stochastic gradient descent, 2017."
H. Zhou, J. Lan, R. Liu, and J. Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. CoRR, abs//195.00167, 2019. URL http:/::/aaai..o////aaaaaaa http:/abb/1955000095
13

---

A
Acknowledgement
This work was supported by the National Key Research and Development Program of China (grant 20212D0200301).
B Comparing EC and NES
EC: discrete 1-bit connection samples
NES: continuous parameter samples
Samples
1.00
Param j
Y=0 0
0.75
Param a
0.50
0.25
Y=1
N
N = 43
0.00
x = 0
==1
0.00
0.25
0.50
0.75
1.00
X
Figure S1: Comparison of search spaces in EC and NES with N 100 samples and two parameters (X, Y). EC samples from a discrete 1-bit search space, while NES samples from a continuous search space.
For readers familiar with evolutionary algorithms, we present the similarities and main differences between our Evolving Connectivity (EC) method and commonly used evolutionary algorithms, namely Natural Evolution Strategies (NES, Wierstra et al.] [2014) and Evolution Strategies (ES, Salimans et al. (2017). This comparison will facilitate better understanding and implementation of these algorithms. EC, ES, and NES share the same unbiased Monte Carlo gradient estimator, gradient update approach, and fitness shaping trick, as described in Section
The primary distinction between these algorithms lies in their search spaces, as illustrated in Figure ST Both ES and NES employ vallve parameters, akin conventional deep learning techniques, and sample the search space using parameterized normal distributions. In contrast, EC utilizes a 1-bit discrete search space parameterized by Bernuuli distributions. This 1-bit formulation reduces the parameter size, endowing EC with unique advantages such faster computation, improved memory efficiency, and reduced energy and cost requirements.
Additionally, there is a minor difference how EC enhances exploration. As described in Section EC clips the elements of connection probability matrix to boost exploration.
C Optimizing 1-bit Networks Using ES and SG
ES and SG continuous optimization techniques designed for continuous parameters. To apply these methods to the training I-bit connection RSNNs proposed in this paper, we utilize discretization methods as described by Zhou et al. [2019]. Specifically, continuous parameter 0, which is then discretized into 1-bit weights W by applying threshold at W (θ)), where H denotes the Heaviside step function. For SG, the tarightt-hrruuhhh estimator utilized as a standard approach [Bengio et al.) 2013). is crucial to acknowledge that employing continuous optimization methods in conjunction with discretization may result in biased gradient estimation, while EC naturally provides unbiased gradient estimation for discrete 1-bit connections.
employ ES and SG to optimize
D On Chip Power Consumption Estimation
We estimate the power consumption of ECC by considering the number of spiking neuron operations and the energy per operation provided by Loihi [Davies et al. 2018), representative neuromorphic
14

---

Table 2: Energy consumpution and network specifics
Parameter
Value
Energy per synaptic spike op P.
23.6 (pJ)
Within-tile spike energy Pa
1.7 (pJ)
Energy per neuron update Pa
(pJ)
Generations G # Population P
1000
10240
Time steps S # Neurons N
33200
256
Spikes per neuron per step R
0.025
Connection per neuron C
128
# Update operations per neuron I
Initially, we computed the estimated energy consumption for single network inference as follows:
device. The data employed for this estimation is presented in Table 2
Eone Pu I
C P.)) R S 2.8mJ
Subsequently, we determined the total energy consumption during the training process:
Ex
Eone G P = 28kJ
is important to note that these calculations are approximations and serve as an initial assessment. In future research, we plan to conduct experiments using neuromorphic chips to empirically validate the energy efficiency RSNNs.
Performance Comparison with PPO
In order to assess the efficacy of the EC-RSNN approach in relation to contemporary deep reinforce- ment learning (RL) techniques, we conducted additional training with GRU and LSTT models using Proximal Policy Optimization (PPO) (Schulman et al. [201]] on the most complex Humanoid task. The results these experiments are presented Figure S3 The data clearly indicates that EC-RSNN surpasses the performance of PPO-LSTM, illustrating a level of performance that is on par with state-of-the-are deep RI. methods.
F
Full SG Results
We present comprehensive analysis of the Surrogate Gradient (SG) results, considering various damping factors (7) and the parameter (3). The outcomes are illustrated in Figure S2
G
Detailed Experiment Settings
G.1 Hardware
All experiments were conducted on single GPU server with the following specifications. Addition- ally, all efficiency experiments, including those that measure wall-clock time, were executed on single GPU within this server.
8x NVIDIA Titan RTX GPU (24GB VRAM)
2x Intel(R) Xeon(R) Silver 41 CPU
252GB Memory
G.2 Hyperparameters
15

---

Humanoid (17-DuF), PPO
0000
In this section, we provide the hyperparameters for our frame- work, Evolving Connectivity (EC), and the baselines, including Evolution Strategies (ES) and Surrogate Gradient (SG). All meth- ods utilize the same hyperparameter set for three locomotion tasks. We also list the settings of neural networks below.
PPO-GRU (PP32)
PIG-STMM (11532)
BC-RENN (1863)
steps
Figure S3: Performance com-
parison among PPO-GRU, PPO- LSTM, and EC-RSNN.
Table 3: Hyperparameters for Evolving Connectivity (EC)
Hyperparameter
Value
Population size
10240
Learning rate
0.15
Exploration probability e
10
Table 4: Hyperparameters for Evolution Strategies
Hyperparameter
Value
Target
Population size N
10240
All
Learning rate 1
0.15
RSNN
0.01
Deep RNN
Noise standard deviation
0.3
RSNN
0.02
Deep RNN
Weight decay
0.1
All
Humanoid (17-DoF)
SuperSpile:: M(x) (0lN DF
Easer et al.: A(z) m MH)
o': Mx)
1900
400
1018 10
0000
1000
7
0.5,
1000
U.S.P 2
0.5.
0.8. 10
0.S,
Gradlont
Figure $2: Surrogate Gradient on the Humanoid task, with several surrogate functions, 0 and 7 parameters. Performance is sensitive to the chosen surrogate function and its parameters.
16

---

Table 5: Hyperparameters for Surrogate Gradient
Hyperparameter
Value
Proximal Policy Optimization (PPO)
Batch size
2048
BPTT length
16
Learning rate
3 x 10-1
Clip gradient norm
0.5
Discount 7
0.99
GAE À
0.95
PPO clip
0,2
Value loss coefficient
1.0
Entropy coefficient
10
Surrogate Gradient
Surrogate function
SuperSpike 10
Surrogate function parameter ,
Table 6: Settings of neural networks
Hyperparameter
Value
Recurrent Spiking Neural Network (RSNN)
Number of neurons da
256
Excitatory ratio
50%
Simulation time per environment step
16.6 ms
Simulation timestep At Synaptic time constant Tsyn Membrane time constant Tm Output time constant Tout
0.5 ms
5.0 ms
10.0 ms
10.0 ms
Input membrane resistance Rin
0.1 Tm
dm
Hidden membrane resistance
Ra
1.0
Ti
Tagn
Output membrane resistance Rout
5.0 Tout
dn
Gated Recurrent Unit (GRU)
Hidden size
256
Long-short term memory (LSTM)
Hidden size
128
H Source Code and Licensing
The source code associated with this paper is licensed un- der the Apache 2.0 License and is publicly available at https://gi thgb. com/ httm/imoneoi/Ben
In our research, we incorporated lo-
comotion tasks and the Brax physics simulator from Freeman et al.) (202T). as well as the JAX framework (Bradbury et al. 2018). Both of these resources are also released under the Apache 2.0 License.
Resistance is set following
Ding et al. 2022 preserve variance.
17


Reviews:
[{"review_id": "WsTJ6BsjWU", "reviewer": "Reviewer_Apb4", "summary": "The authors present a new evolutionary algorithm, evolving connectivity (EC) to train recurrent spiking neural networks (RSNN). The algorithm alleviates the gradient estimation problem of surrogate gradient-based (SG) training methods which are often hard to implement in neuromorphic hardware. Further, by focusing the algorithm only on evolving network connectivity instead of the weight magnitude, EC can reduce all network weights down to 1bit {0,1} alternatives making it further hardware friendly. The paper also highlights the superior accuracy and performance of their training algorithm for a few robotics tasks involving sequential decision making against state-of-the-art Surrogate Gradient and Evolution Strategies based algorithms.", "questions": "1.\tDescribe the RSNN architecture and how it compares to the RNNs used in Experiments in terms of number of layers/params/model size/precision? \n2.\tExperiment with 1-bit RSNN utilizing ES/SG algorithms for training to understand the impact on accuracy and performance?\n3.\tThe combination of ES and RSNN performs notably well on all the benchmarks tested, but it would be good to understand what part of it comes from the network architecture (try another architecture), what part comes from bit precision and lastly what is contributed by the training algorithm? This kind of study will help strengthen the credibility of EC algorithm for other benchmarks and network architectures. ", "limitations": "\nThe authors provide a useful discussion on the memory footprint of the different training algorithms discussed. Please address the concerns related to network architecture and precision impacting accuracy and performance as discussed in weakness section. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present a strong argument for developing RSNNs which focus more on the connectivity probability between layers instead of selecting specific weight magnitudes. This enables the network to implicitly fit many samples without overfitting to any specific subset of data. \n\nThe paper explains the EC algorithm clearly and highlights justifications for its superiority in performance and accuracy compared to other algorithms (ES/SG) for targeted robotics benchmarks. \n\nRemoving the gradient estimation requirement and reducing the weight matrix to 1-bit are significantly useful optimizations that make the algorithm hardware friendly. \n", "weaknesses": "The paper doesn’t clearly describe the RSNN architecture that is adopted for the experiments discussed and how it compares in terms of number of neurons/connections to the Deep RNNs that are used as baseline for performance/accuracy. This makes the comparisons to the ES-RNN results difficult to understand for me. \n\nFurther, comparing the RSNN based experiments only (ES/EC/SG), it appears that only EC algorithm is 1-bit owing to the focus on connectivity only. But this makes me wonder if 1-bit networks applying ES/SG algorithms could compete against EC approach in terms of performance. I understand that by the authors’ assertion that connectivity is more critical than assigning wide ranging weights to a fixed set of connections. However, for experiment’s sake I am curious to understand that if ES/SG algorithms also had the HW friendly assumption of only creating a binary SNN then would their accuracy/performance improve or deteriorate?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1]** *Describe the RSNN architecture and how it compares to the RNNs used in Experiments in terms of number of layers/params/model size/precision?*\n\n**[Response]** \n\nThanks for your valuable question.\nThe table below outlines the similarities and differences between RSNN and baseline models in terms of the number of layers, hidden size, number of parameters, precision, and model size.\nTo ensure a fair comparison, our RSNN (EC) architecture has a similar input-hidden-output structure and number of parameters to the RNNs used in the experiments.\nThe primary difference lies in the precision, with RSNN (EC) utilizing 1-bit precision, while other baseline models employ FP32 precision.\nConsequently, RSNN (EC) has a smaller model size.\nIn our final version, we will include this table in Section 6.1 and further elaborate on the architecture information.\n\n*Table 1: Comparison of Model Architectures*\n| Model         | Hidden size | # Hidden Layers | # Params | Precision | Size (KB) |\n|---------------|-------------|-----------------|----------|-----------|-----------|\n| RSNN (EC)     | 256         | 1               | 193K     | 1-bit     | 24        |\n| RSNN (Others) | 256         | 1               | 193K     | FP32      | 768       |\n| GRU           | 256         | 1               | 386K     | FP32      | 1544      |\n| LSTM          | 128         | 1               | 191K     | FP32      | 764       |\n\n----\n**[Q2]** *Experiment with 1-bit RSNN utilizing ES/SG algorithms for training to understand the impact on accuracy and performance?*\n\n**[Response]** \n\nThanks for your advice. Despite ES and SG being continuous optimization methods built for continuous parameters, we attempted to discretize ES and SG for training 1-bit connection RSNNs proposed in our paper. \nSpecifically, we adopt ES and SG to optimize a continuous parameter $\\theta$ and discretize them to 1-bit weights $\\textbf{W}$. \nWe use threshold at 0 as $\\textbf{W}=H({\\theta})$, where $H$ is the Heaviside step function. For SG, we additionally used the straight-through estimator as a common practice (Bengio et al. 2013). \nPlease note that these approximations and discretization may result in biased gradients for the 1-bit discrete optimization.\n\nThe results in Table 2 show that ES-RSNN (1-bit) and SG-RSNN (1-bit) exhibit learning progress but still remain a significant gap to EC-RSNN (1-bit). Besides, ES and SG performed better on continuous FP32 weights compared to discrete 1-bit. This suggests that continuous optimization methods excel with continuous parameters. \nFor 1-bit discrete connections, EC should be adopted as it is specifically designed for discrete 1-bit optimization and provides unbiased gradients. Detailed results can be found in Fig. S3 in the supplementary PDF.\nWe will add these experiment results in Section 6.\n\n*Table 2: Comparison of Model Architectures*\n| Algorithm | Precision | Return |\n|-----------|-----------|--------|\n| EC-RSNN   | 1-bit     | 13808  |\n| ES-RSNN   | 1-bit     | 10240  |\n| SG-RSNN   | 1-bit     | 6067   |\n| ES-RSNN   | FP32      | 11264  |\n| SG-RSNN   | FP32      | 11505  |\n\n----\n**[Q3]** *Network architecture and precision impact accuracy and performance.*\n\n**[Response]** \n\nThanks for your suggestion. To validate the EC on more network architectures, we conducted experiments using a vanilla RNN trained with EC, meanwhile using ES and PPO as a baseline. The results can be seen in Table 3 and Figure S2 in the supplementary PDF. It demonstrates that EC can effectively train 1-bit deep recurrent neural networks and has the potential for different architectures and quantized neural networks.\n\n*Table 3: Vanilla RNN*\n| Algorithm | Precision | Return |\n|-----------|-----------|--------|\n| EC-RNN    | 1-bit     | 12455  |\n| ES-RNN    | FP32      | 11042  |\n| PPO-RNN   | FP32      | 11133  |\n\nRegarding the impact of precision, Table 2 highlights that ES and SG, not specifically designed for 1-bit precision, encounter optimization challenges, which negatively affect performance. In contrast, EC, designed to optimize 1-bit connections, is not hindered by these challenges and outperforms the RSNN baselines. This evidence supports the robustness and adaptability of the EC approach across varying network architectures and precision levels."}, {"review_id": "Rv5l7KAf9n", "reviewer": "Reviewer_ho2W", "summary": "The paper describes an evolutionary training algorithm to optimize the binary weight matrix of a recurrent spiking neural network. The optimization algorithm is derived using natural evolutionary strategies assuming that the weights follow a Bernoulli distribution with parameter $\\rho$. The resulting weight update is remarkably simple, and despite that, it seems to work very well.\n\nThe algorithm is tested on the classical locomotion tasks of simulated robotics, and the performance is reported in terms of task return and time to solution in wall clock time. On all the task RSNN model trained in this way performs better or similar to the LSTM or GRU baselines trained with ES, and it is always better than the spiking networks trained with surrogate gradient. By leveraging the discrete values of the weight matrix, the implementation is also faster than a floating point evolutionary strategy.", "questions": "I realized that points 1a.b and 2 above would be best addressed with extra simulations. Given the time and effort that it requires, I do not consider those simulations necessary, but I would encourage the authors to address those questions verbally during the rebuttal and consider the feasibility and the cost/benefits of these potential simulations at some point.\n\nFor points 3, 4 and 5. I would encourage the authors to do minor edits.\n\nMore general question: Do the authors believe that this approach can replace gradient descent learning in some other context ? Or is it that ES and EC and bounded to make only sense in this type of toy reinforcement leanring task where the network size is small and the batch size can be super large?\n\n", "limitations": "I see no potential negative impact. The limitations are clearly stated in the paper.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "As said in the paper: inference-only training solutions can be very useful for hardware where the training algorithm cannot be implemented easily. This approach is likely to have important for training or fine-tuning architectures when deployed on unconventional devices.\n\n I would also highlight an important achievement that the authors did not mention enough in my opinion: it appears that this training algorithm is better than all existing training algorithms for spiking networks which is remarkable for something that is practicable and competitive with deep learning solutions to such problems. It could in fact provide opportunities for quantized and spiking architectures which are otherwise bounded in their performance by the approximate gradients that are usually computed with surrogate gradient and straight through. This is solved here since the gradients are unbiased (not mentioned by the authors as far as I see). This was probably why surrogate gradient baselines do not quite reach the performance of LSTM and GRU baselines as seen here as in previous papers, but the algorithm provides a solution to this problem.", "weaknesses": "1) There are a few data points that would be useful to fully evaluate the quality of the training algorithm and separate the benefits of the spiking architecture from the benefits of the algorithm.\na. Is it possible to train an LSTM or a GRU with EC? What is its return after training, is that also better than ES for the spiking neurons?\nb. What is the return obtained with LSTM / GRU and PPO? This is both important to verify that PPO is well-implemented and claim or not that the EC-RSNN is competitive with the regular deep learning approach. (If the performance is low, then maybe there is a problem with the surrogate implementation too).\n\n2) I think it would be valuable to clarify that most surrogate gradient techniques were typically tested mainly without recurrent connections (Slayer, super-spike for instance). I find it admirable that the authors tested so many baselines already, but also unfortunate that they might have missed important details that were reported the be crucial in the presence of recurrent connections: depending on the weight initialization it was shown that the pseudo derivate should be multiplied by a dampening factor < 1 to avoid the accumulation of approximation errors, with Glorot initialization you might be fine without this (see Celotti and Rouat 2022). But what initialization did you use and did you test this?\n\n3) I realized only later that most of the concrete hyperparameters are detailed in the appendix: How many layers, units per layer, batch size, epsilon. Please state clearly in the main text that all this important is available in the appendix when it's relevant. I find it in general even better to put directly these numbers in the main text when possible. (For instance, write $\\epsilon = 0.001$ instead of $\\epsilon \\rightarrow 0$). Same remark when talking about the hardware. Instead of writing \"GPGPU\" or \"identitical hardware\" which I find vague since neuromorphic hardware is often mentioned in the main text. When possible, I would encourage writing just NVIDIA GPU and pointing out the appendix where the hardware reference is provided. On a related note, I would also encourage the authors to prefer a specific reference to the Figure number and panel at line 277 when commenting on their results.\n\n4) In the context of this conference which is broadly addressed to the machine learning community, I think it is important that the surrogate gradient is nothing more than a variant of the straight-through gradient estimator from Bengio 2013 which is much better known outside of the neuromorphic community and was discovered before.\n\n5) I find the first sentence line 20 a bit strange, and I find it weird to reference Pei et al. at this point. AGI is also never used elsewhere in the text (for good reasons since you have more concrete and good results to discuss), so why defining this acronym here ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[S1]** *The gradients of the EC training algorithm are unbiased.*\n\n**[Response]** \n\nThank you for your affirmation and suggestion. \nWe have also pointed out that \"the surrogate gradient leads to inherent inaccuracy in the descent direction\" in line 37.\nWe will further emphasize the contribution of our unbiased gradient property in the introduction of our revised manuscript.\n\n---\n\n**[W1-a]** *Is it possible to train an LSTM or a GRU with EC? What is its return after training? Is that also better than ES for spiking neurons?*\n\n**[Response]**\n\nThank you for your insightful question. While training an LSTM or GRU with EC is theoretically possible, it requires scaling and tweaking the gating mechanism to adapt to 1-bit weights (Xuan Liu et al., 2018). For simplicity, we conduct experiments using a vanilla RNN trained with EC. \n\nAs a result, we obtained 12,455 returns with EC-RNN, compared to 11,042 returns with ES-RNN and 11,133 returns with PPO-RNN. Detailed results can be found in Fig. S3 in the supplementary PDF.\nThe results demonstrate that EC can also train 1-bit deep recurrent neural networks, showcasing its potential for different architectures and quantized neural networks. We will add these experiment results in Section 6.\n\n---\n\n**[W1-b]** *What is the return obtained with LSTM / GRU and PPO?*\n\n**[Response]** Thank you for your question. \n\nWe train PPO-LSTM and PPO-GRU with the 12,960 and 14,312 average final return, while Brax (Freeman et al., 2021) report a PPO return of approximately 11,300. \nAs a result, our EC-RSNN achieves a comparable final return (i.e., 13,808) with PPO, demonstrating competitive performance with deep reinforcement learning. More detailed results can be found in Fig. S4 of the Supplemental PDF.\nWe will add these experiment results in Section 6.\n\n---\n\n**[W2]** *Miss important details that were reported the be crucial in the presence of recurrent connections: Depending on the weight initialization it was shown that the pseudo derivate should be multiplied by a dampening factor $< 1$ to avoid the accumulation of approximation errors. But what initialization did you use and did you test this?*\n\n**[Response]** Thanks for your advice. We conduct further hyperparameter tuning for our SG-RSNN baseline, introducing a dampening factor equal to $0.8$. Our experiment shows a similar maximum performance (i.e., 11,765) with the SG-RSNN baseline (i.e., 11,546) from our paper. The detailed results can be seen in Fig. S5 in the supplementary PDF. We will replace this experiment setup in Section 6.\n\nIn terms of initialization, all parameters are initialized to $0.5$, as a balance point between connection and no connection in the Bernoulli distribution. For a fair initialization, we moved the $1/\\sqrt N$ term in LeCun Normal initialization to a product of hyperparameters, as shown in Table 1 below. We will add the initialization in Section 4 in our revised paper.\n\n*Table 1: Initialization*\n\n| Hyperparameter                     | Value                                          |\n|------------------------------------|------------------------------------------------|\n| Input membrane resistance $R_{in}$   | $0.1 * \\tau_m * \\sqrt{2/d_{in}}$                   |\n| Hidden membrane resistance $R_h$   | $1.0 * \\tau_m / \\tau_{syn} * \\sqrt{2/d_{h}}$          |\n| Output membrane resistance $R_{out}$ | $5.0 * \\tau_{out} * \\sqrt{2/d_{h}}$                  |\n\n---\n\n**[Q2]** *For weaknesses points 3, 4 and 5. I would encourage the authors to do minor edits.*\n\n**[Response]** Thank you for your valuable suggestions. In response to [W3] and [W4], we will move the main information about hyperparameters and hardware to the main text.\nMeanwhile, we will further polish our statements to be more clear.\n\nAs for [W5], we recognize that the reference to AGI in line 20 may seem out of place. In our revised manuscript, we will provide a more appropriate introduction to neuromorphic computing, ensuring a clear and concise presentation of the topic.\n\n---\n\n**[Q3]** *Do the authors believe that this approach can replace gradient descent learning in some other context? Or is it that ES and EC and bounded to make only sense in this type of toy reinforcement learning task where the network size is small and the batch size can be super large?*\n\n**[Response]**\n\nThanks for posing this thought-provoking question. \n\nFirstly, humanoid locomotion is a challenging task in reinforcement learning (RL), as it has high dimensional observation and action space (Duan et al., 2016), and poses a challenge to deep RL algorithms (Haarnoja et al., 2018).\nExtending beyond locomotion tasks, we believe that the EC framework has the potential to be useful in other contexts, such as image classification (Xingwen Zhang et al., 2017) and game playing (Salimans et al., 2017), where evolutionary algorithms have demonstrated success.\n\nSecondly, under large batch size conditions, our EC is much more efficient than ES and SGD.\nOn the one hand, our EC can be efficiently distributed over multiple devices by sending random seeds, while SGD requires computing an average of the gradient of all batches incurring significant communication costs.\nOn the other hand, the 1-bit connections in the EC framework may reduce memory and computation costs to 1/32 compared to the FP32 commonly used by both ES and SGD.\nMoreover, EC requires only a forward pass, while SGD needs both a forward and backward pass.\nNamely, the 10,240 batch size in EC is analogous to $\\frac{10240}{2 \\times 32} = 160$ in SGD in memory usage, which is comparable to commonly-used SGD batch sizes such as 128 and 256.\n\nIn summary, we believe that the EC framework holds promise for a broader range of applications and is capable of training larger networks with larger batches.\nWe will explore different tasks and the efficiency of the EC framework in future work."}, {"review_id": "qMf5TDbLOU", "reviewer": "Reviewer_SW36", "summary": "Facing on the inaccurate and unfriendly limitation of current surrogate gradient-based learning methods for recurrent spiking neural networks (RSNN), this study develops the evolving connectivity (EC) framework for inference-only training. The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions. The performance of the proposed EC is evaluated on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs.", "questions": "1.\tThe novelty of the weight-based parameterization method and the NES methods in EC framework should be explained further.\n2.\tThe energy consumption should be computed to show the efficiency of the proposed framework.\n3.\tHow about the performance comparison with Transformer models?\n", "limitations": "See the weakness.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation is reasonable and the application scene is interesting.\n2.\tThe framework of EC considering the weight reparameterization and connection evolution is reasonable.\n", "weaknesses": "1. It seems that the weight-based parameterization method and the NES method used in EC is not novel, it seems to use the method proposed in other papers.\n2. The energy consumption should be computed to show the efficiency of the proposed framework.\n3. The experiments seems not enough to verify the effectiveness of the proposed model.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1]** *The novelty of the weight-based parameterization method and the NES methods in EC framework should be explained further.*\n\n**[Response]** \n\nThank you for your advice.\nOur work introduces a novel approach to the NES framework by utilizing a 1-bit discrete search space formulation, which distinguishes it from the original NES paper (xNES, SNES, Wierstra et. al) and its derivatives like ES (Salimans et. al). Prior studies employed continuous search spaces parameterized by normal distributions to train real-valued parameters in neural networks, as seen in conventional deep learning.\n\nOur proposed method searches for discrete 1-bit connection-based parameterization within the NES framework, deriving a natural gradient estimator tailored for this discrete space. This approach offers several advantages over the continuous parameter search using ES. As demonstrated in our paper's experiments, our EC method yields superior final performance and accelerates the training process by 2 to 3 times. Additionally, the use of integer arithmetic for 1-bit values, as opposed to floating-point arithmetic for continuous values, reduces computational costs and enhances compatibility with neuromorphic hardware.\n\n---\n\n**[Q2]** *The energy consumption should be computed to show the efficiency of the proposed framework.*\n\n**[Response]**\n\nThank you for your valuable suggestion. In our study, all experiments are conducted on the same NVIDIA Titan RTX GPU, operating at a consistent 100% GPU power (280W). As a result, the energy consumption of the training process is estimated proportional to the computation wall time as shown in Table 1. Our EC framework demonstrates an energy improvement over SG and even more than 2 times to ES. We will incorporate energy consumption calculations in Section 6 to further demonstrate the efficiency.\n\n*Table 1: Estimated training power comsumption*\n| Model   | Run time (h) | Power Consumption (KWh) |\n|---------|--------------|-------------------------|\n| EC-RSNN | 18           | 5.0                     |\n| ES-RSNN | 46           | 12.9                    |\n| SG-RSNN | 20           | 5.6                     |\n\n---\n\n**[Q3]** *How about the performance comparison with Transformer models?*\n\n**[Response]**\n\nThank you for your question. Although Transformer models have demonstrated remarkable performance in numerous domains, their direct application to online reinforcement learning (RL) presents challenges. Specifically, conventional Transformer models encounter stability issues in RL tasks and require modifications for stabilization (Emilio Parisotto et al. 2019). We appreciate your suggestion and consider it a promising avenue for future research."}, {"review_id": "SqmCr5Rabu", "reviewer": "Reviewer_mDuJ", "summary": "This study presents an application of a previously developed approach, NES, to train RSNNs formulated based on connection probabilities. However, there are concerns regarding the ethical aspect of this work, specifically the reproducibility and proper citation of related works (see Weaknesses). Due to these reasons, it is challenging to recommend this manuscript for acceptance in its current form.", "questions": "n/a", "limitations": "See weaknesses", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "n/a\n", "weaknesses": "o\tThe lack of code availability at the time of review poses a significant barrier to verifying the claims made by the authors. \n\no\tAdditionally, it appears that an existing work by Stockl et al., 2021 from Maass group, which applies NES to learn RSNNs formulated in terms of probability skeleton inspired from Billeh et al., is not cited in this manuscript. This omission warrants clarification as it may influence the perceived contribution of this work.\n\no\tBeyond the ethical issues outlined above, this paper has several limitations that need to be addressed. Firstly, the results presented are purely empirical and there is a lack of theoretical contributions. Secondly, the manuscript lacks an analysis of how varying the sample size impacts the results, which leaves unanswered questions regarding the robustness and efficiency of the proposed methodology. Lastly, it is unclear whether the authors have performed a thorough tuning of the hyperparameters, which could significantly impact the performance of the benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Code availability**\n\n**[Response]**\n\nThanks for your advice, but we cannot agree on code availability as the main reason to reject. \nAccording to the conference policy in Call for Papers, NeurIPS strongly **encourage** accompanying code and data to be submitted with **accepted papers** when appropriate, rather than a mandatory requirement.\nAnd, we have already promised in Appendix A that we will release the code upon the paper is accepted.\nHowever, to address your concern, we are glad to send a copy of the code to the Area Chair.\n\n---\n\n**[W2] An similar existing work by Stockl et al., 2021 using NES**\n\n**[Response]**\n\nWe appreciate your attention to the existing work by Stockl et al., 2021, which applies NES to learn RSNNs formulated in terms of probability skeleton inspired by Billeh et al. We acknowledge the relevance of this work and will include it in our revised manuscript.\n\nHowever, our EC framework has several fundamental differences from the approach taken by Stockl et al., 2021 and NES, which we detail below:\n\n1. **Hyperparameter searching vs. parameter training**. Stockl et al. search tens to hundreds of free parameters to characterize the connection skeleton, akin to hyperparameter optimization in deep learning and HyperNEAT in evolutionary computation. In contrast, our EC framework operates more like training 1-bit neural networks in deep learning, searching every one-to-one connection parameter in the RSNN, and is agnostic to network architecture. In our experiments, we train the full 193K RSNN connections using EC, which represents more than $1000\\times$ the number of dimensions than the 44 to 164 hyperparameters for the probabilistic model explored by Stockl et al.\n\n2. **Discrete vs. continuous**. Our work presents a novel approach to the NES framework by utilizing a 1-bit discrete search space formulation, which distinguishes it from the original NES paper (xNES, SNES, Wierstra et al., 2011) and its derivatives like ES (Salimans et al., 2017). These prior works employed continuous search spaces parameterized by normal distributions to train real-valued parameters in neural networks, as seen in conventional deep learning.\n\nOur EC framework also offers several unique advantages:\n\n1. **High performance**. Our results demonstrate that EC outperforms ES on training RSNNs, despite EC-RSNN using 1-bit discrete parameters, which are 1/32 the size of the 32-bit floating point precision employed by ES-RSNN.\n\n2. **Faster training & inference**. As discussed in our paper, the 1-bit connections resulting from the EC framework offer potential for accelerated training and inference. In Fig. 5, we show that EC-RSNN exhibits $2 \\sim 3\\times$ efficiency compared to ES-RSNN.\n\n3. **Scaling to complex tasks**. EC's ability to efficiently search the full 193K RSNN parameter space enables it to tackle complex tasks that require a larger number of network parameters. The Humanoid task is a challenging locomotion task in reinforcement learning community, typically addressed by neural networks with parameter scales ranging from 10K to 100K, as demonstrated by Salimans et al. and Freeman et al. Our EC framework can solve this task with performance comparable to deep RNNs.\n\n---\n\n**[W3-a] Lack theoretical contributions**\n\n**[Response]**\n\nThe focus of our manuscript is on the development and evaluation of the evolving connectivity (EC) framework for training RSNNs, emphasizing its hardware-friendly characteristics and performance. \nWe do not consider it as a requirement to add theory to empirical experiments. \nIt would be helpful if you can provide some detailed theoretical insights.\n\n---\n\n**[W3-b] Impact of sample size on robustness and efficiency**\n\n**[Response]**\n\nIn response to your concern regarding the robustness and efficiency of our proposed methodology, we have taken the following steps to address these concerns:\n\n1. As shown in our paper, to evaluate the robustness of our method, we performed multiple trials (n=3) with different random seeds for each experiment. By reporting the average and standard deviation of the results, we demonstrate the consistency of our approach across various initial conditions.\n\n2. To assess the generalization of our methodology, we conducted experiments using 1-bit deep RNN architectures, which further supports the adaptability of our approach to other network configurations.\n\n3. In terms of efficiency, we have compared our method's performance with existing methods by measuring the wall-clock time, providing evidence for the improved efficiency of our approach.\n\nWe acknowledge the importance of analyzing the impact of sample size on the results; However, we believe our current analysis sufficiently demonstrates the robustness and efficiency of our proposed methodology. \n\n---\n\n**[W3-c] Hyperparameters tuning and benchmark**\n\n**[Response]**\n\nWe appreciate your concern about the thoroughness of hyperparameter tuning, as it is crucial for the performance of the benchmarks. We assure that we have conducted an extensive tuning of the hyperparameters for all baseline models.\n\n1. The PPO implementation in Brax (Freeman et al., 2021) obtains 11,300 for the Humanoid environment. Our baselines (SG-RSNN, ES-GRU, and ES-LSTM) yielded returns of 11,500, 13,000, and 15,000, respectively, outperforming the Brax benchmark.\n\n2. As detailed in our paper, we tested various surrogate function parameters for SG-RSNN and selected the best set as our baseline.\n\n3. We have conducted additional experiments, as in Supplement PDF Fig. S4 and S5, including PPO-LSTM, PPO-GRU, and a broader hyperparameter tuning on SG-RSNN, to confirm that our baseline is fairly constructed. \n\nOur rigorous tuning process contributed to the enhanced performance of the baselines, and we are confident in the thoroughness and effectiveness of our hyperparameter optimization.\n\n---\n\nThe detailed experimental settings and results can be found in the comment text and Supplemental PDF in the overall rebuttal."}]

Author rebuttal:
Thanks for all your suggestions. In light of them, we gathered commonly asked questions and ran a comprehensive set of additional experiments, as detailed below. The result figures are shown in the supplementary PDF. All our additional experiments are configured using the same settings and hyperparameters as in our paper and Appendix A, tested on the most complex Humanoid task (17-DoF) in Brax (Freeman et al., 2021), and averaged over 3 independent random seeds, with standard deviation shown as shaded areas.

1. **EC vs. NES?** The primary distinction is discrete vs. continuous, as demonstrated in Fig. S1. Our work presents a novel approach to the NES framework by utilizing a 1-bit discrete search space formulation, which distinguishes it from the original NES paper (xNES, SNES, Wierstra et al., 2011) and its derivatives like ES (Salimans et al., 2017). These prior works employed continuous search spaces parameterized by normal distributions to train real-valued parameters in neural networks, as seen in conventional deep learning.

2. **Can EC work on other 1-bit networks?** To verify EC's capabilities for training deep RNNs, we conducted experiments using a vanilla RNN trained with EC, meanwhile trained with ES and PPO as baselines. The RNN has 256 tanh units in the hidden layer. For 1-bit EC, the weight magnitudes are 0-1 connection matrix multiplied by LeCun initialization standard deviation, and the weight signs are determined by $+$ for excitatory, $-$ for inhibitory, with the first 128 neurons as excitatory and last 128 as inhibitory. For ES and PPO as baselines, we use real-valued weights with LeCun normal initialization. The results in Fig. S2 demonstrate that EC can effectively train 1-bit deep recurrent neural networks and has the potential for different architectures and quantized neural networks.

3. **Can ES or SGD/SG work on 1-bit RSNN?** Despite that ES and SG are continuous optimization methods built for continuous parameters, we attempted to discretize ES and SG for training 1-bit connection RSNNs proposed in our paper. Specifically, we adopted ES and SG to optimize a continuous parameter $\theta$ and discretize them to 1-bit weights $\textbf{W}$ by thresholding at 0 as $\textbf{W}=H(\theta)$, where $H$ is the Heaviside step function. For SG, we used the straight-through estimator as a common practice (Bengio et al., 2013). Please note that these approximations and discretization may result in biased gradients for the 1-bit discrete optimization.

    Fig. S3 (a) shows that ES-RSNN (1bit) and SG-RSNN (1bit) exhibited learning progress but were outperformed by EC-RSNN (1bit). Figs. S3 (b) and (c) indicate that ES and SG performed better on continuous FP32 weights compared to discrete 1-bit. This suggests that continuous optimization methods excel with continuous parameters, and for 1-bit discrete connections, EC should be adopted as it is specifically designed for discrete 1-bit optimization and provides unbiased gradients.

4. **Is PPO correctly implemented?** To ensure our PPO implementation's accuracy, we trained PPO-LSTM and PPO-GRU and compared with EC-RSNN (Figure S4). The Brax maintainers reported a PPO return of approximately 11,300. Our PPO implementation yielded higher results, with 12,960 (PPO-LSTM) and 14,312 (PPO-GRU) average final return, confirming its correctness and performance. Notably, EC-RSNN achieved a higher final return than PPO-LSTM, demonstrating competitive performance with deep reinforcement learning.

5. **Is SG correctly implemented?** We conducted additional experiments using different dampening factors to thoroughly test our surrogate gradient baseline, as demonstrated in Figure S5. With a dampening factor of $\gamma=0.8$, our experiment showed a similar maximum performance to the SG-RSNN baseline from our paper.

In conclusion, our additional experiments reinforce our findings that EC is an effective novel training framework for RSNNs and that our baselines are robustly constructed.

Meta-review: The authors propose an algorithm to optimize recurrent spiking neural networks (RSNNs) called evolving connectivity (EC), employing natural evolution strategies (NES) to optimize connectivity probabilities. Using this technique, weight values can be reduced to be binary. This avoids the use of surrogate gradient methods. It is evaluated on standard robotic locomotion tasks.

There were some concerns regarding the novelty of the work in the initial reviews. However, the authors could convince the reviewers that their approach provides a significant novel contribution compared to previous work. In general, the reviewers were convinced that the work presents a very interesting algorithm that is quite different form gradient-based approaches and shows excellent performance on the considered tasks. The potential benefit for neuromrophic implementations was also acknowledged.
Decision: Accept (poster)