SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Performance evaluation on locomotion tasks. SG-RSNN is plotted based on the final return, as gradient-based updates are not directly comparable to generations in evolution-based methods. The proposed Evolving Connectivity (EC) framework effectively solves the 17-DoF Humanoid locomotion task, demonstrating competitive performance with deep RNNs and outperforming RSNNs trained using both Surrogate Gradient and Evolution Strategies across all tasks.
calculations take advantage of INT8 arithmetic for performance optimization. As a result, the training process achieves over 180,000 frames per second on a single NVIDIA TITAN RTX GPU. Each experiment's result is averaged over 3 independent seeds, with the standard deviation displayed as a shaded area. For detailed information on hyperparameters and hardware specifications, please refer to Appendix G.
6.2 Performance Evaluation
Firstly, we compare RSNN trained with our EC framework (EC-RSNN) to deep RNNs. To ensure a fair comparison, we utilized the evolution-based training approach ES [Salimans et al., 2017] for deep RNNs, including ES-GRU and ES-LSTM. Typically, SNNs face challenges in surpassing the performance of their DNN counterparts. However, as shown in the experimental results in Fig. 3, our EC-RSNN achieves competitive performance and training efficiency compared to deep RNNs, even surpassing ES-GRU and ES-LSTM on the Walker2d and Hopper tasks. In the complex 17-DoF Humanoid task, our EC-RSNN could also outperform ES-GRU. Moreover, EC-RSNN demonstrates comparable performance to GRU and LSTM trained with PPO, as detailed in Appendix E. It is worth noting that, while GRU and LSTM use densely connected, full precision (32-bit floating-point) weights and hard-coded gating mechanisms, EC-RSNN is constrained on 1-bit sparse weights and adheres to Dale's law for weight signs.
Then, as for the same structured RSNNs, our EC-RSNN outperforms ES-RSNN, especially significantly surpassing ES-RSNN on complex Humanoid and Walker2d tasks. We highlight two aspects that potentially account for the superior performance of our EC framework. On the one hand, schema theory [Whitley, 1994] suggests that selecting over a population of binary strings could combine partial binary string patterns (schemata) to implicitly work on many solutions without evaluating them explicitly, leading to massive parallelism. EC leverages the discrete  $\{0,1\}$ -string space, which may enable more efficient optimization due to implicit parallelism. On the other hand, ES uses fixed-scale noise to perturb a set of deterministic parameters, which may often fall in wide areas in the landscape of the objective function and fail to proceed on sharp maxima [Lehman et al., 2018]. In contrast, EC operates over a probabilistic distribution, providing the flexibility to adjust variance on sensitive parameters and achieving more fine-grained optimization.
Finally, gradient updates cannot be directly converted into evolution generations, thus we only compare the final converged return for surrogate gradient trained RSNNs. Considering that the surrogate gradient approach is sensitive to the selection of the surrogate function and its parameters [Zenke and Vogels, 2021], we choose three commonly-used surrogate functions with different  $\beta$  and  $\gamma$  parameters to thoroughly validate the performance: SuperSpike [Zenke and Ganguli, 2018], the piecewise linear function proposed by Esser et al. [2016], and the derivative of the Sigmoid function [Zenke and Ganguli, 2018]. As shown in Fig. 4, the performance of our EC-RSNNs is consistently higher than SG-RSNNs across all surrogate functions and hyper-parameters on the complex 17-DoF Humanoid task. We adopt the best parameter set of the piecewise linear surrogate function, with  $\gamma = 0.8$ ,  $\beta = 2$  as the baseline in Fig. 3.
Figure 4: Surrogate Gradient on the Humanoid task, with  $\gamma = 0.8$  and several surrogate functions,  $\beta$  parameters. For  $\gamma = 1.0$  results, please refer to Appendix F. Performance is sensitive to the chosen surrogate function and its parameters. In this context, EC surpass all considered parameters of SG.
One possible reason to explain the performance of SG-RSNN is that due to RSNNs having both explicit and implicit recurrence, the choice of surrogate function may determine whether gradients vanish or explode [Zenke and Vogels, 2021]. Moreover, the gradient approximation of SG is inaccurate and may deviate from the steepest descent direction [Li et al., 2021]. In contrast, our EC framework leverages unbiased Monte-Carlo gradient estimation, which approaches the optimal direction given that the population is sufficiently large [Zhang et al., 2017].
6.3 Efficiency Comparison
In this section, we compare the computational efficiency of our proposed EC framework with other baselines, as shown in Fig. 5. To ensure a fair comparison, we evaluate the EC and baseline methods using the same wall-clock computation time and implement them on identical hardware.
As expected, EC-RSNN exhibits slower training than deep RNNs due to the higher complexity of RSNNs, which necessitates simulating multiple timesteps to compute firing and integrate differential equations. Additionally, the deep RNN models employed in this study, specifically LSTM and GRU, are well-established and highly optimized for GPU implementation. Nevertheless, within the same computation time, the performance achieved by EC-RSNN remains competitive with deep RNNs.
Furthermore, we execute both ES and EC for 1,000 generations using the same population size and RSNN architecture. Experimental results in Fig. 5 have shown that our 1-bit EC-RSNN achieves a speedup of approximately  $2 \sim 3 \times$  over 32-bit floating-point ES-RSNN. This finding emphasizes the value of 1-bit connections within the proposed framework. Utilizing integer arithmetic typically results in higher throughput than floating-point arithmetic across most accelerators, and incorporating smaller data types reduces both memory requirements and memory access time. It is also worth mentioning that the 1-bit connections are implemented using INT8 on GPU, which is not fully optimized. By implementing the framework on hardware supporting smaller data types, connection size could be further reduced by up to  $8 \times$ , enabling more significant acceleration and cost reduction.
Additionally, our EC-RSNN demonstrates faster convergence compared to SG-RSNN. This can be attributed to the fact that EC-RSNN is an inference-only method with 1-bit connections, requiring
Figure 5: Efficiency comparison in locomotion tasks. Evolution-based approaches (EC, ES) are executed for 1,000 generations, while surrogate gradient (SG) runs for 250,000 gradient steps to attain a comparable run-time to EC, providing a fair comparison. When training RSNNs, EC attains a  $2 \sim 3 \times$  speedup over ES and demonstrates faster convergence than SG.
only a single forward pass using integer arithmetic, while SG-RSNN demands both a forward and a backward pass employing computationally-intensive floating-point operations.
Finally, we calculated the estimated energy consumption of the EC-RSNN on both GPU and neuromorphic devices, aiming to assess the power efficiency of the EC to train RSNN. In our study, all experiments were consistently conducted on the same NVIDIA Titan RTX GPU, operating at a stable  $100\%$  GPU power (280W). Consequently, the energy consumption of the EC training process on GPU is approximated to be directly proportional to the computation wall time, resulting in a 9 MJ. For neuromorphic device, we have conducted computations pertaining to the estimated energy consumption of EC-RSNN when implemented on the Loihi chip [Davies et al., 2018]. As detailed in Appendix D, the estimated energy consumption of EC is  $28\mathrm{kJ}$ , which indicate a noticeable reduction in power consumption by more than two orders of magnitude.
6.4 General 1-bit Framework
The task of discretizing a neural network into a 1-bit representation presents a considerable challenge, particularly when employing continuous training techniques such as ES and SGD. To illustrate this difficulty, we attempted to discretize both ES and SG for training 1-bit connection RSNNs utilizing the straight-through estimator [Bengio et al., 2013]. Fig. 7 (a) showed that while both ES-RSNN (1bit) and SG-RSNN (1bit) exhibited learning progress, they were surpassed by EC-RSNN (1bit). Fig. 7 (b) and (c) suggest that ES and SG demonstrated superior performance on continuous FP32 weights as opposed to discrete 1-bit representations. These findings imply that continuous optimization methods excel when applied to continuous parameters; conversely, for 1-bit discrete connections, EC is recommended
Figure 6: Performance comparison of vanilla RNN trained with EC, ES, PPO. EC surpasses baselines despite using 1-bit connections.
due to its specific design for discrete 1-bit optimization and the provision of unbiased gradients. A comprehensive description of the discretization for ES and SG methods is provided in Appendix C.
Moreover, Despite EC have demonstrated the efficacy on RSNNs, it serves as a versatile 1-bit training framework to non-spiking networks as well. To substantiate EC's potential in training deep RNN, we conducted a series of experiments employing a standard RNN model trained with EC, while using ES and PPO as baselines. The RNN has 256 tanh units in the hidden layer. For 1-bit EC, the weight magnitudes are 0-1 connection matrix, and the weight signs are separated to excitatory or inhibitory. The results in Fig. 6 demonstrate that EC can effectively train 1-bit deep recurrent neural networks and has the potential for different architectures and quantized neural networks.
In this study, we present the innovative Evolving Connectivity (EC) framework, a unique inference-only approach for training RSNN with 1-bit connections. The key attributes of the EC framework,
Figure 7: Optimization of the 1-bit RSNN proposed in this study using ES and SG. (a) In the discrete 1-bit RSNN settings, EC significantly outperforms ES and SG. (b) (c) Comparison of FP32 and 1-bit precision for ES and SG respectively, indicating that these continuous optimization methods achieve optimal results with FP32 continuous weights.
such as its inference-only nature and scalability, render it particularly suitable for training RSNNs on neuromorphic devices. The use of 1-bit connections significantly reduces memory requirements and computational cost, facilitating faster training on GPUs and paving the way for additional cost reductions in neuromorphic chip production.
We performed extensive experiments on a variety of intricate locomotion tasks, showcasing the competitive performance of our proposed EC framework in comparison to deep RNNs like LSTM and GRU. Moreover, the EC framework outperforms other RSNN training methods such as Evolution Strategies and Surrogate Gradient, both in terms of performance and computational efficiency.
Evolutionary algorithms demand the storage of  $N$  distinct parameter sets for population evaluation, leading to a space complexity of  $O(N|\theta|)$ . In contrast, gradient-based approaches utilize a single parameter set but necessitate the storage of intermediate results at each timestep, resulting in a space complexity of  $O(NHS + |\theta|)$ , where  $H$  corresponds to the number of BPTT timesteps, and  $S$  represents the size of the intermediate results. This situation gives rise to a trade-off: evolutionary methods offer greater memory efficiency for tasks featuring long time horizons, whereas gradient-based techniques with larger parameter sizes and shorter time horizons require less memory. Recognizing this trade-off is crucial in practical applications. Although our proposed EC framework, as an evolutionary algorithm, retains the same space complexity and trade-off, it can achieve a constant term reduction by storing 1-bit connections.
9 Discussions
Neuromorphic hardware. One primary bottleneck in the development of neuromorphic hardware is the lack of effective on-chip learning algorithms to build applications compared to deep learning approaches. This paper proposes a novel EC framework that circumvents the requirement for gradients and demonstrates efficacy on locomotion tasks, providing a potential solution for solving the on-chip learning challenge. Typically, the demand for computing devices falls into two categories, including cloud and edge. In the cloud, our proposed EC framework supports large-scale learning, while at the edge, it enables energy-efficient applications. Therefore, our framework offers a potential approach to further building neuromorphic applications.
Another practical challenge lies in the trade-off between numeric precision and cost. We suggest that it is possible to drastically reduce connections from floating point to 1-bit, providing a novel design principle for the next generation of neuromorphic hardware. Therefore, our method holds the potential to significantly decrease the manufacturing and energy costs of neuromorphic hardware.
Neuroscience. Our EC framework introduces a novel method for neuroscience research. First, as opposed to toy tasks or simulated signals, which are often studied in previous neuromodeling work, we employed RSNN in a complex, real-world like locomotion task. Additionally, controlling signals in neuroscience like 'GO' and 'NOGO' signals in the basal ganglia can be easily integrated into the task by concatenating them to the environment observation vector, enabling the creation of novel neuroscientific tasks. Our work lays the foundation for further investigation of decision-making processes and motor control.
Moreover, our framework provides a novel type of data to analyze: neuron-to-neuron level connection probability. Connection probability is one of the most fundamental properties in brain-wide connectomes. However, obtaining neuron-to-neuron connection probability from the whole mammalian brain is experimentally implausible due to limitations in nowadays neuroconnectomic technology. Our work provides in silico connection data for further analysis, including covariances, motifs, clustered engrams, and dimensional properties.
Finally, our framework is capable of incorporating neuroanatomical and neurophysiological data to construct a novel neurosimulation model, since our framework is able to train arbitrary models in principle. Neuroscientists have discovered that connection probability between two neurons is determined by various factors, including spatial distance, receptive field, and neuron types. Our framework can leverage these findings and conduct in silico experiments with data-driven models.
This work was supported by the National Key Research and Development Program of China (grant 2021ZD0200301).
B Comparing EC and NES
Figure S1: Comparison of search spaces in EC and NES with  $N = 100$  samples and two parameters  $(X, Y)$ . EC samples from a discrete 1-bit search space, while NES samples from a continuous search space.
EC: discrete 1-bit connection samples
NES: continuous parameter samples
For readers familiar with evolutionary algorithms, we present the similarities and main differences between our Evolving Connectivity (EC) method and commonly used evolutionary algorithms, namely Natural Evolution Strategies (NES, Wierstra et al. [2014]) and Evolution Strategies (ES, Salimans et al. [2017]). This comparison will facilitate better understanding and implementation of these algorithms. EC, ES, and NES share the same unbiased Monte-Carlo gradient estimator, gradient update approach, and fitness shaping trick, as described in Section 4.
The primary distinction between these algorithms lies in their search spaces, as illustrated in Figure S1. Both ES and NES employ real-valued parameters, akin to conventional deep learning techniques, and sample the search space using parameterized normal distributions. In contrast, EC utilizes a 1-bit discrete search space parameterized by Bernoulli distributions. This 1-bit formulation reduces the parameter size, endowing EC with unique advantages such as faster computation, improved memory efficiency, and reduced energy and cost requirements.
Additionally, there is a minor difference in how EC enhances exploration. As described in Section 4, EC clips the elements of connection probability matrix to boost exploration.
C Optimizing 1-bit Networks Using ES and SG
ES and SG are continuous optimization techniques designed for continuous parameters. To apply these methods to the training of 1-bit connection RSNNs proposed in this paper, we utilize discretization methods as described by Zhou et al. [2019]. Specifically, we employ ES and SG to optimize a continuous parameter  $\theta$ , which is then discretized into 1-bit weights  $\mathbf{W}$  by applying a threshold at 0 as  $\mathbf{W} = H(\theta)$ , where  $H$  denotes the Heaviside step function. For SG, the straight-through estimator is utilized as a standard approach [Bengio et al., 2013]. It is crucial to acknowledge that employing continuous optimization methods in conjunction with discretization may result in biased gradient estimation, while EC naturally provides unbiased gradient estimation for discrete 1-bit connections.
D On Chip Power Consumption Estimation
We estimate the power consumption of EC by considering the number of spiking neuron operations and the energy per operation provided by Loihi [Davies et al., 2018], as a representative neuromorphic
Table 2: Energy consumption and network specifics
Parameter	Value
Energy per synaptic spike op Ps	23.6 (pJ)
Within-tile spike energy Pw	1.7 (pJ)
Energy per neuron update Pu	81 (pJ)
# Generations G	1000
# Population P	10240
# Time steps S	33200
# Neurons N	256
# Spikes per neuron per step R	0.025
# Connection per neuron C	128
# Update operations per neuron I	4
device. The data employed for this estimation is presented in Table 2. Initially, we computed the estimated energy consumption for a single network inference as follows:
$$
E _ {o n e} = P _ {u} * N * I * S + \left(P _ {s} + C * P _ {w}\right) * N * R * S = 2. 8 m J
$$
Subsequently, we determined the total energy consumption during the training process:
$$
E _ {t o t} = E _ {o n e} * G * P = 2 8 k J
$$
It is important to note that these calculations are approximations and serve as an initial assessment. In future research, we plan to conduct experiments using neuromorphic chips to empirically validate the energy efficiency of RSNNs.
E Performance Comparison with PPO
In order to assess the efficacy of the EC-RSNN approach in relation to contemporary deep reinforcement learning (RL) techniques, we conducted additional training with GRU and LSTM models using Proximal Policy Optimization (PPO) [Schulman et al., 2017] on the most complex Humanoid task. The results of these experiments are presented in Figure S3. The data clearly indicates that EC-RSNN surpasses the performance of PPO-LSTM, illustrating a level of performance that is on par with state-of-the-art deep RL methods.
F Full SG Results
We present a comprehensive analysis of the Surrogate Gradient (SG) results, considering various damping factors  $(\gamma)$  and the parameter  $(\beta)$ . The outcomes are illustrated in Figure S2.
G Detailed Experiment Settings
G.1 Hardware
All experiments were conducted on a single GPU server with the following specifications. Additionally, all efficiency experiments, including those that measure wall-clock time, were executed on a single GPU within this server.
- 8x NVIDIA Titan RTX GPU (24GB VRAM)
- 2x Intel(R) Xeon(R) Silver 4110 CPU
252GB Memory
G.2 Hyperparameters
In this section, we provide the hyperparameters for our framework, Evolving Connectivity (EC), and the baselines, including Evolution Strategies (ES) and Surrogate Gradient (SG). All methods utilize the same hyperparameter set for all three locomotion tasks. We also list the settings of neural networks below.
Figure S3: Performance comparison among PPO-GRU, PPO-LSTM, and EC-RSNN.
Table 3: Hyperparameters for Evolving Connectivity (EC)
Hyperparameter	Value
Population size N	10240
Learning rate η	0.15
Exploration probability ε	10-3
Table 4: Hyperparameters for Evolution Strategies
Hyperparameter	Value	Target
Population size N	10240	All
Learning rate η	0.15	RSNN
0.01	Deep RNN
Noise standard deviation σ	0.3	RSNN
0.02	Deep RNN
Weight decay	0.1	All
Figure S2: Surrogate Gradient on the Humanoid task, with several surrogate functions,  $\beta$  and  $\gamma$  parameters. Performance is sensitive to the chosen surrogate function and its parameters.
Table 5: Hyperparameters for Surrogate Gradient
Hyperparameter	Value
Proximal Policy Optimization (PPO)
Batch size	2048
BPTT length	16
Learning rate η	3 × 10-4
Clip gradient norm	0.5
Discount γ	0.99
GAE λ	0.95
PPO clip	0.2
Value loss coefficient	1.0
Entropy coefficient	10-3
Surrogate Gradient
Surrogate function	SuperSpike
Surrogate function parameter β	10
Table 6: Settings of neural networks
Hyperparameter	Value
Recurrent Spiking Neural Network (RSNN)
Number of neurons dh	256
Excitatory ratio	50%
Simulation time per environment step	16.6 ms
Simulation timestep Δt	0.5 ms
Synaptic time constant τsyn	5.0 ms
Membrane time constant τm	10.0 ms
Output time constant τout	10.0 ms
Input membrane resistance1Rin	0.1 · τm√2/din
Hidden membrane resistance1Rh	1.0 · τm/τsyn√2/dh
Output membrane resistance1Rout	5.0 · τout√2/dh
Gated Recurrent Unit (GRU)
Hidden size	256
Long-short term memory (LSTM)
Hidden size	128
H Source Code and Licensing
The source code associated with this paper is licensed under the Apache 2.0 License and is publicly available at https://github.com/imoneoi/EvolvingConnectivity. In our research, we incorporated locomotion tasks and the Brax physics simulator from Freeman et al. [2021], as well as the JAX framework [Bradbury et al., 2018]. Both of these resources are also released under the Apache 2.0 License.
<sup>1</sup>Resistance is set following Ding et al. [2022] to preserve variance.