SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guan Wang $^{1,2*}$ , Yuhao Sun $^{2,3*}$ , Sijie Cheng $^{1,4}$ , Sen Song $^{2,3\dagger}$
$^{1}$ Deptartment of Computer Science and Technology, Tsinghua University
$^{2}$ Laboratory of Brain and Intelligence, Tsinghua University
$^{3}$ Department of Biomedical Engineering, Tsinghua University
$^{4}$ Institute for AI Industry Research (AIR), Tsinghua University
imonenext@gmail.com, {syh18, csj23}  $@$  mails.tsinghua.edu.cn,
songsen@tsinghua.edu.cn
Recurrent spiking neural networks (RSNNs) hold great potential for advancing artificial general intelligence, as they draw inspiration from the biological nervous system and show promise in modeling complex dynamics. However, the widely-used surrogate gradient-based training methods for RSNNs are inherently inaccurate and unfriendly to neuromorphic hardware. To address these limitations, we propose the evolving connectivity (EC) framework, an inference-only method for training RSNNs. The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions. Our EC framework circumvents the need for gradients and features hardware-friendly characteristics, including sparse boolean connections and high scalability. We evaluate EC on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs, even solving the complex 17-DoF humanoid task. Additionally, the EC framework demonstrates a two to three fold speedup in efficiency compared to directly evolving parameters. By providing a performant and hardware-friendly alternative, the EC framework lays the groundwork for further energy-efficient applications of RSNNs and advances the development of neuromorphic devices. Our code is publicly available at https://github.com/imoneoi/EvolvingConnectivity.
Inspired by the remarkable information processing power of the human brain, neuromorphic computing aims to design and implement computational systems that mimic the architecture and functionality of biological neural networks, paving the way for a new era of intelligent machines. Specifically, the highly recurrently connected and spike-emitting brain network inspired recurrent spiking neural networks (RSNNs), which employ discrete, spike-based signals for transmitting information through recurrent connections in an event-driven manner. RSNNs can serve as realistic models of the brain incorporating the latest anatomical and neuro-physiological data [Billeh et al., 2020], leading to the discovery of general principles of computing, e.g., noise robustness [Chen et al., 2022]. Furthermore, RSNNs can exhibit self-organized critical properties [Poil et al., 2012] and serve as a reservoir for complex temporal dynamics [Jaeger, 2001, Maass et al., 2002], making them suitable for sequential tasks like robotics [Lele et al., 2020], path-finding [Rueckert et al., 2016], and others.
Despite the importance of RSNNs, developing end-to-end training algorithms remains a challenge. Inspired by the success of deep learning, a line of research [Wu et al., 2018, Shrestha and Orchard, 2018, Bauer et al., 2022] uses error-backpropagation [Rumelhart et al., 1985] with carefully chosen surrogate gradients to address the non-differentiability problem and achieve performance comparable to deep learning. However, incorporating surrogate gradients into RSNNs introduces two concerns. Algorithmically, the surrogate gradient leads to inherent inaccuracy in the descent direction [Li et al., 2021] and sensitivity to function scale selection [Zenke and Vogels, 2021]. At the implementation level, gradient-based training is incompatible with prominent neuromorphic devices [Pei et al., 2019, Davies et al., 2018, Mayr et al., 2019] due to the requirement of accessing the full network state over every timestep [Werbos, 1990]. Consequently, this raises a critical research question: can we design a training method for RSNNs that bypasses the need for gradients without compromising performance?
In response to this challenge, we observe that connection probability distribution information from brain connection maps across various species exhibit similarities [Haber et al., 2023] and can be used to construct large-scale neural computing models [Billeh et al., 2020, Schmidt et al., 2018]. Existing studies demonstrate that networks with binary connections alone can achieve high performance, rivaling that of weighted networks [Gaier and Ha, 2019, Frankle and Carbin, 2018, Malach et al., 2020]. Moreover, computing boolean connections relies primarily on integer arithmetic rather than resource-intensive floating-point operations, enabling simpler on-chip implementation and enhanced energy efficiency. Considering these features, we reformulate the architecture of RSNNs, where connections are sampled independently from parametric Bernoulli distributions, and adopt Natural Evolution Strategies (NES) [Wierstra et al., 2014] to optimize this parametric probability distribution, providing a scalable, inference-only, and effective algorithm for tuning parameters.
In this paper, we introduce the evolving connectivity (EC) framework for training RSNNs. The EC framework reformulates RSNNs as boolean connections with homogeneous weights and employs NES to evolve the connection probabilities of the network. To evaluate effectiveness and efficiency, we conduct extensive experiments on locomotion tasks, a set of widely-used sequential decision-making tasks [Brockman et al., 2016, Freeman et al., 2021]. Experimental results show that our proposed EC method achieves performance comparable to deep recurrent neural networks, surpassing the asymptotic performance of surrogate gradients. Despite using a GPU, which is not specifically optimized for our framework, the EC method yields a speed improvement of  $2 \sim 3 \times$  compared to directly evolving parameters using Evolution Strategies [Salimans et al., 2017] and exhibits better efficiency than surrogate gradients. Additionally, our EC method can effectively train deep recurrent neural networks (RNNs), demonstrating versatility for different architectures and potential for quantized deep neural networks.
- Novel framework: We propose a novel inference-only training framework for RSNNs by reformulating weight-tuning as connection probability searching through evolution-based algorithms.
- High performance: Our method can solve the complex 17-DoF humanoid locomotion task with RSNN, achieving performance on par with recurrent neural networks and outperforming gradient-trained RSNNs.
- Hardware-friendly: By producing RSNNs with sparse 1-bit boolean connections, as well as enabling inference-only training and high scalability, our method is highly compatible with neuromorphic devices. This compatibility holds promising potential for further energy-efficient applications of RSNNs.
2.1 Training Recurrent Spiking Neural Networks
The study of training algorithms for RSNNs endeavors to unravel the mechanism that enables the brain to learn. For decades, biological plasticity rules, especially spike-timing-dependent plasticity (STDP) [Bi and ming Poo, 1998], have been considered as a foundational of RSNNs training [Diehl and Cook, 2015, Kheradpisheh et al., 2018, Mozafari et al., 2019]. Recently, with the success of deep learning, surrogate gradient-based approaches [Wu et al., 2018, Shrestha and Orchard, 2018, Bauer et al., 2022] have been predominately utilized in training RSNNs. Surrogate gradients share conceptual similarities with the straight-through estimator [Bengio et al., 2013] used in deep learning.
Both techniques employ continuous surrogate functions to compute gradients of non-differentiable functions during the backward pass, while preserving the original non-differentiable functions in the forward pass. Slayer [Shrestha and Orchard, 2018] adopted an exponential decaying surrogate function with a spike response model (SRM), while STBP [Wu et al., 2018] incorporated BPTT in multi-layer SNN training and proposed several surrogate function choices. Chen et al. [2022] trained an anatomical and neurophysiological data constrained RSNN and demonstrated its robustness and versatility. However, gradient-based approaches face the new problem of being difficult to implement on neuromorphic devices. To address the limitation, E-prop [Bellec et al., 2020] utilize eligibility trace to compute truncated gradient, becoming an exemplary training algorithm on neuromorphic devices like Loihi2 [Davies et al., 2018] and SpiNNaker2 [Mayr et al., 2019], but have a strict restriction on the temporal relationship of SNN models and still lags behind surrogate gradient methods in performance. Our approach tackles the gradient-effectiveness dilemma from the beginning, by introducing the inference-only training framework.
2.2 Weight-agnostic Neural Networks
Training a neural network typically involves assigning appropriate values to the network's weights. However, the weight-agnostic neural network (WANN) [Gaier and Ha, 2019] has demonstrated that a network's topology can be highly informative, similar to that of a weighted neural network. Additionally, research on the lottery ticket hypothesis (LTH) [Frankle and Carbin, 2018, Zhou et al., 2019, Ramanujan et al., 2020] suggests that an over-parameterized network contains an effective subnetwork, even when using its initial parameter weights. Several theoretical studies have also shown that a sufficiently large, randomly-weighted deep neural network contains a subnetwork capable of approximating any function [Malach et al., 2020, Fischer et al., 2022]. Our approach, furthering the existing research on connection-encoded networks, proposes a framework that utilizes connection probability to parameterize the network.
2.3 Deep Neuroevolution
Deep neuroevolution utilizes evolutionary algorithms for training deep neural networks. Natural Evolution Strategies (NES) [Wierstra et al., 2014] have laid the foundation for gradient estimation in this domain. A well-known variant, Evolution Strategies (ES) [Salimans et al., 2017], has been employed for training deep neural networks in sequential decision-making tasks. ES optimizes a single set of continuous network weights by applying Gaussian perturbations and updating the weights using the NES gradient estimator. Following the success of ES, numerous evolutionary algorithms have been proposed for training deep neural networks. For example, Such et al. [2018] showed that deep neural networks could be effectively trained using simple genetic algorithms that mutate weights. Furthermore, Conti et al. [2018] integrated exploration and novelty-seeking into ES to overcome local minima. While the majority of prior research has primarily focused on continuous parameters, our proposed framework presents a novel approach by concentrating on the search for connection probability distributions. This shift in perspective offers new possibilities for evolving recurrent spiking neural networks in a hardware-friendly manner.
3 Preliminaries: Recurrent Spiking Neural Networks
RSNN is a class of spiking neural networks which incorporate feedback connections. In this paper, we adopt a typical RSNN architecture from the reservoir network [Jaeger, 2001, Maass et al., 2002] for sequential tasks. It is worth noting that, despite that we adopt a specific RSNN model as an example, our framework can be broadly applied to search for connectivity distributions in any type of RSNN, as it does not depend on network-specific assumptions and only relies on evaluating the network with a set of parameters.
According to Dale's law, our network consists of an excitatory neuron group and an inhibitory neuron group. Each neuron is modeled as a leaky integrate and fire (LIF) neuron. A neuron will fire a spike when its membrane potential  $u$  exceeds the threshold, and the membrane potential will hard-reset to 0. More specifically, our model defines the dynamics of membrane potential  $u$  and synaptic current  $c$  as follows:
$$
\tau_ {m} \frac {\mathrm {d} \mathbf {u} ^ {(g)}}{\mathrm {d} t} = - \mathbf {u} ^ {(g)} + R \mathbf {c} ^ {(g)} \tag {1}
$$
Figure 1: Architecture of evolving connectivity (EC). The connectivity  $\theta_{k}$  of the population is sampled from the global distribution  $B(\pmb{\rho})$  and then evaluated in parallel. The RSNN consists of excitatory and inhibitory neurons, simulated using 1-bit firing calculations and the LIF model.
where  $g = \{Exc, Inh\}$  denotes the excitatory and inhibitory group, respectively, and  $R$  denotes the resistance.
The current input was modeled using exponential synapses to retain information for a short duration, commonly adopted in robotic tasks [Tang et al., 2020, Naya et al., 2021].
$$
\frac {\mathrm {d} \mathbf {c} ^ {(g)}}{\mathrm {d} t} = - \frac {\mathbf {c} ^ {(g)}}{\tau_ {s y n}} + \sum_ {g _ {j}} I _ {g _ {j}} \sum_ {j} \mathbf {W} _ {i j} ^ {\left(g _ {i} g _ {j}\right)} \delta \left(t - t _ {j} ^ {s \left(g _ {j}\right)}\right) + \mathbf {I} _ {e x t} \tag {2}
$$
where  $t_j^{s(g_j)}$  denotes the spike time of neuron  $j$  in neuron group  $g_j$ ,  $\delta$  denotes Dirac delta function.  $I_g$  was set to define the connection strength of excitatory and inhibitory synapse respectively, where  $I_{Exc} > 0$  and  $I_{Inh} < 0$ . Weight  $\mathbf{W}^{(g_i g_j)}$  was defined as a matrix with non-negative elements, connecting group  $g_j$  to group  $g_i$ . Besides, the external input signal  $\mathbf{I}_{ext}$  is extracted using linear projection of observation  $\mathbf{x}$ .
Discretizing the LIF differential equations (Eq. 1 and 2) with  $\Delta t$  as a time-step, we could obtain the following difference equation for our RSNN model:
$$
\mathbf {c} ^ {(t, g)} = d _ {c} \mathbf {c} ^ {(t - 1, g)} + \sum_ {g _ {j}} I _ {g _ {j}} \mathbf {W} ^ {\left(g _ {i} g _ {j}\right)} \mathbf {s} ^ {(t - 1, g _ {j})} + \mathbf {I} _ {e x t} ^ {(t, g)} \tag {3}
$$
$$
\mathbf {v} ^ {(t, g)} = d _ {v} \mathbf {u} ^ {(t - 1, g)} + R \mathbf {c} ^ {(t, g)} \tag {4}
$$
$$
\mathbf {s} ^ {(t, g)} = \mathbf {v} ^ {(t, g)} > \mathbf {1} \tag {5}
$$
$$
\mathbf {u} ^ {(t, g)} = \mathbf {v} ^ {(t, g)} \left(\mathbf {1} - \mathbf {s} ^ {(t, g)}\right) \tag {6}
$$
where  $d_{c} = e^{-\frac{\Delta t}{\tau_{syn}}}$  and  $d_v = e^{-\frac{\Delta t}{\tau_m}}$  are two constant parameters.
The output vector  $\mathbf{0}$  is extracted using linear projection of neuron firing rates in a short time period  $\tau$ , i.e.,
$$
\mathbf {o} ^ {(t)} = \sum_ {\tau} k (\tau) \sum_ {g} \mathbf {W} _ {o u t} ^ {(g)} \mathbf {s} ^ {(t - \tau , g)} \tag {7}
$$
where  $\mathbf{W}_{out}^{(g)}$  denotes the output weight, and  $k$  is a function averaging the time period  $\tau$ .
4 Framework
In this section, we present our proposed Evolving Connectivity (EC) framework, which is depicted in Fig. 1. Our approach consists of three main steps: (1) reformulating the neural network architecture from weight-based parameterization to connection probability distribution, (2) employing the Natural Evolution Strategies (NES) method to optimize the reformulated parameter space, and (3) deterministically extracting the final parameters from the distribution.
Reformulation. A sparse connected weight matrix can be described into a weight matrix  $\mathbf{w}$  and a connection mask  $\theta$ . Traditionally, we can use Erdős-Rényi random matrix to describe the connection
mask, in which connections are independently drawn from a Bernoulli distribution, i.e.,
$$
\mathbf {W} _ {i j} = \mathbf {w} _ {i j} \cdot \boldsymbol {\theta} _ {i j}, \text {w h e r e} \boldsymbol {\theta} _ {i j} \sim B (\rho) \tag {8}
$$
Inspired by the success of subnetworks in deep neural networks [Ramanujan et al., 2020], we can reformulate this in a connection framework. In this view, we aim to find a connection probability matrix,  $\rho = (\rho_{ij})$ , where each element represents the connection probability between two neurons, and set all weights  $\mathbf{w}_{ij}$  to be unit size, i.e.,
$$
\mathbf {W} _ {i j} = \boldsymbol {\theta} _ {i j}, \text {w h e r e} \boldsymbol {\theta} _ {i j} \sim B (\boldsymbol {\rho} _ {i j}). \tag {9}
$$
**Optimization.** We optimize  $\rho$  to maximize the expected performance metric function  $R(\cdot)$  across individual network samples drawn from the distribution, which can be expressed as the following objective function:
$$
\boldsymbol {\rho} ^ {*} = \underset {\boldsymbol {\rho}} {\arg \max } J (\boldsymbol {\rho}) = \underset {\boldsymbol {\rho}} {\arg \max } \mathbb {E} _ {\boldsymbol {\theta} \sim B (\boldsymbol {\rho})} [ R (\boldsymbol {\theta}) ] \tag {10}
$$
To optimize this objective, we employ Natural Evolution Strategies (NES) [Wierstra et al., 2014]. NES provides an unbiased Monte Carlo gradient estimation of the objective  $J(\pmb{\rho})$ , by evaluating the performance metric  $R_{k}$  on multiple individual samples  $\pmb{\theta}_{k}$  drawn from  $B(\pmb{\rho})$ . Specifically, NES estimates the gradient of  $J(\pmb{\rho})$  with respect to the parameters of the distribution  $\pmb{\rho}$  by computing the expectation over samples from  $B(\pmb{\rho})$ :
$$
\begin{array}{l} \nabla_ {\boldsymbol {\rho}} J (\boldsymbol {\rho}) = \mathbb {E} _ {\boldsymbol {\theta} \sim B (\boldsymbol {\rho})} [ \nabla_ {\boldsymbol {\rho}} \log P (\boldsymbol {\theta} | \boldsymbol {\rho}) R (\boldsymbol {\theta}) ] (11) \\ = \mathbb {E} _ {\boldsymbol {\theta} \sim B (\boldsymbol {\rho})} \left[ \frac {\boldsymbol {\theta} - \boldsymbol {\rho}}{\boldsymbol {\rho} (\mathbf {1} - \boldsymbol {\rho})} R (\boldsymbol {\theta}) \right] (12) \\ \approx \frac {1}{N} \sum_ {k = 1} ^ {N} \frac {\boldsymbol {\theta} _ {k} - \boldsymbol {\rho}}{\boldsymbol {\rho} (\mathbf {1} - \boldsymbol {\rho})} R _ {k} (13) \\ \end{array}
$$
Thus, we obtained an inference-only estimation of the gradient by simply sampling and evaluating the metric. Then gradient descent can be carried over the estimated gradients, following the NES approach [Wierstra et al., 2014]. Furthermore, as in Williams [1992], we scale the step size proportionally to the variance, i.e.  $\alpha = \eta \cdot \mathrm{Var}[B(\pmb {\rho})]$ , and obtain the update rule:
$$
\boldsymbol {\rho} _ {t} = \boldsymbol {\rho} _ {t - 1} + \alpha \nabla_ {\boldsymbol {\rho}} J (\boldsymbol {\rho}) \approx \boldsymbol {\rho} _ {t - 1} + \frac {\eta}{N} \sum_ {k = 1} ^ {N} \left(\boldsymbol {\theta} _ {k} - \boldsymbol {\rho}\right) R _ {k} \tag {14}
$$
In addition, all elements of  $\rho$  clipped in the interval  $[\epsilon, 1 - \epsilon]$ , where  $\epsilon \to 0^{+}$ , to guarantee a minimal level of exploration within the search space. In our experiment, we set  $\epsilon = 0.001$  and initialize  $\rho = 0.5$ . Additionally, we adopted a similar fitness shaping trick, center rank transform, as discussed in Salimans et al. [2017], which sort the return of a population, then linearly rescale the sorting indexes into a fixed interval  $[-0.5, 0.5]$ .
Once a sufficient number of updates have been performed, the final parameter  $\theta$  can be deterministically obtained as the parameter with maximum probability, which is subsequently used for deployment. In the context of a Bernoulli distribution, this process is equivalent to thresholding  $\rho$  at 0.5 to produce a boolean matrix containing only  $\{0,1\}$  values.
5 Properties of EC Framework
In this section, we further discuss the important properties of our proposed EC framework in implementation, thanks to leveraging the connection probability distribution.
Inference only. The most significant challenge in neuromorphic devices is the absence of an effective hardware-friendly learning algorithm [Li et al., 2023]. Most neuromorphic chips, such as Loihi [Davies et al., 2018], Tianjic [Pei et al., 2019], TrueNorth [Akopyan et al., 2015], SpiN-Naker2 [Mayr et al., 2019], lack support for directly calculating error backpropagation, which is crucial for surrogate gradient methods. The inference-only EC framework enables an alternative method for training on these inference chips.
1-bit connections. EC employs 1-bit sparse connections throughout training and deployment, replacing the traditional floating-point weight matrix. Therefore, the 1-bit connections permit the use of more economical integer arithmetic instead of costly floating-point cores. This approach not only accelerates computations on devices like GPUs, but also holds a promise of driving the creation of novel 1-bit connection neuromorphic computing hardware.
Scalability. The inference-only property of the EC framework implies no data dependence between evaluations, which allows for the distribution of sampled parameters  $\theta_{k}$  across independent workers and the collection of their reported performance measures, making EC highly scalable. Moreover, the random seed for sampling the population can be transmitted across nodes instead of the connections  $\theta_{k}$  to minimize communication overhead, facilitating scalar-only communication.
6.1 Experimental Setups
Tasks. We focus on three robotic locomotion tasks in our experiments, Humanoid, Walker2d, and Hopper, as they are commonly used for sequential decision-making problems in the reinforcement learning domain [Brockman et al., 2016, Freeman et al., 2021]. As illustrated in Fig.
2, these tasks involve controlling robots with varying degrees of freedom (DoF), to perform certain actions that maximize the return within a fixed-length episode  $T$ . In the EC framework, the performance metric function can be defined as the expected return over episodes  $R(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}}[\sum_{t=0}^{T} r_t]$ , and a single evaluation  $R_i$  corresponds to the return of one episode using network parameter  $\theta_k$ .
Figure 2: Locomotion tasks illustrated from left to right: Humanoid (17-DoF), Walker2d (6-DoF), and Hopper (3-DoF).
RSNN architecture. In our study, we employed a RSNN consisting of an input layer, a hidden recurrent layer with LIF neurons, and an output layer, as delineated in the Preliminaries section. To achieve an equilibrium within the excitatory inhibitory balance, input and hidden layers were evenly divided into excitatory and inhibitory group.
Baselines. To thoroughly evaluate the effectiveness of our framework and its advantages over prior methods, we compare our EC framework with deep RNNs, as well as RSNN trained with Surrogate Gradients (SG) and Evolution Strategies (ES). Detailed network architecture and precision is illustrated in Table 1.
For deep RNNs, we employ widely-used recurrent deep neural networks, specifically long-short term memory (LSTM) [Hochreiter and Schmidhuber, 1997] and gated recurrent unit (GRU) [Cho et al., 2014], trained using Evolution Strategies (ES) [Salimans et al., 2017] as our baselines.
For RSNN, we employ the same structured but excitatory/inhibitory not separated RSNN trained with ES and Surrogate Gradient (SG). ES is directly applied to search the weight matrix  $\mathbf{W}$  of RSNNs, optimizing the performance metric in a gradient-free manner. In contrast, the Surrogate Gradient is combined with Proximal Policy Optimization (PPO) [Schulman et al., 2017], a prominent reinforcement learning method, to optimize the weight  $\mathbf{W}$  for sequential decision-making tasks where returns cannot be differentiated.
Table 1: Comparison of model architectures.
Model	Hidden size	# Params	Precision	Size (MB)
RSNN	256	193K	1-bit(EC) / FP32(Others)	24KB(EC) / 768KB(Others)
GRU	256	386K	FP32	1544KB
LSTM	128	191K	FP32	764KB
Implementation Details. Our EC framework and all baselines are implemented using the JAX library [Bradbury et al., 2018] and just-in-time compiled with the Brax physics simulator [Freeman et al., 2021] for efficient GPU execution. The population is vectorized automatically, and 1-bit firing