Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks
Authors: Korneel Van den Berghe
Stein Stroobants
Vijay Janapa Reddi
Guido De Croon
Venue: NeurIPS 2025 oral ()
Keywords: Neuromorphics
Spiking Neural Networks
Reinforcement Learning
Surrogate Gradients
Robotics
Sequential
Temporal
Primary area: reinforcement_learning
TL;DR: We improve training of spiking neural networks for energy-efficient robotic control by analyzing surrogate gradient slopes and introducing a privileged policy-guided method, achieving a 2.1× performance boost and strong real-world results.

Abstract: Neuromorphic computing systems are set to revolutionize energy-constrained robotics by achieving orders-of-magnitude efficiency gains, while enabling native temporal processing. Spiking Neural Networks (SNNs) represent a promising algorithmic approach for these systems, yet their application to complex control tasks faces two critical challenges: (1) the non-differentiable nature of spiking neurons necessitates surrogate gradients with unclear optimization properties, and (2) the stateful dynamics of SNNs require training on sequences, which in reinforcement learning (RL) is hindered by limited sequence lengths during early training, preventing the network from bridging its warm-up period.

We address these challenges by systematically analyzing surrogate gradient slope settings, showing that shallower slopes increase gradient magnitude in deeper layers but reduce alignment with true gradients. In supervised learning, we find no clear preference for fixed or scheduled slopes. The effect is much more pronounced in RL settings, where shallower slopes or scheduled slopes lead to a $\times2.1$ improvement in both training and final deployed performance. Next, we propose a novel training approach that leverages a privileged guiding policy to bootstrap the learning process, while still exploiting online environment interactions with the spiking policy. Combining our method with an adaptive slope schedule for a real-world drone position control task, we achieve an average return of 400 points, substantially outperforming prior techniques, including Behavioral Cloning and TD3BC, which achieve at most –200 points under the same conditions. This work advances both the theoretical understanding of surrogate gradient learning in SNNs and practical training methodologies for neuromorphic controllers demonstrated in real-world robotic systems.

Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks
Korneel Van den Berghe Delft University of Technology
Stein Stroobants
Delft University of Technology
Vijay Janapa Reddi Harvard University
G.C.H.E. de Croon
Delft University of Technology
Abstract
Neuromorphic computing systems are set to revolutionize mnergy-constrainee efficiency gains, while enabling native
robotics by achieving temporal processing. Spiking Neural Networks (SNNs) represent a promising algorithmic approach for these systems, yet their application complex control tasks faces two critical challenges: (1) the non-differentiable nature of spiking neurons necessitates surrogate gradients with unclear optimization properties, and (2) the statefuul dynamics of SNNs require training on sequences, which in reinforcement learning (RL) hindered by limited sequence lengths during early training, preventing the network from bridging its warm-up period.
We address these challenges by systematically analyzing surrogate gradient slope settings, showing that shallower slopes increase gradient magnitude in deeper layers but reduce alignment with true gradients. In supervised learning, we find no clear preference for fixed or scheduled slopes. The effect is much more pro- nounced in RL settings, where shallower slopes scheduled slopes lead to x2.1 improvement in both training and final deployed performance. Next, we propose novel training approach that leverages privileged guiding policy bootstrap the learning process, while still exploiting online environment interactions with the spiking policy. Combining our method with adaptive slope schedule for real-world drone position control task, we achieve an average return of 400 points, substantially outperforming prior techniques, including Behavioral Cloning and TD3BC, which achieve at most -200 points under the same conditions. This work advances both the theoretical understanding of surrogate gradient learning in SNNs and practical training methodologies for neuromorphic controllers demonstrated in real-world robotic systems.
Introduction
Spiking Neural Networks (SNNs) class of neuromorphic algorithms [1] that offer native temporal processing and significantly outperform conventional deep learning architectures in terms of energy efficiency across wide range of applications 2 4 However, training SNNs for control tasks remains challenging due to the non-differenniaabe nature of spiking neurons, which complicates gradient-based optimization. Surrogate gradients (5] are a popular solution this issue. Gygax and Zenke [6] demonstrate that the gradient can severely deviate from its true value depending on the surrogate gradient chosen. Yet, how these deviations affect learning in deep stacked architectures remains largely unexplored.
"Corresponding email: korneel. vonneel.anddeneergheee hotmail.he
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Moreover, leveraging the temporal processing capabilitiees of SNNs requires training on sequences rather than individual transitions. critical component of this process is warm-up period, a number of timesteps during which the hidden states of the network are stabilized before gradients are applied. Reinforcement Learning (RI.) is a framework for sequential decision-making which has been used to achieve superhuman performance on Atari games (7] and drone racing [8]. However, at its core, most RL algorithms assume the underlying process to be a Markov Decision Process (MDP), where the next state depends only on the current state and action. This assumption is often not satisfied in robotics [0] Frame-stacking acts as a work-around by explicitly adding a history of observations to approximate an MDP [1 B [4]. This approach is inefficient and redundant for SNNs, which are inherently stateful and capable of encoding temporal dependencies internally.
A major challenge arises in robotic environments, such as drone control. Subpar initial policies often lead to early episode termination (e.g., due to crashing), preventing the collection of sufficiently long sequences to bridge the warm-up period. The agent cannot act long enough to gather data that would allow it to improve. Addressing this issue key to unlocking the potential of sequence- training with SNNs in real-world control tasks.
In this work, we analyze the effect of surrogate gradient settings on the gradient being propagated throughout the network and investigate the role of surrogate gradients across a spectrum of learning regimes, from supervised learning to online RL. We introduce a novel RL algorithm tailored for continuous control with Spiking Neural Networks, explicitly leveraging their inherent temporal dynamics without relying on frame stacking. As a case study, we demonstrate the efficacy of our approach by training a low level spiking neural controller for the Crazyflie quadrotor [15]. The controller trained entirely in simulation and successfully transfers to the real-world platform, bridging the reality gap without the need for observation or action history augmentation. This work contributes both practical insights and theoretical implications for training temporal-aware, energy-efficient embodied agents.
2
Related Work
Several approaches have been proposed for combining Reinforcement Learning (RL) with Spiking Neural Networks (SNNs). Early work focused on biologically plausible learning rules, such as Timing-Depent
Plasticity (STDP), which have been successfully applied to tasks like source seeking [16] and maze navigation [17]. While these methods demonstrate the potential of SNNs for solving control tasks, they often lack the scalability and efficiency of modern deep RL techniques.
To benefit from the advances in deep RL., surrogate gradients can used to enable end-to-end training of SNNs with standard RL algorithms. The Deep Spiking Q-Network (DSQN) algorithm ||] adapts the DQN algorithm 7 to the spiking domain. Although DSQN achieves robust performance and demonstrates significant energy savings neuromorphic hardware [18]. it is trained on single transitions and resets the network state at every step. Therefore, it resets the network state at every environment transition, preventing the model from capturing temporal dependencies across time. Moreover, as a value-based method, DSQN is inherently limited to discrete action spaces and is not well-suited for continuous control.
More generally, value-based RL algorithms that leverage recurrent architectures, such as R2D2 19 rely on a warm-up period to stabilize hidden states before applying updates. This technique assumes the agent can gather long sequences of experience. However, in robotic control environments, such as drone flight, poorly initialized policies often result in early termination (c.g., due crashing), preventing the network from bridging this period. This limits the practicality of recurrent value-based methods in high-risk real- world environments.
To address these limitations, policy- based approaches have been explored. SNN-PPO (0) applies the Proximal Policy Optimization algorithm to train SNNs in an on-policy manner using entire trajectory sequences. This method has been shown to successfully solve a variety of continuous control tasks from the MuJoCo suite (2]. However, on-policy methods like SNN-PPO can be challenging to use when sample cost is high. Furthermore, they do not propose a solution for bridging the warm-up period.

---

Off-polica approaches enable reusing past experience stored in the replay buffer. PopSAN 22 an off policy, actor-critic method designed for continuous control with SNNs. By leveraging conventional ANN as the critic, PopSAN benefits from the stability of standard deep RI. while using an energy efficient SNN-based actor. achieves strong performance on several MuJoCo tasks in simulation and reports up to 140x reduction in energy consumption compared to non-eeuromorphic implementations [23]. Nonetheless, similar to DSQN, PopSAN requires the reset of the hidden states between transitions, limiting its ability to capture temporal dependencies across timesteps.
3 Methods
3.1 Online TD3BC with Jump-Starting Privileged Actor
To address the aforementioned challenges in sequential SNN training, we adapt the Jump-Start Reinforcement Learning (ISRL) framework [24]. leverages a pre-trained guide policy to create a curriculum of starting conditions for a secondary policy. We implement privileged, non-spiking actor, trained for few epochs through TD3 [25], training is stopped when the policy can hover the drone for the warm-up period consistently. Its primary function is to bridge the critical warm-up period required by the SNN. Both the guiding policy and the non-spiking critic receive privileged information in the form of action histories.
replay buffer is continuously populated throughout training with transitions generated by both the guiding actor (during the initial warm-up phase) and the spiking policy (during subsequent interactions). This hybrid approach bridges the early training period, where the spiking policy still gathers short sequences. Therefore, supervised learning from the guiding policy interactions is effective during early training.
The guide controller, which receives a privileged set of observations, including the action history, is used for the first 500 N timesteps, after which the spiking policy interacts for the remaining N steps, linearly increasing N until the guide policy is only used for the warm-up period. Initially, the buffer is filled mostly with experience from the guiding policy. This introduces a risk of training spiking policy that closely resembles the guide policy, which can be undesirable as the guide is usually not an expert policy. Therefore, the choice of the BC term weight is crucial to prevent the spiking policy from overfitting to the guide policy.
Once spiking policy gains baseline performance, and generates sufficiently long rollouts, want to leverage the reward information, for which online RL is better suited. Inspired by TD3BC 26).. we incorporate a Behavioral Cloning (BC) term into the RL objective function that utilizes the guide policy demonstrations stored in the replay buffer. The BC weight À decays exponentially, shifting from imitation to reward- based optimization. The BC term in our approach serves two purposes: avoiding large changes in policy behavior, improving training stability, and leveraging the demonstration data efficiently.
We call this approach TD3BC+ISRI, pseudocode can be found in subsection 8.4 maximized can be defined as:
The objective to be
Jr EYnD
Σ
1 122 (Qoi (Sr.s, To(s,, h,,)) >
- (o(Ss) hy,;) ar. a..11))) (1)
Where the history, h is defined as:
h = (5,,0, Sr. S7,4,11,,
(2)
h := 23.
(3)
Qor in the first term represents the first critic network, as used in TD3. The variable r denotes sequence sampled from the replay buffer, with a length of Nsae = 100. Within this sequence, Sy, and ar,, correspond to the :th observation and action, sampled from the replay buffer, respectively. The hyperparameter À controls the strength of the BC regularization and decays exponentially over time; the BC coefficient starts at A 0.2 and decays exponentially (A + 0.99X) after each epoch, shifting emphasis from imitation to reward optimization as training stabilizes. When the guiding
3

---

policy is of high quality, a larger À is preferred. Finally, Lign equals zero during the warm-up period (Dwarm-up up)). which in our case 50 steps, corresponding to 0.5 seconds in the drone control task. We zero-initialize hidden states, which causes only marginal performance differences [9] The critic is not time variant and thus does not need a warm-up period.
3.2 Spiking Neural Networks for Continuous Control
Spiking Neural Networks leverage bio-inspired neuron models, which transmit information through binary spikes. The membranc potential is charged by input current Lin. which incrementally charges its membrane potential U. Over time, the potential decays at rate determined by the leak factor 3. When the membrane potential exceeds defined threshold Uthr. the neuron emits spike, and its potential subsequently reset. We employ a soft-reset spiking mechanism, which described by:
1I,
if Uth 0, otherwise
Ult + 1 (U[(] Iin[t + Uthr: where s
Hence, the spiking function f() is the non-differentiable Heaviside function and calls for surrogate gradients [5] to enable gradient based optimization methods. During the backward pass, we act if this Heaviside function was a parametrized sigmoid function, a [kkz), which the true when k + co. For computational efficiency, the gradient of this parametrized sigmoid is approximated To avoid exploding gradients, the actual gradient of the fast sigmoid is normalized by k which leaves us with the derivative in Equation 5)
by the gradient of fast sigmoid (fs(k x) = I+k-|a
kor
17
d fa(k (]kk.x)=-
K
de
(1 (1+k- 2
(1+k-
where larger relates steeper slope of the parametrized sigmoid, reducing the range inputs for which a gradient exists.
SNNs work in spike- domain. To encode continuous values spikes and vice versa, population based encoding and decoding are used. Specifically, the first linear layer encodes continuous input values into currents, and final layer decodes the output spikes into motor commands. This approach has been proven successful in previous work 288
Our controller is a feedforward SNN with input size 18, two hidden layers and 4 output neurons. The two hidden layers of the network are of size 256 and 128 respectively and make use of Leaky Integrate-and-Fii neurons. This architecture enables analysis of surrogate gradient effects across multiple layers, sufficient capacity for continuous control, and is small enough for deployment on the resource constrained Crazyflie.
3.3 Surrogate Gradients in Deep Networks
Training SNNs via backpropagation requires the use of surrogate gradients approximate the derivative the non-differentiable spike function. critical hyperparameter in this process is the slope &t of the surrogate function, which determines the sensitivity of the gradient near the spiking threshold.
We adopt the gradient of fast sigmoid function, see Equation 5) and examine slope configurations ranging from shallow (k = 1) to steep (k: = 100). We analyze 4 layer SNN, with all layers having 64 neurons, layer 0 corresponds to the input layer, while layer relates to the output. The results on the plots are the average rumning the test 100 times. As shown in Figure la steeper slopes closely resemble the Dirac delta function but restrict non-zero gradients to narrow input range. In contrast, shallower slopes lead to a broader range of zon- gradients. This increases the average gradient magnitude, particularly in deeper layers (Figure 1b). Note that the average gradient magnitude is influenced by the proportion of neurons with zero gradients in each layer.
While the surrogate gradient's 's slope affects the quantity of weight updates per backward pass, it also introduces noise into the gradient computation (6]. Since the true gradient for deeper network layers does not exist, analyze the relationship between steep surrogate gradient (A 100), which

---

Surrogate Gradient
Average Gradient Magnitude
Average Gradient Magnitude
Average Cosine Similarity
100
Slope Values
Slape
10
Cosine Similarity
0.8
Gradient
Slope 10
Slope Values
Slope Values
Slope 25
10"
0.8
Slope
10
Slope 10
Stope
Slope
Slope 100
Slope
0.4
Slope
Dircc Della
0.2
20
0.00
10
Slope 100
Slope 100
-1.5
-055 Membrane potential, U
-1.0
Layer
Layer
(a) The slope of the surrogate dient dictates the range of inputs for which gradient exists.
(b) A more shallow slope carries the gradient deeper through the work, suffering less from vanishing gradients.
(c) A shallower slope introduces bias and variance, computed using the cosine similarity.
Figure 1: Surrogate gradient san---manndd--iinnneeeee tradeoff in deep SNNs. 1a Shallow slopes (k=1) provide gradients over wider input range than steep slopes (k=100). (Th) This prevents vanishing gradients in deeper layers, critical for multi-layer SNN training. (Ic) However, shallow slopes introduce directional noise (cosine similarity - 0), which aids RL exploration but hinders supervised convergence. Network: layers of 64 neurons; layer 0 first hidden. layer 4 = output.
approximates the true gradient VW, more truthfully than a shallow slope, and shallow surrogate gradients, denoted as VW, using cosine similarity:
VW, VW;
cosine similarity
(6)
VW Wiill
Figure shossows that the cosine similarity for shallow slopes reduce to O, therefore, the weight updates in deeper networks become essentially random.
We propose an adaptive surrogate gradient scheduling approach. Shallow slopes, which introduce stochasticity into the gradient computation, can facilitate the update of broader set of connections in deeper networks, enhancing exploration in RL, and increase the gradient magnitude. Conversely, steep slopes yield more precise gradient estimates, which are advantageous once the agent's performance stabilizes and fine-tuned optimization is desired. This mechanism provides a natural means of balancing exploration and exploitation throughout training.
Three schedules are analyzed:
Fixed: The surrogate gradient slope is held fixed throughout training.
Interval: The surrogate gradient slope is made steeper according to a fixed time schedule.
Adaptive: The surrogate gradient slope is adapted as function of the achieved reward.
The adaptive slope scheduler tracks both the reward value and its derivative (see Equation 7). allowing the slope to increase when performance improves and stabilize once it saturates. This keeps the gradient noisy when exploration is needed and sharp when fine-tuning. Note that the term depending on rt-s is largely responsible for maintaining the slope when maximum performance is reached, and thus prevents destroying progress.
9
h2
[0.5m + 0.5r
10
where k, is the slope at time f, Tt-i is the reward score at time t - i and r/=; is the first order derivative of the reward score at time Furthermore, ie is clamped between and 100, the same limits as used in Figure 3
3.4 Asymmetric Actor-Critic
While the actor is spiking, the critic a feedforward ANN, which receives observed states and an action history of 32 timesteps. This asymmetric setup has been demonstrated to successfully leverage
5

---

the improved training stability of ANN (22 23 The hump-starting actor used in this work, is also an ANN network with the same privileged inputs as the critic. This critic only used stabilize training, and is not deployed on the drone at inference time.
4
Experimental Setup
4.1 Quadrotor Position Control Task
We validate approach on low-level quadrotor control, challenging domain where (1) carly crashes prevent long sequences needed for SNN warm-up, directly testing our TD3BC+JSRL. solutiionn and (2) the temporal dynamics stress-test surrogate gradient choices across extended rollouts. he Crazyflie 2.1 platform [15] provides 18-dimensional state vectors of position, velocity, orientation, and angular velocity. The controller outputs motor commands a, IR at 100//z, matching the onboard sensor update rate.
Figure 2 illlstrrae the information flow between the Crazyflie platform and the spiking controller.
Controller
Us Sgg
ma Mag Wag, M944
0. 1
15. r.
CrazyFile
Figure 2: The spiking controller receives (2r, ys 2), linear velocity (Uz, Byy U2), orientation angles (8, 9, 1)), and angular velocities (p, r) inputs and outputs motor commands (m1, M2, Ma, m4).
Episodes terminate after 500 timesteps seconds) or upon crashes. This limit creates the core challenge our method addresses: early training policies crash within 100 steps, preventing data collection beyond the 50-step warm-up period required for stable SNN gradients.
Training completes in 6 hours on an M4 Pro MacBook (24GB RAM). We compare four approaches (Table 1):
Table 1: Method comparison. Only TD3BC+JSRL (ours) combines online learning with warm up bridging and demonstration leveraging, enabling sequence-basee based SNN training despite early crashes.
Method
Type
Leverages Reward
Warm-up
Uses Demos
BC
Offlin
X
X
TD3BC
Offine
X
X
TD3
Online
X
TD3BC+JSRL (Ours)
Online
V
4.2 Simulation Environment and Reward Structure
The controller is trained in simulation, relying dynamics model which has demonstrated sim-to- real bridging capabilities [14].
We employ linear curriculum learning approach where the reward structure increases difficulty as training advances. The total reward at each timestep computed using Equation 8
Ft Crs Crpllpt 1 Crull
Uses Cryllat Qdes Cra Cralla: Crabll
(8)
where Cr. is survival bonus encouraging long episodes, Crp: Crv. Cry penalize deviations from desired states, such as Pies. and Cra Crab penalizes deviations from hover throttle Crab- We apply curriculum learning by linearly interpolating penalty coefficients from lenient to strict values over training. The reward is explained in detail in subsection 8.2

---

Results
5.1
Surrogate Gradients During Training
We now analyze the effect of surrogate gradient slope choices both fully supervised, Behavioral Cloning (BC), and fully online training algorithms, TD3 [25]. To eliminate the effect of the warm-up during training for this analysis, do the following. We stack history of observations, process multiple forward passes per observation and reset the SNN in between subsequent actions, similar previous RL implementations for SNNs. The dataset used for BC has been gathered with the privileged actor introduced in subsecttion 3.1] The following experiments do not use reward curriculum, nor learn from the temporal dimension.
While slope choice barely affects supervised learning, online RL strongly prefers shallower slopes whose induced gradient noise enhances exploration, similar to parameter noise [29], albeit with higher variability across runs. Poor intermediate updates can corrupt the replay buffer with low-quality experiences, hindering convergence. Shallower slopes introduce noise in the policy update, and therefore heighten this risk, making training less stable.
We observe that scheduled slope settings firstly reduces the number of epochs to reach reward of 100 by factor x4.5 compared to steeper slopes. Secondly, final performance the trained agents lie in the same regime as fixed slope experiments.
Achieved (higher better)
Epochs Reach Reward of 100 (lower better)
Methods
175
Methods
400
Bispo: 50
Siepe: 2
Stepe:
150
125
Adapt
Best Reward
Epochs 100
100
200
100
EC
TDO
TOL
(b) For BC, final performance remains largely invariant to the slope setting, confirming the robustness of super- vised training. In RL, however, shallow slopes yield superior asymptotic rewards due to their exploration- promoting noise. Scheduled slopes, specifically adap- tive scheduling performs competitively, approaching the performance of optimally tuned shallow slopes while improving training stability.
(a) In BC. convergence speed is largely unaffected by the surrogate slope. the noisier updates from shal- slopes are compensated by broader gradient prop- agation. In contrast, RI. henefits significantly from shallower slopes, which enhance exploration through gradient noise. Scheduled slopes achieve competitive convergence speeds without manual tuning.
Comparison between fixed and scheduled surrogate gradient slopes for both BC and RL settings. While BC performance is largely dope-invarian RL benefits from shallower or scheduled slopes. which improve exploration and training efficiency. Scheduling effectively balances stability and performance, reducing the need for extensive slope tuning.
Figure 3:
While using an interval or adaptive schedule for the surrogate gradient slope does not always result in the best performing network, we find that can stabilize training. Moreover, achieves a near-optimal performance both training speed and final performance. As result, such schedules eliminate the need for exhaustive hyperparameter sweeps across different slope settings to identify the best performing ones.
5.2 Training on Sequences with Reward Curriculum
To fully leverage the temporal processing capabilities of SNNs in RL, we extend training from single transitions to full sequences (subsection 3.T) and introduce a reward curriculum that gradually increases penalties on position, velocity, and action magnitude to encourage stable, robust behavior.

---

All experiments use an adaptive slope scheduler for surrogate gradient slope adjustment. We compare BC, TD3BC, TD3, and TD3BC+JSRL. BC and TD3BC are offline RL algorithms trained on dataset sequences, gathered with the guiding policy used in our TD3BC+JSRL setup, while TD3 and TD3BC+JSRL, are online algorithms sampling sequences from the environment. Unlike TD3, which starts from scratch, TD3BC+ISRL. benefits from privileged guiding policy during the first n steps of rollouts, All approaches are evaluated under the same reward curriculum.
Comparison of Test Reward under Curriculum
Compartson of Test Interaction Length under Curriculum
50C
TD38C TDXBCHJBRL
Test Interacction Longth
500
Teat Reward
500
200
-200
102
20k 40K
6022
aCk
1O0K
Training Steps
20k
6.k
80% 100K
Training Stops
(a) Offline methods (BC and TD3BC) struggle the reward function adapts, failing to generalize beyond the initial dataset, due the lack of online interac- tions. TD3BC+JSRL able to adapt to the curriculum throughout training.
(b) While all methods, except TD3, eventually learn fly, TD3-trained policies frequently terminate carly duc to unstable exploration. TD3BC+JSRL achieves longer and more stable flight trajectories.
Comparison of training approaches under reward curriculum. All experiments were run with different seeds. As curriculum increases difficulty of the lask. we find that only TD3BC+JSRL able adapt to the changing curriculum. During testing, we average the results from 20 runs.
Figure 4:
TD3BC+JSRI. achieves 400-point average reward under curriculum learning, while BC, TD3BC, and vanilla TD3 fail to exceed 200 points (Figure 4a). This 600-point gap validates both components of approach: leveraging demonstrations (vs. TD3's from-scratch learning) and online adaptation (vs. BC/TD3BC's static datasets).
As the curriculum tightens, BC and TD3BC, trained on static datasets containing rollouts of the guiding actor, fail generalize, as shown in Figuur 4a| While TD3 has access to evolving interactions, struggles to gather meaningful sequences early in training, often resulting in premature episode terminations, as seen on Figure 4b) failing to bridge
warm-up period. Our proposed TD3BC+JSRL approach, which leverages new rollouts guided by the policy, demonstrates robust learning even under challenging curriculum. The inclusion of online rollouts enables the policy to adapt to the evolving reward landscape, leading to substantial performance improvements.
The hyperparameters, curriculum and training details used for the experiment are part of the supple- mentary materials, subsection 8.2
5.3 Bridging the Reality Gap
We quantitatively evaluated the computational efficiency of our sequential SNN approach using Neu- roBench [2]. The trained SNN is compared a feedforward ANN that achieves similar performance, trained using TD3. This controller requires explicit action history to control the drone successfully, Results show that temporally-traine tational traits. Despite a higher memory footprint, SNNs benefit from activation sparsity and primarily use energy-efficient accumulates (ACs) instead of energy- hunggyy multiply-ate hatiiply--ccuuulttes (MACs), making them well-suited for neuromorphic deployment. While the number of dense operations of the SNN much higher than the ANN to which we compare, the nature of the underlying ACs opens opportunity for energy efficient compute. Using the methodology proposed by Davies et al. [30, we can compute rough energy consumption estimate of 9.7 x 10 3m. see subsection 8.3 for further details.
SNNs match ANN performance while exhibiting distinct compu-

---

Table 2: NeuroBench performance comparison between ANN (64-64 architecture) and temporally-triie trainee SNN (256-128 architecture) using TD3BC+JSRL, While dense operations are computed ignoring activation and connection sparsity, effective operations reflect the spansity aware number of operations, as defined in
NeuroBench 2
Footprint (kb)
Activation Sparsity
SynOps Dense
SynOps Eff MACSs
SynOps Eff ACs
Requires History
Model
Reward
ANN
447
55.3
0.0
13.7 x 103
13.7 x 103
0.0
True
SNN
446
158.3
0.79
37.9 10y
4.6 x 10
12.2 x 10
False
Figure 5: When deployed on the Crazyflie, the spiking actor exhibits oscillatory hehavior but always successfully performs maneuvers such as circular flight.
When deployed on the Crazyflie, the SNN controller exhibits slightly oscillatory behavior. However, the SNN is still able to execute complex maneuvers like circles (see Figure 5). figure-eight and square trajectories without failure. ANN controllers, benefiting from full action history, show smoother control. However, when comparing to our SNN to an ANN which does not receive action history, we find that the ANN can no longer control the drone, as shown in Table 3
To improve stability, future work could incorporate angular velocity penalties in the reward function, throttle deviation outputs rather than absolute throttle settings, train for longer, increased control frequency, as prior work has shown smoother SNN control at higher rates 14 28
Compared to ANN controllers from Eschmann et al. [4], SNNs trained with our approach achieved lower position error under ideal conditions but had reduced reliability. Notably, while ANNs without action history failed to control the drone. A demonstration video is available online"
Table 3: Comparison of neural network models for position control and trajectory tracking tasks. The models include an ANN with action history, an ANN without action history and an SNN trained with TD3BC+ISRI. without action history. The mean position error of the deployed SNNs is benchmarked against the best-performing ANN policy [14]. Position error is measured the average xy xy-plane error (in meters), and trajectory tracking is evaluated the average error across figure-eight and square-following tasks. The ANN without action history did manage to stabilize the quadrotor.
ANN action history
ANN action history
SNN action history [Ours]
14
Position Error [m]
0.1
0.25
0.04
Trajectory Error [m]
0.21
NA
0.24
5.4
Ablation Study
To analyze the contribution of each component of the algorithm, an ablation study is performed. We first remove only the BC-term, then only the jump-start period. Lastly, we analyze the result with no BC-term and Jump-Start.
Removing the BC term allows the SNN controller to eventually show improvements in performance, but convergence becomes substantially slower, approximately 15 times more training steps required to achieve a reward 100. In contrast, eliminating the jump-starting actor entirely prevents

---

The implementation and experiment scripts used in this work are publicly available at The repository contains detailed instructions to reproduce all the SNN from collecting sequences long enough to bridge the warm-up up period, resulting in the network failing learn stable flight altogether. As expected, removing both terms, leads similar behavior. We find the BC term to improve training efficiency when the jump-start period allows bridging the warm-up period, and fills the buffer with several demonstrations from the guiding policy.
Table 4: Ablation study analyzing the impact of the BC-term and Jump-Start period on training performance. Reported values show the average final reward and the number of environment steps required reach reward of 100.
BC-term
Jump-start
Reward
Steps to 100
Yes
Yes
412 + 6.72
2200 + 124
No
Yes
334 + 25.59
33030 = 1600
Yes
No
32 + 1.2
NA
No
No
36 - 4.1
NA
6 Conclusion
This work addresses key challenges in training Spiking Neural Networks (SNNs) for Reinforcement Learning (RL) by combining theoretical insights into surrogate gradient slope settings with novel RL approach for sequential training. We demonstrate that the surrogate gradient slope plays a critical role in optimization dynamics, affecting both gradient alignment and the gradient magnitude. We find that shallower slopes increase the gradient magnitude at the cost of alignment with the true gradient. In supervised learning approaches, these effects balance out. However, in RL approaches, we find a strong preference towards shallower slopes. We propose adaptive slope scheduling strategies that improve training stability and performance. We demonstrate that adaptive slope scheduling improves training efficiency by x4.5 compared to fixed slope of 100.
To address the limitations of standard RL algorithms in temporally extended tasks, we introduce jump-start framework that leverages privileged policies to bootstrap training. This enables effective sequence-based learning in unstable control tasks, such as drone flight, where subpar controllers fail to generate experience long enough to bridge the warm-up period, which is necessary to efficiently train stateful networks. We find our method to enable SNNs to achieve a final average reward of 400 points, while other methods fail to surpass final performance of 200 points. We demonstrate the performance of our SNN receiving only the latest state information be competitive to an ANN which receives explicit action history. A comparable ANN without this action history fails to control the Crazyflie successfully.
Our method depends on the availability of a guiding policy. Although this policy can be any function, leveraging privileged observations and does need be deployable at inference, must still produce stable behavior early in training, which might not always be trivial to obtain.
Code Availability
reported results.
7
Acknowledgments
Author Guido de Croon gratefully acknowledges support from the Dutch Research Council (NWO) under grant number 20663 of the VICI personal grant programme.
References
[1] Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. 10:1659-1671, 1997.
10

---

[2] Jason Yik, Korneel Van den Berghe, Douwe den Blanken, Younes Bouhadjar, Maxime Fabre, Paul Hueber, Weijie Ke, Mina Khoci, Denis Kleyko, Noah Pacik-Nelson, et al. The neu- robench framework for benchmarking neuromorphic computing algorithms and systems. Nature communications, 16(1))1545, 2025.
[3] Rui-Jie Zhu, Qihang Zhao, Guoqi Li, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2.302. 13939, 2023.
[4] Xinyu Shi, Jianhao Ding, Zecheng Hao, and Zhaofei Yu. Towards energy efficient spiking neural networks: An unstructured pruning framework. The Twelfth International Conference on Learning Representations, 2024.
[5] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks.
[6] Julia Gygax and Friedemann Zenke. Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks.
[7] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
[8] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subrama- nian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223-22) 228, 2022.
[9] Greg Wayne, Chia-Chun Hung. David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwi nssaabawi 'sska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, Mevlana Gemici, Malcolm Reynolds, Tim Harley, Josh Abramson, Shakir Mohamed, Danilo Rezende, David Saxton, Adam Cain, Chloe Hillier, David Silver, Koray Kavukcuoglu, Matt Botvinick, Demis Hassabis, and Timothy Lillicrap. Unsupervised predictive memory goal-directed agent. 2018
[10] Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber. Recurrent policy gradients.
[11] Ding Chen, Peixi Peng, Tiejun Huang, and Yonghong Tian. Deep reinforcement learning with spiking q-learning. arXiv.org, 2022.
[12] Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, and Huajin Tang. Human-level control through directly trained deep spiking g-networks. IEEE transactions cybernetics, pages 1-12, 2022.
[13] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR.
[14] Jonas Eschmann, Dario Albani, and Giuseppe Loianno. Learning to fly in seconds. 11 2023.
[15] Wojciech Giernacki, Mateusz Skwierzzyíski, Wojciech Witwicki, Pawet Wroński, and Piotr Kozierski. Crazyflie 2.0 quadrotor as a platform for research and education in robotics and control engineering. In 2017 22nd International Conference on Methods and Models in Automation and Robotics (MMAR), pages 37-42, 2017.
[16] Rázvan V Florian. A reinforcement learning algorithm for spiking neural networks. Symposium on Symbolic and Numeric Algorithms for Scientific Computing, pages 299-306, 2005.
[17] Nicolas Frémaux, Henning Sprekeler, and Wulfram Gerstner. Reinforcement learning using continuous time actor-critic framework with spiking neurons. PLOS Computational Biology, 9(4):1-21, 04 2013.
11

---

[18] Mahmoud AKl, Yulia Sandamirskaya, Florian Walter, and Alois Knoll. Porting deep spiking q-networks to neuromorphic chip loihi. In International Conference on Neuromorphic Systems 2021, pages 1-7, 2021.
[19] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. ICLR, 2019.
[20] Luca Zanatta, Francesco Barchi, Simone Manoni, Silvia Tolu, Andrea Bartolini, and Andrea Acquaviva. Exploring spiking neural networks for deep reinforcement learning in robotic tasks. Scientific Reports, 14(1): 3464483 2024.
[21] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033, 2012.
[22] Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos Michmizos. Deep reinforce- ment learning with population-coded spiking neural network for continuous control. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 2016-2029, PMLR, 16-18 Nov 2021.
[23] Guangzhi Tang, Neclesh Kumar, and Konstantinos P. Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. 3 2020.
[24] Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-start reinforcement learning. 4 2022.
[25] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International conference on machine learning, pages 1587-1596. PMLR, 2018.
[26] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning.
[27] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 30(6):1514-15A4 1541, 2018.
[28] Stein Stroobants, Christophe De Wagter, and Guido CHE De Croon. Neuromorphic attitude estimation and control. IEEE Robotics and Automation Letters, 2025.
[29] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abheel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: (tff::170.009000 2017.
[30] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam. Shweta Jain, Yuyun Liao, Chit-Kwan Lin. Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnb Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, and Hong Wang. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 18, 2018.
[31] David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough, 10 2021.
[32] Guan Wang, Yuhao Sun, Sijie Cheng, and Sen Song. Evolving connectivity for recurrent spiking neural networks. Advances in Neural Information Processing Systems, 36:2991-3007, 2023.
[33] Federico Paredes- Vallés, Jesse Hagenaars, Julien Dupeyroux, Stein Stroobants, Yingfu Xu, and Guido de Croon. Fully neuromorphic vision and control for autonomous drone flight, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state the main contributions of the paper. A surrogate gradient analysis, and an RL approach for temporal learning for SNN.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limited stability of the controller, and propose directions of research to potentially resolve the issue.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[Yes]
Justification:
We describe the assumptions made the analysis of the surrogate gradient.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The methodology has been described, and specific implementation details are provided. Supplementary materials will be added, giving insight into specifics such as the reward curriculum.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The repository containing the code publically available, including manual
to reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//mdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The required details make sense of the results are part of the core paper. Additional details for reproducability will be included in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We believe that where relevant, the error bars appropriately defined.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were ran on an Apple MacBook Pro M4Pro, with 24GB RAM.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttt::///errtte http///wwww....bb.b.......ccccaa
Answer: [Yes]
Justification: ment.
All experiments were done in simulation or in controlled real-world environ-
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper presents analysis of surrogate gradients and reinforcement learning method for SNNs.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Apart from the Crazyflic simulator, all work has been done by the authors.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper describes analysis and method but does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19

---

Supplementary Materials
8.1 Effect of Surrogate Gradients on Gradient Alignment
When training spiking neural networks (SNNs), the non-diffcrentiabbiity of the spiking function is often handled by replacing with smooth surrogate function. In this section, we formalize how the slope of the surrogate gradient affects weight updates in deeper networks, supporting the main paper's s findings on cosine similarity decay. We replace the binary spike with a sigmoid to obtain a tractable closed-fomm baseline, however, believe the qualitative findings hold for spiking neurons as well.
Network Setup. We consider an ANN feedforward network with two hidden layers and sigmoid activations. Let E R" denote the input vector. Each layer performs an affine transformation followed by nonlinearity:
21 Wix + b a1 o(z))
22 = W2a1 + b = o (22)
Z3 W3a2 + by a3 z3
Here, (()) I+e- is the sigmoid activation. The output layer is linear layer, not followed by an activation.
Backpropagation and Surrogates. During backpropagation, we can compute å for neuron in the layer / (going from to 3 for the input to output layer) as:
n++1
(1(+118(411) by
o2(2))
now the weight update computed as:
0L aw! ;
(1-1) 65
aj
When computing the gradient of the sigmoid, replace the true derivative o2(2) with surrogate gradient:
o' d'kkz) = - (1- (k=))
(x)))
k
where & controls the slope of the surrogate. Since the surrogate gradient involves both scaling the input by (to sharpen the activation) and differentiating with respect to that input, the resulting gradient scales with & as well, due to the chain rule. Without compensating for this, gradient magnitudes can explode as & increases. We therefore divide by & to stabilize gradient flow regardless of the steepness."
Bias in Gradient Magnitude.
The gradient of the second layer weights becomes:
ØL
(1)
Σ W
(=
aw
A=1
Compared to the true gradient using o' (z), ), the surrogate gradient generally yields:
(x) 2 o' d'(x) for most z
This introduces a positive bias gradient magnitude, as:
OL. aL
bias unless & 1, which recovers the original derivative.
>
0
(ow
(w ji
20

---

Effect on Deep Layers. first-layer gradient becomes:
In deeper networks, this overestimation accumulates. For example, the
OL
R2 Σ
W
nB Σ
W 3 W gh
ok(2(")
aw Dij
where the sigma' larger than their true counterparts. Compared to its counterpart using the true derivative, the surrogate gradient not only increases magnitude but distorts direction.
Besides increasing magnitude, the surrogate gradient also distorts direction. To quantify this, compute the cosine similarity between gradients using surrogate and true derivatives:
VW. VWV
cosine similarity = |
|VW
As shown in Figure of of the main paper, cosine similarity decays toward 0 in deeper layers when using shallow slopes (e.g.. 1). This indicates that surrogate- based gradients become increasingly misaligned with the true gradient, effectively randomizing weight updates.
8.2 Simulator and Training Details
The design of the reward structure and termination conditions plays critical role in shaping the learned policy. Overly strict rewards can prevent learning altogether, while rewards that are too lenient often lead unsafe suboptimal behavior. In practice, tuning the reward function is crucial for the success of reinforcement learning algorithms [3..
The reward function is described as:
Y1
Crs
Crp Ipt
Pdes 1|2 Crollve - Udeal" Cryllat - q Crallat - Crab 11
(9)
where:
Pt E IR3 the current position (x, y, z), Pdes is the desired hover position
Uz R is the linear velocity (Ux, Uys Uz), Vdes is the desired velocity
q R represents the orientation angles (0, Φb, v)), ques the desired orientation
ar R is the action (motor commands)
Cra is the survival bonus to discourage early termination
Cras is fixed action baseline offset
To encourage stable and progressively more precise control, a reward curriculum is applied by linearly increasing the penalties during training. The initial and final values are summarized in Table
Crp
Crv
Cra
Crg
Crs
Crab
Start
1.0
0.01
0.14
0.25
1.0
0.667
End
3.5
0.10
0.50
0.25
1.0
0.667
Table 5: Reward parameters used during training. Penalties increase linearly across epochs, updated in six steps.
Drone Dynamics. The simulated drone uses simple physics model. Motor thrust is derived from RPM via second-order polynomial:
T = + rpm + rpm?
The motor dynamics are modeled using a first-order low-pass filter:
rpm rum
Arpm
where represents the motor time constant. Body-frame dynamics are integrated and then transformed to the world frame, described by Eschmann et al. [4).
21

---

Table 6: Training hyperparameters for all methods.
Parameter
BC
TD3BC
TD3BC+JSRL [Ours]
TD3
Common Parameters
Hidden sizes
[256, 128]
[256,
256]
[256, 128]
[256, 128]
Learning rate
le-3
le-3
le-3
le-3
Buffer size
1M
1M
2M
2M
Slope start
2
2
2
Slope schedule
adaptive
adaptive
adaptive
adaptive
Scheduling order
3
3
Method-Specifte Parameters
Warm-up steps
50
50
50
Policy noise
0.0
0.0
0.2
Noise clip
0.5
0.5
a (TD3BC)
2.0
2.0
(Target)
0.01
0,01
0.01
(Discount)
0.99
0.99
0.99
(BC coef)
0.2
BC decay factor
0.99
Training epochs
300
300
1000
1000
Steps per epoch
2M
1M
Training Hyperparameters. Tabveloprovidees an overview of the hyperparameters used across all methods. Common parameters are listed first, followed by method-specific values.
8.3 Energy Consumption Analysis
Although we do not have access to neuromorphic hardware small enough to fit on the Crazyflie, the
energy consumption can be estimated using the method proposed by Davies et al. [30], also used in
that the total energy per inference would be 9.7 10-5 mJ. which is in line with the result from Wang et al. [32] (normalized per timestep), and also in line with real-world measurements from Paredes- Paredes-Valles et al. [33), who measured 7 x 10-3 mJJ for much larger and deeper vision
32
network deployed on a neuromorphic system.
An overview of this calculation is given below.
Table Simulation Parameters and Energy Values
Parameter
Value
Energy per synaptic spike op P, Within-tike tile spike energy Pw Energy per neuron update P.. Layer sizes [Nin, N₁, N2, Nout] Activation Sparsity AS
23.6 (pJ)
1.7 (pJ)
81 (pJ)
18, 256, 128, 4 0.79
The total energy per inference is estimated as:
E [N (P, + N Pw)]
[Pa (1 (1-AS) Ni + N2. - (Pa + N1 Pb)]
[Ps (1- AS) N2 + Nout (Ng Pw)
9.7 10 mJ
Note: The multiply operations of the input encoding are ignored. The output is a linear layer that accumulates spikes, without explicit output neurons, therefore no energy for neuron updates P, are accounted for.
22

---

Discussion
comparison of a conventional method (traditional controller or RNN) versus our method on the Teensy would not be fair. For realizing the energy efficiency, our method depends on neuromorphic hardware that leverages sparsity and asynchronous compute, while conventional methods would not be able to profit as much. Therefore, the hardware implementation section of our paper aims at demonstrating the feasibility neuromorphie control, not at showing improved energy efficiency.
8.4 Pseudo Code: TD3BC+JSRL (Sequence-Basee Training)
Algorithm TD3BC+JSRL for Online Spiking Policy Training
1: Initialize: Replay buffer D, environment &, guiding policy Teul
2: Initialize: Spiking actor me, crities Qø. Qæz target networks wer Q Qag
3: Hyperparameters: a (TD3BC weight), Anc (BC coefficient), T (soft update rate), 7 (discount),
Gexplore (exploration noise), wwarrmup (warm-up steps), (BC decay)
4: for each training iteration do
5: Collect trajectories
6: for each rollout do
7:
Reset environment: 8y 3 for timestep t to T 1 do
8:
9:
if twarmup then + (5t) T8(8))
10:
D Privileged guiding policy
b Warm up spiking hidden states
12:
else
a: - no c, CEEENN( NO,, Tesplore clip(a,, 2,
14:
15:
end
16:
Execute a,, observe (se+1, de) end for
17:
18:
Slice trajectory into overlapping sequences as, $441)f/ with stride A
19:
Add sequences 7 to D
20:
end for
21:
Network updates
22:
for each gradient step do
23:
Sample mini-batch of sequences {T from D
24:
for each sequence = {(,,, a, Tz, di, 8141) 1
do
25:
Critic update: for all timesteps t [1,
26:
31 = ri + 7(1 de) miny Qay (St+1, Tgr (8t+1)
27:
6 6j V (s;;
Il, for 1,2
28:
à. < (8)): Q. < Qo, (Nr, ât)
29:
Actor update: only for t 2 twarmup
30:
LBc |0: 12
31:
A = a mean(
32:
- 0 V-(- + ABCCBC)
33:
end for
34:
Soft update targets:
35:
0' - 70 (1-
36:
0f - TØ, (1- for j = 1,2
37:
end for
38:
Decay BC coefficient:
ABC <- AACC
39:
if curriculum learning enabled then
40:
cuupdate_currculunn
41:
end if
42:
end for
43: return trained spiking policy me
23


Reviews:
[{"review_id": "7Z4wxGz3x5", "reviewer": "Reviewer_HZCM", "summary": "The authors propose a method for training spiking neural networks via surrogate gradients in a variety of problem settings, from supervised learning to online RL. Because spiking neural networks aren’t differentiable, it necessitates the use of surrogates for the gradients being applied to the network. The authors also highlight the temporal nature of spiking neural networks aligns with the sequential decision making of RL methods. The authors present a thorough analysis of the effects of surrogate gradients between different layers of the network that inform their choice of a surrogate.", "questions": "- Where do the hidden states fit into the architecture of the SNN? I see a description of the SNN architecture in Sec. 3.2 but it only mentions the inputs and outputs when interacting with the CrazyFlie.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- The authors compare against a variety of methods that have been previously applied to spiking neural network training.\n\n- The analysis of surrogate gradients help inform the design of the RL algorithm proposed. \n\n- Sec. 4 could be better motivated. For example, I do not really understand the purposes of the experiments in 4.1 and 4.2. These sections focus on describing what is being done, but it is not really clear why we need to analyze the surrogate gradients.\n\n- The method appears to be named TD3BC+JSRL, but this name is not introduced until Sec. 4.3. I think it would be better to introduce it in Sec. 3 (or earlier) and describe exactly what are the components of TD3BC+JSRL.\n\n- I don’t have a clear understanding of how difficult these domains in the paper are because I am unfamiliar with them. This makes it difficult for me to judge the significance of the results.", "was_revised": true, "final_justification": "I believe this is a good paper that targets an interesting area in neural network training (spiking networks). It builds off theoretically justified work in smaller settings and helps motivate future theory work on spiking networks and surrogate models. I like the ideas, but wish there was more unified benchmark domains to understand the significance of the work. I'm confident the authors will address presentation issues. To me this is the biggest drawback of the current paper. However, if I saw this as a poster at NeurIPS, I would engage with the authors. The subject and approach are interesting to me, and I'd like to see more papers like this at NeurIPS overall, even with the weaknesses discussed.", "rebuttal": "We thank the reviewer for the thoughtful and specific feedback. We thank your positive feedback on the analysis of the surrogate gradients to help inform the RL algorithm design.\n\nWe acknowledge that Section 4 should be better motivated. The analysis of the surrogate gradients and the scheduling of this surrogate gradient aims at emphasizing the importance of this parameter in successfully training SNNs in both supervised and reinforcement learning settings.\n\nConventionally, the slope setting for the surrogate gradient is not explicitly discussed or analyzed in current literature. However, we find that the setting for this gradient is a crucial parameter for training behavior and the final performance of the SNN. In RL settings, this effect is even more prevalent, as we show.\n\nConcerning mathematical rigor, the theoretical foundations of surrogate gradients have been rigorously analyzed by Gygax and Zenke (Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks), who provide mathematical proofs for how surrogate gradients approximate true gradients in small networks. Our work extends their theoretical framework empirically to address three critical gaps:\n\n- Extending the analysis to deep networks: We empirically demonstrate how these deviations compound through layers, showing that shallower slopes enable more gradient updates but worsen the approximation of the true spiking function—an effect that intensifies in deeper architectures.\n- Revealing RL-specific dynamics: We show that in RL settings, the gradient noise from shallow slopes actually aids exploration, creating a unique trade-off not present in supervised learning.\n- Providing a practical solution: Our adaptive slope scheduling method brings robustness and eliminates the need for extensive hyperparameter sweeps.\n\nWhile Gygax and Zenke provide the mathematical foundation for single neurons and small networks, extending these proofs to deep networks in RL settings remains an open challenge. We believe our empirical analysis and practical insights provide valuable contributions that complement the existing theory and can inspire future theoretical extensions to deep networks and RL contexts.\n\nWe discuss that no single optimal setting for the surrogate gradient slope exists, but that this depends on task performance, learning paradigm, etc. We find that scheduling the slope of the surrogate gradient can consistently compete with the optimal slope setting across all experiments.\n\nIn Section 4.1, we aim to provide the reader with a fundamental understanding of the effect of the surrogate gradient slope setting on the gradient that is propagated throughout the network. In Section 4.2, we discuss how this affects training across both supervised learning (BC) and online RL (TD3), which we believe cover the two extremes of the spectrum we consider in our article.\n\n\nFor the camera-ready version, we will restructure the method section to clearly introduce the final proposed method, TD3BC+JSRL, formally. We understand the authors' concern and believe this will improve the readability of the article.\n\nWe recognize the need for a more common benchmark to display the performance of our method, but believe that the chosen task of low-level quadrotor control covers several challenging aspects which our controller needs to handle. From a robotics standpoint, we believe quadrotors pose an ideal use case for neuromorphic solutions due to the high control frequency required to control these platforms and the stringent requirements on computational resources.\n\nFrom a learning perspective, the task is interesting as the platform is inherently unstable and very sensitive to control inputs, which makes it even challenging to hover. We focus on the lowest level control (motor-level) where related work usually focuses on a more abstracted level of control, with classic control solutions translating the higher-level controls (e.g., orientation) to motor commands. End-to-end control remains challenging with current solutions.\n\n\nWhere do the hidden states fit into the architecture of the SNN?\nSNNs leverage neurons which charge membrane potential. This can be seen as a type of self-recurrency which is passed through a step function to compute the output of the neuron (and consequently release the built-up charge).\n\n---\n\nWe sincerely thank you for your thorough review. Your feedback has helped us identify areas where our presentation can be clearer. We are confident that the camera-ready version will address your concerns while maintaining the paper's core contributions to practical SNN training for robotics."}, {"review_id": "oY9DfJK4T3", "reviewer": "Reviewer_FSHc", "summary": "Spiking neural networks (SNN) have advantages for energy-constrained applications since it’s mostly sparse. The paper focuses on applying spiking neural networks (with surrogate gradients) in reinforcement learning to control a small drone. The authors introduce a hybrid approach between behavioral cloning (BC) and reinforcement learning (RL) where BC is applied using a guiding policy that produces good transitions at the beginning of learning where the contribution of BC fades away with time and the RL components becomes dominant. The paper validates the viability of their approach by real-world deployment of the algorithm into a small drone.", "questions": "- What does “The critic is not time variant and thus does not need a warm-up period” mean? The objective clearly applies the warmup and prevents any updates for both the actor and critic.\n- Why is the critic ANN instead of SNN?\n- The authors have in line 180: “We stack a history of observations, process multiple forward passes per observation and reset the SNN in between subsequent actions, similar to previous RL implementations for SNNs”. I’m confused because the authors argued that with SNNs we don’t need to have frame stacking. How can we reconcile both statements?", "limitations": "The authors mention the limitations of their approach in the last section in the paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The authors address an important problem, especially since it enables effective robot policies at small scale where energy consumption becomes a significant problem (e.g., tiny robots).\n- The approach is validated by deploying on a real hardware which shows its promise.\n\n**Weaknesses:**\n\n- The role of BC is not clear. Why are SNNs unstable at the beginning of learning? The network should learn the representations and hidden states requires to perform better from the beginning.\n- The mathematical part is not rigorous. Equation 1 has hard-coded numbers 100 and 50, where clearly those are hyperparameters of the algorithms that need to be tuned for each problem, right? Also, what is the significance of the value 100? Are all episodes constrained to have lengths of 100s?\n- Unclear writing in the empirical evaluation lacks many details and it makes it hard to follow. The task should be described well, followed by the experiment setup and the results, and ending with a comprehensive discussion. I don’t think the authors follows this standard way of writing; instead the information is scattered and hard to follow.\n- Many crucial components of the papers are missing:\n    - The reward function definition\n    - The task being addressed in the experiment is not defined.\n    - Missing algorithm pseudocode. It’s not clear what algorithm steps are. The information is scattered all over the paper.\n    - Standard bars in Figure 3.\n    - Video of the performance of the drone moving in circles, eight-figures and squares. I checked the supplementary materials but couldn’t find any video. The only evidence is an edited photo that adds the drone in multiple places in the photo. I don’t think is a clear evidence of the behavior.\n    - Comparison with recurrent ANN approaches.\n    - Ablation study to understand the importance of each component in the TD3BC+JSRL.\n    - Energy consumption on the drone when using TD3BC+JSRL against conventional approach.\n\n\n**Minor issues:**\n\n- The acronym JSRL is used in the paper without definition. It is referring to jump-start reinforcement learning, right?\n- It’s not clear why SNNs have higher memory foot print as mentioned in line 232.\n- Trajectory error in ANN with no action history cannot be infinity as reported in Table 2. It has to be a number that is measured by the authors. Please update it with the actual value.\n- What is the meaning of altered drone in Table 2? How is different from the original drone?\n\n\nOverall, I don't think the manuscript is ready in its current form for publication. I encourage the authors to keep working on this important problem and strengthen their paper by incorporating the feedback as needed.", "was_revised": true, "final_justification": "The authors made efforts to address my concerns. I increased the score accordingly. However, some concerns still remain.", "rebuttal": "We thank the reviewer for the detailed and elaborate review. We thank your positive feedback on our hardware deployment as this is usually undervalued. \n\n**What is the role of BC?**\n\nAs SNNs are notoriously difficult to train effectively, other works on SNNs for control resort to supervised learning (BC). To compare our methods to this learning paradigm, we have included behavioral cloning in our analysis. \n\nIn our own algorithm we use a behavioral cloning term in the objective function to guide the SNN agent towards effective control behavior. This speeds up learning significantly. However, relying on BC alone does not allow for surpassing the performance of the sub-optimal guiding agent, as expected. So the role of BC in our algorithm is to initially achieve a decent baseline performance, while still allowing for further improvement using the TD3 term. \n\n**Why are SNNs unstable at the beginning of learning?**\n\nWhen initializing an SNN, the hidden states are initialized to zero. This causes issues during the first few timesteps, before the hidden states have converged to more realistic values for real flight. When training in closed-loop settings, we find that the network is not able to achieve a baseline performance where its own flights are sufficiently long to cover this stabilization period, and to enter a phase where training is performed under realistic hidden states. \n\nIn drone control, where bad controllers cause the agent to crash and therefore generate very short sequences to train on, the network is unable to train effectively. In other environments such as grid-search environments, a bad controller might not be able to solve the task, but will still lead to lengthy sequences where training on these sequences delivers mostly gradient updates calculated under realistic hidden states conditions. \n\nNote, however, that during testing the SNN is not jump-started and takes control from the first timestep. The jump-starting using the guiding policy is only required for training, in the early phases of training, where the SNN itself can not yet bridge the warm-up period. \n\n**Mathematical Rigor and Surrogate Gradient Insights**\n\nWe acknowledge the concerns about mathematical rigor. The theoretical foundations of surrogate gradients have been rigorously analyzed by Gygax and Zenke in *Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks*, which we cite. Their work provides the mathematical framework for understanding how surrogate gradients deviate from true gradients in small networks and cause sign reversal.\n\nOur contribution builds upon this theoretical foundation by:\n- Extending the analysis to deep networks: We empirically demonstrate how these deviations compound through layers, showing that shallower slopes enable more gradient updates but worsen the approximation of the true spiking function, an effect that intensifies in deeper architectures. In this analysis we also shift our focus from sign reversal to general gradient alignment, using the cosine similarity. \n- Revealing RL-specific dynamics: We show that in RL settings, the gradient noise from shallow slopes actually aids exploration, creating a unique trade-off not present in supervised learning\n- Providing a practical solution: Our adaptive slope scheduling method brings robustness and eliminates the need for extensive hyperparameter sweeps\n\nWe emphasize that our finding about this trade-off can be explained intuitively: shallower surrogate gradient slopes allow gradients to flow to more neurons (enabling more updates), but at the cost of deviating further from the true binary spiking behavior. This fundamental trade-off compounds through network layers, as demonstrated in our empirical analysis and supported by Gygax' and Zenke's theoretical framework. This intuitive understanding is crucial for practitioners working with deep SNNs.\n\nOur slope scheduling method represents a key innovation that brings extra robustness to SNN training, preventing the need for exhaustive parameter sweeps that would be prohibitive for time-consuming RL experiments. This is particularly critical for hard continuous control tasks like drone flight, where the SNN must exploit its temporal dynamics over multiple timesteps, a challenge that distinguishes our work from simpler discrete control scenarios.\n\nWhile Gygax and Zenke provide the mathematical foundation for single neurons and small networks, focused on sign reversal, extending these proofs to deep networks in RL settings remains an open challenge. Although we agree that a mathematical proof would be valuable, we believe the intuitive understanding and empirical evidence we provide already constitutes a significant contribution and opens important avenues for future theoretical investigation.\n\n**Hyperparameters**\n\nRegarding Equation 1: It is indeed correct that 100 and 50 are hyperparameters, representing sequence length and number of warm-up steps respectively. These values were determined through preliminary experiments, balancing computational efficiency with learning effectiveness. For the camera-ready version, we will denote these correctly as hyperparameters.\n\n**Episode length and sampling strategy**\n\nEpisodes are constrained to 500 timesteps, as discussed in section 3.2. We split up the episode into sequences of 100 timesteps, sampled starting at 0, 50, 100 ... 400 timesteps respectively. From a successful episode of 500 timesteps, we can thus sample 9 unique sequences.\n\n**Experimental Details**\n\nThe task of the experiment is explained in section 3.2. The section also elaborates on the algorithm steps. However, we agree that this could be emphasized more clearly, and a pseudocode could improve the article's readability. The reward function has been described in the supplementary materials. It is the standard reward function from the Learning To Fly training simulator. For improved readability, we will include it in the main section of the camera-ready version.\n\nThe drone was flown in the facilities of our institution. Large logos are scattered around the room, which would conflict with the anonymity of the submission. We do, however, understand the need for transparency and upon potential publication, we will release video material demonstrating the proposed method in a real-world scenario. The edited photo was meant as a trajectory visualization, not as primary evidence, we apologize for the confusion.\n\nFurthermore, we compared the SNN to conventional ANN solutions since the SNN also does not have explicit recurrence in the model architecture. \n\nAn ablation study would indeed improve understanding of the components of the method and will be considered for the camera-ready version. We will analyze the sensitivity of the components of our algorithm by removing the BC term, removing the jump-start period, and using varying degrees of jump-starting agent levels, from novice to expert policies.\n\n**Minor Issues:**\n- JSRL indeed stands for Jump-Start Reinforcement Learning.\n- SNNs have a higher memory footprint as the hidden layers are larger, and the spiking neurons hold a memory state as well.\n- The trajectory error of the ANN was reported as infinity, as the ANN without action history during our testing never managed to fly the full trajectory and would always crash. We see that quantifying this as infinite is not correct, and will modify this in the camera-ready paper to reflect the fact that this network is not able to carry out the task (by replacing infinity with \"-\" and adding a footnote). \n- We understand the confusion regarding the altered drone, and should have described this in further detail. In this case, the regular 16mm long brushed motors of the Crazyflie are replaced by 20mm brushed motors that increase the thrust by about 40%. This way, the Crazyflie is much more stable while flying with the added weight from the Teensy 4.0 that was used to run our algorithm and greatly increased the practicality of testing. Besides, it also shows that our method is robust against a change in the dynamics in the real-world. We will discuss this in more detail in the camera-ready paper. \n\n**Critic architecture and training**\n\nThe critic is a feedforward ANN, and does not encode any temporal information. The ANN critic is updated using the standard TD3 loss function which does not use a warm-up period. The critic is therefore updated on all 100 timesteps of the sampled sequence, leading to improved training efficiency. \n\nAdditionally, ANNs display more stable training behavior, which we exploit by using an ANN critic instead of an SNN. Only the SNN will be deployed on the robot, and thus needs to benefit from the efficiency provided by the SNN. \n\n**Frame Stacking Clarification**\n\nWe understand the confusion and see the need for further clarification. \n\nCommon RL pipelines for SNNs reset the hidden states between predictions (and thus observations), therefore removing any temporal information capabilities of the SNN. However, a lot of information can be encoded in the hidden states of the spiking neurons. To leverage these, we need to stabilize them. These common RL pipelines pass a single observation several times through the network to output one single prediction. Before passing the next observation, the network is reset again. In the surrogate gradient analysis, we use the classic RL for SNN approach, and pass the network a history of observations to give it temporal context. In our own method, TD3BC+JSRL we enable training the agent using sequences, which removes the need for this frame stacking.\n\nWe sincerely thank you for your thorough review. Your feedback has helped us identify areas where our presentation needs to be clearer. We are confident that the camera-ready version will address your concerns while maintaining the paper's core contributions to practical SNN training for robotics."}, {"review_id": "wmBEhEWvks", "reviewer": "Reviewer_ps6o", "summary": "This paper focuses on the challenges of training Spiking Neural Networks (SNNs) for reinforcement learning (RL), particularly the non-differentiable nature of spiking neurons and the difficulty of training on sequences. The authors analyze the impact of surrogate gradient slope settings, finding that shallower slopes improve training and performance in RL. They propose a novel training approach using a privileged guiding policy to bootstrap learning, combined with an adaptive slope schedule. Applied to a drone position control task, this method achieves an average return of 400 points, outperforming prior techniques. The work advances understanding of surrogate gradients in SNNs and provides practical training methods for neuromorphic controllers, demonstrating successful real-world deployment on a Crazyflie quadrotor.", "questions": "1. The adaptive slope scheduler (Eq. 5) empirically improves training, but the paper lacks a theoretical explanation for why it works better than fixed slopes in some cases. Could the authors provide a convergence analysis (e.g., under what conditions the scheduler stabilizes training)? Also, could the authors provide a sensitivity analysis (e.g., how do choices of the weighting terms in Eq. 5 affect performance)?\n2. The paper focuses on SNN-vs-SNN comparisons (e.g., TD3BC+JSRL vs. BC), but how does the method fare against ANN equivalents (to quantify energy efficiency vs. performance trade-offs)?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. The paper presents a solid empirical evaluation, including comparisons with multiple baselines (BC, TD3BC, TD3) and ablation studies on surrogate gradient slopes. The results are well-supported by quantitative metrics (e.g., reward scores, gradient alignment analysis) and qualitative observations (e.g., real-world drone control).\n2. The methodology is sound, combining theoretical insights (e.g., surrogate gradient analysis) with practical innovations (e.g., jump-start RL with a guiding policy). The use of adaptive slope scheduling is particularly well-motivated.\n3. The work addresses critical challenges in training SNNs for RL, particularly in robotics (e.g., unstable early training, non-differentiability). The proposed method achieves 2.1 times performance gains over baselines, demonstrating practical impact.\n\nWeaknesses\n1. While the adaptive slope scheduler is promising, the paper does not fully explain why it outperforms fixed slopes in some cases but not others. A deeper theoretical analysis (e.g., convergence guarantees) would strengthen the claims.\n2. The computational cost of training (e.g., wall-clock time vs. ANN baselines) is not discussed, which is important for neuromorphic applications.", "was_revised": true, "final_justification": "The authors partially solved my concerns. After reading the rebuttals and the reviews from other reviewers, I would like to keep the score.", "rebuttal": "We thank the reviewer for their actionable feedback and insights. We thank you for your positive feedback on our empirical evaluation including multiple baselines.\n\n**Adaptive Slope Scheduler Analysis**\n\nThe adaptive slope scheduler finds its strength in finding the optimal slope setting for the training setup, by incorporating knowledge about the current training performance (is the network still improving?) and test performance (is the network already at top performance?). Using these factors, it aims to find the optimal slope setting. This addresses a fundamental and intuitive trade-off we discovered: shallower surrogate gradient slopes allow more neurons to receive gradient updates, but at the cost of worse approximation of the true binary spiking function, an effect that compounds through network layers. This finding is supported by Gygax and Zenke's theoretical framework: *Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks*, which discusses sign reversal when using surrogate gradients. We extend this finding empirically to deep networks in RL settings, and by shifting focus from sign reversal to gradient alignment using the cosine similarity.\n\nImportantly, our slope scheduling method brings critical robustness to SNN training, eliminating the need for extensive hyperparameter sweeps that would be prohibitive for time-consuming RL experiments. This innovation is particularly valuable for hard continuous control tasks like drone flight, where the SNN must exploit its temporal dynamics over multiple timesteps to maintain stability, a challenge that distinguishes our work from simpler control scenarios.\n\nHowever, this slope setting lags behind slightly due to the update based on the historic trend of the training performance. When we pick the perfect slope from the beginning, we find that this can in fact outperform the adaptive scheduling, but that the adaptive scheduling is always competitive with the optimal slope. Instead of having to perform large hyperparameter sweeps, including the slope setting, the adaptive scheduler thus finds this optimal slope setting automatically, although sometimes delayed.\n\nRegarding convergence analysis: While a full theoretical proof is beyond this paper's scope, we can provide intuition for why adaptive scheduling works:\n\n- **Early Training (low reward):** The scheduler selects shallow slopes ($k$ between 1 and 10), maximizing gradient flow to all neurons despite poor alignment, enabling broad exploration\n- **Mid Training (improving reward):** $k$ increases, gradually trading exploration for exploitation\n- **Convergence (high reward):** The proportional term in the slope scheduler maintains $k$ at optimal levels, preventing degradation\n\nThis mirrors established RL principles (epsilon-greedy, temperature scheduling) but operates in gradient space.\n\nAlthough we agree that mathematical convergence guarantees would strengthen our claims, we believe the intuition-based understanding and empirical evidence we provide already constitutes a significant contribution. Our work reveals fundamental insights about surrogate gradient behavior in deep SNNs and provides a practical solution that works robustly across different scenarios. This opens important avenues for future theoretical investigation into adaptive surrogate gradient methods.\n\nWe believe the method we propose is intuitive and grounded in theoretical findings of previous work by Gygax and Zenke in *Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks*. They analyze the sign reversal of the gradient which occurs when the shallow surrogate gradient slope is used. We shift our focus from sign reversal to general gradient alignment, using the cosine similarity. To improve understanding of our methods, we will analyze the sensitivity of the components of our algorithm by removing the BC term, removing the jump-start period and using varying degrees of jump-starting agent levels, from novice to an expert policy.\n\n**Computational Cost Considerations**\n\nThe computational cost of our approach compared to the baseline SNN is not discussed currently, but will be described more elaborately in the camera-ready version. In section 4.3 we mention that all experiments were trained on an Apple MacBook Pro with an M4Pro chip and 24GB of RAM. The SNN experiments took roughly 10 hours to train, with a single agent. The ANN solution was trained using thousands of parallel agents, using an RL library optimized in C, which allowed training in several minutes. This ANN network was then converted back to Python for further analysis. While we acknowledge this difference, the focus of the article has been on the proof of concept, rather than the optimization of the runtime of our algorithm in Python.\n\n**SNN vs ANN Comparison**\n\nThe reviewer correctly points out that most of our paper consists of SNN-vs-SNN comparison, and we do not look at the performance of NNs with explicit recurrency (RNN, LSTM, GRU, ...). We have chosen to compare the SNN only to a regular MLP (the ANN in Sec. 4.4) since although the feedforward SNN has implicit recurrency from the membrane potential, there is no explicit recurrency to other neurons in the same layer. In this comparison, we focus on the trained agent, rather than the training pipeline, as ANN training allows for using many open-source optimized libraries, while the SNN equivalent requires mostly custom solutions, which are not optimized for speed.\n\nThe biggest difference between the ANN and SNN is the need for an action history in ANN, while SNN can learn to fly the drone in a streaming fashion, where we do not need to keep an explicit action history, but rather just stream the state observations.  \nWith the ANN, it was necessary to input the action history along with the state observations to result in stable flight.  \nFurthermore, the SNN leverages mostly Accumulate operations, which are significantly cheaper in hardware to perform, compared to the costly Multiply Accumulate (MAC) operations, found in the ANN. We will draw the connection between the energy consumption and the operations more clearly in the camera-ready version by referring to the NeuroBench harness.\n\n---\n\nWe sincerely thank you for your thorough review. Your feedback has helped us identify areas where our presentation can be clearer. We are confident that the camera-ready version will address your concerns while maintaining the paper's core contributions to practical SNN training for robotics."}, {"review_id": "QP9kRPBS5J", "reviewer": "Reviewer_eoYk", "summary": "This paper investigates the training of Spiking Neural Networks (SNNs) for continuous control tasks using reinforcement learning. The authors identify two primary challenges: understanding the optimal surrogate gradients used to train SNNs, and overcoming the difficulty of training networks in RL environments with the limited length in early training.\n\nTo address these, the paper provides a systematic analysis of how the slope of the surrogate gradient function affects gradient propagation and learning performance. Secondly, it proposes a training framework, TD3BC+JSRL, which combines several existing techniques. This method uses a privileged \"guide\" policy to bootstrap the learning process and generate longer initial trajectories, an off-policy RL algorithm (TD3), and a behavioral cloning term to leverage the guide's demonstrations. The authors demonstrate their approach on a simulated drone position control task and show successful transfer to a real-world Crazyflie platform.", "questions": "1) What is the network architecture of your work? RNN? MLP? Please provide a full discription of it. \n\n2) The paper's motivation heavily relies on the energy efficiency of SNNs. It is encouraged to further investigate the energy efficiency on such edge situation: a flying robot, where batteries are heavy, so that energy is a much more important aspect to consider, compared to other edge computation scenarios.\n\n3) Could you please compare your result with other SNN - RL works, to better clarify your contribution? For example, how about a E-prop trained SNN, or a Evolution trained SNN? Here's some papers that might relevant: Evolving connectivity for recurrent spiking neural networks; A solution to the learning dilemma for recurrent networks of spiking neurons.", "limitations": "The primary limitation of this paper is its novelty and insufficient baseline comparisons. The proposed TD3BC+JSRL framework is a composite of well-established techniques. And the experimental evaluation is limited to variants of the TD3 algorithm.  The paper would be significantly stronger if it included direct comparisons against other relevant SNN-RL methods mentioned in the related work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strength\n\n1) A promising research direction. The work addresses the important and challenging problem of applying energy-efficient SNNs to real-world robotics, which is a promising research direction. Notably, flying robotics are highly energy sensitive, therefore it best suits the energy efficient SNNs.\n\n2) Thorough Empirical Analysis: The analysis of surrogate gradient slopes (Section 4.1 and 4.2) is a valuable empirical study. It provides practical insights into how this key hyperparameter affects training stability and performance, which will be useful for other researchers in the field.\n\n3) Valid Experimental Results: The proposed method achieves impressive performance on the drone control task, significantly outperforming the baseline methods (BC, TD3, and TD3BC) under the same conditions. The successful sim-to-real transfer is a notable achievement.\n\n# Weakness:\nThe primary weakness of this paper is its limited novelty. While the empirical results are strong, the core methodological contributions are incremental and rely heavily on combining existing methods without introducing a new, fundamental concept.\nThe proposed TD3BC+JSRL framework is a composite of well-established techniques. Additionally, the use of surrogate gradients is the standard approach for training SNNs. The paper's analysis is an exploration of this existing technique, not the introduction of a new one.\nThe main contribution is the specific application and successful integration of these existing components to the problem of training recurrent SNNs. While this is a solid piece of engineering and produces good results, it does not constitute a significant scientific advance.", "was_revised": true, "final_justification": "The response and discussion from authors have clarified my questions, and I decided to raise my score to 5, Accept. \n\nThere are two major reasons for the raise of my rating:\nA) SNN is inheriently temporal, therefore the best application for it is temporal controlling, for instance, flying control as the authors did. It  could be a possible significant application for SNNs, rather than tackling the tasks that is already accomplished by feedforward ANN architectures.\nB) The authors provide a novel training technique for SNN: scheduling the surrogate gradient, and provide a thorough and detailed analysis on it. It helps the stability of training SNN, especially on scenario of the paper.\n\nTherefore, after a revision, clarifying technical details and highlighting the significance, the paper is qualified enough to be accepted.", "rebuttal": "We thank the reviewer for the detailed and actionable feedback. We thank you for your feedback on our sim-to-real transfer as this is a time-consuming process which usually goes undervalued.\n\n**Core Novelty and Contributions**\n\nThe reviewer mentions a reasonable concern about the novelty and contributions of our work. We feel the need to further clarify this in this section and acknowledge the need to emphasize this in the camera-ready paper.\n\nA core novelty of our work is a scheduled approach to setting the slope, parameterizing this surrogate gradient. This slope scheduling method is an innovative component that brings critical robustness to SNN training, eliminating the need for extensive hyperparameter sweeps that would be prohibitive for time-consuming RL experiments. To build an understanding of the underlying effect of this parameter on the gradient propagated throughout the network, we include the surrogate gradient analysis.\n\nOur analysis reveals a fundamental trade-off: shallower surrogate gradient slopes enable more gradient updates (allowing more neurons to learn) but worsen the approximation of the true binary spiking function, an effect that compounds through network layers. This finding is supported by the theoretical framework of Gygax and Zenke in their paper: *Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks*, who rigorously analyzed the sign reversal which may occur when using surrogate gradients in small networks. We extend this understanding to deep networks, showing that not only sign reversal probability increases, but cosine similarity between the true and surrogate gradient decreases with a decreasing slope setting. We display how this affects RL settings more than supervised learning settings, providing crucial insights for practitioners.\n\nFinally, we find that scheduling the surrogate gradient to adapt during training allows for near-optimal performance in all test cases. Where the optimal surrogate gradient setting varies significantly, the scheduled approach succeeds in approaching the optimal surrogate gradient consistently.\n\nWe emphasize a second key insight: in scenarios where the quality of the SNN controller directly impacts the length of collected sequences, such as UAV control, where poor policies lead to crashes and thus very short trajectories, the SNN struggles to converge to effective behavior. These short sequences do not provide enough temporal context to stabilize the hidden states, which is necessary for learning. This challenge is particularly acute in hard continuous control tasks of unstable systems like drone flight, where the SNN must exploit its temporal dynamics over multiple timesteps to maintain stability, distinguishing our work from simpler discrete control scenarios.\n\nTo address this, we leverage a jump-starting agent. The function of this jump-starting agent differs significantly from the original publication, where its function is to increase the states seen by the trained agent. In our approach, the jump-starting agent is responsible only for bridging the warm-up period of the SNN, such that the hidden states can stabilize and that the generated sequences, which are used for training, are lengthy enough to learn from. We will more clearly emphasize this difference in the final version of the paper by introducing the privileged jump-starting actor purely as a stabilizing agent, rather than making the close comparison to the original Jump-Start RL paper. Furthermore, we will include an ablation study where we remove this jump-starting actor.\n\n**Network Architecture**\n\nThe network architecture is an MLP with stateful spiking neurons (Leaky-Integrate and Fire neurons). These neurons can to some extent be interpreted as a hidden layer with a diagonal recurrency matrix, which is subsequently passed through a binary threshold function.  \nAll networks contain 2 hidden layers, with 256 and 128 neurons for the first and second layer respectively. Although some of this information was given in the Supplementary Materials, we see that this is still lacking important data. We will extend the network description in the Supplementary Materials and clarify the network architecture in Sec. 3.3, along with the definition of the policy.\n\n**Energy Efficiency Considerations**\n\nThrough the analysis of the network using NeuroBench we aim to investigate the energy efficiency considerations for edge situations. The NeuroBench framework includes metrics which have been demonstrated to align closely with the energy consumption on real neuromorphic systems.  \nFor the real-world task specifically, we chose a task where energy efficiency is of prime importance: The CrazyFlie drone weighs only 27g, the batteries themselves weigh 7.1g, with flight times constrained to 3-5 minutes. Especially these platforms will benefit most from asynchronous event-based algorithms that can be deployed on tiny efficient neuromorphic hardware.\n\n**Comparison with Other SNN-RL Approaches**\n\nWe acknowledge that comparing the method to other RL approaches for SNNs would be valuable. However, the motivation of this work came from the fact that existing work using RL and SNNs ignores the temporal dimension of SNNs by training on single transitions. Ignoring this temporal dimension, we found that the approaches which have been covered in literature earlier (such as DSQN) lead to large SNNs which require multiple forward passes per prediction to stabilize the hidden states. Not only did these not fit into the memory of the processor on the CrazyFlie, these networks were also not able to run at 100Hz. E-prop is designed for online learning, failing to reach the same performance as backpropagation through time, and is often considered to have limited scalability compared to RL. Evolutionary trained SNNs were initially tried, but failed to generate usable controllers. Our method was therefore the only method that successfully trains an SNN to fly the CrazyFlie using the temporal dimension.\n\nWe sincerely thank you for your thorough review. Your feedback has helped us identify areas where our presentation can be clearer. We are confident that the camera-ready version will address your concerns while maintaining the paper's core contributions to practical SNN training for robotics."}]

Author rebuttal:

Meta-review: The authors investigate the challenges of training Spiking Neural Networks (SNNs) for continuous control in reinforcement learning (RL) setups and the impact of surrogate gradients. They propose a novel training approach and apply it to a drone control task.
Strengths:
- tackles an important problem with practical relevance
- approach is validated on real hardware
- solid empirical evaluation and comparison to other methods, and ablation studies
- it shows clear performance gains
Most concerns have been addressed and clarified during the discussion period. The authors provided an energy consumption analysis in the rebuttal, having a more detailed analysis would have been beneficial. Due to the many strengths of the manuscript with only minor remaining weaknesses, I propose acceptance.
Decision: Accept (oral)