Title: Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neuromorphics

Spiking Neural Networks

Reinforcement Learning

Surrogate Gradients

Robotics

Sequential

Temporal
Abstract: Neuromorphic computing systems are set to revolutionize energy-constrained robotics by achieving orders-of-magnitude efficiency gains, while enabling native temporal processing. Spiking Neural Networks (SNNs) represent a promising algorithmic approach for these systems, yet their application to complex control tasks faces two critical challenges: (1) the non-differentiable nature of spiking neurons necessitates surrogate gradients with unclear optimization properties, and (2) the stateful dynamics of SNNs require training on sequences, which in reinforcement learning (RL) is hindered by limited sequence lengths during early training, preventing the network from bridging its warm-up period.

We address these challenges by systematically analyzing surrogate gradient slope settings, showing that shallower slopes increase gradient magnitude in deeper layers but reduce alignment with true gradients. In supervised learning, we find no clear preference for fixed or scheduled slopes. The effect is much more pronounced in RL settings, where shallower slopes or scheduled slopes lead to a $\times2.1$ improvement in both training and final deployed performance. Next, we propose a novel training approach that leverages a privileged guiding policy to bootstrap the learning process, while still exploiting online environment interactions with the spiking policy. Combining our method with an adaptive slope schedule for a real-world drone position control task, we achieve an average return of 400 points, substantially outperforming prior techniques, including Behavioral Cloning and TD3BC, which achieve at most –200 points under the same conditions. This work advances both the theoretical understanding of surrogate gradient learning in SNNs and practical training methodologies for neuromorphic controllers demonstrated in real-world robotic systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the training of Spiking Neural Networks (SNNs) for continuous control tasks using reinforcement learning. The authors identify two primary challenges: understanding the optimal surrogate gradients used to train SNNs, and overcoming the difficulty of training networks in RL environments with the limited length in early training.\n\nTo address these, the paper provides a systematic analysis of how the slope of the surrogate gradient function affects gradient propagation and learning performance. Secondly, it proposes a training framework, TD3BC+JSRL, which combines several existing techniques. This method uses a privileged \"guide\" policy to bootstrap the learning process and generate longer initial trajectories, an off-policy RL algorithm (TD3), and a behavioral cloning term to leverage the guide's demonstrations. The authors demonstrate their approach on a simulated drone position control task and show successful transfer to a real-world Crazyflie platform.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n1) A promising research direction. The work addresses the important and challenging problem of applying energy-efficient SNNs to real-world robotics, which is a promising research direction. Notably, flying robotics are highly energy sensitive, therefore it best suits the energy efficient SNNs.\n\n2) Thorough Empirical Analysis: The analysis of surrogate gradient slopes (Section 4.1 and 4.2) is a valuable empirical study. It provides practical insights into how this key hyperparameter affects training stability and performance, which will be useful for other researchers in the field.\n\n3) Valid Experimental Results: The proposed method achieves impressive performance on the drone control task, significantly outperforming the baseline methods (BC, TD3, and TD3BC) under the same conditions. The successful sim-to-real transfer is a notable achievement.\n\n# Weakness:\nThe primary weakness of this paper is its limited novelty. While the empirical results are strong, the core methodological contributions are incremental and rely heavily on combining existing methods without introducing a new, fundamental concept.\nThe proposed TD3BC+JSRL framework is a composite of well-established techniques. Additionally, the use of surrogate gradients is the standard approach for training SNNs. The paper's analysis is an exploration of this existing technique, not the introduction of a new one.\nThe main contribution is the specific application and successful integration of these existing components to the problem of training recurrent SNNs. While this is a solid piece of engineering and produces good results, it does not constitute a significant scientific advance.\n\nQUESTIONS:\n1) What is the network architecture of your work? RNN? MLP? Please provide a full discription of it. \n\n2) The paper's motivation heavily relies on the energy efficiency of SNNs. It is encouraged to further investigate the energy efficiency on such edge situation: a flying robot, where batteries are heavy, so that energy is a much more important aspect to consider, compared to other edge computation scenarios.\n\n3) Could you please compare your result with other SNN - RL works, to better clarify your contribution? For example, how about a E-prop trained SNN, or a Evolution trained SNN? Here's some papers that might relevant: Evolving connectivity for recurrent spiking neural networks; A solution to the learning dilemma for recurrent networks of spiking neurons.\n\nLIMITATIONS:\nThe primary limitation of this paper is its novelty and insufficient baseline comparisons. The proposed TD3BC+JSRL framework is a composite of well-established techniques. And the experimental evaluation is limited to variants of the TD3 algorithm.  The paper would be significantly stronger if it included direct comparisons against other relevant SNN-RL methods mentioned in the related work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the challenges of training Spiking Neural Networks (SNNs) for reinforcement learning (RL), particularly the non-differentiable nature of spiking neurons and the difficulty of training on sequences. The authors analyze the impact of surrogate gradient slope settings, finding that shallower slopes improve training and performance in RL. They propose a novel training approach using a privileged guiding policy to bootstrap learning, combined with an adaptive slope schedule. Applied to a drone position control task, this method achieves an average return of 400 points, outperforming prior techniques. The work advances understanding of surrogate gradients in SNNs and provides practical training methods for neuromorphic controllers, demonstrating successful real-world deployment on a Crazyflie quadrotor.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. The paper presents a solid empirical evaluation, including comparisons with multiple baselines (BC, TD3BC, TD3) and ablation studies on surrogate gradient slopes. The results are well-supported by quantitative metrics (e.g., reward scores, gradient alignment analysis) and qualitative observations (e.g., real-world drone control).\n2. The methodology is sound, combining theoretical insights (e.g., surrogate gradient analysis) with practical innovations (e.g., jump-start RL with a guiding policy). The use of adaptive slope scheduling is particularly well-motivated.\n3. The work addresses critical challenges in training SNNs for RL, particularly in robotics (e.g., unstable early training, non-differentiability). The proposed method achieves 2.1 times performance gains over baselines, demonstrating practical impact.\n\nWeaknesses\n1. While the adaptive slope scheduler is promising, the paper does not fully explain why it outperforms fixed slopes in some cases but not others. A deeper theoretical analysis (e.g., convergence guarantees) would strengthen the claims.\n2. The computational cost of training (e.g., wall-clock time vs. ANN baselines) is not discussed, which is important for neuromorphic applications.\n\nQUESTIONS:\n1. The adaptive slope scheduler (Eq. 5) empirically improves training, but the paper lacks a theoretical explanation for why it works better than fixed slopes in some cases. Could the authors provide a convergence analysis (e.g., under what conditions the scheduler stabilizes training)? Also, could the authors provide a sensitivity analysis (e.g., how do choices of the weighting terms in Eq. 5 affect performance)?\n2. The paper focuses on SNN-vs-SNN comparisons (e.g., TD3BC+JSRL vs. BC), but how does the method fare against ANN equivalents (to quantify energy efficiency vs. performance trade-offs)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSpiking neural networks (SNN) have advantages for energy-constrained applications since it’s mostly sparse. The paper focuses on applying spiking neural networks (with surrogate gradients) in reinforcement learning to control a small drone. The authors introduce a hybrid approach between behavioral cloning (BC) and reinforcement learning (RL) where BC is applied using a guiding policy that produces good transitions at the beginning of learning where the contribution of BC fades away with time and the RL components becomes dominant. The paper validates the viability of their approach by real-world deployment of the algorithm into a small drone.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The authors address an important problem, especially since it enables effective robot policies at small scale where energy consumption becomes a significant problem (e.g., tiny robots).\n- The approach is validated by deploying on a real hardware which shows its promise.\n\n**Weaknesses:**\n\n- The role of BC is not clear. Why are SNNs unstable at the beginning of learning? The network should learn the representations and hidden states requires to perform better from the beginning.\n- The mathematical part is not rigorous. Equation 1 has hard-coded numbers 100 and 50, where clearly those are hyperparameters of the algorithms that need to be tuned for each problem, right? Also, what is the significance of the value 100? Are all episodes constrained to have lengths of 100s?\n- Unclear writing in the empirical evaluation lacks many details and it makes it hard to follow. The task should be described well, followed by the experiment setup and the results, and ending with a comprehensive discussion. I don’t think the authors follows this standard way of writing; instead the information is scattered and hard to follow.\n- Many crucial components of the papers are missing:\n    - The reward function definition\n    - The task being addressed in the experiment is not defined.\n    - Missing algorithm pseudocode. It’s not clear what algorithm steps are. The information is scattered all over the paper.\n    - Standard bars in Figure 3.\n    - Video of the performance of the drone moving in circles, eight-figures and squares. I checked the supplementary materials but couldn’t find any video. The only evidence is an edited photo that adds the drone in multiple places in the photo. I don’t think is a clear evidence of the behavior.\n    - Comparison with recurrent ANN approaches.\n    - Ablation study to understand the importance of each component in the TD3BC+JSRL.\n    - Energy consumption on the drone when using TD3BC+JSRL against conventional approach.\n\n\n**Minor issues:**\n\n- The acronym JSRL is used in the paper without definition. It is referring to jump-start reinforcement learning, right?\n- It’s not clear why SNNs have higher memory foot print as mentioned in line 232.\n- Trajectory error in ANN with no action history cannot be infinity as reported in Table 2. It has to be a number that is measured by the authors. Please update it with the actual value.\n- What is the meaning of altered drone in Table 2? How is different from the original drone?\n\n\nOverall, I don't think the manuscript is ready in its current form for publication. I encourage the authors to keep working on this important problem and strengthen their paper by incorporating the feedback as needed.\n\nQUESTIONS:\n- What does “The critic is not time variant and thus does not need a warm-up period” mean? The objective clearly applies the warmup and prevents any updates for both the actor and critic.\n- Why is the critic ANN instead of SNN?\n- The authors have in line 180: “We stack a history of observations, process multiple forward passes per observation and reset the SNN in between subsequent actions, similar to previous RL implementations for SNNs”. I’m confused because the authors argued that with SNNs we don’t need to have frame stacking. How can we reconcile both statements?\n\nLIMITATIONS:\nThe authors mention the limitations of their approach in the last section in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors propose a method for training spiking neural networks via surrogate gradients in a variety of problem settings, from supervised learning to online RL. Because spiking neural networks aren’t differentiable, it necessitates the use of surrogates for the gradients being applied to the network. The authors also highlight the temporal nature of spiking neural networks aligns with the sequential decision making of RL methods. The authors present a thorough analysis of the effects of surrogate gradients between different layers of the network that inform their choice of a surrogate.

STRENGTHS AND WEAKNESSES:
- The authors compare against a variety of methods that have been previously applied to spiking neural network training.

- The analysis of surrogate gradients help inform the design of the RL algorithm proposed. 

- Sec. 4 could be better motivated. For example, I do not really understand the purposes of the experiments in 4.1 and 4.2. These sections focus on describing what is being done, but it is not really clear why we need to analyze the surrogate gradients.

- The method appears to be named TD3BC+JSRL, but this name is not introduced until Sec. 4.3. I think it would be better to introduce it in Sec. 3 (or earlier) and describe exactly what are the components of TD3BC+JSRL.

- I don’t have a clear understanding of how difficult these domains in the paper are because I am unfamiliar with them. This makes it difficult for me to judge the significance of the results.

QUESTIONS:
- Where do the hidden states fit into the architecture of the SNN? I see a description of the SNN architecture in Sec. 3.2 but it only mentions the inputs and outputs when interacting with the CrazyFlie.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-18