Title: Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spiking Neural Networks

Reinforcement Learning
Abstract: Spiking Neural Networks (SNNs) offer low-latency and energy-efficient decision making on neuromorphic hardware, making them attractive for Reinforcement Learning (RL) in resource-constrained edge devices. However, most RL algorithms for continuous control are designed for Artificial Neural Networks (ANNs), particularly the target network soft update mechanism, which conflicts with the discrete and non-differentiable dynamics of spiking neurons. We show that this mismatch destabilizes SNN training and degrades performance. To bridge the gap between discrete SNNs and continuous-control algorithms, we propose a novel proxy target framework. The proxy network introduces continuous and differentiable dynamics that enable smooth target updates, stabilizing the learning process. Since the proxy operates only during training, the deployed SNN remains fully energy-efficient with no additional inference overhead. Extensive experiments on continuous control benchmarks demonstrate that our framework consistently improves stability and achieves up to $32$% higher performance across various spiking neuron models. Notably, to the best of our knowledge, this is the first approach that enables SNNs with simple Leaky Integrate and Fire (LIF) neurons to surpass their ANN counterparts in continuous control. This work highlights the importance of SNN-tailored RL algorithms and paves the way for neuromorphic agents that combine high performance with low power consumption. Code is available at https://github.com/xuzijie32/Proxy-Target.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the challenge of applying Spiking Neural Networks (SNNs) to continuous control reinforcement learning (RL) tasks—a domain where conventional Artificial Neural Networks (ANNs) dominate due to their compatibility with smooth gradient-based updates. The authors point out an incompatibility between the discrete, event-driven nature of SNNs and the soft-update mechanism of target networks in RL, which relies on continuous and differentiable dynamics. To solve this problem, the paper introduces a proxy target framework that decouples the discrete behavior of SNNs from the smooth updating required in RL. This proxy serves as a differentiable surrogate for SNN-based target networks, enabling stable and effective learning. Experimental results across multiple continuous control benchmarks (e.g., MuJoCo tasks) show that the proposed proxy target mechanism can improve the performance of SNN agents—achieving up to 32% performance gains over baseline SNNs—without requiring major changes to the overall learning algorithm or spiking neuron models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. **Timely and Relevant Problem**.\nThe paper tackles the critical gap in making SNNs viable for continuous control RL tasks on edge devices. Given the growing interest in neuromorphic computing for energy-constrained agents (e.g., robotics), this work is both relevant and potentially impactful.\n\n2. **Novel Technical Contribution**.\nThe proxy target mechanism is a clean and elegant solution to the incompatibility between spike-based updates and target network soft updates. By introducing a continuous and differentiable proxy to interface with the RL loss, the paper bridges a gap that has previously hindered the use of SNNs in continuous action spaces.\n\n3. **Empirical Validation**.\nThe experiments are extensive and compare different spiking neuron models under standard RL benchmarks. The gains in learning stability and final performance are significant, demonstrating the effectiveness of the proposed technique across neuron types.\n\n## Weaknesses\n\n1. **Energy Efficiency Claims Not Substantiated**.\nWhile the paper argues that SNNs are better suited for edge deployment due to energy efficiency, no empirical or simulated hardware evaluation is provided to validate this claim. In fact, modern NPUs and embedded accelerators (e.g., those discussed in https://arxiv.org/pdf/2503.22567) can run small ANNs at milliwatt-level power budgets—potentially lower than neuromorphic chips like Intel’s Loihi-2 (~1W) (https://open-neuromorphic.org/neuromorphic-computing/hardware/loihi-2-intel/). The authors should temper their claims or support them with energy and latency measurements.\n\nQUESTIONS:\n1. Can the authors provide empirical evidence or simulated estimates to support the claim that SNNs offer superior energy efficiency for edge deployment compared to ANN baselines running on modern NPUs? Specifically, how does the energy consumption of your SNN-based agent compare (in hardware, simulator, or profiling estimates) to that of ANN agents with similar model sizes on commercially available low-power accelerators such as those described in https://arxiv.org/pdf/2503.22567?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Proxy Target in the SNN-RL training framework to bridge the gap between the discrete nature of SNNs and the continuous control RL. While traditional RL algorithms rely on smooth target network updates, it conflicts with the non-differentiable dynamics of SNNs, leading to unstable training. The proposed method incorporates a differentiable proxy actor used only during training to stabilize learning while preserving the energy efficiency of SNNs during inference. Various experiments demonstrate significant performance gains, while the average performance across different environments even surpasses standard ANN baselines with simple LIF neurons.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper addresses the critical challenge of training instability in SNN-based reinforcement learning by introducing a well-motivated proxy target framework that effectively mitigates the mismatch between discrete spiking dynamics and continuous control requirements.\n2. The experimental evaluation is thorough, covering multiple environments and incorporating different neuron types (LIF, CLIF, ...). Results consistently show improved performance, reduced variance, and, notably the SNN-based RL outperforms ANN baselines. Moreover, the study also demonstrates the energy efficiency of SNNs by estimating inference energy consumption.\n3. It is impressive to see SNNs outperform ANN in RL field.\n\nWeaknesses\n1. While the proxy actor effectively stabilizes reinforcement learning with an SNN actor, the additional training cost introduced by the implicit update mechanism is not thoroughly discussed. As illustrated in Figure 1, the implicit update appears to involve multiple backpropagation steps per iteration, in contrast to the single-step soft parameter update in traditional frameworks. This may lead to increased training time and computational overhead.\n\n2. In line 212, the authors state that the proxy is a multilayer feedforward network. However, the effect of specific architectural choices (such as the number of layers and hidden units) is not clearly justified and studied. Additionally, the effect of the number of proxy update iterations (hyperparameter K) is not explored.\n\n3. In the discussion on energy consumption, it would be beneficial to report the number of SOPs alongside the estimated energy values, as SOPs offer a more general measure of computational complexity. Since energy estimates can vary significantly depending on the hardware, including a comparison of SOPs would enhance the generalizability of the energy efficiency claims.\n\nQUESTIONS:\n1. The authors claimed that “Simple neuron makes the best.”, Could the authors discuss more about why simple neurons perform better in this specific situation?\n\n2. There are typos and informal writings in the manuscript. For example, in Figure 1, “inplicit” should be “implicit”; “Multi Iterations” should be “Multi-iteration” or “Multiple Iterations”.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the challenge of applying continuous-control reinforcement learning algorithms to Spiking Neural Networks (SNNs), pointing out that standard RL mechanisms—especially the soft update of target networks—are incompatible with the discrete, non-differentiable nature of SNN spikes. The authors propose a proxy target framework in which a continuous, differentiable proxy network is used during training to smooth updates and bridge the gap between SNNs and RL algorithms originally designed for ANNs. Experiments demonstrate that SNNs trained with this framework achieve improved stability and up to 32% higher average performance in standard MuJoCo environments, even surpassing traditional ANNs in some settings. The framework preserves the energy-efficiency benefits of SNNs at inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Identification of a Previously Unaddressed Limitation: The paper provides a clear diagnosis of a previously underappreciated source of instability in SNN-based RL for continuous control—the incompatibility of non-differentiable spiking output with soft target updates. This is demonstrated empirically (see Figure 2b), which shows abrupt, discrete changes in SNN target outputs that undermine stable learning.\n\n- Methodological Contribution: The proposed \"proxy target\" mechanism is simple yet effective. By using a continuous, differentiable actor only for target updates during training, it permits smooth transitions needed for stable RL optimization without sacrificing SNN deployment benefits. The update mechanism is derived clearly, and implementation details are provided.\n\n- Comprehensive Experimental Validation: The authors provide thorough empirical evidence across five MuJoCo environments, using multiple spiking neuron types (LIF, CLIF, DN). Learning curves (Figure 4) illustrate consistent improvement in convergence speed and final return, with shaded regions for statistical reliability. Table 1 compares max average returns to multiple strong SNN baselines and ANN-based RL, showing the proposed PT-LIF outperforms all baselines in average performance gain (5.84%).\n\nWeaknesses:\n- Limited Theoretical Analysis: While the mechanism is well-motivated and empirically validated, the manuscript lacks rigorous theoretical support. The smoothing effect of the proxy is justified mainly by intuition and plotting, without formal stability/convergence claims (see Section 4, pages referencing Figures 2c-d). This limits broader generalizability to other RL algorithms or network types.\n\n- Ablation Shortcomings: The experimental section would benefit from ablations isolating the effects of proxy update frequency, the batch size (N) used for proxy training, or the depth/width of proxy architectures to clarify the mechanism’s robustness and optimal configurations. While Figure 2c-d and Figure 4 highlight core effects, missing such ablations leaves open questions about sensitivity and practical guidance.\n\n- Missing Key Baselines: Although several state-of-the-art SNN RL methods are compared in Table 1, additional direct baselines such as SNNs with architectural/topological trims but without proxy, or variants using different activation surrogates, would further clarify the gain is from the proxy and not other confounds.\n\n- Proxy-Online SNN Gap Quantification: While Figure 2d visually indicates the proxy network closely tracks the SNN actor, a quantitative, environment-wide measure of this approximation gap is not provided. Including such a metric (e.g., proxy-SNN output distance during learning) across seeds/environments would clarify how tightly the proxy must follow the SNN to ensure learning stability.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper solves a critical mismatch between SNNs and continuous control RL: SNNs' discrete spikes clash with RL's need for smooth optimization, causing training instability. This paper introduces the Proxy Target Framework to resolve this. During training, a continuous network (the proxy) replaces the SNN target network, providing the smooth updates necessary to stabilize learning. Crucially, the efficient SNN is still used during inference, preserving energy benefits. This approach significantly boosts stability and performance.

STRENGTHS AND WEAKNESSES:
**Strengths:**
1. This paper pinpoints the core conflict between discrete SNN dynamics and the continuous soft-update mechanisms in RL.
2. By using the proxy only during training, it stabilizes the learning process of SNNs.


**Weaknesses:**
W1. The auxiliary proxy network increases computational cost and complexity during the training phase, even if inference remains efficient. This overhead should be analyzed.

W2. The solution is tailored for Off-policy RL algorithms using target networks (e.g., DDPG, TD3). Its applicability to On-policy methods (e.g., PPO) is unclear.

W3. Achieving performance superior to ANNs might require high spike firing rates or many timesteps, potentially diminishing the claimed energy efficiency benefits (sparsity and latency).

W4. The mechanism for updating the proxy network to match the SNN likely introduces significant interaction costs between the two models. This dual update process could become a training bottleneck and requires further discussion on optimization.

QUESTIONS:
see weakness (W1-W4)

LIMITATIONS:
Increased Training Overhead & Limited Task Generality
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-05