Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control
Authors: Zijie Xu
Tong Bu
Zecheng Hao
Jianhao Ding
Zhaofei Yu
Venue: NeurIPS 2025 poster ()
Keywords: Spiking Neural Networks
Reinforcement Learning
Primary area: neuroscience_and_cognitive_science

Abstract: Spiking Neural Networks (SNNs) offer low-latency and energy-efficient decision making on neuromorphic hardware, making them attractive for Reinforcement Learning (RL) in resource-constrained edge devices. However, most RL algorithms for continuous control are designed for Artificial Neural Networks (ANNs), particularly the target network soft update mechanism, which conflicts with the discrete and non-differentiable dynamics of spiking neurons. We show that this mismatch destabilizes SNN training and degrades performance. To bridge the gap between discrete SNNs and continuous-control algorithms, we propose a novel proxy target framework. The proxy network introduces continuous and differentiable dynamics that enable smooth target updates, stabilizing the learning process. Since the proxy operates only during training, the deployed SNN remains fully energy-efficient with no additional inference overhead. Extensive experiments on continuous control benchmarks demonstrate that our framework consistently improves stability and achieves up to $32$% higher performance across various spiking neuron models. Notably, to the best of our knowledge, this is the first approach that enables SNNs with simple Leaky Integrate and Fire (LIF) neurons to surpass their ANN counterparts in continuous control. This work highlights the importance of SNN-tailored RL algorithms and paves the way for neuromorphic agents that combine high performance with low power consumption. Code is available at https://github.com/xuzijie32/Proxy-Target.

Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control
Zijie Xu Peking University Beijing, China 100871 xu22dsst pku. edu
Tong Bu Peking University Beijing, China 100871 putong300pku. edu. .n
Zecheng Hao
Jianhao Ding
Zhaofei Yu*
Peking University Beijing, China 100871 haozecheng@pku.cc edu. cn
Peking University
Peking University Beijing, China 100871 yuzf 120pku. edu. cn
Beijing, China 100871
djh019980alummi. pku edu. en
Abstract
Spiking Neural Networks (SNNs) offer low-latency and energy-efficient decision making on neuromorphic hardware, making them attractive for Reinforcement Learning (RL) in resource-constrained edge devices. However, most RL algo- rithms for continuous control are designed for Artificial Neural Networks (ANNs), particularly the target network soft update mechanism, which conflicts with the discrete and non-differentiable dynamics of spiking neurons. We show that this mismatch destabilizes SNN training and degrades performance. To bridge the gap between discrete SNNs and continuous-contr algorithms, we propose a novel proxy target framework. The proxy network introduces continuous and differen- tiable dynamics that enable smooth target updates, stabilizing the learning process. Since the proxy operates only during training, the deployed SNN remains fully energy erffgy-efficient with no additional inference overhead. Extensive experiments on continuous control benchmarks demonstrate that our framework consistently im- proves stability and achieves up 32% higher performance across various spiking neuron models. Notably, to the best our knowledge, this is the first approach that enables SNNs with simple Leaky Integrate and Fire (LIF) neurons to surpass their ANN counterparts in continuous control. This work highlights the importance of SNN-tailored RL algorithms and paves the way for neuromorphic agents that combine high performance with low power consumption. Code is available at bttpps://githue www.
Target
Introduction
Reinforcement Learning (RL), combined with Artificial Neural Networks (ANNs), has become cornerstone of modern artificial intelligence, achieving remarkable success in diverse domains such as game playing Mnnh 2013 Silver et al.] 2016 Mnih al., 2015). autonomous driving (Kiran et al.)
2021 Sallab et al.[ Shalev-Shwartz et 2016), and large language model training [Ouyang
et al. 2022 Bai et al. 2022 Shao et al. 2024 Among these, continuous control problems have
drawn particular attention due to their close alignment with real-world robotic and embodied AI applications [Kober et al. 2013 Gu al. 2017 Brunke al. 2022 However, the high computational
cost and power demands of ANN-based RL algorithms limit their deployment on edge devices such drones, wearables, and 1oT sensors Abadía et al. 202 Tang et al. 2020 Yamazaki et al. 2022
CCoresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Actor-Critic Frameworks
Parameters Copying
Parameters Copying
Behavior Imitating
Agent
Observation
Target Actor
Target Actor
Proxy Actor
Reward
Action
Actor
Actor
Actor
Target
Target Crilic
Target Crilic
Critic
Critic
Critic
Critic
Parameters
Parameters
Parameters
Copying
Copying
Copying
Environment
(a) Actor-Crite Critic ANN
(b) Actor-Critic SNN
(c) Proposed Framework
ANN
Vanilla
SNN
(d) SNN Performance Ratio Compared with
Figure 1: Overview of the training framework and performance comparison. (a)-(c) are different training paradigms. (a) Actor-Critic framework in ANNs, (b) The Actor-Critic framework for SNNs, (c) the proposed proxy target framework for SNNs. (d) Performance ratio of SNNs relative ANNs across five random seeds and five environments. The middle orange line denotes the median, the box spans from the first to the third quartile, and the whiskers extend the farthest data within 1.5 inter-quartile range from the box.
Inspired by biological neural systems, Spiking Neural Networks (SNNs) offer sparse, event-driven computation with ultra-low latency and energy consumption on neuromorphic hardware (DeBole
al. 2019 Davies al. 2018). These properties make SNNs attractive for RL applications
resource-constrainee edge devices (Yamazaki et al. 2022]. Recent works have attempted to integrate
SNNs into continuous-control RI algorithms via hybrid frameworks (Tang al. 2020
2021 Zhang
ct al. [224 Chen et al. 2024 Zhang et al., [2022]. where spiking actor network (SAN) is co-trained
with ANN critic using
Backpropagation (STBP) [Wu et al. 2018 Fang et al.
2021]. as illustrated in Fig. [b). With well-chosen hyperparameters, such frameworks have shown
that SNNs can approach or even surpass the performance of ANNs some tasks.
However, most of these studies simply retrofit SNNs into existing ANN-centric RL frameworks without adapting the algorithms to SNN dynamics. Since ANNs and SNNs exhibit fundamentally different computational characteristics, remains unclear whether RL algorithms designed for continuous, differentiable activations well-suited for discrete, event-driven networks.
key issue arises from the target network soft update mechanism, a core component widely used in off-policy RL algorithms to stabilize training by gradually updating target networks (Sutton and Barto 2018 Lillicrap, 2015 Fujimoto et al. 2018 Haarnoja et al. 2017 This mechanism relies on continuous, smooth output changes- property violated the non-differenniable binary nature of SNN spikes. This can cause abrupt output shifts, leading to unstable optimization objective, and oscillatory updates. Such instability not only makes the model highly sensitive to random seed initialization but also hampers convergence and undermines reliability in real-world deployment.
To address this mismatch between discrete spikes and continuous-comte updates, we propose proxy target framework for SNN-based RI. (Fig. Mc)). Instead of using an SNN target actor, we introduce differentiable proxy actor network that imitates the behavior of the online spiking actor network.

---

The proxy target network can alter its output smoothly and continuously, stabilizing the learning process and improving performance, as demonstrated in Fig. [d). Since the proxy target network only used for auxiliary training, the proposed approach retains SNN's advantages of low-latency and energy efficiency during inference real world applications. Our main contributions are summarized as follows:
We identify a critical mismatch between discrete SNN outputs and the continuous target network soft update mechanism used in off-poliey RL, showing how this conflict destabilizes training and degrades performance.
We propose proxy target framework that replaces the spiking target network with continuous, differentiable proxy, enabling smooth target updates and stable optimization.
We introduce an implicit gradient- beasee update rule that aligns the proxy with the online SNN, mitigating target output gaps and giving precise optimization goals.
Extensive experiments across multiple neuron models and continuous control benchmarks demonstrate consistent stability improvements and up to 32% higher average performance. To the best of our knowledge, this is the first approach where SNNs with simple Leaky antegrate-and-Fire (LIF) neurons surpass ANN performance in continuous control.
2 Related works
2.1 Learning rules of SNN-based RL
Synaptic plasticity. Inspired by the plasticity of biological synapses, several works have inte- grated SNNs into reinforcement learning via reward-modulated spike-tining-dependees
plasticity (R-STDP) [Florian| 2007 Frrmmuxx and Gerstner, 2016 Gerstner al. 2018 Frémaux et al.) 2013
Yang et al. 2024). These approaches are biologically plausible and energy- effigyeeffiieet,, but have limited performance on complex tasks.
ANN-SNN conversion. With the progress of ANN-based deep RL and ANN-SNN conversion
2021 Kumar et al. 2025] convert well-trained Deep Q-Networks (DQNs) Mnih 2013 Mnih al.
algorithms [Coo et al. 2015 Bu et al. 2022a b). some studies [Patel et al. [001 Tan et al.
2015
into SNNs. Such conversion- based methods achieve lower energy consumption during inference, but
require ANN pre-training.
Gradient- -base direct training. To avoid ANN pre-training, several works [Liu al. 2022 Chen et al. 2022, Qin et al., 2022, Sun et al. 2022 2025
directly train SNNs for RL using STBP Wu
al. 2018 while Bellee et al. [2020] introduced e-prop with eligibility traces to learn policies through the policy gradient algorithm [Sutton et al., 1999]. These approaches achieve competitive results in discrete action spaces, but they cannot be extended continuous control tasks.
2.2 Hybrid framework of spiking actor network.
In continuous-comtro problems where the action space is continuous, hybrid frameworks have been extensively explored. TTag et al. [2020) first proposed an SNN-based actor co-trained with an ANN critic in the Actor-Critic framework Konda and Tsitsiklis 1999]. [Tang et al. (202T) demonstrated that population encoding improves the performance of spiking actor networks. Subsequent works enhanced these frameworks through various mechanisms, such as utilizing dynamic neurons [Zhang
ctt al. 2022], incorporating lateral connections [Chen et al. 2024], adding bio-plausible topologies [Zhang et al. 2.24]. and integrating dynamic thresholds Ding et al. 2022
While these hybrid approaches report performance comparable to or exceeding their ANN counter- parts, two key limitations remain. First, they often rely on complex neuron models (e.g., current- based LIF or second-order dynamic neurons), increasing computational cost and training difficulty. Second, the RL algorithms themselves are not modified to account for SNN-specific dynamics, which may cause instability and suboptimal convergence. In contrast, our proxy target framework is tailored to the discrete, event-driven nature of SNNs, achieving superior stability and performance with only simple LIF neurons.
3

---

Preliminaries
In continuous control settings, the action space a continuous vector (e.g., torque values). Most continuous control algorithms adopt the Actor-Critic framework with deterministic policy (Sutton To avoid ambiguity, we use training steps to denote RL time steps and simulation steps to denote internal SNN simulation time steps.
3.1 Reinforcement Learning
Reinforcement Learning (RL) involves an agent interacting with an environment. The agent observes the current state performs action a and receives reward r, while the environment transitions to the next state s'. The agent's ss objective to learn a policy Tg. parameterized by Φ, that maximizes the expected return.
and Barto 2018), where the actor Tg outputs actions a = (s))) and the critic Qo evaluates them with parameters 0 [Konda and Tsitsiklis, 1999]. The actor is updated by the deterministic policy gradient [Silver et al. 2014]:
voJ(o)= = VoQo(s, la=x(s) V Vo)
The critic is updated via temporal-differenee (TD) learning [Sutton 1988 using the Bellman equa- tion [Bellman 1966]
Qo(s,a) y: y y=r+ ( Ts (s'),
(2)
where y is the discount factor and (T Q+ denote target networks.
3.2
Target network soft update
The target networks ( Qo share the same architecture as their online counterparts (To, Qp) but updated more slowly provide stable learning targets. Their parameters are updated by the Polyak function with smoothing factor T:
- (1- +' 10 (1-
These soft updates play a crucial role in off-policy continuous-control algorithms. As shown in Eqs. 2), the actor and critic are jointly optimized through bootstrapping, which can cause oscillatory updates due to their mutual dependence. The target networks mitigate this by producing slowly changing targets, thereby stabilizing training and preventing divergence.
3.3 Spiking Neural Networks
Spiking neuron model. In an SNN, each neuron integrates presynaptic spikes into its membrane potential and emits a spike when the potential exceeds a threshold. The Leaky Integrate and Fire (LIF) neuron [Gerstner and Kistler 2002 is one of the most widely used models, governed by the following dynamics:
7 = w' + = 7!
(4)
S O Vnn,, = S.) St. Vresets
where I is the input current, H is the accumulated membrane potential, S is the binary output spike, V the membrane potential after the firing process. W and are the weights and the biases, Va. Vieser- and Ả the threshold voltage, the reset voltage and the membrane leakage parameter, respectively. All subscripts (.), and all superscripts (-)6 denote simulation step and layer / respectively. O(.) is the Heaviside function.
Spiking actor network. The spiking actor network (SAN) consists of population encoder with Gaussian receptive fields [Tang al.) 2021], a multi-layer SNN, and a decoder that uses membrane potentials of non-firing neurons as continuous outputs [Chen et al. 2024). The SAN is trained using STBP with a surrogate gradient function. Detailed forward and wack formulations are provided in Appendix A.2

---

4 Methodology
In this section, propose novel proxy target framework address the incompatibility between the discrete dynamics of spiking neurons and the continuous target network soft update mechanism in RL. Section 4.1 analyzes the instability caused by discrete target outputs and introduces proxy target network with continuous dynamics. Section 4.2 presents an implicit imitation mechanism that aligns the proxy network with the online SNN through gradient-based optimization. Section 4.3 summarizes the overall training procedure.
ANN target for ANN
SNN target for SNN
Proxy target for SNN
Target Outputs
Target Outputs
Target Outputs
MSE Error
200 400
600
200
400
600
200
400
600
2000
4000
Update Steps
Update Steps
Update Steps
Training Steps
(a) ANN Target
(b) SNN Target
(c) Proxy Target
(d) Target Errors
Effects of different target network update mechanisms. (a)-(c) show output trajectories of different target networks during updates, where each line denotes a normalized output dimension within (-1, 1). (a) ANN target network exhibits smooth transitions; (b) SNN target network produces discrete and irregular output jumps; (c) the proposed proxy target achieves continuous and stable transitions. (d) Mean squared error between target and online networks during training in the InverredDoublleennduuum environment.
Figure 2:
4.1 Addressing discrete targets by proxy network
Performance degradation due to discrete target outputs. In the standard Critic framework, the target network is updated using the Polyak function (Eq. 3 which assumes that small parameter updates lead to smooth output transitions. This assumption holds for ANNS with continuous activation functions but fails for SNNs, whose firing function is binary and non-differentiable. To illustrate this effect, we construct target networks corresponding to trained online networks using identical architectures and neuron models. The target parameters are updated according to Eg, 3 with T = 0.005 (the most commonly used setting), while the online network is frozen. Figures (b) show the target outputs during updates: the ANN target evolves smoothly, whereas the SNN target exhibits frequent discontinuous jumps. Although both targets eventually converge to their online counterparts, the discrete shifts in the SNN target (Fig. (Fig.2(b) cause erratic transitions that propagate instability to the critic's optimization objectives, resulting in oscillatory and unreliable learning dynamics (Fujimoto) et al. 2018).
Smoothing target outputs by proxy network. As illustrated in Fig. [c). to restore smoothness, we introduce a proxy target network that replaces the discrete spiking neurons of SNNs with continuous activation functions of ANNs. As shown in Fig. (c). the proxy network produces gradual output transitions during updates, effectively eliminating the discrete jumps observed in SNN targets. This design enables stable soft updates and prevents abrupt shifts in the target outputs, thereby improving the stability of the overall Actor Critic learning process.
4.2 Addressing target output gaps by implicit updates
Performance degradation due to target output gaps. Although the proxy network achieves smooth updates, directly substituting spiking neurons with continuous activations (c.g., RcLU) introduces output gap between the proxy and the online SNN. This discrepancy prevents the proxy target from accurately reproducing the output of the online SNN, distorting the critic's learning targets and reducing overall policy performance.
Aligning proxy network by implicit updates. Since the approximation errors cannot be eliminated by explicitly copying the weights of the online SNN, we propose an implicit proxy update method.
5

---

Proxy Network (updating
Spiking Actor Network (frozen Y
Output-guided
population encoder
population decoder
state
action
state
action
Implicit update
Figure 3: Architecture of the proposed proxy network and the spiking actor network. The proxy actor updated implicitly by imitating the behavior of the online spiking actor network, ensuring stable and accurate target updates.
As shown in Fig. E] unlike the explicit soft update that directly averages parameters, approach computes updates in the output space, which gradually reduces the gap between the online SNN and the proxy target. Let the proxy actor o have parameters Φb For each input state s, the proxy output is implicitly updated toward the SNN actor output
Proxy have parameters @' and the online spiking actor SNN
Proxy (s) TT'
7') Proxy
(6)
where is smoothing coefficient similar to Eq. 3 corresponding parameter according to Eq. [6] instead perform gradient-based optimization that achieves a similar effect:
Since difficult to directly update the
+ T n (s)
Proxy (s)
Prixy (s)
T
V
Prony (s)) n
(s))
(7)
2
2
where || . 112 denotes the squared l2 norm. Thus, the proxy network can be updated by gradient descent that minimizes the proxy loss:
Ippoay N
Σ
Proxy (s:))
15 SSNNN (s:))
(8)
where N denotes the batch size, are the states sampled from the replay buffer in RL algorithm. This proxy update mechanism acts form of implicit imitation learning, aligning the proxy network with the SNN actor while maintaining smooth output transitions, as demonstrated in Theorem
Theorem Let the proxy network n
Praxy be updated by minimizing the loss Tapracy in Eq. During the output change satisfies
each update, as the proxy learning rate ir prayy
||
Proxy (s) Maa
Proxy (s) (r +
where H'sa and K'ew denote parameters before and after the update, respectively. Hence, minimizing Lppaay ensures sufficiently small and smooth policy updates, promoting stable optimization.
Since the proxy network is a multi-layer feedforward model, universal approximator (Hornik et al. 1989]. it can asymptotically match the SNN actor's output by minimizing Eq, g To further Fig-2(d) demonstrate this empirically, Fig. dd) shows the mean-squared output gap between the proxy and the SNN actor during training. While the SNN target occasionally diverges from the online SNN, the proxy network remains well-aligned throughout, validating that the proposed approach effectively mitigates target output gaps and provides precise and stable optimization goals for RL training.
4.3 Overall training framework
The proposed proxy target framework shown in Fig. [c.. The proxy actor network contains continuous activations of ANN that replace the discontinuous SNN target actor network. Instead of explicitly updating network parameters, the proxy actor is implicitly optimized imitate the behavior of the online SNN actor by minimizing the loss in Eq. 8 During each update episode, the proxy actor optimized for K iterations to reliably approximate the discrete SNN outputs, compensating for the greater representational difficulty. Mcanwhile, the target critic is updated explicitly by copying

---

Algorithm Proxy Target framework
1: Initialize SNN actor network SNN (s), ANN critic network Q a) with parameters Φ, 0
2: Initialize proxy actor &
Proxy (s) and ANN target critic Qg (s, a) with parameters o' and #'
3: Initialize replay buffer DD
4: for each iteration do
5: Execute action a according to T (s) and store the transition (s, a, r, D
6: if proxy larget update then
7: for k = 1 to K do
8: Sample minibatch of N transitions (si, dz Ti, s() from D
9:
Update proxy actor parameters o' by minimizing:
I Iprozy
1 N
Σ
Proxy
(8:) =
SNN (s:)
end for
11:
end if
if ANN target update then
13:
Update ANN target critic parameters O' by the Polyak function: 0' < + (1 +)0'
end if
15:
if ANN critic update then
16:
Compute target values y; using proxy actor TT
Proxy and target critic QANN
17:
Update ANN critic by minimizing: Lecitic
t/4 E, (Q (s; a.) y.)
18:
end if
19:
if SNN actor update then
20:
Update SNN actor by maximizing: J N E, QANN
21:
end if
22:
end for
weights using the Polyak function, as both the critic and target critic are conventional ANNs. The complete training procedure summarized in Algorithm
As demonstrated Theorem T and Fig. 2 cc)) the proxy actor not only produces smooth output transitions but also closely tracks the SNN actor's behavior"] The proxy target framework effectively alleviates the instability caused by discrete and imprecise targets in traditional SNN-RI. frameworks, resulting in
stable training process within the Actor-Critic framework.
worth noting that the proposed mechanism preserves the energy efficiency SNNs, as the proxy network and the critic network are used exclusively during training and are discarded during deployment, introducing no additional computational overhead during deployment.
5
Experiments
5.1
Experimental setup
The proposed proxy target framework (PT) was evaluated across multiple coatinuous-contin tasks
in the MuJoCo simulator [Todorov et al. 2012 Todorov 2014b) using the OpenAl Gymnasium benchmark suite [Brockman] 2016 Towers et al. 2024 including InvertedDoubbeeendduuuu
(IDP) [Todorov] 2014a], Ant-v4 [Schulman al.[ HalfCheetah-ve [Wawrzyński] [209)].
Hopper-v4 [rezz et al. 2012 and Walker2d-vé All environments follow the default configurations without modifications.
The experiments were carried out with different spiking neuron models, such as the LIF neuron, the current- based LIF neuron (CLIF) Tang al. [2021] and the dynamic neuron (DN) Zhang et al.
Minor fluctuations the proxy network output (Fig. 2 ()) resemble the stochasticity introduced by soft
target updates with noise injection in DRL Fujimoto et al. 2018 which can further reduce overfitting in value estimation.

---

a
b
(c)
(d)
(e)
Figure 4: Continuous control tasks the MuJoCo environments on OpenAl Gymnasium. (a)
InvercdDDubleePmndllunv.y, (b) Ant-v4, (c) HalfCheetahh-v,, (d) Hopper-v4, (e) Walker2d-v4,
The LIF and CLIF neuron parameters follow Tang initialized as in /Zhan et al. [0022].
2022
2021 while the DN parameters
We tested the proposed algorithm in conjunction with the TD3 algorithm [Fujimoto al. 2018), all detailed parameter settings are provided in Appendix[A.4 [A.4, For fair comparison, all spiking actor networks share the same architecture, encoding, and decoding schemes provided in Appendix A.2 All SNNs have a simulation step of unless otherwise noted. All reported data this section are reproduced results across five random seeds.
5.2
Results across different spiking neurons
IDP-v4
Ant-v4
HalfChectah-v4
Hopper-v4 v4
Walker2d-v4
10
AR (1e3)
PT+LIF
5
2
LIF
10
AR (1e3)
PT+CLIF
2
CLIF
0-
AR (1c3)
7.5
PT+DN
2
2
DN
2.5
0
0
TS (1c4)
TS (le6)
TS (1e6)
TS (le6)
TS (1e6)
Figure 5: Learning curves of the proxy target framework (PT) and the vanilla Actor-Critic framework with the LIF neuron, the CLIF neuron and the DN. AR denotes average returns, and TS denotes training steps. The shaded region represents half standard deviation over different seeds. Curves are uniformly smoothed for visual clarity.
Increasing performance. Fig. shows the learning curves the proposed proxy target framework and the vanilla Actor-Critic framework with different spiking neurons. The proxy target framework improves the performance of different spiking neurons, demonstrating general applicability in delivering both faster convergence and higher final returns across different neuron types and environments.
Improving stability. Fig. a) shows the performance variance (after training) of the proxy target framework and the vanilla Actor-Critie framework with different spiking neurons. The proxy target framework reduces the variance of different spiking neurons, demonstrating its capability to stabilize

---

with proxy target network
without proxy target network
30
25.0
24.6
1.0
Average variance (%)
Normalized
perfomance
20
15.8
11.4
11.6
0.5
10
6.9
0.0
LIF
CLIF Spiking neurons
DN
0.0
0.5
1.0
Normalized training steps
(a) Variance with Different Neurons
(b) Proxy Target in ANN
Figure 6: (a) Average variance of different neurons after training. The average variance is computed by averaging the standard deviation ratio with seeds, across all environments. (b) Normalized learning curves across all environments of the ANN integrated with the proposed proxy network across all environments. The performance and training steps normalized linearly (0, 1). Curves are uniformly smoothed for visual clarity.
training. This is crucial for real-world deployments, where retraining costs high and consistent behavior is required.
5.3 Exceeding state-of-the-art
To quantify relative improvements, we define the average performance gain (APG) as:
performance(enn)_.
-1) 100%,
APG
envs
baseline(env)
CIVY Eenvss
where lenvs| denotes the total number of environments, performance and baseline(env) are the performance of the algorithm and the baseline in that particular environment. comppars our proxy target framework with ANN-based RL, the ANN-SNN conversion method [Bu et al. 2025 (100 simulation steps), and other state-of-the-
SNN-based RL algorithms, including pop-SAN (Tang et al. 2021]. MDC-SAN (Zhang et al. 2022]. and ILCC-SN [Chen et al., 2024 With the proxy network, simple LIF-based SNN surpasses all baselines, including those using complex neuron dynamics connection structures, and achieves higher average returns than standard ANNs. Although performance varies across tasks, the average gain across all environments and neurons indicates the general applicability of the proxy target framework.
Table 1: Max average returns over 5 random seeds with different spiking neurons, and the average performance gain against the ANN baseline, where denotes one standard deviation.
Method
IDP-v4
Ant-v4
HaliChectah-v4
Hopper-v4
Walker2d-v4
APG
ANN (TD3)
7503
3713
4770 +: 1014
10857 +: 475
3410 + 164
4340 +: 383
0.00%
ANN-SNN
3859
4440
3550 +: 963
8703 + 658
3098 + 281
4235 + 354
21.11%
Vanilla LIF
9347 +
4294 + 1170
9404 + 625
3520 + 94
1862 + 1450
10.54%
Pop-SAN
9351 +
4590 + 1006
9594 + 689
2772 1263
3307 + 1514
6..6%
MDC-SAN
9350 +
4800 994
9147 +
3446 + 131
3964 + 1353
0.37%
ILC-SAN
9352
5584 272
9222 -
3403 + 148
4200 717
4.64%
PT-LIF
9351 +
5014 + 1074
9663 + 426
3526 + 112
4564 + 555
5.46%
PT-DN
9350 +
5400 277
9347 + 666
3507 + 144
4277 + 650
5.06%
PT-IFF
9348 + 5383 + 250
10103 + 607
3385 4: 157
4314 + 423
5.84%
5.4 Simple neurons perform best
Interestingly, the simplest LIF neuron achieves the highest overall performance under the proxy target framework. This contrasts with previous findings where complex neuron models generally perform better. Once the SNN surpasses its ANN counterpart, the primary performance bottleneck shifts from the neuron model the RL algorithm itself. Hence, introducing more complex spiking dynamics may unnecessarily increase training difficulty and even degrade performance.

---

5.5 SNN-friendly design
Fig- b(b) shows the normalized performance of ANN with and without the proxy network. The proxy target framework cannot improve the performance in ANNs, confirming that the observed benefits arise from addressing SNN-speciifs challenges rather than providing stronger RL algorithm. This validates the SNN-friendly design of our framework.
5.6 Energy efficiency
Finally, we evaluate inference energy consumption across models. The comparison includes traditional ANN-based TD3 model, a baseline spiking actor network using vanilla LIF neurons, and PT-LIF model. The consumption is estimated as the same way as Merolla et al. (014 where
(MAC) operation costs J.97JJ on modern NPUs Millar et al. 2025 and synaptic operation (SOP) costs 77FJ Hu et al., 2021].
Table 2:
Energy consumptions of different tasks per inference for the spiking actor network with LIF neurons, where the energy unit is nano-joule (nJ).
Method
IDP-v4
Ant-v4
HalfCheetah-vd
Hopper-v4
Walker2d-v4
Average
ANN (TD3)
72.33
295.60
283.41
274.26
283.41
281.78 (71014.4 MACs)
Vanilla LIF
8.14
11.78
15.13
7.21
18.82
12.21 (158.6 x 10 SOPs)
PT-LIF
9.01
12.18
13.46
6.86
13.93
11.09 (144.0 10 SOPs)
As shown in Tab the ANN (TD3) model consumes significantly more energy, while both spiking models demonstrate dramatically lower energy consumption. Specifically, our proposed PT-LIF model achieves the lowest average consumption while maintaining better stability and performance. Moreover, PT-LIF's average firing rate (32%) slightly lower than that the vanilla LIF model (33%), further improving energy efficiency. These results highlight the superior energy efficiency of the proposed method, making compelling for deployment on energy mmrgyyconsttrined platforms.
Conclusion
In this work, we identified critical mismatch between the discrete dynamics of SNNs and the continuous requirement of the target network soft update mechanism in the Actor-Critic framework. To address this, we proposed novel proxy target framework that enables smooth target updates and faster convergence. Experimental results demonstrate that the proxy network can stabilize training and improve performance, enabling simple LIF neurons to surpass ANN performance in continuous control.
In contrast to previous works which retrofit SNNs into ANN-centric RL frameworks, this work opens a door to investigate and design SNN-friendly RL algorithms which is tailored to SNN's specific dynamics. In the future, more SNN-specific adjustments could be applied to SNN-based RL algorithms to improve performance and energy efficiency real-world, resource-constrainee RL applications.
Limitation. While this work designs a proxy target framework that suitable for SNN-based RL, still remains at the simulation level. The next step may involve implementing it on edge devices and enabling decisions-makee making in the real world.
Acknowledgement
This work is funded by National Natural Science Foundation of China (62422601, U24B20140, and 62088102) 102), Beijing Municipal Science and Technology Program (Z241 10000422400) and Beijing Nova Program (20230484362, 20240484703), and National Key Laboratory for Multimedia Information Processing.
Eneegy per MAC is estimated from the maximum inference throughput (mJ 1) across seven NPUs: MAX78k (C/R). GAP8, NXP MCP-MCX9447 -X- (S/P), and MILK-V [Millar et al. 2025 NPU initialization energy is excluded.
10

---

References
Ignacio Abadía, Francisco Naveros, Eduardo Ros, Richard Carrillo, and Niceto R Luque. A cerebellar-base solution the nondeterministic time delay problem in robotic control. Science Robotics, 6(58):eab12756, 2021.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212..08773 2022.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, LI(1):3625, 2020.
Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.
G Brockman. Openai gym. arXiv preprint arXiv: 2016.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-basee control to safe reinforcement learning. Anmual Review of Control, Robotics, and Autonomous Systems, 5(1):41 1-444, 2022.
Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Optimized potential initialization for low- latency spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 36, 2022a.
Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ANN-SNN conversion for high-accuracy and ultra-lowllatency spiking neural networks. In International Conference on Learning Representations, 2022b.
Tong Bu, Maohua Li, and Zhaofei Yu. Inference-scale complexity in ANN-SNN conversion for hugh-performance and low-power applications. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24387-24397, 2025.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 2015.
Ding Chen, Peixi Peng. Tiejun Huang, and Yonghong Tian. Deep reinforcement learning with spiking q-learning. arXiv preprint arXiv:2201.09754 2022.
Ding Chen, Peixi Peng, Tiejun Huang, and Yonghong Tian. Fully spiking actor network with intralayer connections for reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, (0()))2881-2892) 2024.
Mike Davies, Narayan Srinivasa, Tsung-Har Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 2018
Michael V DeBole, Brian Taba, Arnon Amir, Filipp Akopyan, Alexander Andreopoulos, William P Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, al. TrueNorth: Accelerating from zero to 64 million neurons in 10 years. Computer, 2019.
Jianchuan Ding, Bo Dong, Felix Heide, Yufei Ding, Yunduo Zhou, Baocai Yin, and Xin Yang. Bio- logically inspired dynamic thresholds for spiking neural networks. Advances in Neural Information Processing Systems, 35:6090-6103,, 2022.
Tom Erez, Yuval Tassa, and Emanuel Todorov. anfinit-hhorizon model predictive control for periodic tasks with contacts. Robotics: Science and Systems VII. 2012.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEFF/CVF international conference on computer vision, pages 2661 -2671,
2021.
11

---

Räzvan V Florian. Reinforcement learning through modulation of tewettimmiineeeeeee synaptic plasticity. Neural Computation, 19(6):1468-1500 2007.
Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike-tining-depeneee plasticity, and theory of three-factor learning rules. Frontiers in Neural Circuits, 9:85, 2016.
Nicolas Frémaux, Henning Sprekeler, and Wulfram Gerstner. Reinforcement learning using a continuous time actor-critic framework with spiking neurons. PLoS Computational Biology, (4): e1003024, 2013.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. International Conference on Machine Leurning, pages 1587-1596. PMLR, 2018.
Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge University Press, 2002.
Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Corneil, and Johanni Brea. Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules. Frontiers in Neural Circuits, 12:53, 2018.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE International Conference on Robotics and Automation, pages
IEEE, 2017.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy- based policies. International Conference on Machine Learning, pages 1362-1361. PMLR, 2017.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural Networks, 2(5):359-366, 1989.
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions
Neural Networks and Learning Systems, 34(8):5200-5205 2021.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, 23(6):4909-4926, 2021.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 12:333312311 2013.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Processing Systems, 12, 1999.
Aakash Kumar, Lei Zhang, Hazrat Bilal, Shifeng Wang, Ali Muhammad Shaikh, Lu Bo, Avinash Rohra, and Alisha Khalid. Dsqn: Robust path planning of mobile robot based on deep spiking q-network. Neurocomputing, 634: 129916, 2025.
TP Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint arXiv:150900297. 2015.
Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, and Huajin Tang. Human-level control through directly trained deep spiking q-networks. IEEE Transactions on Cybernetics, 53(11))7187-7198 2022.
Paul Merolla, John Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668 663, 2014.
Josh Millar, Yushan Huang, Sarab Sethi, Hamed Haddadi, and Anil Madhavapeddy. Benchmarking ultra-low-power μ npus. arXiv preprint 2503.225003.2567 2025.
12

---

Volodymyr Mnih. Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5600 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-53) 2015.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27744, 2022.
Devdhar Patel, Hananel Hazan, Daniel Saunders, Hava T Siegelmann, and Robert Kozma. Im- proved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied atari breakout game. Neural Networks, 120: 300:108-115 2019.
Lang Qin, Rui Yan, and Huajin Tang. A low latency adaptive coding spiking framework for deep reinforcement learning. arXiv preprint arXiv:2211.1170.. 2022.
Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. arXiv preprint urXiv:/704.0253 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: arXiv: 1506.02438, 2015.
Shai Shalev-Shward Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:6610.03295 2016.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pages 387-395. Pmlr, 2014.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
Yinqian Sun, Yi Zeng, and Yang Li. Solving the spike feature information vanishing problem in spiking deep q network with potential based normalization. Frontiers in Neuroscience, 16:953368, 2022.
Yinqian Sun, Feifei Zhao, Zhuoya Zhao, and Zeng. Multi-compartment neuron and population encoding powered spiking neural network for deep distributional reinforcement learning. Neural Networks, 182:106898, 2025.
Richard S Sutton. Learning predict by the methods of temporal differences. Machine Learning, 9-44, 1988.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
Weihao Tan, Devdhar Patel, and Robert Kozma. Strategy and benchmark for converting deep q- networks to event-driven spiking neural networks. Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 9816-9824. 2021.
13

---

Guangzhi Tang, Neelesh Kumar, and Konstantinos P Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy- mapless navigation with neuromorphic hardware. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 6090-6097. IEEE, 2020.
Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos Michmizos. Deep reinforcement learning with population-coded spiking neural network for continuous control. Conference on Robot Learning, pages 2016-2029. PMLR, 2021.
Emanuel Todorov. Convex and saalyttiaall-invertbbb dynamics with contacts and constraints: Theory and implementation in mujoco. In 2014 IEEE International Conference on Robotics and Automation, pages 6054-6061. IEEE, 2014a.
Emanuel Todorov. Convex and malytically-inveriin dynamics with contacts and constraints: Theory and implementation mujoco. In 2014 IEEE International Conference on Robotics and Automation, pages 6054 6061. IEEE, 2014b.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 20/2 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-500 6026-5033. IEEE, 2012.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, KG Arjun, et al. Gymnasium: A standard interface for reinforcement learning environments. CoRR, 2024.
Pawel Wawrzyúski. A cat-like robot real-time learning to run. In Adaptive and Natural Computing Algorithms: 9th International Conference, ICANNGA 2009, Kuopio, Finland, April 23-25, 2009, Revised Selected Papers 9, pages 380-390. Springer, 2009.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for
training gigh-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le. Spiking neural networks and their applications: A review. Brain Sciences. 12(7):863, 2022.
Zhile Yang. Shangqi Guo, Ying Fang, Zhaofei Yu, and Jian Liu. Spiking variational policy gradient for brain inspired reinforcement learning. IEEE Transactions Patlern Analysis and Machine Intelligence, 2024.
Duzhen Zhang, Tielin Zhang, Shuncheng Jia, and Bo Xu. Multi-sacle dynamic coding improved spiking actor network for reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 59-67, 2022.
Duzhen Zhang, Qingyu Wang, Tielin Zhang, and Bo Xu. spiking actor network for efficient deep reinforcement learning, arXiv preprint arXiv: 2003003.0.. 2024.
topology improved
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction sections, we clearly point out the contributions of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
14

---

The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We point out the limitations in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
15

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide detailed experimental setups in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide codes with sufficient instructions in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new spun-s benchmark).
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
htt.ps.cc/pulii//uuudeeeee Sssiae/isssieiisiiiiiiiiiiia osiliiolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show setups and hyper-parametee in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The learning curves figure and the main result table show the standard devia- tions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
17

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We give the experiments compute resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We think there is societal impact be emphasized.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
18

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We think there is no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. www.coo/caaooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
19

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work dose not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: All important and original component dose not involve LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standaaddard components.
Please refer to LLM policy for what should or should not be described.
20

---

A Appendix
A.1 Proof of Theorem
Proxy be updated by minimizing the loss Lproxy in Eq. During Theorem Let the proxy network a each update, as the proxy learning rate pracy + O, the output change satisfies
Provy T (s)
(s) (s))) - 0,
Proxy
where Sasa and Know denote parameters before and after the update, respectively. Hence, minimizing Tppuay ensures sufficiently small and smooth policy updates, promoting stable optimization.
Proof 1 By standard gradient descent, we have:
[1 Wrrozy +0
lim
S'ta)ll
lim
ITprozy
Volaaa [prory) 0.
Irarezy +0
Under the assumption that T first-order Taylor expansion:
Prozy
(s) is continuously differentiable with respect d' we apply
lim Epprozy
Prozy
(s)
Prozy (rozy(s)
lim Irprony +0
[Chew dota) Vo
TT ola
Prory ()ary(s) =0.
A.2 Spiking actor network architecture
The spiking actor network (SAN) consists of population encoder with Gaussian receptive fields, multi-layer SNN with population output, and a decoder with non-firing neurons.
A.2.1 Forward propagation of the SAN
In the state encoder, each imput dimension consists of Nm, soft reset IF neurons with different Gaussian receptive fields with trainable parameters and a. The neurons receive stimulation Ag at every time step and outputs spikes 9 according to:
1 (s 11) 2
Ag = exp
(10)
2
V.nh Vin
S:n + +Ag
(11)
where Vg is the threshold of the encoding populations.
The last layer of the SNN consists of Nout neurons for each action dimension, respectively. The decoder layer is made up of non-spiking integrate neurons connected the last layer of SNN:
your yout + Wout St + bout
(12)
where Wrout and gout are weights and biases. The final output action determined by the membrane potential in the last time step = Vouut The detailed forward propagation of the spiking actor network is shown in Algo. 2
A.2.2 Back propagation of the SAN
OL SAN parameters are trained by the gradient with respect to the output action du The output decoder can be updated by:
where Vigut
8L
8L
@Vout
JW oW OL
Ja
Bw.our
(13)
av
du
Bbour
Then, the main SNN is trained by STBP with the rectangular surrogate function defined as:
@'(x) =
Z
(14)
0,
else
21

---

9: end for Algorithm 2 Forward propagation of spiking actor network
1: Input: M,-dimensional observation
2: Compute the stimulation of input populations:
(s - 1)²²
Ag exp
3: for t=I...., do
4:
Compute the output spikes of the population encoder:
Vinhm V S + Ag
St O(Vi VE)
5:
for l=I.... do
6: Update neurons in layer at timestep
7:
end for
8:
Update the decoder neurons' membrane potential:
Voua Vi-1 + Wout S + bout
10: Output: Ma-dimensional action a = Vpu
where w is the window size.
85 0Ag
Next, the gradient of the encoder stimulation Ag is written in Eq to to simplify the gradient computation.
15
Note that
is manually set
aL
OL
OS
OL
(15)
DAg
O5
OA
as
Finally, the trainable parameters a and in the encoder can updated by:
aL
0L
0Ag
AL
1 Ag
8Ag
0Ag
(16)
8L
81. BAg
8L (a-1)²
AF
Øo BAg
DAg
A.3
Other Spiking Neuron Models
Section 3.3 already shows the LIF neuron model, this section will show two other spiking neuron models conducted in the experiments.
A.3.1
Current-Based LIF neuron model
In the current-based LIF (CLIF) neurons proposed Tang et al. [202]], the input current Eq redefined as:
is
= a + w's +b'
(17)
where a is the current leakage parameter. While other dynamics of CLIF neurons are the same as those LIF neurons.
A.3.2 Dynamic neuron model
Zhang et al. [2022) designed second-order dynamic neurons (DN) for continuous control. The DN consists membrane potential V/ and resistance item U to simulate hyperpolarization. Its dynamics is shown as follows:
dV! =1²² Vi U.
(18)
dt
dU! 0 Vt e.u!
(19)
22

---

where 8, and Oy are the conductivities of V and U, respectively. Once firing spike, the membrane potential V is reset to Veset and the resistance U is added by 6..
With a firs-order Taylor expansion, the iterative DNN can be written as:
C = a C +
S + b':
V (1-5 -
Vi-1 +
Vieseti
U U + S.
Vdelta = V,¹ V
U. C₁:
(20)
Uaeta 0. V 0n U
V V delm
U U: + Udeta
S θ (V) Ven)
A.4
Experiment details
A.4.1
Compute Resources
We conduct the experiments on an RTX 3090 GPU and an Intel(R) Xeon(R) Platinum 8362 CPU.
A.4.2
Spiking Neuron Parameters
The LIF and CLIF neuron parameters are shown in Tab which are the same as those in Tang et al. 202T] except that the LIF neuron has no current leakage parameter. The DN parameters shown
in Tab determined by the pre-learning process proposed in Zhang et al. 2022
Table 3: Parameters of LIF and CLIF Tang et
2021
neurons
Parameter
LIF
CLIF
[Tng et al.
2021
Membrane leakage parameter
0.75
0.75
Threshold voltage Vea.
0.5
0.5
Reset voltage Vreset
0
Current leakage parameter
0.5
Table 4: Parameters of the DN [Zhang et al. 2022
Parameter
Value
SNN time steps Threshold voltage Ven
5
0.5
Current leakage parameter
0.5
Conductivity of membrane potential O,,
-0.122
Conductivity of hidden state O,,
0.529
Reset voltage Vreset
0.021
spike effect to hidden state 0,
0.132
A.4.3 Specific Parameters for the Proxy Target Framework
5abs5shows hyper-parameters of the proxy target framework for different spiking neurons. To capture the behavior of the SNN, the hidden sizes of the proxy network set wider than that of its online SNN. Since different spiking neurons exhibit different dynamics and learning speed, hidden size and learning rate of the proxy network vary across spiking neurons. All other parameters are kept the same.
Siine the InvertedDoulleeenduumm eavironment is relatively easier, there is no needs for such wide proxy network. Thus set the hidden size is (512, 512) specifically for that environment for the CLIF neuron.
23

---

Table 5: paper--arameters of the proxy network framework with different spiking neurons
Parameter
LIF
CLIF (Tang et al.
2021]
DN Zhang et al. 2022
Proxy network architecture
(512, 512)
(800, 600)
(512, 512)
Proxy network activation
ReLU
ReLU
ReLU
Proxy network learning rate
1 10-3
3 10-3
3 10-3
Proxy network optimizer
Adam
Adam
Adam
Proxy update iterations K
3
Proxy update batch size N
256
256
256
A.4.4 Spiking Actor Network Parameters
All hyper-parameters of the spiking actor network shown in Tab This is the same in wide
range of previous studies Tang et al. 2021 Zhang et al. 2022 Chen et al. 2024
Table 6: dlyer-pprameen of the spiking actor network
Parameter
Value
Encoder population per dimension Nin
Encoder threshold VE
0.999
Network hidden units
(256, 256)
Decoder population per dimension Nout
10
Surrogate gradient window size w
0.5
A.4.5 RL algorithm parameters
We conduct the experiment based on the TD3 algorithm (Fujimoto et al. 2018 with hyper-parametert shown in Tab
Table 7: Hyper-parameters of the implemented TD3 algorithm Fujimoto et al. 2018
Parameter
Value
Actor learning rate
3 10-4
Actor regularization
None
Critic learning rate
3 10-4
Critic regularization
None
Critic architecture
(256, 256)
Critic activation
ReL.U
Optimizer
Adam
Target update rate
5 10-3
Batch size N Discount factor
256
0.99
Iterations per time step Reward scaling
1.0
1.0
Gradient clipping
None
Replay buffer size Exploration noise (0 o)
106
NO 0.1)
Actor update interval
Target policy noise A (0, ō)
(0.00.2)
Target policy noise clip
0.5
A.4.6 Experiment environments
Fig. shows various MuloCo environments Todorov et al. 2012 Todorov 2014b on OpenAI
Gymnasium benehmarks [Brockman] 2016,] Towers al. 2024 including InvertedDoubleeenduuuum
(IDP) [ooorov 20 2014a), Ant [Schulman et al. 2015 HalfCheetal (Wawzynski 2009). Hopper
24

---

[Erez et al. 2012] and Walker2d. All environment setups used the default configurations without
modifications.
It worth noting that the state vector ranges from -0o to oc, it is normalized to (-1, 1) by a tanh func- tion. In addition, since the action has the minimum and maximum limits, the output of actor network is normalized to (-1,1) 1) by a tanh function and then linearly scaled to (Min action, Max action).
A.5 Pseudo codes for the proposed proxy target framework in conjunction with TD3
We present the detailed pseudocode of the general proxy target framework in Algo in Section Specifically, Algo shows how implement the proxy target framework in the TD33 algorithm Fujimoto et al. 2018]. It is worth noting that the original TD3 algorithm updates the target actor with delay. However, in our framework, the proxy actor updated without delay because of its inherently
slow update pace.
Algorithm 3 Proxy target framework with TD3
1: Initialize SNN actor network (NNN(s) ANN critic networks QANN a). QAN a) with weights Φ, 0, and 02
T
Proxy 2: Initialize proxy actor T
(s) (s) and ANN target critics DANN(s, a). QAN (s, a) with weights o',
81 and 6
3: Initialize replay buffer D
4: for each iteration do
5: Execute action
" (NNN(s) (s) c, c,c (s,a, a, r, 8') in D
(0, +,,c~N(0,,0)
and observe reward r and next state s'
6: Store the transition (s,
7: for k = 1 to K do
8: Sample minibatch of N transitions (s;, a,, s/) from
9: Update proxy actor network parameters ø' by minimizing the loss:
1
Ltarget
Σ
Prony Ø'
(st)
SNN (s;
end for = ra + min 20, QANN (st, āz), āg = Proxy (sx)) +1s))+e +G, (V)
11:
Update ANN critics by minimizing the critic loss: Teeittic = ト/4 E, (QOAN (8i, as) - y1
12:
13:
if / mod d then
(st, (1 1T (s:) Update SNN actor by maximizing the objective function: J 1/2 E, QANN
Update ANN critic target parameters explicitly by the Polyak function: 8 + T8, + (1)) 16: end if
17:
end for
A.6
Additional experiments results
A.6.1 Additional results in terms of performance
In the main text, we show that our proxy target framework can increase performance for various
spiking neurons. Fig. 7 shows the normalized learning curves of our proxy target framework for different spiking neurons. In addition, Tab. & Tab. 9 and Tab 10 show the maximum average returns and the average performance gains of the proxy network against vanilla SNN with LIF neuron, CLIF neuron, and dynamic neuron, respectively.
25

---

with proxy target network
without proxy target network
ANN DN
LIF
CLIF
Noarmalized Performance
1.0
0.8
0.6
0.4
0.2
0.0
0.5 Normalized training steps
1.0 0.0
0.5
1.0 0.0
0.5
1.0
Normalized training steps
Normalized training steps
Figure 7: Normalized learning curves of the proposed proxy target framework with different spiking neurons across all environments. The performance and training steps are normalized linearly based on ANN performance. Curves are uniformly smoothed for visual clarity.
Table 8: Max average returns over random seeds with LIF neurons.
Method
IDP
Ant
HallCheetah
Hopper
Walker2d
APG
Vanilla LIF
9347 +
4294 + 1170
9404 = 625
3520 + 94
1862 + 1450
32.15%
PT-LIF
9348 1
5383 250
10103 607
3385 157
4314 + 423
Table 9: Max average returns over random seeds with CLIF neurons.
Method
IDP
Ant
HalfCheetah
Hopper
Walker2d
APG
Vanilla CLIF
9351 +
4590 1006
9594 + 689
2772 + 1263
3307 1514
15.03%
PT-CLIF
9351 +
5014 + 1074
9663 426
3526 + 112
4564 555
Table 10: Max average returns over 5 random seeds with dynamic neurons.
Method
IDP
Ant
HalfChoctah
Hopper
Walker2d
APG
Vanilla DN
9350 + 1
4800 1 994
9147 1 231
3446 1 131
3964 1: 1353
4.87%
PT-DN
9350 + 1
5400 1 277
9347 1 666
3507 1: 144
4277 1 650
A.6.2
Additional results ANN
show the normalized learning curves of the proxy target framework with ANN in Fig. b(. Here, show the detailed learning curves and maximum average returns of environments Fig. g and respectively.
Tab
IDP-v4
Ant-v4
HalfChectah-ve v4
Hopper-v4
Walker2d-v4
3
AR (1e3)
2
5
PT+ANN
ANN
TS (le4)
TS (1e6)
TS (1e6)
TS (le6)
TS (le6)
Figure 8: Learning curves of utilizing the proxy target framework in ANN. The PT represents the proxy target framework, AR denotes average returns, and TS training steps. The shaded region represents half a standard deviation over 5 different seeds. Curves are uniformly smoothed for visual clarity.
26

---

Table 11: Max average returns over random seeds with ANN (TD3).
Method
Ant
HalfCheetah
Hopper
Walker2d
APG
ANN
7503 3713
4770 + 1014
10857 = 475
3410 +: 164
4340 + 383
8.38%
PN-ANN
5653 4540
4234 998
10708
3435 145
4106 + 366
27


Reviews:
[{"review_id": "ZxvoqKSve9", "reviewer": "Reviewer_PQzp", "summary": "This paper solves a critical mismatch between SNNs and continuous control RL: SNNs' discrete spikes clash with RL's need for smooth optimization, causing training instability. This paper introduces the Proxy Target Framework to resolve this. During training, a continuous network (the proxy) replaces the SNN target network, providing the smooth updates necessary to stabilize learning. Crucially, the efficient SNN is still used during inference, preserving energy benefits. This approach significantly boosts stability and performance.", "questions": "see weakness (W1-W4)", "limitations": "Increased Training Overhead & Limited Task Generality", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. This paper pinpoints the core conflict between discrete SNN dynamics and the continuous soft-update mechanisms in RL.\n2. By using the proxy only during training, it stabilizes the learning process of SNNs.\n\n\n**Weaknesses:**\nW1. The auxiliary proxy network increases computational cost and complexity during the training phase, even if inference remains efficient. This overhead should be analyzed.\n\nW2. The solution is tailored for Off-policy RL algorithms using target networks (e.g., DDPG, TD3). Its applicability to On-policy methods (e.g., PPO) is unclear.\n\nW3. Achieving performance superior to ANNs might require high spike firing rates or many timesteps, potentially diminishing the claimed energy efficiency benefits (sparsity and latency).\n\nW4. The mechanism for updating the proxy network to match the SNN likely introduces significant interaction costs between the two models. This dual update process could become a training bottleneck and requires further discussion on optimization.", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged that you recognize the effectiveness of the proposed proxy target in terms of training stability and performance. We will do our best to address your concerns in detail.\n\n>W1. The auxiliary proxy network increases computational cost and complexity during the training phase, even if inference remains efficient. This overhead should be analyzed.\n\n\nThank you for this suggestion. To quantify the computational cost during training, we measured the training time required for 100,000 steps on the Ant-v4 environment using an RTX 3090 GPU and an Intel Xeon Platinum 8362 CPU. We then computed the relative training time compared to the baseline vanilla LIF model.\n\n**Table R1:** The relative training time (compared with vanilla LIF) and the average performance gain (against ANN) across different SNN models. \n||Vanilla LIF|MDC-SAN|ILC-SAN|PT-LIF (one-step proxy update)\\*|PT-LIF (default)|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: |\n|Relative Training Time|1.00|1.41|1.79|**1.35**|2.24|\n|Average Performance Gain|-10.54%|0.37%|4.64%|5.44%|**5.84%**|\n\n\nThe trade-off between training cost and performance is a long-standing issue. All SOTA SNN algorithms require more training time than the vanilla LIF model. However, our PT-LIF achieves the highest performance gain with reasonable training overhead. Notably, with a lightweight one-step proxy update configuration, our PT-LIF requires even less training time than other SOTA models (e.g. MDC-SAN, ILC-SAN), while still surpassing them in terms of performance. \n\n\nIn addition, our model remains inference efficient. In contrast, other SNN variants (e.g. MDC-SAN, ILC-SAC) introduce architectural or neuronal complexities that increase inference energy consumption. \n\n\n\\*: To reduce training costs, we can adopt a one-step proxy update setting for LIF neuron, with the proxy update iteration $K=1$ and proxy learning rate $lr_{proxy}=3\\cdot10^{-3}$ (where the default setting is $K=3$ and $lr_{proxy}=3\\cdot10^{-3}$). This lightweight configuration significantly reduces computation without major performance loss. Please refer to Table R1 in our response to Reviewer JUY9 for detailed performance under this setting across all environments.\n\n>W2. The solution is tailored for Off-policy RL algorithms using target networks (e.g., DDPG, TD3). Its applicability to On-policy methods (e.g., PPO) is unclear.\n\nThank you for this insightful question. This work focuses on resolving the conflict between smooth target network updates and the non-differentiable dynamics of spiking neural networks (SNNs), a challenge that arises specifically in off-policy reinforcement learning (RL) algorithms such as DDPG and TD3, which employ target networks. Since on-policy algorithms like PPO do not use target networks, the proposed solution does not directly apply to them and falls outside the current scope of our study. We will revise the introduction to explicitly state that the proposed approach is tailored for off-policy algorithms.\n\nWe would also like to highlight that off-policy algorithms remain a central focus in RL research due to their better sample efficiency, as they can reuse past experience from the replay buffer. Hence, we believe that addressing the challenges of applying SNNs in these algorithms is of great practical value.\n\nMoreover, adapting on-policy methods to SNNs also has its own challenges. For example, methods like PPO rely on smooth policy updates, while the discrete nature of SNNs can lead to abrupt output shifts in the policy network that may destabilize training. We are interested in exploring potential solutions in our future work.\n\n\n> W3. Achieving performance superior to ANNs might require high spike firing rates or many timesteps, potentially diminishing the claimed energy efficiency benefits (sparsity and latency).\n\n\nThank you for this insightful comment. We would like to clarify that achieving performance superior to ANNs does not necessarily require higher spike firing rates or longer timesteps in our method. To support this point, we report the average firing rates in Table R2. The firing rate of our model remains within a typical range and is even lower than that of the baseline SNN in most cases.\n\n**Table R2:** Comparison of the average firing rates of spiking neurons in different environments over $5$ random seeds.\n\n| Method   | IDP-v4   | Ant-v4   |HalfCheetah-v4|Hopper-v4|Walker2d-v4|Average|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: | :----:|\n|LIF|0.22|0.31|0.40|0.25|0.47|0.33| \n|PT-LIF|0.24|0.33|0.37|0.25|0.40|**0.32**| \n\nFurthermore, we would like to clarify that the number of timesteps is consistently set to 5 across all experiments, including those for the baseline SNN and other existing algorithms.\n\nThese results indicate that our method maintains moderate spike activity and low latency while outperforming ANN baselines. Furthermore, as shown in Table 2 of the main manuscript, our model achieves lower energy consumption than both the baseline SNN and ANN models, preserving energy efficiency.\n\nTo avoid any potential misunderstanding, we will revise the manuscript to explicitly emphasize the timestep setting and include the firing rate statistics in the energy analysis section of the manuscript (Section 5).\n\n\n>W4. The mechanism for updating the proxy network to match the SNN likely introduces significant interaction costs between the two models. This dual update process could become a training bottleneck and requires further discussion on optimization.\n\nThank you for this insightful question. To address the concern clearly, we would like to respond from three perspectives (energy efficiency, optimization performance, and clarification of the dual update process).\n\n* Energy Efficiency:\nWe would like to emphasize that the proposed framework requires less inference energy compared to the ANN or SNN baseline (as shown in Table 2 of the main manuscript), and thus remains inference efficient. Moreover, the proposed method introduces reasonable additional computational overhead during training. As discussed in our response to Question W1, the overall training cost remains comparable to existing methods.\n* Optimization Performance:\nWe would like to clarify that the proxy network is not a training bottleneck in terms of performance. On the contrary, it is introduced to alleviate a key bottleneck observed in traditional SNN-RL frameworks. Specifically, we found that directly using an SNN as the target network can lead to unstable updates due to its discrete and abrupt output dynamics, which hinders optimization. The proxy network helps to smooth the learning targets and improves the stability and performance of the learning process. For theoretical justification, please refer to our response to Reviewer i6Dh, Question 1. We also showed empirical evidence supporting the effectiveness of the proxy network in the experiments section of the main manuscript.\n* Clarification of the Dual Update Process:\nThe presence of a dual update process in our framework is consistent with standard actor-critic RL algorithms such as DDPG and TD3. These algorithms contain actors, critics, and corresponding target networks (or proxy targets). In our method, the only distinction lies in how we update the target actor (proxy). Instead of performing a soft update for the SNN target actor, we introduce an implicit update from the proxy target to the SNN actor. Importantly, the actor network remains unaffected during this process, as its parameters are frozen. This update scheme is fully compatible with the conventional actor-critic paradigm and has been empirically shown to improve training stability and overall performance.\n\nWe will revise the manuscript to include a more detailed explanation of this mechanism. Please feel free to let us know if there are any remaining concerns."}, {"review_id": "40aAyuV1YC", "reviewer": "Reviewer_i6Dh", "summary": "This paper studies the challenge of applying continuous-control reinforcement learning algorithms to Spiking Neural Networks (SNNs), pointing out that standard RL mechanisms—especially the soft update of target networks—are incompatible with the discrete, non-differentiable nature of SNN spikes. The authors propose a proxy target framework in which a continuous, differentiable proxy network is used during training to smooth updates and bridge the gap between SNNs and RL algorithms originally designed for ANNs. Experiments demonstrate that SNNs trained with this framework achieve improved stability and up to 32% higher average performance in standard MuJoCo environments, even surpassing traditional ANNs in some settings. The framework preserves the energy-efficiency benefits of SNNs at inference.", "questions": "n/a", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Identification of a Previously Unaddressed Limitation: The paper provides a clear diagnosis of a previously underappreciated source of instability in SNN-based RL for continuous control—the incompatibility of non-differentiable spiking output with soft target updates. This is demonstrated empirically (see Figure 2b), which shows abrupt, discrete changes in SNN target outputs that undermine stable learning.\n\n- Methodological Contribution: The proposed \"proxy target\" mechanism is simple yet effective. By using a continuous, differentiable actor only for target updates during training, it permits smooth transitions needed for stable RL optimization without sacrificing SNN deployment benefits. The update mechanism is derived clearly, and implementation details are provided.\n\n- Comprehensive Experimental Validation: The authors provide thorough empirical evidence across five MuJoCo environments, using multiple spiking neuron types (LIF, CLIF, DN). Learning curves (Figure 4) illustrate consistent improvement in convergence speed and final return, with shaded regions for statistical reliability. Table 1 compares max average returns to multiple strong SNN baselines and ANN-based RL, showing the proposed PT-LIF outperforms all baselines in average performance gain (5.84%).\n\nWeaknesses:\n- Limited Theoretical Analysis: While the mechanism is well-motivated and empirically validated, the manuscript lacks rigorous theoretical support. The smoothing effect of the proxy is justified mainly by intuition and plotting, without formal stability/convergence claims (see Section 4, pages referencing Figures 2c-d). This limits broader generalizability to other RL algorithms or network types.\n\n- Ablation Shortcomings: The experimental section would benefit from ablations isolating the effects of proxy update frequency, the batch size (N) used for proxy training, or the depth/width of proxy architectures to clarify the mechanism’s robustness and optimal configurations. While Figure 2c-d and Figure 4 highlight core effects, missing such ablations leaves open questions about sensitivity and practical guidance.\n\n- Missing Key Baselines: Although several state-of-the-art SNN RL methods are compared in Table 1, additional direct baselines such as SNNs with architectural/topological trims but without proxy, or variants using different activation surrogates, would further clarify the gain is from the proxy and not other confounds.\n\n- Proxy-Online SNN Gap Quantification: While Figure 2d visually indicates the proxy network closely tracks the SNN actor, a quantitative, environment-wide measure of this approximation gap is not provided. Including such a metric (e.g., proxy-SNN output distance during learning) across seeds/environments would clarify how tightly the proxy must follow the SNN to ensure learning stability.", "was_revised": true, "final_justification": "Thanks for the rebuttal. Combining the comments with other reviewers, I will maintain my original score.", "rebuttal": "We would like to thank you for your constructive questions and suggestions! We will conduct a detailed clarification and discussion for your concerns.\n\n>Limited Theoretical Analysis: While the mechanism is well-motivated and empirically validated, the manuscript lacks rigorous theoretical support. The smoothing effect of the proxy is justified mainly by intuition and plotting, without formal stability/convergence claims (see Section 4, pages referencing Figures 2c-d). This limits broader generalizability to other RL algorithms or network types.\n\nThank you for the valuable suggestion. Here we provide a theoretical analiysis demonstrating that the proxy network enables smoother policy updates than the SNN target.\n\n**Theorem 1:** For proxy network $\\pi_{\\phi'}^{Proxy}$ updated by minimizing the loss $L_{proxy}$ in Eq.11, during each update, as the proxy learning rate $lr_{proxy}$ approaches to zero, $\\left\\|\\pi_{\\phi_{new}'}^{Proxy}(s)-\\pi_{\\phi_{old}'}^{Proxy}(s)\\right\\|$ also approches to zero. Where $\\phi_{old}'$ and $\\phi_{new}'$ are the parameters of the proxy network before and after the update, respectively.\n\n**Proof 1:** By standard gradient descent, we have:\n$$\\lim_{lr_{proxy}\\to0}\\left\\|(\\phi_{new}'-\\phi_{old}')\\right\\|=\\lim_{lr_{proxy} \\to 0} \\left \\|lr_{proxy} \\cdot \\nabla_{\\phi_{old}'} L_{proxy}\\right\\|=0.$$\nUnder the assumption that $\\pi_{\\phi'}^{Proxy}(s)$ is continuously differentiable with respect to $ϕ'$, we apply a first-order Taylor expansion:\n$$\\lim_{lr_{proxy}\\to0}\\left\\|\\pi_{\\phi_{new}'}^{Proxy}(s)-\\pi_{\\phi_{old}'}^{Proxy}(s)\\right\\|= \\lim_{lr_{proxy}\\to0}\\left\\|(\\phi_{new}'-\\phi_{old}')\\nabla_{\\phi'}\\pi_{\\phi_{old}'}^{Proxy}(s)\\right\\|=0.$$\n\nThis result shows that the proxy network supports sufficiently small policy updates. In contrast, due to the discrete nature of SNN target, it cannot produce sufficiently small output changes. Thus, the smoothing effect of proxy is superior, which enables smooth and stable optimization.\n\nWe will include the above theoretical analysis in the main text after Section 4.2.\n\n>Ablation Shortcomings: The experimental section would benefit from ablations isolating the effects of proxy update frequency, the batch size (N) used for proxy training, or the depth/width of proxy architectures to clarify the mechanism’s robustness and optimal configurations. While Figure 2c-d and Figure 4 highlight core effects, missing such ablations leaves open questions about sensitivity and practical guidance.\n\nThank you for this valuable question. We conducts ablations isolating the effects of proxy update iterations (K), the batch size (N) used for proxy training, the depth and width of proxy architectures.\n\n**Table R1:** Maximum average returns and average performance gain (APG) compared to the default settings. \n||Ant-v4|HalfCheetah-v4|Hopper-v4|Walker2d-v4|APG|\n| -------- | :--------: | :--------: | :--------: | :--------: | --------: |\n|Without Proxy|$4294 \\pm 1170$|$9404 \\pm 625$|$3520 \\pm 94$|$1862 \\pm 1450$|$-20.01$%|\n|K=2|$5532 \\pm 2714$|$8461 \\pm 4213$|$3419 \\pm 1677$|$4229 \\pm 2083$|$-3.62$%\n|K=3|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$0.00$%|\n|K=5|$5587 \\pm 2745$|$9750 \\pm 4782$|$2374 \\pm 1648$|$4131 \\pm 2033$|$-8.46$%\n|N=128|$4967 \\pm 1086$|$9688 \\pm 786$|$3517 \\pm 106$|$3521 \\pm 1479$|$-6.59$%\n|N=256|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$0.00$%|\n|N=512|$5374 \\pm 2641$|$10064 \\pm 4931$|$2380 \\pm 1655$|$2940 \\pm 1775$|$-15.53$%\n|depth=1|$5062 \\pm 960$|$9272 \\pm 625$|$3453 \\pm 141$|$4170 \\pm 340$|$-3.89$%\n|depth=2|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$0.00$%|\n|depth=3|$5176 \\pm 2559$|$9728 \\pm 4795$|$3435 \\pm 1687$|$4067 \\pm 2018$|$-2.96$%\n|width=(256,256)|$4491 \\pm 2424$|$8464 \\pm 4177$|$3475 \\pm 1708$|$4242 \\pm 2118$|$-7.95$%\n|width=(512,512)|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$0.00$%|\n|width=(800,600)|$5459 \\pm 2675$|$9616 \\pm 4734$|$3495 \\pm 1715$|$2841 \\pm 1715$|$-7.26$%|\n\nThese results suggest that the proposed proxy network is robust to moderate variations in hyperparameters and consistently outperforms the baseline without the proxy. The default configuration (K=3, N=256, depth=2, width=(512,512)) achieves strong and stable performance across different tasks. Meanwhile, setting these parameters too low or too high tends to degrade performance.\n\n\n>Missing Key Baselines: Although several state-of-the-art SNN RL methods are compared in Table 1, additional direct baselines such as SNNs with architectural/topological trims but without proxy, or variants using different activation surrogates, would further clarify the gain is from the proxy and not other confounds.\n\nThanks for the suggestion. We added several new baselines. For SNNs with different architects, we conducted experiments using vanilla LIF neuron, pop-SAN[1], MDC-SAN[2]. For SNNs with topological modifications, we conducted experiments of ILC-SAN[3]. For SNN with different surrogates, we  added experiments on LIF neuron with tanh surrogate function without proxy. In addition, we also added a new baseline for converting the pre-trained ANN to an SNN (using the method proposed in [4] with 100 time steps).\n\n**Table R2:** Max average returns and average performance gain (APG) against ANN.\n\n|Method|IDP-v4|Ant-v4|HalfCheetah-v4|Hopper-v4|Walker2d-v4|APG|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: |--------: |\n|ANN|$7503 \\pm 3713$|$4770 \\pm 1014$|$10857 \\pm 475$|$3410 \\pm 164$|$4340 \\pm 383$|$0.00$%|\n|ANN-SNN|$3859\\pm4440$|$3550\\pm963$|$8703\\pm658$|$3098\\pm281$|$4235\\pm354$|$-21.11$%|\n|Vanilla LIF with rectangular surrogate|$9347 \\pm 1$|$4294 \\pm 1170$|$9404 \\pm 625$|$3520 \\pm 94$|$1862 \\pm 1450$|$-10.54$%|\n|Vanilla LIF with tanh surrogate|$9348 \\pm 1$|$4486 \\pm 2380$|$9019 \\pm 4436$|$3361 \\pm 1654$|$4147 \\pm 2036$|$-0.83$%|\n|pop-SAN|$9351 \\pm 1$|$4590 \\pm 1006$|$9594 \\pm 689$|$2772 \\pm 1263$|$3307 \\pm 1514$|$-6.66$%|\n|MDC-SAN|$9350 \\pm 1$|$4800 \\pm 994$|$9147 \\pm 231$|$3446 \\pm 131$|$3964 \\pm 1353$|$0.37$%|\n|ILC-SAN|$9352 \\pm 1$|$5584 \\pm 272$|$9222 \\pm 615$|$3403 \\pm 148$|$4200 \\pm 717$|$4.64$%|\n|PT-CLIF|$9351 \\pm 1$|$5014 \\pm 1074$|$9663 \\pm 426$|$3526 \\pm 112$|$4564 \\pm 555$|$5.46$%|\n|PT-DN|$9350 \\pm 1$|$5400 \\pm 277$|$9347 \\pm 666$|$3507 \\pm 144$|$4277 \\pm 650$|$5.06$%|\n|PT-LIF|$9348 \\pm 1$|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$5.84$%|\n\nThe proposed algorithms (i.e. PT-CLIF, PT-DN, and PT-LIF) still outperform other baselines.\n\nWe will include the new results in Table 1 of the main text to further clarify the gains from the proxy.\n\n\n>Proxy-Online SNN Gap Quantification: While Figure 2d visually indicates the proxy network closely tracks the SNN actor, a quantitative, environment-wide measure of this approximation gap is not provided. Including such a metric (e.g., proxy-SNN output distance during learning) across seeds/environments would clarify how tightly the proxy must follow the SNN to ensure learning stability.\n\nThank you for the suggestion. We evaluated the mean squared output error between the target network and the online SNN during training (with the same inputs), and averaged them over 4 different environments (i.e., Ant-v4, HalfCheetah-v4, Hopper-v4, and Walker2d-v4) over 5 random seeds.  The average approximation gap is relatively smaller with the proxy network in all training steps, demonstrating that the proxy can more closely follow the SNN to ensure learning stability.\n\n**Table R3:** The average approximation gap between the target network and the SNN actor in different training steps.\n|Method|$0.2$M|$0.4$M|$0.6$M|$0.8$M|$1.0$M|\n| -------- |:--------: | :--------: | :--------: | :--------: | :--------: | \n|LIF|0.082|0.067|0.062|0.067|0.060|\n|PT-LIF|0.079|0.067|0.053|0.051|0.052|\n\n\n**References** \n\n\n[1] Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos Michmizos. Deep reinforcement learning with population-coded spiking neural network for continuous control. In Conference on Robot Learning, pages 2016–2029. PMLR, 2021.\n\n[2] Duzhen Zhang, Tielin Zhang, Shuncheng Jia, and Bo Xu. Multi-sacle dynamic coding improved spiking actor network for reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 59–67, 2022.\n\n[3] Ding Chen, Peixi Peng, Tiejun Huang, and Yonghong Tian. Fully spiking actor network with intralayer connections for reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 36(2):2881–2893, 2024a.\n\n[4] Tong Bu, Maohua Li, and Zhaofei Yu. Inference-Scale Complexity in ANN-SNN Conversion for High-Performance and Low-Power Applications. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24387-24397, 2025."}, {"review_id": "rh5fCOA937", "reviewer": "Reviewer_JUY9", "summary": "This paper introduces a Proxy Target in the SNN-RL training framework to bridge the gap between the discrete nature of SNNs and the continuous control RL. While traditional RL algorithms rely on smooth target network updates, it conflicts with the non-differentiable dynamics of SNNs, leading to unstable training. The proposed method incorporates a differentiable proxy actor used only during training to stabilize learning while preserving the energy efficiency of SNNs during inference. Various experiments demonstrate significant performance gains, while the average performance across different environments even surpasses standard ANN baselines with simple LIF neurons.", "questions": "1. The authors claimed that “Simple neuron makes the best.”, Could the authors discuss more about why simple neurons perform better in this specific situation?\n\n2. There are typos and informal writings in the manuscript. For example, in Figure 1, “inplicit” should be “implicit”; “Multi Iterations” should be “Multi-iteration” or “Multiple Iterations”.", "limitations": "See weaknesses.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. This paper addresses the critical challenge of training instability in SNN-based reinforcement learning by introducing a well-motivated proxy target framework that effectively mitigates the mismatch between discrete spiking dynamics and continuous control requirements.\n2. The experimental evaluation is thorough, covering multiple environments and incorporating different neuron types (LIF, CLIF, ...). Results consistently show improved performance, reduced variance, and, notably the SNN-based RL outperforms ANN baselines. Moreover, the study also demonstrates the energy efficiency of SNNs by estimating inference energy consumption.\n3. It is impressive to see SNNs outperform ANN in RL field.\n\nWeaknesses\n1. While the proxy actor effectively stabilizes reinforcement learning with an SNN actor, the additional training cost introduced by the implicit update mechanism is not thoroughly discussed. As illustrated in Figure 1, the implicit update appears to involve multiple backpropagation steps per iteration, in contrast to the single-step soft parameter update in traditional frameworks. This may lead to increased training time and computational overhead.\n\n2. In line 212, the authors state that the proxy is a multilayer feedforward network. However, the effect of specific architectural choices (such as the number of layers and hidden units) is not clearly justified and studied. Additionally, the effect of the number of proxy update iterations (hyperparameter K) is not explored.\n\n3. In the discussion on energy consumption, it would be beneficial to report the number of SOPs alongside the estimated energy values, as SOPs offer a more general measure of computational complexity. Since energy estimates can vary significantly depending on the hardware, including a comparison of SOPs would enhance the generalizability of the energy efficiency claims.", "was_revised": true, "final_justification": "The authors have successfully addressed all my concerns. It is remarkable that SNNs outperform ANNs in reinforcement learning tasks, which expands the application scope of SNNs beyond their well-known energy efficiency advantages. I will maintain my recommendation for acceptance.", "rebuttal": "We sincerely appreciate your recognition of our work. We are delighted that you noted that \"It is impressive to see SNNs outperform ANN in RL field\". We will strive to address your relevant questions.\n\n\n>While the proxy actor effectively stabilizes reinforcement learning with an SNN actor, the additional training cost introduced by the implicit update mechanism is not thoroughly discussed. As illustrated in Figure 1, the implicit update appears to involve multiple backpropagation steps per iteration, in contrast to the single-step soft parameter update in traditional frameworks. This may lead to increased training time and computational overhead.\n\n\nThank you for this valuable question. To reduce the training cost associated with multiple backpropagation steps during the proxy update, we can adopt a one-step proxy update setting for LIF neuron, with the proxy update iteration $K=1$ and proxy learning rate $lr_{proxy}=3\\cdot10^{-3}$ (where the default setting is $K=3$ and $lr_{proxy}=3\\cdot10^{-3}$). This lightweight configuration significantly reduces computation without major performance loss (only a 0.4% performance degradation).\n\n**Table R1:** Maximum average returns and average performance gain (APG) against ANN with different settings of the proposed PT-LIF. \n| Method                         | IDP-v4          | Ant-v4          | HalfCheetah-v4  | Hopper-v4      | Walker2d-v4    | APG     |\n| ------ | :--------: | :--------: | :--------: | :--------: | :--------: | :------: |\n| ANN                            | $7503 \\pm 3713$ | $4770 \\pm 1014$ | $10857 \\pm 475$ | $3410 \\pm 164$ | $4340 \\pm 383$ | $0.00$% |\n| PT-LIF (default)               | $9348 \\pm 1$    | $5383 \\pm 250$  | $10103 \\pm 607$ | $3385 \\pm 157$ | $4314 \\pm 423$ | $5.84$% |\n| PT-LIF (one-step proxy update) | $9348 \\pm 0$    | $5032 \\pm 535$  | $9440 \\pm 553$  | $3504 \\pm 159$ | $4661 \\pm 263$ | $5.44$% |\n\n\nWe further measured the training time required for 100,000 steps on the Ant-v4 environment using an RTX 3090 GPU and an Intel Xeon Platinum 8362 CPU. We then computed the relative training time compared to the baseline vanilla LIF model.\n\n**Table R2:** The relative training time (compared with vanilla LIF) and the average performance gain (against ANN) across different SNN models. \n||Vanilla LIF|MDC-SAN|ILC-SAN|PT-LIF (one-step proxy update)\\*|PT-LIF (default)|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: |\n|Relative Training Time|1.00|1.41|1.79|**1.35**|2.24|\n|Average Performance Gain|-10.54%|0.37%|4.64%|5.44%|**5.84%**|\n\n\nThe trade-off between training cost and performance is a long-standing issue. All SOTA SNN algorithms require more training time than the vanilla LIF model. However, our PT-LIF achieves the highest performance gain with reasonable training overhead. Notably, with a lightweight one-step proxy update configuration, our PT-LIF requires even less training time than other SOTA models (e.g. MDC-SAN, ILC-SAN), while still surpassing them in terms of performance. \n\n\nIn addition, our model remains inference efficient. In contrast, other SNN variants (e.g. MDC-SAN, ILC-SAC) introduce architectural or neuronal complexities that increase inference energy consumption.\n\n\n>In line 212, the authors state that the proxy is a multilayer feedforward network. However, the effect of specific architectural choices (such as the number of layers and hidden units) is not clearly justified and studied. Additionally, the effect of the number of proxy update iterations (hyperparameter K) is not explored.\n\nThank you for this valuable suggestion. We conduct further experiments on the effects of proxy update iterations (K), the depth and width of proxy architectures.\n\n**Table R3:** Max average returns and average performance gain (APG) against the default settings. \n||Ant-v4|HalfCheetah-v4|Hopper-v4|Walker2d-v4|APG|\n| -------- | :--------: | :--------: | :--------: | :--------: | --------: |\n|Without Proxy|$4294 \\pm 1170$|$9404 \\pm 625$|$3520 \\pm 94$|$1862 \\pm 1450$|$-20.01$%|\n|K=2|$5532 \\pm 2714$|$8461 \\pm 4213$|$3419 \\pm 1677$|$4229 \\pm 2083$|$-3.62$%\n|K=3|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$0.00$%|\n|K=5|$5587 \\pm 2745$|$9750 \\pm 4782$|$2374 \\pm 1648$|$4131 \\pm 2033$|$-8.46$%|\n|depth=1|$5062 \\pm 960$|$9272 \\pm 625$|$3453 \\pm 141$|$4170 \\pm 340$|$-3.89$%\n|depth=2|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$0.00$%|\n|depth=3|$5176 \\pm 2559$|$9728 \\pm 4795$|$3435 \\pm 1687$|$4067 \\pm 2018$|$-2.96$%\n|width=(256,256)|$4491 \\pm 2424$|$8464 \\pm 4177$|$3475 \\pm 1708$|$4242 \\pm 2118$|$-7.95$%\n|width=(512,512)|$5383 \\pm 250$|$10103 \\pm 607$|$3385 \\pm 157$|$4314 \\pm 423$|$0.00$%|\n|width=(800,600)|$5459 \\pm 2675$|$9616 \\pm 4734$|$3495 \\pm 1715$|$2841 \\pm 1715$|$-7.26$%|\n\nThese results suggest that the proposed proxy network is robust to moderate variations in hyperparameters, as all these settings could outperform the baseline without the proxy. The default configuration (K=3, depth=2, width=(512,512)) achieves strong and stable performance across different tasks. Meanwhile, setting these parameters too low or too high tends to degrade performance.\n\n>In the discussion on energy consumption, it would be beneficial to report the number of SOPs alongside the estimated energy values, as SOPs offer a more general measure of computational complexity. Since energy estimates can vary significantly depending on the hardware, including a comparison of SOPs would enhance the generalizability of the energy efficiency claims.\n\nThank you for the valuable suggestion. We show the number of SOPs in Table R4, the proposed method also has low SOPs on average.\n\n**Table R4:** The average number of SOPs per inference in different environments across $5$ random seeds. Where the unit is $1,000$ SOPs.\n| Method   | IDP-v4   | Ant-v4   |HalfCheetah-v4|Hopper-v4|Walker2d-v4|Average|\n| -------- |:--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|LIF   |105.8 |\t152.9 |\t196.6 \t|93.6 \t|244.4 |\t158.6| \n|PT-LIF| 117.1 |\t158.2 |\t174.9 \t|89.1 \t|180.9 \t|144.0 |\n\nWe will add the number of SOPs alongside the estimated energy values in the energy consumption part of the main text to offer a more general measure of computational complexity.\n\n\n>The authors claimed that “Simple neuron makes the best.”, Could the authors discuss more about why simple neurons perform better in this specific situation?\n\nThank you for this insightful question. Once the SNN surpasses the ANN in performance, the training bottleneck shifts from the capacity of the SNN actor to the reinforcement learning algorithm itself. Therefore, incorporating more complex spiking neurons may introduce unnecessary training complexity, potentially degrading performance.\n\n\n>There are typos and informal writings in the manuscript. For example, in Figure 1, “inplicit” should be “implicit”; “Multi Iterations” should be “Multi-iteration” or “Multiple Iterations”.\n\nThanks for your detailed review of the manuscript. We will correct these errors and thoroughly revise the manuscript."}, {"review_id": "yt1fZbKQhL", "reviewer": "Reviewer_tLiq", "summary": "This paper investigates the challenge of applying Spiking Neural Networks (SNNs) to continuous control reinforcement learning (RL) tasks—a domain where conventional Artificial Neural Networks (ANNs) dominate due to their compatibility with smooth gradient-based updates. The authors point out an incompatibility between the discrete, event-driven nature of SNNs and the soft-update mechanism of target networks in RL, which relies on continuous and differentiable dynamics. To solve this problem, the paper introduces a proxy target framework that decouples the discrete behavior of SNNs from the smooth updating required in RL. This proxy serves as a differentiable surrogate for SNN-based target networks, enabling stable and effective learning. Experimental results across multiple continuous control benchmarks (e.g., MuJoCo tasks) show that the proposed proxy target mechanism can improve the performance of SNN agents—achieving up to 32% performance gains over baseline SNNs—without requiring major changes to the overall learning algorithm or spiking neuron models.", "questions": "1. Can the authors provide empirical evidence or simulated estimates to support the claim that SNNs offer superior energy efficiency for edge deployment compared to ANN baselines running on modern NPUs? Specifically, how does the energy consumption of your SNN-based agent compare (in hardware, simulator, or profiling estimates) to that of ANN agents with similar model sizes on commercially available low-power accelerators such as those described in https://arxiv.org/pdf/2503.22567?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. **Timely and Relevant Problem**.\nThe paper tackles the critical gap in making SNNs viable for continuous control RL tasks on edge devices. Given the growing interest in neuromorphic computing for energy-constrained agents (e.g., robotics), this work is both relevant and potentially impactful.\n\n2. **Novel Technical Contribution**.\nThe proxy target mechanism is a clean and elegant solution to the incompatibility between spike-based updates and target network soft updates. By introducing a continuous and differentiable proxy to interface with the RL loss, the paper bridges a gap that has previously hindered the use of SNNs in continuous action spaces.\n\n3. **Empirical Validation**.\nThe experiments are extensive and compare different spiking neuron models under standard RL benchmarks. The gains in learning stability and final performance are significant, demonstrating the effectiveness of the proposed technique across neuron types.\n\n## Weaknesses\n\n1. **Energy Efficiency Claims Not Substantiated**.\nWhile the paper argues that SNNs are better suited for edge deployment due to energy efficiency, no empirical or simulated hardware evaluation is provided to validate this claim. In fact, modern NPUs and embedded accelerators (e.g., those discussed in https://arxiv.org/pdf/2503.22567) can run small ANNs at milliwatt-level power budgets—potentially lower than neuromorphic chips like Intel’s Loihi-2 (~1W) (https://open-neuromorphic.org/neuromorphic-computing/hardware/loihi-2-intel/). The authors should temper their claims or support them with energy and latency measurements.", "was_revised": true, "final_justification": "I read the rebuttal carefully. The authors answered my questions. So I decide to raise my score.", "rebuttal": "We are pleased that you appreciate our proposed concept of the proxy target. Additionally, we are delighted to see the significant progress modern NPUs have made in reducing the energy consumption of ANNs. We would like to address your concerns as follows.\n\n>While the paper argues that SNNs are better suited for edge deployment due to energy efficiency, no empirical or simulated hardware evaluation is provided to validate this claim.\n\nThank you for this valuable suggestion. Recent studies have demonstrated that SNNs can achieve higher energy efficiency in FPGAs through empirical hardware simulations [1]. We will add this to the Introduction part of the manuscript to support the energy efficiency claim.\n\n>In fact, modern NPUs and embedded accelerators (e.g., those discussed in [2]) can run small ANNs at milliwatt-level power budgets—potentially lower than neuromorphic chips like Intel’s Loihi-2 (~1W)[3].\n\nIntel’s Loihi-2 [2] is not designed to achieve extremely energy efficient, as it has to compromise for compatibility with complex neurons and brain-inspired learning rules. For instance, it supports graded spike events (up to 32 bit) and three-factor learning rules. Thus, we believe it is fairer to compare the energy consumption of ANNs running on NPUs with SNNs running on FPGAs. The detailed comparison will be shown in the answer to the next question. \n\n>Can the authors provide empirical evidence or simulated estimates to support the claim that SNNs offer superior energy efficiency for edge deployment compared to ANN baselines running on modern NPUs? Specifically, how does the energy consumption of your SNN-based agent compare (in hardware, simulator, or profiling estimates) to that of ANN agents with similar model sizes on commercially available low-power accelerators such as those described in [2]?\n\n**ANN inference energy in modern NPUs.** We estimated the energy consumption of each MAC based on the data provided in [2], using the maximum inference per $mJ$ across seven NPUs (i.e. MAX78k (C), MAX78k (R), GAP8, NXP-MCXN947, HX-WE2 (S), HX-WE2 (P),  MILK-V), not including NPU initialization.\n\n**Table R1:** Estimates of energy consumption per MAC in NPU, with data provided in [2]. \n||NAS|ResNet|SimpleNet|Autoenc|YoloV1_small|\n| -------- |:--------: | :--------: | :--------: | :--------: | :--------: | \n|MMACs|74.2512|37.7812|38.0006|0.5455|43.8294|\n|Maximum Inferences time per $mJ$|2.75|9.31|4.13|47.06|5.15|\n|Energy Consumption per MAC ($pJ$)|4.90|4.61|6.37|38.95|3.97|\n\nWe then use the minimum estimate (i.e., 3.97 $mJ$ per MAC) to calculate the energy consumption for our baseline ANN model.\n\n**Table R2:** MACs and estimated energy consumption per inference for ANN models running on NPUs in different environments.\n\n|    | IDP-v4   | Ant-v4   |HalfCheetah-v4|Hopper-v4|Walker2d-v4|Average|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| MACs  |68608|74496|71424|69120|71424|71014.4|\n|Energy Consumption ($nJ$)|272.33|295.60|283.41|274.26|283.41|281.78|\n\n\n**SNN inference energy estimations.** As shown in the main text, we estimate the energy consumption of SNNs in the same way as [4], where the synaptic operation (SOP) costs $77fJ$ [5,6].\n\n\n**Table R3:** SOPs and estimated energy consumption for vanilla LIF in different environments.\n|    | IDP-v4   | Ant-v4   |HalfCheetah-v4|Hopper-v4|Walker2d-v4|Average|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|SOPs (1k)|105.8 |\t152.9 |\t196.6 \t|93.6 \t|244.4 |\t158.6| \n|Energy Consumptions ($nJ$)|8.14|11.78|15.13|7.21|18.82|12.21|\n\n**Table R4:** SOPs and estimated energy consumption for the proposed PT-LIF in different environments. \n|    | IDP-v4   | Ant-v4   |HalfCheetah-v4|Hopper-v4|Walker2d-v4|Average|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|SOPs (1k)| 117.1 |\t158.2 |\t174.9 \t|89.1 \t|180.9 \t|144.0 |\n|Energy Consumptions ($nJ$)|9.01|12.18|13.46|6.86|13.93|11.09|\n\n\n**Comparison between ANNs running on NPUs and SNNs.** Table R5 concludes the estimated inference consumption by summarizing the results in Table R2, R3, R4. \n\n**Table R5:** Inference energy consumption for ANNs running on NPUs and SNNs, where the energy unit is nano-joule (nJ).\n\n| Method   | IDP-v4   | Ant-v4   |HalfCheetah-v4|Hopper-v4|Walker2d-v4|Average|\n| -------- | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|ANN|272.33|295.60|283.41|274.26|283.41|281.78|\n|LIF|8.14|11.78|15.13|7.21|18.82|12.21|\n|PT-LIF|9.01|12.18|13.46|6.86|13.93|**11.09**|\n\nTable R5 demonstrates that SNN models consume less energy than the ANN models running on NPUs. \n\n\nThank you again for your suggestion. We will include the energy consumption data for NPUs, based on the information provided in [2], in Table 2 of the manuscript. Please feel free to let us know if you have any remaining concerns.\n\n\n**References**\n\n[1] Ali, Asmer Hamid, Mozhgan Navardi, and Tinoosh Mohsenin. \"Energy-Aware FPGA Implementation of Spiking Neural Network with LIF Neurons.\" arXiv preprint arXiv:2411.01628 (2024).\n\n[2] Millar, Josh, et al. \"Benchmarking Ultra-Low-Power $\\mu$NPUs.\" arXiv preprint arXiv:2503.22567 (2025).\n\n[3] Davies, Mike. \"Taking neuromorphic computing to the next level with Loihi2.\" Intel Labs’ Loihi 2.1 (2021).\n\n[4] Merolla, Paul A., et al. \"A million spiking-neuron integrated circuit with a scalable communication network and interface.\" Science 345.6197 (2014): 668-673.\n\n[5] Qiao, Ning, et al. \"A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses.\" Frontiers in neuroscience 9 (2015): 141.\n\n[6] Hu, Yangfan, Huajin Tang, and Gang Pan. \"Spiking deep residual networks.\" IEEE Transactions on Neural Networks and Learning Systems 34.8 (2021): 5200-5205."}]

Author rebuttal:

Meta-review: The authors propose a novel method for actor-critic reinforcement learning with SNNs. They introduce a proxy target network with continuous dynamics that is updated using implicit updates of weights. This stabilizes training. The method is tested on a set of continuous control tasks.
Strengths:
- A weak point in previous SNN methods is demonstrated
- The method is novel, simple, and effective.
- Good experimental validation. The method improves over the SOTA on average

The initial concerns of the reviewers were based on 
- potentially high computational costs during training
- potentially high spike rates
- limited theoretical analysis
- missing ablations and analysis of varying meta parameters
- missing key baselines.
Most concerns have been settled by the authors during the discussion phase, in particular concerning computational costs and spike rates - which seem reasonable - missing ablations and analysis of varying meta parameters, and missing key baselines. One reviewer rated with 2, but did not respond to the final comments of the authors and also not to my specific question to him. So I would weight her/his rating less strongly.

The authors also provided a theoretical results, which I find however close to trivial. I do not think that it adds much to the paper. I also find the performance improvements not overly convincing, as their method improves on average over many tasks (the mean performance), but for several tasks other methods are better and standard deviations are typically large.

Despite these weak points, the manuscript pinpoints an interesting problem and proposes an innovative solution to it. I therefore weakly recommend acceptance.
Decision: Accept (poster)