CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning
Authors: Zijie Xu
Xinyu Shi
Yiting Dong
Zihan Huang
Zhaofei Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Spiking Neural Networks
Batch Normalization
Reinforcement Learning
Primary area: applications to neuroscience & cognitive science
TL;DR: Batch Normalization for Spiking Neural Networks in Reinforcement Learning.

Abstract: Spiking Neural Networks (SNNs) offer low-latency and energy-efficient decision-making on neuromorphic hardware by mimicking the event-driven dynamics of biological neurons. However, the discrete and non-differentiable nature of spikes leads to unstable gradient propagation in directly trained SNNs, making Batch Normalization (BN) an important component for stabilizing training. In online Reinforcement Learning (RL), imprecise BN statistics hinder exploitation, resulting in slower convergence and suboptimal policies. While Artificial Neural Networks (ANNs) can often omit BN, SNNs critically depend on it, limiting the adoption of SNNs for energy-efficient control on resource-constrained devices. To overcome this, we propose Confidence-adaptive and Re-calibration Batch Normalization (CaRe-BN), which introduces (i) a confidence-guided adaptive update strategy for BN statistics and (ii) a re-calibration mechanism to align distributions. By providing more accurate normalization, CaRe-BN stabilizes SNN optimization without disrupting the RL training process. Importantly, CaRe-BN does not alter inference, thus preserving the energy efficiency of SNNs in deployment. Extensive experiments on both discrete and continuous control benchmarks demonstrate that CaRe-BN improves SNN performance by up to $22.6$% across different spiking neuron models and RL algorithms. Remarkably, SNNs equipped with CaRe-BN even surpass their ANN counterparts by $5.9$%. These results highlight a new direction for BN techniques tailored to RL, paving the way for neuromorphic agents that are both efficient and high-performing. Code is available at https://github.com/xuzijie32/CaRe-BN.

Published as a conference paper at ICLR 2026
CARE-BN: PRECISE MOVING STATISTICS FOR STABI- LIZING SPIKING NEURAL NETWORKS IN REINFORCE- MENT LEARNING
Zijie Xu, Xinyu Shi, Yiting Dong, Zihan Huang, Zhaofei Yu'
Peking University Beijing, 100871, China
ABSTRACT
Spiking Neural Networks (SNNs) offer low-latency and energy eeffgy-efficient decision- making neuromorphic hardware by mimicking the event-driven dynamics of biological neurons. However, the discrete and non-differentiable nature of spikes leads to unstable gradient propagation in directly trained SNNs, making Batch Normalization (BN) an important component for stabilizing training. In online Reinforcement Learning (RL), imprecise BN statisties hinder exploitation, re- sulting in slower convergence and suboptimal policies. While Artificial Neural Networks (ANNs) can often omit BN, SNNs critically depend adoption of SNNs for energy-efficient control on To overcome this, we propose Confidence-adaptive and Re-calibration Batch Nor- malization (CaRe-BN), which introduces (i) confidence-guidee adaptive update strategy for BN statistics and (ii) a calibration mechanism to align distributions. By providing more accurate normalization, CaRe-BN BN stabilizes SNN optimiza- tion without disrupting the RL training process. Importantly, CaRe-BN does not alter inference, thus preserving the energy efficiency of SNNs in deployment. Ex- tensive experiments on both discrete and continuous control benchmarks demon- strate that CaRe-BN improves SNN performance by up 22.6% across differennt spiking neuron models and RL algorithms. Remarkably, SNNs equipped with CaRe-BN even surpass their ANN counterparts by 5.9%. These results high- light new direction for BN techniques tailored to RL, paving the way for neuro- morphic agents that are both efficient and high-performing. Code is available https: : /ttptttt.t..////////////...... com xuzi jie32/ CaRe- BN
limiting the devices.
INTRODUCTION
Spiking Neural Networks (SNNs) have emerged as promising class of neural models that more closely mimic the event-driven computation of biological brains (Maass 1997 Gerstner et al. 2014). This event-driven property makes SNNs particularly well suited for deployment on neuromorphic
hardware platforms (Davies et al. 2018 DeBole al.] 2019). enabling low-latency and energy- efficient inference.
In parallel, Reinforcement Learning (RL) has achieved remarkable success across a wide range of domains (Mnih et al. 2015 Lillicrap et al.) 20155: HHarnoja et al.) [0018)). Among these, complex control tasks have received significant attention due to their alignment with real-world scenarios and their strong connection to embodied Al and robotic applications (Kober et al. 2013 Gu al. 2017]: Brunke et al.] 2022). Combining the strengths of SNNs with RL (SNN-RL) offers the potential to train agents that not only learn complex behaviors but also execute them with extremely low energy consumption (Yamazaki et al., 2022). This makes SNN-RL particularly appealing for robotics and autonomous systems deployed resource-constrained edge devices.
However, training SNNs is challenging. Due to the discrete spike dynamics and the reliance on surrogate gradients approximate the backward pass, directly trained SNNS often suffer from un- stable gradient propagation, including vanishing or exploding gradients (Zheng et al. 2021). Batch
"Corresponding author; yuzf1 pepu.eucuuc

---

Published as a conference paper at ICLR 2026
Vanilla BN
CaRe-BN
Real
Estimation
Real
Estimation
Update Itenations
[+]
(8)
(b)
Figure 1: Real and estimated input activation distributions in BN layers. Between each gradient update iterations, distributions change rapidly in (a) and (c), while remaining stable in (b) and (d).
Normalization (BN) (loffe & Szegedy 2015) plays crucial role in stabilizing SNN training by reg- ulating activation statistics and improving gradient flow, mitigates such instability and contributes to klae-offthh--an performance (Duan et al. 2022 Jiang al. 2024).
While effective in supervised learning, BN suffers a severe breakdown in online RL because mov- ing statistics cannot be estimated precisely under nonstationary dynamics. As shown in Figure D traditional BN struggles to track the true statistics: When distributions shift rapidly (Figure Ta)), estimates lag behind; when distributions These inaccuracies lead agents to select suboptimal actions and generate poor trajectories, which are
relatively static (Figure [b)), estimates contain noise.
then reused for wraiiing-furtor compounding the problem and hindering policy improvement.
This issue is especially critical for SNNs. Traditional online RL algorithms usually remove BN lay- in their networks (Sutton & Barto 2018 Fujimoto et al., 2018: Haarnoja al. 2017]] Schulman let al., 2017). Unlike ANNs that can train stably without BN, SNNs rely heavily on normaliza- tion stabilize membrane potentials and surrogate-gradient backpropagation. Removing BN from SNN-based RL leads to severe gradient instability and substantial performance degradation.
In this work, we address this issue by proposing Confidence-adaptiie and Re-calibration Batch Normalization (CaRe-BN), a BN strategy tailored for SNN-based RL. CaRe-BN introduces two complementary components: (i) Confidence-ddaptive update (Ca-BN),
monfidence-weighte moving estimator of BN statistics that ensures unbiasedness and optimal variance reduction; and (ii) Re-calibration (Re-BN), a periodic correction scheme that leverages replay buffer resampling to refine inference statistics. Together, these mechanisms enable precise, low-variance estimation of BN statistics under the nonstationary dynamics SNN-RL (Fi With more accurate moving statistics, CaRe-BN stabilizes SNN optimization without disrupting the online RL process.
We evaluate CaRe-BN on a variety of control tasks, including the Atari benchmark (Bellemare et al. 2013 Machado al. 2018) for discrete action spaces and the MuJoCo suite (Todorov et al. 2012 Todorov 2014b) for continuous control. The results show that CaRe-BN not only resolves the issue of imprecise BN statisties but also accelerates training and achieves state-of-thee performance. Remarkably, SNN-based agents equipped with CaRe-BN even outperform their ANN counter- parts by 5.9%, without requiring complex neuron dynamics specialized RL frameworks.
RELATED WORKS
2.1
BATCH NORMALIZATION IN SPIKING NEURAL NETWORKS
Batch Normalization (BN) was originally proposed for ANNs to mitigate internal covariate shift during training (loffe Szegedy 2015), thereby accelerating convergence and improving perfor-
2

---

Published as a conference paper at ICLR 2026
mance (Santurkar et al. 2018) To address unstable training in SNNs, several extensions of BN have been developed (Zheng et al. 2021 Duan et al. 2022; Kim & Panda 2021 Jiang et al. 2024). While these methods are effective in supervised tasks, they are designed under the assumption of static distributions. This assumption is violated in online RI., where distributions shift continually the agent interacts with the environment, making these BN variants ill-suited for SNN-RL..
2.2 SPIKING NEURAL NETWORKS IN REINFORCEMENT LEARNING
Early work in SNN-RL primarily relied on synaptic plasticity rules, particularly
Seeee-iiinnn----nndeee Plasticity (R-STDP) and its variants (Florian 2007 Frémaux & Gerstner
2016 Gerstner et al.) 2018 Frémaux et al. 2013 Yang et al. 2024). Another research direction
focused on ANN-to-SS SNN conversion: while Patel al. (2019): g Tan et al. (2021); Kumar et al. (202)) converted Deep Q-Networks (DQNs) MMih 2013 Mnih et al. 2015) into SNNs for discrete control, recent work by [xu et al. (2026) has greatly reduced conversion errors continuous control. To enable direct gradient-basee based training, Liu et al. (2022); Chen et al. (2022) : Qin et al. (2022); ) Sun al.) (2022) applied Spatio- Temporal Backpropagation (STBP) (Wu et al. 2018) to train DQNs, while Bellec et al. (2020) introduced e-prop with eligibility traces to train policy networks using policy gradient methods (Sutton et al. 1999).
For continuous control tasks, hybrid frameworks have been extensively explored (Tang et al. 2020 2021 Zhang et al. 2022; Chen et al.] [2024]] Zhang et al.] [024]] Ding et al., [022] Chen et al, 2024b). These approaches typically employ Spiking Actor Network (SAN) co-trained with deep
ANN critic in the Actor-Cin Critic framework (Konda & Tsitsiklis 1999). Empowered by the recent proxy target framework (Xu et al. (2025). simple SNNs are capable of outerforming their ANN counterparts. However, none of these methods address the challenge of normalization SNN-based RL. The absence of proper normalization often leads unstable updates and slower convergence.
3 PRELIMINARIES
3.1 SPIKING NEURAL NETWORKS
Spiking Neural Networks (SNNs) communicate through discrete spikes rather than continuous acti- vations. The most widely used neuron model is the Leaky Integrate-and and-Fire (LIF) neuron, whose membrane potential dynamics are described as:
=1 S:) He + S. Vreset (I) where C₁, H,, Sy, and V/2 denote the input current, the accumulated membrane potential, the binary output spike, and the post-firing membrane potential at time step t, respectively. The parameters Ven. Vreset. and À represent the firing threshold, reset voltage, and leakage factor, respectively. (-)) the Heaviside step function.
11, = AV,-1 + Ct: S, = O(H,- - Vaa,,
3.2 REINFORCEMENT LEARNING
RL is a framework in which an agent learns to maximize cumulative rewards by interacting with an environment. The agent maps states observations) actions, with the learning loop consisting of two steps: (i) the agent selects an action, receives reward, and transitions to the next state; and (ii) the agent updates its policy by sampling mini-batches of past experiences.
Because the policy continuously evolves during training, the data distribution is inherently non- stationary. This poses challenges for batch normalization methods, which rely on the assumption of stationary distribution.
3.3 BATCH NORMALIZATION
Batch Normalization (BN) IIffe & Szegedy 2015) is a widely used technique to stabilize and accelerate the training of deep neural networks. Given an activation E Rd iteration i, BN normalizes it using the mean and variance computed over a mini-batch = {xt {x!....,.?)
PB
oB
Σ
(of pB)
j=1
3

---

Published as a conference paper at ICLR 2026
32 3i PB
Vi 7 + 3
where is a small constant for numerical stability, and 7, B are learnable affine parameters. During inference, moving statistics (f4, o3) are used in place of batch statistics (1s, of).
In supervised learning, this discrepancy between training (mini-batch statistics) and inference (mov- ing statistics) is usually tolerable, as imprecise moving estimates do not directly affect gradient updates. However, in online RI, inaccurate moving statistics degrade policy exploitation, leading to unstable training dynamics and even divergence.
4 METHODOLOGY
(Ht of)
(pa Mu o ..))))))
Var
Var
(A, a
(Arr1 of-1))
Compute Statistics
CA-EMA
CA-EMA
(D,,
(Di,).
EMA
EMA
Ca-BN
Ca-BN
Re-BN
Figure 2: The statistics estimation scheme of CaRe-BN. In this framework, Ca-BN is applied at
every update step, while Re-BN is performed periodically. A denotes the squared error, Var repre- sents the variance computed according to Eq. 9 EMA refers to the exponential moving average in Eq. I1. and CA-EMA denotes the sonfidence-adaptive update defined in Eqs. Sland 6
As illustrated in Figure 2 propose Confidence Confidence-adaptiee and Recalibration Batch Normalization (CaRe-BN) address challenge of approximating moving statistics in online RL. Section 4.1 analyzes the limitations of traditional BN in online RL, where statistics are often estimated impre- cisely. Section 4.2 introduces confidence-adaptive update mechanism (Ca-BN), which dynami- cally adjusts statistics estimation based on the reliability of the current approximation. Section 4.3 presents the recalibration mechanism (Re-BN), which periodically corrects accumulated estima- tion errors. Finally, Section 4.4 integrates these components into the full CaRe-Bi BN framework and demonstrates its use online RI. algorithms.
4.1 ISSUES IN APPROXIMATING MOVING STATISTICS
Online RL introduces stronger distribution shifts. Unlike supervised learning, where the data distribution is typically assumed to be static, online RL involves continuous interaction between the agent and the environment. This results non-stationary data distribution, which turn causes activation statistics to drift over time.
Inaccurate statistics degrade RL performance. Supervised learning only requires the final mov- ing statistics to be accurate, as inference is performed after training. In contrast, online RL requires reliable statistics throughout training. When statistics are imprecise, the agent selects suboptimal actions during exploration and exploitation, generating poor trajectories that further degrades policy updates.
The key of the problem lies in accurately estimating inference-ting time statistics under shifting dis- tributions. Hence, it essential to design estimators that adapt to distributional changes while minimizing approximation error during training.
is worth noting that most conventional ANN-based RL algorithms do not employ BN Lillicrap 2015 Sutton & Barto, 2018), as shallow ANNs often leam stable representations without nor- malization. In contrast, BN is indispensable for stabilizing SNNs training. Therefore, addressing this issue is particularly critical for SNN-hased RI.
4

---

Published as a conference paper at ICLR 2026
4.2 CONFIDENCE-ADAPHO UPDATE OF BN STATISTICS (CA-BN)
Conventional BN approximates population statistics using an exponential moving average (EMA) of the batch mean and variance:
e + - -a) op4i
(1- to?
(4)
where is the momentum parameter. This update rule faces fundamental noise-delay trade-off. As shown in Figure [ low momentum yiclds stable but slow adaptation to distribution shifts, while high momentum adapts quickly but amplifies the noise from small-batch estimates. This trade-off is particularly harmful in online RL, where accurate normalization is critical for stable policy learning.
Inspired by the Kalman estimator (Kalman I960), derive confidence-guidee mechanism that adaptively reweights estimators to minimize the mean-squared error (MSE) of BN statistics.
Theorem Let (Hi, of) and ( be two unbiased estimators of the population parame- ters (u),0?²) 2 Taking them as random variables, the optimal linear estimator is
Pi = (L==(kk P + Kr
KP D(1 (1/1-1) + D11 us)
(5)
Doo 0, 1//-11)
07 = 21=11- (1-11; KL-KK K2)0 + K.
K = D(o ( + D0o o??)
(6)
o?)
where K.f and Kę are confidence confidene--guieed adaptive weights, and (-) denotes generalized variancd
Proof Since both Pajt-1 and μr are unbiased for pt: any linear combination ju; = =1 (1-
Kus is also unbiased. The variance is
(ū, pT] (1 K)² (Pppj=11 24) + K² D(m p.).
Minimizing over K yields the optimal K = KW. The variance update (Eq. 6 follows analogously.
Assumption The activations in iteration are modeled x; NN2, ²) following the stan- dard Gaussianity assumption in BN.
Confidence of mini-batch statistics. For batch of size N, the sample mean μ4 and variance of satisfy
Pi N (1i :)) (N 1)o?
o.² XN-1-
(8)
Since μ4 and of? 2 are unknown, we adopt the common upproximation using μ, and o/. thus:
D(4 - p4) of2
20 20 N N-1
Dia;²" -
N
Confidence of previous estimates. Since the true statistics n and of are unknown, direct com- putation of [D[ Âds-1) and ID - - is infeasible. To approximate them, view the minibatch statistics μ4 and of as a stochastic sample drawn from the unknown hypothetical distri- butions induced by nę and o.²² Thus, the squared deviations (μu: (dj;-1)²² and (a 62 1)
serve as unbiased but noisy probes of D(p (mj=-) and D(o o 1))
Because these single-miniban estimates exhibit high variance, we maintain smoothed recursive estimators updated using an exponential moving average with momentum parameter a::
D(K Papt-1) 2 D.,
D( -² 1).-)))== DV
(10)
+ -1- a + alu; 1)
+ alo -
0 D))
(11)
Combining Eqs. 5 11 obtain the confidence-adappe update scheme When distributional shifts are rapid, D. and D grow large, increasing K" and Ky and accelerating adaptation. Con- versely, when statistics stable, these terms shrink, lowering KJt and KỤ and reducing noise from small mini-batches.
'The confidence is defined as the inverse of the generalized variance: confidence score = 1 As BN statistics fluctuate without monotonic trends, we define fuaj--1 14-1 and 0 0;-1-
5

---

Published as a conference paper at ICLR 2026
4.3 RE-CALIBRATION MECHANISM OF BN STATISTICS (RE-BN)
While the confidence-adaptive update provides online estimates of BN statistics during training, these estimates may still drift from the true population values due stochastic mini-batch noise. The most accurate approach would be to recompute exact statistics by forward-propagating the entire dataset after each update (Wu & Johnson 2021). However, this is computationally infeasible in RL, as it would require processing millions of samples at every step.
A more practical alternative is to periodically re-calibrate BN statistics using larger aggregated batches. Specifically, at fixed intervals Teal, we draw M calibration batches (Bi,..., Bar] from the replay buffer. For each batch Bj, we compute its mean My and variance 03. The recalibrated BN statistics are then given by:
M Σ 1=1
M
A
P3
0,
1 M
Σ j=1
+
+
-
(12)
M This recalibration requires additional forward passes, but the extra overhead is upper bounded by Tai times the total training cost. Since we set Teat >> M, the computational overhead remains negligible, while significantly improving the accuracy of BN statistics.
4.4
INTEGRATING WITH RL
The proposed Confidence- and Re-calibration Batch Normalization (CaRe-BN) BN) integrates two complementary mechanisms: the confidence-adaptive update Section 4.2 which provides an online estimation of batch normalization (BN) statistics, and the re-calibration procedure in Sec-
tion 4.3 which corrects accumulated bias. The overall integration within an online RL framework outlined in Algorithm
Algorithm General RL Algorithm with CaRe-BN
1: Initialize the agent networks and the replay buffer.
2: for each iteration do
3: Select an action and store the transition (inference BN statistics).
4: Update the agent by sampling a minibatch of N transitions (mini-batch BN statisties).
5: Update the moving BN statistics as:
D + (1 (1- + a(4 -ii1))²
D. + (1 - (1-a)D1, + a(o 0)-
DH p4 +
a
N fii-
6,- D of N-1
20t
0-1
A
D
20 D +
N-T
6: if Re-calibration then
7: Sample M minibatches of N transitions each and update BN statistics using Eq. 12
8: end if
9: end for
It important note that the inference procedure of CaRe-BN remains identical to that of con- ventional BN. At inference time, the CaRe-BN layer seamlessly fused into synaptic weights, introducing no additional inference overhead during deployment.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
We evaluate CaRe-BN on RL tasks covering both discrete and continuous action spaces. All envi- ronments default settings, and performance is evaluated by averaging the rewards in 10 trials.
For discrete action spaces, we consider four widely used Atari 2600 games from the Arcade Learning Environment (ALE) (Bellemare et al. 2013 Machado al. 2018): Pong, Breakout, SpaceInvaders,

---

Published as a conference paper at ICLR 2026
Freeway, and Seaquest. We adopt deep Q-learning framework (Miih et al. 2015) and train deep Spiking Q-Network (Liu et al. 2022) that receives RAM-based observations and outputs state-action values.
For continuous control, we evaluate on five standard MuJoCo benchmarks Todorov et al. 2012 Todoroy 2014b) provided in the OpenAl Gymnasium suite (Brockman 2016 Towers et al. 2024]: InverredDoubleeeenduua (IDP) Todorovy] 2014a). Ant (Schulman et 2015). HalfChee- tah (Wawrzyński 2009), Hopper (Erez al. 2012), and Walker2d. We employ hybrid frame- work in which spiking actor network is co-trained with a deep critic network using several RL algorithms, including Deep Deterministic Policy Gradient (DDPG) (Lillicrap)
2015). Twin Delayed DDPG (TD3) (Fujimoto et al. 2018), and Soft Actor- Critic (SAC) (Haarnoja et al. 2018a).
To evaluate the generality of CaRe-BN, we experiment with multiple spiking neuron models: the Leaky Integrate- taaite--nd-Fiir Fire (LIF) neuron (Gerstner & Kistler 2002). the Current- based LIF (CLIF) neuron (Tang al. 2021). and the Dynamic Neuron (DN) model (Zhang et al. 2022), with detailed dynamics provided in the Appendix. All SNN agents are trained via
Backpropa-
gation (STBP) (Wu et al. 2018)) with the CaRe-BN module inserted between every pair adjacent layers. For fair comparison, all models share the same hyperparameters, fully listed in the Appendix.
During each RI. environment step, the SNN agent performs a single forward inference composed of 5 simulation time steps, after which all neuron states are reset.
5.2 MORE PRECISE BN STATISTICS LEAD TO BETTER EXPLORATION
In online RL, the quality of explo- ration directly affects subsequent pol- icy updates. As discussed in Sec- 1.. traditional BN methods strug- gle maintain accurate moving statis- ties, which can lead to suboptimal ex- ploration behavior.
Wasserstiin Distance
Layer 1
Layer
Layer 3
0.24
0.16
118-
0:14
Calle-NN
BN
Training Steps (1o4)
Tranning Stops
Training Steps (lel)
To quantify this effect, we compute the Wasserstein distance between the true feature distribution and the Gaus- sian distribution estimated by BN. Fig- ure 3 shows that CaRe-BN consis- tently reduces this discrepancy across all layers throughout training, produc- ing more precise normalization.
Figure 3: Wasserstein distance between estimated BN statistics and the true distribution across layers, mea- sured with CLIF neurons and the TD3 algorithm in the environment. Shaded areas denote half a standard deviation over five runs. Curves are uniformly smoothed for visual clarity.
IDP-v4
Ant-v4
HalfCheetuh-v4
Hoppar-v4
Walker2d-v4
Explonaion Recuns
4.
CaRe-BN
0.2
0.4
0.0
0.4 0.6 Training Steps (1e6)
0.5 Training Steps (166)
0.6
0.4 08 Tmining Stops (1o6)
Training (te4s
Training Staps (1e6)
Figure 4: Exploration returns of BN and CaRe-BN with CLIF neurons and the TD3 algorithm across five MuJoCo tasks. Shaded areas represent half standard deviation across five random seeds. Curves are uniformly smoothed for visual clarity.
The impact of improved statistics is reflected in exploration performance. As shown in Figure 4 CaRe-BN consistently achieves higher exploration returns. Since CaRe-BN does not directly modify the gradient update process, the observed improvement in exploration performance is solely due to its more precise estimation of BN statistics. This leads to better exploration policies, which in turn generate higher-quality qualiity trajectories for updating the agent. As result, CaRe-BN BN forms positive feedback loop: improved statistics - better exploration - higher-quality experiences - better policy.
7

---

Published as a conference paper at ICLR 2026
5.3 ADAPTABILITY OF CARE-BA
To evaluate the adaptability of CaRe-Bi BN. we test it across different RL algorithms (DQN (Mmih
2015), DDPG (Liliccapp 2015). TD3 Fujjmoto et al., 2018). and SAC (Harmoja et al.
201 8ab) and spiking neuron models (LIF, CLIF ("Tan et al. 2021). and DN (Zhagg et al. 2022
IDP-v4
Ant-v4
HaliCheetah-v4
Hopper-v4
Walker2d-v4
Returns (163)
Average
CaRe-NN-LIF
2.5
10
Returs (1e3)
Average
Call--N+CLIF
CLIP
Average Returns (lc3
CoR:-aN-DN
0.0 0.2
04 0.6 08
1.0
00 0.2
04
06 0.5 1.0
00 02
0.4
06
0.5 1.0
0.0 02 0,4 06 08 1.0
Training Steps (1o4)
Training Steps (1o6)
Training (1e6)
Training Steps (1e6)
Training Stops (1e6)
Training with TD3 Algorithm
IDP-v4
HalCChottsh-v4
Hopper-v4
Walker2d-v4
Returns (eb)
Average
CaRt-BNNLLIP
LIF
Retanns (c3)
Averape
CaRe e
Retura (1e))
Average
DN
02
0.6
1.0
Training Steps (1e4)
Training Steps (1e6)
Training Steps (1e6)
Training Steps (1e6)
(b) Training with DDPG Algorithm
Figure 5: Learning curves of SNN-based agents in continuous control trained with TD3 (top) and DDPG (bottom). Since the DDPG algorithm (in both ANN and SNN) diverges in the Ant-v4 envi- ronment, these curves are not shown. Shaded areas represent half standard deviation across five random seeds. Curves are uniformly smoothed for visual clarity.
Better final return. Figure the learning curves for SNN models with and without CaRe-BN. In most cases, CaRe-BN consistently outperforms standard SNNs, converging faster and achieving higher final returns. These improvements are robust across different spiking neurons and RL algo- rithms, confirming that CaRe-BN enhances performance in diverse settings.
Lower variance. Figure (a) and (b) display the relative variance of the final policy. Compared to standard SNNs, CaRe-Hi BN significantly reduces the variance SNN-RL training, and even achieves lower variance than ANN baselines (i.e., 17.71% for DDPG and 21.24% for TD3). This indicates that CaRe-BN not only enhances performance but also improves the stability and reproducibility.
Generalizing across different RL domains. Beyond continuous control, we also evaluate CaRe- BN in discrete-action settings using the deep spiking Q-network. As shown in Figure SNN agents equipped with CaRe-BN achieve markedly improved performance across Atari tasks. These results demonstrate the strong generalization capability CaRe-BN across diverse RL domains.
Curres with SAC shown in Figure in the Appendix.
8

---

Published as a conference paper at ICLR 2026
CaRe-BN
Vanilla
23,9
24.6
Performance (%)
Variazee (%0
Past CANMITAEEE
95.3
6.3
LIF
CLIP
LIF
CLa
BN
Ca
Rs BN CaRs BN
Trining Step (d) CaRe-BN ANNS
(e) Variance in DDPG
(b) Variance in TD3
(c) Ablation Study
Figure 6: (a), (b) Relative variance percentage of final policy returns, computed by averaging the standard deviation ratio across five random seeds, for environments. (c) Normalized maximum performance across all environments for the ablation study, using CLIF neurons and TD3 algorithm. (d) Normalized learning curves across all environments for ANNs implementing CaRe-BN, The dashed lines represent DDPG and the solid lines represent TD3. Performance and training steps normalized linearly. Curves are uniformly smoothed for visual clarity.
Pong-v5
Breakout-v3
Freeway-v5
Seaquest-v5
Spacelnvaders-vv
Average Returs
20
1000
300
-15
500
200
Vanilla SNN
100
0.0
1.5 Training Stops (le6)
15 Training Steps (1o6)
9.5
1.5 Training Steps (1o6)
03
0.5
1.0
2.0
0.5
1.0
20
Training Steps (1e6)
Training Steps (1c6)
Figure 7: Learning curves of SNN-based agents in discrete control. Shaded areas represent half standard deviation across three random seeds. Curves uniformly smoothed for visual clarity.
5.4 EXCEEDING SOTA
To further validate the effectiveness of CaRe-BN, we compare it with existing state-of-the-art (SOTA) SNN-RL methods and various batch normalization strategies for SNNs. The evaluation conducted using the TD3 algorithm (Fujimoto et al. 2018) (a strong SOTA bascline for continu- control) and the CLIF neuron model (Tang al. (021) (the most commonly used neuron type in recent SNN-RL studies). The ANN-SNN conversion baseline follows the SOTA method proposed in Bu al. (2025). For direct-trained SNNs, include pop-SAN (Tang al., (02T), MDC-SAN Zhang et al.) 3022). and ILC-SAN (Chen et al. 2024a). Additionally, we test several BN algorithms for SNNs, including tdBN (Zheng et al.) 2021). BNTT (Kim & Panda 202T. TEBN (Duan al., 2022], and TABN (Jiang et al. 2024). The performance is summarized in Table I where the average performance gain (APG) is defined as;
performance(eny) baseline(env)
-1) 100%,
APG
(13)
lenvs
enV Censs
where jenvs| denotes the total number of environments, and performance(env) and baseline(env) represent the performance of the evaluated algorithm and the ANN baseline in each environment, respectively.
Compared with other SNN-RL methods: CaRe-BN significantly outperforms previous SNN-RL approaches, demonstrating that normalization plays more crucial role than architectural modifica- tions in improving SNN-RI. performance.
Compared with other BN methods: Compared existing SNN-specific BN variants, CaRe-BN performs superior, establishing new state-of-the-art normalization strategy for SNN-RL.
Compared with ANNs: Notably, CaRe-BN trained with TD3 outperforms its ANN counterparts by 5.9% on average This highlights that with proper normalization, SNNs can not only match but
As shown in Figure 9 in the Appendix, SNNs equipped with CaRe-BN also outperform their ANN coun- terparts when trained with SAC (Haarnoja all 2018a)

---

Published as a conference paper at ICLR 2026
Table 1: Max average returns over 5 random seeds with CLIF spiking neurons, and the average per-
formance gain (APG) against ANN bascline, where + denotes one standard deviation. All modules trained using the TD3 algorithm. All directly trained SNN modules have 5 simulation time steps.
Method
IDP-v4
Ant-v4
HallCheetah-v4
Hopper-v4
Walker2d-v4
APG
ANN
7503 + 3713
4770 1014
10857 + 475
3110 164
4340 +: 383
0.00%
ANN-SNN
3859 1440
3550 963
8703 + 658
3098 281
4235 +: 354
21.111
Pop-SAN
9351 1
4590 1006
9594 689
2772 1263
3307 1514
-6.66%
MDC-SAN
9350
4800 994
9147
3446 131
3964 1353
0.37%
ILC-SAN
9352
5584 272
9222 + 615
3403 148
4200 + 717
4.64%
WBN
9346 2
4403 1134
9402 527
3592 46
3464 +: 970
2.28%
BNTT
9347
4379 941
9466 659
3524 = 161
3689 1247
-1.22%
TEBN
9349
4408 1156
9452 + 539
3172 = 135
1235 + 381
0.69%
TABN
9348
4382 + 753
9784 169
3585 83
4537 + 398
3.25%
CaRe-BN
9348 +
5373 + 159
9563 + 442
3586 + 49
4296 + 268
5.90%
exceed the performance of traditional ANN-based RL agents, while retaining their energy-efficient advantages.
5.5 ABLATION STUDIES
We conduct ablation studies by separately evaluating the effects of the Confidence-adaptive update (Ca-BN) and the Re-calibration mechanism (Re-BN), shown in Figure (c). The results demon- strate that both the adaptive estimation and recalibration mechanisms are beneficial on their own. However, their combination provides the most significant improvement. Specifically, Ca-BN ad- dresses the mismatch between training and inference statistics, while Re-BN corrects accumulated errors, further stabilizing training. By integrating both components, CaRe-BN achieves more precise and consistent normalization, leading to superior overall performance.
5.6 SNI-FRIENDLY DESIGN
Dispite the stunning improvement in SNNs. we also evaluate CaRe-BN on standard ANNs trained with TD3 and DDPG, as shown in Figure 6 (d). The results indicate that ANNs with CaRe-BN perform similarly to their baseline counterparts without CaRe-BN. This outcome is expected for the following reasons: (i) Shallow ANNs can already train stably and effectively without normalizatior f so adding CaRe-BN does not provide significant improvements. (ii) While CaRe-BN provides more precise estimates of BN statistics, this does not negatively impact the RL training process. These results further underscore that the improvements observed are not due to stronger RL mechanism, but rather to the SNN-specific normalization strategies.
CONCLUSION
In this work, introduced CaRe-BN, the first batch normalization method specifically designed for SNNs in RL. By addressing the instability of conventional BN in online RI, CaRe-BN enables SNNs outperform their ANN counterparts in continuous control tasks. Importantly, CaRe-BN is lightweight and easy to integrate, making it a seamless drop-in replacement for existing SNN-RL. pipelines without introducing additional computational overhead.
Beyond its technical contributions, CaRe-BN brings SNN-RL one step closer to practical deploy- ment. By stabilizing training and improving exploration, it unlocks the potential of SNNs act as both energy-efficient and gegh-perffrmancee agents in real-world continuous control applications. We believe this work underscores the importance of normalization strategies tailored the unique dynamics of SNNs and opens new avenues for bridging the gap between neuromorphic learning and reinforcement learning at scale.
'In RL, networks typically consist of two hidden layers with 256 neurons.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (U24B20140, 62422601), Beijing Municipal Science and Technology Program (Z241 (20100000224404 Beijing Nova Program (20230484362, 20240484703), National Key Laboratory for Multimedia Informa- tion Processing, and Beijing Key Laboratory of Brain-inspired Spiking Large Models.
REFERENCES
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neu- rons. Nature Communications, 11(1):3625, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
G Brockman. Openai gym. arXiv preprint arXiv:1606.01540 2016
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):41 1-444, 2022.
Tong Bu, Maohua Li, and Zhaofei Yu. Inference-scale complexity in ann-snn conversion for high- performance and low-power applications. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24387-24397, 2025.
Ding Chen, Peixi Peng, Tiejun Huang, and Yonghong Tian. Deep reinforcement learning with spiking g-learning. arXiv preprint (eXXi:22010097:55 2022.
Ding Chen, Peixi Peng, Tiejun Huang. and Yonghong Tian. Fully spiking actor network with in- tralayer connections for reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 36(2))6881-2888 --2933, 2024a.
Ding Chen, Peixi Peng. Tiejun Huang, and Yonghong Tian. Noisy spiking actor network for explo- ration. arXiv preprint arXiv:2403.04112 2024b.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 2018
Michael V DeBole, Brian Taba, Arnon Amir, Filipp Akopyan, Alexander Andreopoulos, William Risk, Ieff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, et al. TrueNorth: Accelerating from zero to 64 million neurons 10 years. Computer, 2019.
Jianchuan Ding, Bo Dong, Felix Heide, Yufei Ding, Yunduo Zhou, Baocai Yin, and Xin Yang. Biologically inspired dynamic thresholds for spiking neural networks. Advances in Neural Infor- mation Processing Systems, 35:6090-6100 2022.
Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. Advances in Neural Information Processing Systems, 35:34377-34390, 2022.
Tom Erez, Yuval Tassa, and Emanuel Todorov. iafinite-horizon model predictive control for periodic tasks with contacts. Robotics: Science and Systems VII, 2012.
Räzvan V Florian. Reinforcement learning through modulation of spike-timing-deepedden synaptic plasticity. Neural Computation, (966):146-11500 2007.
Nicolas Frémaux and Wulfram Gerstner. Neuromodulated teakee-tmii--depeenecde plasticity, and theory of three-factor learning rules. Frontiers in Neural Circuits, 9:85, 2016.

---

Published as a conference paper at ICLR 2026
Nicolas Frémaux, Henning Sprekeler, and Wulfram Gerstner. Reinforcement learning using tinuous time actor-critic framework with spiking neurons. PLoS Computational Biology, 9(4): e1003024, 2013.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge University Press, 2002.
Wulfram Gerstner, Wemer M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Corneil, and Johanni Brea. Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules. Frontiers in Neural Circuits, 12:53, 2018.
Shixiang Gu, Ethan Holly. Timothy Lillicrap. and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 3389-3396 IEEE, 2017.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep anergy-based policies. In International Conference Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018b.
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 34(8):5200-5205, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456. PMLR, 2015.
Haiyan Jiang, Vincent Zoonekynd, Giulia De Masi, Bin Gu, and Huan Xiong. Tab: Temporal aceu- mulated batch normalization in spiking neural networks. In The Twelfth International Conference on Learning Representations, 2024.
Rudolph Emil Kalman. A approach linear filtering and prediction problems. 1960.
Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in Neuroscience, 15:773954, 2021.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal Robotics Research,
2013.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Process- ing Systems, 12, 1999.
Aakash Kumar, Lei Zhang, Hazrat Bilal, Shifeng Wang. Ali Muhammad Shaikh, Lu Bo, Avinash Rohra, and Alisha Khalid. Dsqn: Robust path planning of mobile robot based deep spiking q-network. Neurocomputing, 634:129999 129916, 2025.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1r09 2015.
TP Lillicrap. Continuous control with deep reinforcement learning.
arXiv preprint
arXiv: 150i:0500.00971, 2015.
12

---

Published as a conference paper at ICLR 2026
Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, and Huajin Tang. Human-level control through directly trained deep spiking q q-networks. IEEE Transactions on Cybernetics, 53(11):1187-7198, 2022.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 10(9):155911771 1997.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Paul Merolla, John V Arthur, Rodrigo Alvarez-lcaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with scalable communication network and interface. Science, 345 (6197):668-61) 673, 2014.
Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529-533 2015.
Devdhar Patel, Hananel Hazan, Daniel J Saunders, Hava T Siegelmann, and Robert Kozma. Im- proved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to atari breakout game. Neural Networks, 120:108-11 -15, 2019.
Ning Qiao, Hesham Mostafa, Federico Corradi, Mare Osswald, Fabio Stefanini, Dora Sumislawska, and Giacomo Indiveri. A reconfigurable on-line learning spiking neuromorphic processor com- prising 256 neurons and 128K synapses. Frontiers in Neuroscience, 9:141, 2015.
Lang Qin, Rui Yan, and Huajin Tang. A low latency adaptive coding spiking framework for deep reinforcement learning. arXiv preprint arXiv:2211. 2022.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in Neural Information Processing Systems, 31, 2018.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint (rXiv:1506.02438,, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 0os/1707.06347,20
Yinqian Sun, Yi Zeng, and Yang Li. Solving the spike feature information vanishing problem in spiking deep q network with potential based normalization. Frontiers in Neuroscience, 16:953368, 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
Weihao Tan, Devdhar Patel, and Robert Kozma. Strategy and benchmark for converting deep q- networks to event-driven spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.
2021.
Guangzhi Tang, Neelesh Kumar, and Konstantinos Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy -fficieetf mapless navigation with neuromorphic hardware. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6090- 6097. IEEE, 2020.
13

---

Published as a conference paper at ICLR 2026
Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos Michmizos. Deep reinforcement learning with population-coded spiking neural network for continuous control. Conference on Robot Learning. pp. 2016-2029, PMLR, 2021.
Emanuel Todorov. Convex and analytically-inve -invertiiee dynamics with contacts and constraints: The- ory and implementation in mujoco. In 2014 IEEE International Conference on Robolics and Automation (ICRA). pp. 6054-6061 IEEE, 2014a.
Emanuel Todorov. Convex and analytically- invertible dynamies with contacts and constraints: The- ory and implementation in mujoco. In 2014 IEEE International Conference Robotics and Automation (ICRA), pp. 6054-6061. IEEE, 2014b.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 5026- 5033. IEEE, 2012.
Mark Towers, Ariel Kwiatkowski, Jordan K Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, KG Arjun, et al. Gymnasium: standard interface for reinforcement learning environments. CoRR, 2024.
Pawel Wawrzyński. A cat-like robot real-time learning to run. In Adaptive and Natural Computing Algorithms: 9th International Conference, ICANNGA 2009, Kuopio, Finland, April 23-25, 2009, Revised Selected Papers 9, pp. 380-390. Springer, 2009.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training nigh-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
Yuxin Wu and Justin Johnson. Rethinking" batch" in batchnorm. arXiv preprint arXi:2100..07576, 2021.
Zijie Xu, Tong Bu, Zecheng Hao, Jianhao Ding, and Zhaofei Yu. Proxy target: Bridging the gap between discrete spiking neural networks and continuous control. In The Thirty-ninth Anmual Conference on Neural Information Processing Systems, 2025.
Zijie Xu, Zihan Huang, Yiting Dong, Kang Chen, Wenxuan Liu, and Zhaofei Yu. Error amplification limits ann-to-snn conversion in continuous control. arXiv preprint arXiv:2601.21778, 2026.
Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le. Spiking neural networks and their applications: A review. Brain Sciences, 12(7):863, 2022.
Zhile Yang, Shangqi Guo, Ying Fang, Zhaofei Yu, and Jian K Liu. Spiking variational policy gradient for brain inspired reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Duzhen Zhang, Tielin Zhang, Shuncheng Jia, and Bo Xu. Multi-sacle dynamic coding improved spiking actor network for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 59-67, 2022.
Duzhen Zhang, Qingyu Wang, Tielin Zhang, and Bo Xu. Biologically-plausibbe topology improved spiking actor network for efficient deep reinforcement learning. arXiv preprint 2024.
Hanle Zheng, Yujie Wu, Lei Deng. Yifan Hu, and Guoqi Li. Going deeper with directly-traines larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11062-11070, 2021.
14

---

Published as a conference paper at ICLR 2026
A ETHICS STATEMENT
Our submission follows the ICLR Code of Ethics. We do not identify any specific ethical concerns in this work.
B REPRODUCIBILITY STATEMENT
Source code provided in the supplementary materials. We also provide our full implementation and experimental configurations in the Appendix. All experiments were conducted on a single NVIDIA RTX 4090 GPU, but the code also be executed on CPU-only devices, albeit with longer training times. These materials ensure that the reported results can be reproduced and verified by the community.
C USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used solely for polishing the presentation of this paper, such correcting typos, improving grammar. All ideas, derivations, algorithm design, and experiments were conceived and implemented independently without reliance on LLMs.
D APPENDIX
D. SNN ARCHITECTURES
D.1.1 DEEP SPIKING -ENTWORKK ARCHITECTURE
The deep spiking Q-network consists of an SNN that receives the 128-dimensional RAM input using direct coding. The network contains two hidden layers, each with 256 LIF neurons. The Q-values are obtained by reading out the membrane potentials of the output layer, which uses non-leaky, non-firing neurons provide stable value estimates.
D.1.2 SPIKING ACTOR NETWORK ARCHITECTURE
The spiking actor network (SAN) consists of population encoder with Gaussian receptive fields, multi-layer SNN with a population output, and a decoder with non-firing neurons.
Forward Propagation of the SAN. In the state encoder, each input dimension represented by Ni soft-reset IF neurons with Gaussian receptive fields. These fields have trainable parameters and a. The neurons receive stimulation Ag at every time step and output spikes S according to:
1 (s 22
Ag = exp
(14)
0²
Vinn = Vi
Sin + AE Stn O(Vin VE),
-Sin
(15)
where VE is the threshold for the encoding populations.
The final layer of the SNN consists of Nau neurons, corresponding to each action dimension. The decoder layer consists of non-spiking integrate-and-frr neurons connected the last layer of the SNN:
Vouut
Veal + Wrout S.. + bout
(16)
where Wrout and gout are the weights and biases, respectively. The final output action is determined by the membrane potential at the last time step, a Vartt A detailed description of the forward propagation in the spiking actor network is provided in Algorithm
15

---

Published as a conference paper at ICLR 2026
Algorithm 2 Forward propagation of the Spiking Actor Network (SAN)
Input: -ddieennsio observation 2: Compute input population stimulation:
Ag exp
1 (}-1)²
3: fort I....,T do
4: Compute encoder membrane potential and spikes:
Vith Vi=1 S + AE,
S O(Vi" Vv)
5: forl=1,..,, do
6: Update neurons layer at timestep
7: end for
8:
Update decoder membrane potential:
Voui
V + Wout St. + fout
9:
end for Output: Ma-dimensional action = Virut
10:
Backpropagation of the SAN. The SAN parameters are optimized using gradients with respect ØL
to the output action a = Vp given da For the decoder:
aL
0L
av.ut
ow
TWouT Vou
(17)
8L
du
Obour.
The main SNN is trained using spatio-temporal backpropagation (STBP) (Wu et al. 2018), with the
rectangular surrogate gradient function defined as:
(xx)) = 2w S x S w,
(18)
0, otherwise,
where w denotes the window size.
Next, we derive the gradient the encoder stimulation Ag, as shown in Eq. 19 For simplicity,
a5 is manually 1, which is a common surrogate assumption simplify gradient the term BÄg
computation:
0L
0L
as dAg
aL as
Σ
(19)
0Ag
t=1
Finally, the trainable parameters μ and of the encoder can be updated as:
0Ag
8L.
5-u
AF,
0p
BAE
BAg
(20)
OL
aL
BAg
OL
(8-1)² r
Ag.
0Ag
0o
aAg
D.2 SPIKING NEURON MODELS
introduced the LIF neuron model. Here, we provide the detailed dynamics of the spiking Section 3.1
neuron models used in our experiments.
D.2.1 LIF NEURON MODEL
The dynamics of the LIF neuron are defined in Eq. 1 where the input current is computed as:
C = W + 8¹.
(21)
where W and denote the synaptic weights and biases, respectively.
16

---

Published as a conference paper at ICLR 2026
D.2.2 BARRENT-BASED LIF (CLIF) NEURON MODEL
the current-based LIF (CLIF) neuron proposed in Tang et al. 2021), the input current in Eq. modified as:
= =c W + +b',
(22)
where Ay is the current leakage parameter. All other dynamics of CLIF neurons are identical those of standard LIF neurons.
D.2.3 DYNNAMIC NEURON MODEL
The second-ored order Dynamic Neuron (DN) model proposed in (Zhng et al.) 2022) is designed to capture richer temporal dynamics for continuous control. Each DN maintains membrane potential V and resistance variable U to model hyperpolarization effects. The neuron dynamics are governed by:
av
dt = (V,))² -V,- -7-U- Ut 1.
(23)
du dt V, Q,
(24)
where 0, and 0n denote the conductance parameters of V and U, respectively. When the neuron fires, the membrane potential V is reset to Veeset, and the resistance variable U is incremented by 0,. Using first-order Taylor expansion, the iterative update of the DN model can be written as:
C
Ct_ + +W'st- + b4;
V = [1 - S VL_
+ +
Vreset
U. U!_1 + S 0n:
Vaeta = Vi2² - -V-
W U C/
(25)
Udeta = 0, V - 0n Ul.
V Vi + Vactaa
U U + Usela :
S. θ (V: Van)
D.3
EXPERIMENT DETAILS
D.3.1
COMPUTE RESOURCES
All experiments were conducted on an RTX 4090 GPU (except for the training time study in Ap- pendix D.5.I)
D.3.2 SPIKING NEURON PARAMETERS
The parameters for the LIF and CLIF neurons are listed in Table These are the same as those used in Tang et (2021), except that the LIF neuron does not include current leakage parameter.
Table 2: Parameters of LIF and CLIF
(Tag et al.
2021) neurons
Parameter
LIF
CLIF
Tang et al.
2021
Membrane leakage parameter À
0.75
0.75
Threshold voltage Vaa
0.5
0.5
Reset voltage Vreset
0
0
Current leakage parameter
0.5
The parameters the DN model are listed in Table 3 All values are obtained using the pre-learning procedure described in Zhhnn al. (2022
17

---

Published as a conference paper at ICLR 2026
Table 3: Parameters of the DN (Zhang et al.
2022)
Parameter
Value
SNN time steps
Threshold voltage Via
0.5
Current leakage parameter
0.5
Conductivity of membrane potential 0,
00.12
Conductivity of hidden state 0,,
0.529
Reset voltage Vreset
0.021
spike effect to hidden state 0,
0.132
D.3.3 SPECIFIC PARAMETERS FOR CARE-BN
Table lists the hyperparameters of CaRe-BN. The recalibration frequency Tre is set equal to the evaluation frequency used in the RL algorithms. All hyperparameters are kept consistent across different spiking neuron models and RL algorithms.
Table 4: papeee-rraeeterss of the CaRe-BN
Parameter
Value
Momentum
0.8
Recalibration frequency The
5000
Recalibration batchs M
100
D.3.4 SPIKING ACTOR NETWORK PARAMETERS
All thyper-paramett of the spiking actor network are listed in Table 5 These settings are consistent
with those used in a wide range of previous studies (Tang et al. 2021 Zhang et al. 2022 Chen al. 2024a).
Table 5: Hyper- parrae-paraeeters of the spiking actor network
Parameter
Value
Encoder population per dimension Nin
10
Encoder threshold VE
0.999
Network hidden units
(256, 256)
Decoder population per dimension Nout
10
Surrogate gradient window size
0.5
D.3.5 RL ALGORITHM PARAMETERS
The experiments are conducted using DQN Mnnih et al. 2015)), DDPG (Lillicrap) 2015). TD3 (Fujimoto al, 2018), and the SAC (Haarnoja al. 2018a) algorithms, with their respective hy- perparameters listed in Tables
and
D.3.6 EXPERIMENT ENVIRONMENTS IN CONTINUOUS CONTROL
Figure 8 illustrates various MuJoCo environments Todorov al.] 2012 Todorov 2014b) from the OpenAl Gymnasium benchmarks (Brockman) 2016 Towers et al. 2024). including Inverted- DoublePendulum (IDP) (Todorov] 2014a), Ant (Schulman al. 2015). HalfCheetah (Wawrzyíski, 2009). Hopper (Erez et al. 2012) and Walker. All environments used the default configurations without modification.
Note that the state vectors, which can range from -00 to oc, are normalized to (-1,1) using a tanh function. Similarly, since the actions have minimum and maximum limits, the outputs of the
18

---

Published as a conference paper at ICLR 2026
Table 6: Hyper-parameters of the implemented DQN algorithm (Mnih et al. 2015)
Parameter
Value
Learning rate
1 10-4
Network architecture
(256, 256)
Optimizer
Adam
Target update interval
2000
Batch size N
128
Discount factor
0.99
Iterations per time step
1.0
Reward scaling
1.0
Gradient clipping
None
Replay buffer size
106
Max epsilon
1
Min epsilon
0.1
Epsilon decay steps
20000
Table 7: Hyper-parameters of the implemented DDPG algorithm (Lillicrap) 2015
Parameter
Value
Actor learning rate
1 . 10-4
Actor regularization
None
Critic learning rate
1 10-3
Critic regularization
weight decay =0.01
Critic architecture
(400, 300)
Critic activation Optimizer
Relu
Adam
Target update rate
1 10-3
Batch size N Discount factor
256
0.99
Iterations per time step
Reward scaling
1.0
Gradient clipping
None
Replay buffer size
106
Exploration niose (0,
NO0, 0.1)
(0) IDP-v4
(b) Ant-v4
(c) HalfCheetah-v4
Hopper-v4
(e) Walker2d-v4
Figure 8: Several continuous control tasks of the MuJoCo environments on OpenAI Gymnasium. (a) InverteDDoubdeennduuum v4, (b) Ant-v4, (c) HalfCheetah-v4, (d) Hopper-v4, (c) Walker2d-v4,
19

---

Published as a conference paper at ICLR 2026
Table 8: Hyper-parameters of the implemented TD3 algorithm Fujimoto et al.
2018
Parameter
Value
Actor learning rate
3 10
Actor regularization
None
Critic learning rate
3 10
Critic regularization
None
Critic architecture
(256, 256)
Critic activation Optimizer
Relu Adam
Target update rate
5 10-3
Batch size N
256
Discount factor
0.99
Iterations per time step
1.0
Reward scaling
1.0
Gradient clipping
None
Replay buffer size Exploration niose N( o)
106
NO0, 0.1)
Actor update interval d
2
Target policy noise N (0,
NO, 0.2)
Target policy noise clip
0.5
Table Hyper-parameters of the implemented SAC algorithm HHarnojj et al. Sa)
Parameter
Value
Actor learning rate
10
Actor regularization
None
Critic learning rate
3. 10-4
Critic regularization
None
Critic architecture
(256, 256)
Critic activation
Relu
Optimizer
Adam
Target update rate
10-3
Batch size N
256
Discount factor ?
0.99
Iterations per time step
1.0
Reward scaling
1.0
Gradient clipping
None
Replay buffer size
106
Actor update interval
Entropy target
dim( A)
Alpha learning rate
3. 10
20

---

Published as a conference paper at ICLR 2026
actor network are first normalized to (-1, 1) via tanh function and then linearly scaled to the corresponding (Min action, Max action) range.
D.4 ADDITIONAL EXPERIMENTAL RESULTS
D.4.1 ADDITIONAL RESULTS WITH SAC
In the main text, we demonstrated that CaRe-BN surpass its ANN counterparts using the TD3 al- gorithm. We further train the SNN agent using SAC, a stronger modern off-policy RL algorithm. As shown in Figure SNNs equipped with CaRe-BM BN also have the potential to surpass their ANN counterparts under SAC.
Ant-v4
Hopper-v4
3000
Avazage Retums
20000
3000
1000
Vanills SNN
500
0.0
0.2
0.5
08
100
0.0
0.2
11.8
0.5
1.0
Training Steps (1o6)
Training Steps (1c6)
Figure 9: Learning curves of the SNN-based agents using SAC algorithm. Shaded areas represent half standard deviation across five random seeds. Curves uniformly smoothed for visual clarity.
D.4.2 ADDITIONAL RESULTS ADAPTABILITY
In the main text, we demonstrated that CaRe-BN improves performance across various spiking neu- models and RL algorithms. Additionally, Tables 12
13 14 and report the maximum
average returns and the average performance gains of CaRe- BN compared to vanilla SNNs across different spiking neurons and RL algorithms.
Table 10: Max average returns over 5 random seeds in DDPG with LIF neurons.
Method
IDP
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
9352 + 1
7954 + 356
3035 + 127
2931 + 1395
0.00%
CaRe-BN
9351
8199 + 305
3512 79
3347 321
8.24%
Table 11: Max average returns over 5 random seeds in DDPG with CLIF neurons.
Method
IDP
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
9352
8205 + 376
2566 + 1270
2224 1607
0.00%
CaRe-BN
9352 =
7972 + 245
3247 = 100
3709 = 321
22.62%
Table 12: Max average returns over 5 random seeds in DDPG with DNs.
Method
IDP
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
9351 + 3
8069 + 897
3134 + 134
3238 + 633
0.00%
CaRe-BN
9351 + 2
7731 + 457
3418
159
3438 + 399
2.76%
D.4.3 ADDITIONAL COMPARISON WITH ANNS
Fig TO shows the normalized learning curves of CaRe-BN within different spiking neurons.
21

---

Published as a conference paper at ICLR 2026
Table 13: Max average returns over random seeds in TD3 with LIF neurons.
Method
IDP
Ant
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
9347 = 1
4243 = 949
9073 + 946
3507 = 85
2807 + 1834
0.00%
CaRe-BN
9346 =
5083 = 356
8813 533
3489 = 118
4556 + 497
15.74%
Table 14: Max average returns over random seeds in TD3 with CLIF neurons.
Method
IDP
Ant
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
9351 = 1
4590 + 1006
9594 + 689
2772 + 1263
3307 + 1514
0.00%
CaRe-BN
9348 = 5373 159
9563 + 442
3586 49
4296 + 268
15.20%
SNN with Calte-RN
SNN without Calte-NN
ANN
Norrmalzzed Performance
0.8
0.6
0.2
0.0
0.2
9.4
0.5
0.8
1.0
3.
0.2
0.4
0.5
0.8
1.0
0.0
0.2
04
0,5
0.5
1.0
Normalized Training Steps
Nommalized Truining Steps (b) CLIF Neuron
Normalized Training Steps
(a) LIF Neuron
(c) Dynamic Nenron
Figure 10: Normalized learning curves across all environments of the TD3 algorithm with different spiking neurons across all environments. The performance and training steps normalized linearly based on ANN performance. Curves are uniformly smoothed for visual clarity.
D.4.4 ADDITIONAL COMPARISON WITH OTHER SNN-BN MECHANISMS
Tab. 16, shows the performance of different BN variants and CaRe-BN with the LIF neuron model in TD3 algorithm.
Table 16: Max average returns over 5 random seeds with LIF neuron, and the average performance gain (APG) against ANN baseline, where + denotes one standard deviation.
Method
IDP-v4
Ant-v4 HalfCheetah-v4
Hopper-v4
Walker2d-v4
APG
ANN (TD3)
7503
3713
4770 +: 1014
10837 475
3410
4340 383
0.00%
Vanilla LIF
9347 +
4243 + 949
9073 + 946
3507 + 85
2807 1834
-7.08%
tdBN
9346 +
4876 + 577
8845 = 526
3601 - 29
4098 + 408
1.65%
BNTT
9348 +
5244 321
9339 + 874
3593
3480 1450
1.22%
TEBN
9347
1408 1156
9452 + 539
3472
4235 381
0.69%
TABN
9347
4431 + 1353
9173 + 595
3474 = 183
3818 1133
-1.64%
CaRe-BN
9346
5083 + 356
8813 + 533
3489 118
4556 + 497
3.92%
Table 15: Max average returns over random seeds in TD3 with DNs.
Method
IDP
Ant
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
9350 +
4800 + 994
9147 + 231
3446 + 131
3964 4: 1353
0.00%
CaRe-BN
9349 = 2
5444 + 161
9581 638
3470 = 115
4084 + 362
4.37%
22

---

Published as a conference paper at ICLR 2026
D.4.5 ADDITIONAL RESULTS IN ANN
We shows the normalized learning curves of the CaRe-BN with ANN in Fig (d). Here, show the detailed learning curves and maximum average returns of environments in Fig Fig Tab and Tab. 19 respectively.
D.4.6 ADDITIONAL RESULTS WITH DIFFERENT SNN SIMULATION TIME STEPS.
We future study the impact of SNN simulation time steps. As shown Table T7 SNNs generally benefit from larger simulation time steps, and CaRe-BN achieves stronger results when using SNN simulation steps (up to 6.32% improvement over ANNs). However, we report the main results using SNN simulation time step of 5, following the standard configuration adopted in prior SNN-based RI. studies (Tang et al. 2021 Zhang et al. 2022: Chen et al. 2024a).
Table 17: Max average returns over random seeds of CaRe-BN with CLIF spiking neurons trained using the TD3 algorithm, and the average performance gain (APG) against ANN baseline, where + denotes standard deviation.
SNN time steps
IDP-v4
Ant-v4
HalfCheetah-vé
Hopper-v4
Walker2d-v4
APG
953
4924 + 171
7635 392
3588 10
3885 1365
-2.80%
3
9285 = 100
5078 + 325
8190 567
3522 = 89
4391 + 282
2.03%
5
9348 +
5373 + 159
9563 + 442
3586 + 49
4296 + 268
5.90%
8
9354 +
5417 1: 421
9989 + 278
3479 = 95
4311 + 348
6.32%
IDP-y4
HalfCheetab-v4
Hopper-v4
Walker2d-v4
Average Rotunn (1e5)
CaRt: BN-ANN
00
02
0.4
0.6
038
1.0
000
0.2 04 06
0.8
1.0
0.0
0.2
04
0.6
0,8 1.0
Trainin Stops (lof)
Training Steps (le6)
Training Steps (le6)
Training Stops (1c6)
Figure 11: Learning curves of utilizing CaRe-BN in ANN with DDPG algorithm. The shaded region represents half a standard deviation over 5 different seeds. Curves are uniformly smoothed for visual clarity.
IDP-v4
Ant-v4
HalfCheetah-v4
Hopper-v4
Walker2d-v44
Retums (1e3)
5
6
2
ANN
0.0
0.0 Training
0.2
03
Tmining Steps
Trinning Steps (lef)
Training Steps (1e6)
Training Steps (1e6)
Figure 12: Learning curves of utilizing CaRe-BN ANN with TD3 algorithm. The shaded region represents half standard deviation over 5 different seeds. Curves are uniformly smoothed for visual clarity.
23

---

Published as a conference paper at ICLR 2026
Table 18: Max average returns over random seeds in DDPG with ANN.
Method
IDP
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
9357 + 4
8604 = 241
3486 = 162
3385 + 408
0.00%
CaRe-BN
9360 + 0
8887 + 408
3475 = 135
3328 + 882
0.33%
Table 19: Max average returns over 5 random seeds in TD3 with ANN.
Method
IDP
Ant
HalfCheetah
Hopper
Walker2d
APG
Vanilla SNN
7503 = 3713
4770 + 1014
10857 + 475
3410 + 164
4340 383
0.00%
CaRe-BN
9360 + 0
5014 + 1122
10458 = 1271
3436 - 114
3021 = 1360
-0.69%
D.5
ENERGY CONSUMPTIONS
D.5.1 TRAINING COSTS
To assess the computational overhead introduced by CaRe-BN, we measure the training time and GPU memory usage on an RTX 3090 GPU paired with an Intel(R) Xeon(R) Platinum 8358P CPU. The results are summarized Table 20 As shown, CaRe-BN does not introduce significant addi- tional training time memory consumption compared with other BN variants.
Table 20: Training costs of different BN mechanisms the Ant-v4 environment, trained with TD3 algorithm and CLIF neurons. Training time corresponds to the total wall-clock time required for 5000 RL steps, including exploration, replay sampling. target computation, and gradient updates.
Training costs
tdBN
BNTT
TEBN
TAB
CaRe-BN
Training time for 5000 updates (s)
242
266
251
264
247
GPU memory (MiB)
437
437
441
441
437
D.5.2
INFERRING COSTS
Table 21: Energy consumption per inference (in nJ) for the spiking actor network with CLIF neurons, trained using TD3 across various tasks.
Method
IDP-v4
Ant-v4
HalfCheetah-va
Hopper-v4
Walker2d-v4
Average
ANN
1715.20
1862.40
1785.60
1728.00 16.59
1785.60 18.13
1775.36
SNN with CaRe-BN
12.94
17.36
17.37
16.48
We evaluate the energy consumption of SNNs equipped with CaRe-BN during inference. Energy is estimated following the methodology of Merolla et al. (2014), where each floating-poin operation (FLOP) assumed consume 12.5 pJ and each synaptic operation (SOP) consumes fJ (Qiao et al. 2015 Hu et al. 202T). As shown in Table 21 the ANN baselines require substantially more energy per inference. contrast, the SNN models with CaRe-BN demonstrate dramatically reduced energy consumption across all evaluated tasks. These results highlight the strong energy efficiency of SNNs and underscore their potential for deployment on resource-constrained platforms.
24


Reviews:
[{"review_id": "uoJucUYcdI", "reviewer": "Reviewer_gu7R", "summary": "The paper targets a practical instability in online RL with SNNs: BatchNorm (BN) statistics are hard to estimate under non-stationary data and replay, leading to train–inference mismatch and noisy updates. It proposes CaRe-BN, a two-part BN strategy: (i) Confidence-adaptive BN (Ca-BN), which linearly fuses the previous moving estimate and the current mini-batch estimate using variance-based gains (a Kalman-style, minimum-MSE update under stated assumptions); and (ii) Re-calibration (Re-BN), which periodically re-estimates inference statistics from larger replay samples to correct accumulated drift. CaRe-BN is drop-in and leaves the inference path unchanged. Integrated with TD3/DDPG and LIF/CLIF neurons, experiments on MuJoCo report consistent gains and, on average, SNN+CaRe-BN outperforming a vanilla ANN-TD3 baseline. The paper also argues that “more precise BN statistics” improve exploration quality.", "questions": "1. Theory & Theorem 1 assumptions (target inconsistency and variance modeling).\nCould you formally define the target distribution for BN statistics in online RL and quantify the mismatch between moving BN (inference) and batch BN (replay) (e.g., TV/Wasserstein distance over activations)? Please provide bias/variance bounds for your fused estimator under distribution shift, report effective sample size N_\"eff\" with autocorrelation estimates for replayed batches, and clarify how you separate process noise vs. observation noise when computing the confidence gains (consider an adaptive Kalman/EM treatment instead of a single residual EMA).\n2.“Better exploration” claim: metrics, confounders, and causal evidence.\nWhat is the exact definition of exploration return? To establish causality, could you (i) align random seeds and action-noise schedules, (ii) report standard exploration metrics—state coverage/visitation counts (or pseudo-counts), occupancy entropy, trajectory diversity, feature-space coverage, and (iii) add Actor-only vs. Critic-only CaRe-BN ablations and noise-aligned runs? Please also include tests on exploration-hard benchmarks (e.g., Atari) and comparisons to intrinsic-motivation baselines (ICM, RND).\n3.External validity beyond MuJoCo and advantage over strong ANN-RL.\nBeyond MuJoCo, can you evaluate on vision-based and non-stationary settings (context/reward switches, parameter drift) and compare against stronger ANN baselines (e.g., SAC; TD3/SAC with LN/GN/RMSNorm or Batch Renorm/AdaBN; distributional critics; entropy/parameter-noise regularization)? If the paper claims energy advantages, please report deployment-phase energy/latency (or credible proxies/ simulation power consumption estimation) and disentangle training vs. inference costs; otherwise, consider narrowing the energy-efficiency claims.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear problem framing with a simple, principled fix. The paper isolates BN statistic mismatch as a key bottleneck in SNN-RL and proposes a plug-and-play estimator with an intuitive confidence-weighted fusion + periodic re-calibration backed by a tractable derivation.\n\n2. Consistent empirical gains with informative ablations. Across several MuJoCo tasks and spiking neuron types, CaRe-BN improves stability/returns, and ablations suggest Ca-BN and Re-BN are complementary, supporting the design rationale.\n\n3. Practical applicability and minimal engineering overhead. The method does not alter the inference graph, is easy to integrate into standard actor–critic pipelines, and is plausibly extensible beyond SNNs to other online/temporal settings where BN statistics drift.", "weaknesses": "1. Theory–data distribution mismatch in Theorem 1 (target inconsistency between moving BN and batch BN)\nThe theorem assumes that the “moving/prior” BN estimator and the “mini-batch/observation” BN estimator are unbiased estimates of the same target distribution with correctly specified variances. In online RL with replay, this assumption is typically violated: the moving BN statistics track the current inference distribution, while the batch BN statistics are computed from a replay mixture of (often older) policies, with temporal correlation and potentially heavy-tailed activations. Consequently, the two estimators do not share the same target, unbiasedness/i.i.d. fail, and the linear fusion with gain 𝐾 is not guaranteed to be valid or optimal (and may introduce bias). \n\n2. Limited experimental scope beyond MuJoCo \nAlthough the paper compares against common SNN normalization baselines (e.g., BNTT), the evaluation remains restricted to MuJoCo continuous control. To substantiate generality and the claimed benefits under stronger non-stationarity, the method should also be tested on exploration-hard and vision-based benchmarks (e.g., Atari).\n\n3. Exploration claim lacks causal evidence and proper metrics\nSection 5.2 infers “more precise BN statistics → better exploration” primarily from an “exploration return” curve without defining the metric, controlling confounders (action-noise schedule, value-estimation bias, policy smoothness), or reporting standard exploration measures (state-coverage/visitation, occupancy entropy, trajectory diversity, feature-space coverage). No experiments are provided on exploration-hard tasks or against classic intrinsic-motivation baselines (ICM, RND, NGU/Plan2Explore, count-based variants). The authors need to further analyze the reason why the proposed method can improve the efficiency of exploration. \n\n4. Unestablished advantage over strong ANN-RL and unmeasured energy claims\nThe only ANN comparison uses a vanilla ANN-TD3 (in the appendix). Modern, stronger ANN-RL baselines (e.g., SAC; TD3/SAC with LN/GN/RMSNorm or Batch Renorm/AdaBN; distributional critics; entropy/parameter-noise regularization) are not evaluated, so superiority over ANN remains unproven. Moreover, the touted energy advantage of SNNs is theoretical and specific to neuromorphic hardware; no deployment-phase energy/latency measurements (or credible proxies) are reported, nor is training vs. inference cost disentangled. Open discussion: beyond theoretical energy efficiency on neuromorphic hardware, what concrete advantages does SNN + CaRe-BN have over strong ANN-RL methods (e.g., sample efficiency under non-stationarity, stability under distribution shifts, low-latency control, small-batch robustness), and can the authors provide controlled evidence for them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1iHKVrmaqI", "reviewer": "Reviewer_Yg5y", "summary": "The paper proposes a method to improve the estimation of Batch Normalization (BN) statistics. The authors apply this improved BN to spiking neural networks (SNNs) and report that it outperforms other BN variants on six reinforcement learning (RL) tasks.", "questions": "1. Figure 1: What does the “update step” represent—training parameter updates or an SNN inference time step?\n\n2. I could not find details of the network architectures used. Please describe the models (layers, widths, neuron types, readouts, etc.).\n\n3. How are SNN time steps integrated with the RL policy? Do you aggregate/compress time steps to produce the actor/critic outputs? Which components are SNNs and which components are not?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is relatively easy to follow.\n\n2. Multiple tasks are evaluated.", "weaknesses": "1. The claim that “due to the discrete and non-differentiable nature of spikes, directly trained SNNs rely heavily on BN to stabilize gradient updates” is misleading. BN is not designed to solve non-differentiability in SNNs; please revise this.\n\n2. It is unclear whether the contribution is primarily for ANNs or SNNs. Figure 1 suggests general applicability to ANNs, yet experiments are focusing on SNNs—please clarify the intended scope and update the paper.\n\n3. Section 5.2 / Figure 3: The evidence in Figure 3 appears insufficient to conclude that the method yields more precise BN statistics, and contributes to better reward. We can only observe that CaRe-BN gets obviously higher expected rewards on three tasks out of five. It is unknown what is the root cause of such observation. \n\n4. How many time steps are used in the SNNs? How sensitive is performance to this hyperparameter, and what justifies the chosen value to be used in the baseline setting? Also, ablation study on the time step should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJQxJeL7Ot", "reviewer": "Reviewer_EiPv", "summary": "This paper highlights the misalignment between the true and computed activation distributions when batch normalization is applied in spiking neural networks (SNNs). In particular, the work discusses issues with existing batch normalization methods in SNN models that can negatively impact performance in reinforcement learning (RL) tasks.\n\nInspired by the Kalman filter, the authors propose a confidence-guided approach that more accurately estimates batch statistics. They also discuss a recalibration method to align the computed batch statistics with those of the training data.\n\nThe proposed method is evaluated on several RL tasks and compared against multiple baselines from both ANN- and SNN-based RL frameworks. Furthermore, an ablation study is conducted to demonstrate the performance improvements contributed by each individual component of the method.", "questions": "Line 204: The authors mention that “conventional ANN-based RL algorithms do not employ BN.” This raises an important question — is batch normalization (BN) truly necessary in SNN-based RL frameworks? How does the proposed SNN model perform without BN? A comparison or ablation along these lines would strengthen the justification for including BN in SNNs.\n\n\nLine 233: The authors provide a proof for computing the optimal K value. It appears that \\mathbb{D} is computed recursively. Could the authors please elaborate on this derivation and clarify the recursive relationship involved?\n\n\nThe definition and interpretation of the measure \\mathbb{D} are unclear. Could the authors provide more explanation regarding what \\mathbb{D}  quantifies and how it relates to the overall estimation or confidence process?\n\n\nThe rationale behind approximating \\mathbb{D} as shown in Equation (9) is not clearly justified. Please clarify why this specific approximation is valid and under what assumptions it holds.\n\n\nThe terminology “confidence-based adaptive scheme” is somewhat confusing. Why is the deviation from the actual value interpreted as confidence? A clearer explanation of how “confidence” is defined and why it is used in this context would be helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Most of the paper is easy to follow. \n\nThe problem highlighted is an interesting issue in BN for reinforcement learning. \n\nI believe the discussion of related work is adequate and also provides the necessary background to understand the problem.", "weaknesses": "I believe the novel part of the method is the confidence adaptive updation of batch statistics; however, the second part, recalibration, is a modification of the existing method. \n\nA few things are not clear, which are mostly related to the proposed method, which can be found in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zqxFjcy14g", "reviewer": "Reviewer_eBqg", "summary": "This work presents Confidence-adaptive and Re-calibration Batch Normalization (CaRe-BN) to stabilize Spiking Neural Networks (SNNs) in Reinforcement Learning (RL). Addressing imprecise BN statistics in online RL that hinder SNN performance, CaRe-BN integrates a confidence-guided adaptive update for BN statistics and a periodic recalibration mechanism, ensuring accurate normalization without disrupting RL training or compromising SNN energy efficiency at inference. Evaluated on MuJoCo continuous control tasks, it boosts SNN performance by up to 22.6% across neuron models/RL algorithms, even outperforming ANNs by 5.9%, advancing BN techniques for SNN-RL.", "questions": "see Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First, it addresses SNN-RL's BN statistic imprecision via dual mechanisms, stabilizing training without disrupting RL processes or losing SNN energy efficiency .\n\nSecond, it shows strong adaptability, boosting SNN performance by up to 22.6% across diverse neuron models and RL algorithms .\n\nThird, it outperforms ANNs by 5.9%, proving SNNs’ potential for efficient, high-performance control .", "weaknesses": "First, it only evaluates on MuJoCo continuous control tasks, lacking tests on more complex or real-world RL scenarios .\n\nSecond, it fails to report comparisons of training time and memory with existing BN methods for SNN-RL .\n\nThird, it doesn’t explore CaRe-BN’s performance with more advanced spiking neuron models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)