The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning
Authors: Zihao Wu
Hongyao Tang
Yi Ma
Jiashun Liu
YAN ZHENG
Jianye HAO
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
non-stationary data distributions
plasticity loss
Primary area: reinforcement learning

Abstract: Deep reinforcement learning (RL) suffers from plasticity loss severely due to the nature of non-stationarity, which impairs the ability to adapt to new data and learn continually. Unfortunately, our understanding of how plasticity loss arises, dissipates, and can be dissolved remains limited to empirical findings, leaving the theoretical end underexplored. To address this gap, we study the plasticity loss problem from the theoretical perspective of network optimization. By formally characterizing the two culprit factors in online RL process: the non-stationarity of data distributions and the non-stationarity of targets induced by bootstrapping, our theory attributes the loss of plasticity to two mechanisms: the rank collapse of the Neural Tangent Kernel (NTK) Gram matrix and the Θ(1/k) decay of gradient magnitude. The first mechanism echoes prior empirical findings from the theoretical perspective and sheds light on the effects of existing methods, e.g., network reset, neuron recycle, and noise injection. Against this backdrop, we focus primarily on the second mechanism and aim to alleviate plasticity loss by addressing the gradient attenuation issue, which is orthogonal to existing methods. We propose Sample Weight Decay (SWD) --- a lightweight method to restore gradient magnitude, as a general remedy to plasticity loss for deep RL methods based on experience replay. In experiments, we evaluate the efficacy of SWD upon TD3, SAC with SimBa architecture in MuJoCo and DeepMind Control Suite tasks. The results demonstrate that SWD effectively alleviates plasticity loss and consistently improves learning performance across various configurations of deep RL algorithms, UTD, network architectures, and environments, achieving SOTA performance on challenging DMC Humanoid tasks.

Published as a conference paper at ICLR 2026
THE RANK AND GRADIENT LOST IN NON- STATIONARITY: SAMPLE WEIGHT DECAY FOR MIT- IGATING PLASTICITY LOSS IN REINFORCEMENT LEARNING
Zihao Wu Hongyao Tang'; Yi Jiashun Liu', Yan Zheng"; Jianye Hao
School of Computer Software, Tianjin University
2 IInternational Joint Institute of Tianjin University, Fuzhou The Hong Kong University of Science and Technology
ABSTRACT
Deep reinforcement learning (RL) suffers from plasticity loss severely due to the nature of nonnstationaaity, which impairs the ability to adapt to new data and learn continually. Unfortunately, our understanding of how plasticity loss arises, dis- sipates, and can be dissolved remains limited to empirical findings, leaving the theoretical end underexplored. To address this gap, we study the plasticity loss problem from the theoretical perspective of network optimization. By formally characterizing the two culprit factors in online RL process: the non-stationarity data distributions and the non-stationarity of targets induced by bootstrapping, theory attributes the loss of plasticity to two mechanisms: the rank collapse of the Neural Tangent Kernel (NTK) Gram matrix and the O decay of gradient mag- nitude. The first mechanism echoes prior empirical findings from the theoretical perspective and sheds light on the effects of existing methods, e.g., network reset, neuron recycle, and noise injection. Against this backdrop, we focus primarily on the second mechanism and aim to alleviate plasticity loss by addressing the gradi- ent attenuation issue, which is orthogonal to existing methods. We propose Sam- ple Weight Decay (SWD)- a lightweight method to restore gradient magnitude, general remedy plasticity loss for deep RL methods based experience replay. In experiments, we evaluate the efficacy of SWD upon TD3, Double DQN and SAC with SimBa architecture in MuJoCo, ALE and DeepMind Control Suite tasks. The results demonstrate that SWD effectively alleviates plasticity loss and consistently improves learning performance across various configurations of deep RI. algorithms, UTD, network architectures, and environments, achieving SOTA performance challenging DMC Humanoid tasks, We make code available
INTRODUCTION
Deep reinforcement learning (RL) has achieved remarkable success across a variety of domains, including robotics (Akkaya et al., 2019), game playing (Berner et al., 2019) and LLM post-training that endows language models with the ability to generate human-like replies for breaking the Turing test (Biever, 2023). The core driver behind these advancements of deep RL lies in the combina- tion of RL and deep neural networks. With the powerful expressive capacity and adaptive learning ability, the neural networks can effectively approximate and optimize value functions and policies under the RL training regime. However, recent studies have identified a critical yet often overlooked challenge Plasticity Loss: as training progresses, the learning ability of neural networks gradu- ally diminishes (Elsayed & Mahmood, 2024; Nikishin et al., 2022). To address this phenomenon, researchers in the RL community have proposed different metrics and remedies mainly from em- pirical perspectives, such as Network Reset (Nikishin al., 2022), Neuron Recycling (Sokar et al., 2023), Noise Injection (Nikishin et al., 2023a). However, these existing works all rely on empir- ical intuitions and lack clear theoretical grounding, leaving a significant gap between empiricism
"Corresponding authors: tanghooggaao@rju.cdu..omm yanzheng@ijji wij.hennggu..cuuccnn...
'https://gi thub. wwt.://haaaaaeaaa........

---

Published as a conference paper at ICLR 2026
and theory. Despite the significance of this issue, explaining plasticity from the theoretical perspec- tive and developing principled algorithms remain highly challenging due to the complexity of the underlying mechanisms of plasticity loss in the context of deep RL.
To analyze the optimization dynamics of Reinforcement Learning (RL) agents, we develop a structured theoretical frame- work rooted in a core insight: due the
Modian
ICM
Moan
Optimailly Gap
BAC+SIND
940
660
7:0
600
640
860
720
660
625
660
190
185
dynamic nature of the optimization pro- cess in RL, the loss function evolves with each optimization iteration effeetivvel initiating new optimization "task" in
(a) SWD for SimBa-SAC in DMC tasks
Median
IQM
Meen
Optimality Gep
TD3+SWD
each round. Critically, the initial opti- mization point for the updated loss fune- tion in the current round is exactly the ter- minal point from optimizing the previous round's 's loss function. This sequential ini- tialization mechanism raises fundamental questions about its potential adverse im- pacts on optimization performance, and this line of inquiry underpins the entire logic of our theoretical analysis. Based on this insight, we arrive at key conclusion: RL agents inherently confront two critical challenges that exert profound adverse effects on loss function optimization. The first is the potential rank deficiency of the Neural Tangent Kernel (NTK) (Jacot et al., 2018) core factor that governs the network's fitting capacity, specifically its ability to approximate the optimal value function in RL.The second a gradient magnitude decay. which directly regulates the neural network's fitting rate and dictates the time required to escape saddle points.
3500 4650 4566
3900 4300 4400
4038 1200 1926
1-00
1900
2105 2410
(b) SWD for TD3 in MuJoCo tasks
Modian
IOM
Mcan
Optimaityy Gap
Double DON-SWO
Doubte
4500 3500
4000 4580
4000 2000 3500 2720 2008
(c) SWD for Double DQN ALE tasks
Figure 1: Aggregate Reliable metrics (Agarwal et al., 2021) with 95% Stratified Bootstrap CIS.
Our theoretical results reveal two causal mechanisms for the occurrence of plasticity loss. The first mechanism echoes prior empirical findings from the theoretical perspective and sheds light on the effects of existing methods. Differently, we focus primarily on the second mechanism, which has not been well explored, and aim alleviate plasticity loss by addressing the gradient attenuation issue from an orthogonal angle to existing methods. In this paper, we design an anti-decay sampling strategy compensation measure. We observe that gradient decay is governed by the linearly de- caying term R. where & represents the number of learning iteration. In response to this, we construct set of linearly weighted coefficients, where the sampling probability decreases linearly with the age of the samples. Specifically, we propose Sample Weight Decay (SWD) - a lightweight method tailored to mitigate plasticity loss in deep Reinforcement Learning (RL) algorithms. SWD effectively maintains the gradient magnitude at a appropriate scale, ensuring stable learning dynamics.
Building on the SimBa-SAC (Lee et al., 2025a; Haarnoja et al., 2018). TD3 (Fujimoto et al., 2018) and Double DQN (Hasselt et al., 2016) 6) algorithms as base algorithm, SWD significantly enhances learning stability and performance in continuous control tasks and pixel-based tasks. To validate its effectiveness, we evaluated SWD across three well-established online reinforcement learning (RL. benchmarks: the MuJoCo (Brockman, 2016) Arcade Learning Environment (Bellemare et al., 2013) and the DeepMind Control (DMC) Suite (Tassa et al., 2018). 8). For evaluation protocol, we adopted the Interquartile Mean (IQM) as the core performance metric, while leveraging GraMa (Liu et al., 2025) as the key indicator quantify plasticity. As illustrated in Figure 1, SWD consistently delivers state-of-the-art (SOTA) performance.
The contributions of this paper are summarized follows:
We have developed a unified theory to account for plasticity in deep reinforcement learn- ing (RL), thereby shedding clear light on the origins of such plasticity, bridging the gap between empirical practice and theoretical research.
We propose SWD, thcoretically grounded plug-and-play method to different RL algo- rithms for mitigating plasticity loss and improving learning performance.
The experiments demonstrate the efficacy of SWD in improving learning stability and per- formance. Additionally, SWD achieves state-of-the-ans (SOTA) performance in challenging DMC Humanoid tasks.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Plasticity loss refers to the phenomenon in neural network training where the model gradually loses its ability to adapt to new data, objectives, or tasks during the learning process (Dohare et al., 2024). This usually reflects that the network becomes overly specialized to the early stages of training, resulting in reduced learning capacity, slower convergence, or even a collapse in later stages of training (Nikishin et al., 2022: 2023a). To gain better understanding of plasticity loss and address effectively, many efforts have been made to conduct various empirical investigations and propose different solutions (Ash & Adams, 2020; Lewandowski et al., 2023; Kumar et al., 2023; Ceron et al,, 2023; Asadi et al., 2023; Ellis et al., 2024; Chung et al., 2024; Tang & Berseth, 2024; Frati et al., 2024; Ceron al., 2024).
Sokar et al. (2023) first identified the dormant neuron phenomenon in deep reinforcement learning (RL) networks, where neurons progressively fall into an inactive state and their expressive capacity diminishes over the course of training. To address this issue, they proposed Recycle Dormant neu- rons (ReDo)- - strategy that continuously detects and recycles dormant neurons throughout the training process. In a separate line of work, Nikishin et al. (2023a) proposed Plasticity Injection, technique that boosts network plasticity without altering trainable parame- ters introducing biases into predictive outputs. More recently, Liu et al. (2025) introduced Reset guided by Gradient Magnitude (ReGraMa), which addresses neuronal activity loss in deep RL agents by transitioning from activation statistics to mradient-based neuron reset strategies, maintaining net- work plasticity through GraMa metrics. While these approaches have empirically validated their effectiveness in combating plasticity loss, they predominantly operate at the model level modify- ing network architectures without addressing the fundamental theoretical questions: why plasticity loss occurs and how different underlying mechanisms contribute this phenomenon. This presents a significant gap between empiricism and theory.
a
This theoretical gap motivates our work, which targets the fundamental gradient decay mechanism identified through our theoretical analysis. Our proposed Sample Weight Decay (SWD) approach operates at the strategic level - focused on weighting experience replay - and provides prin- cipled means of compensating for the ((1/k) gradient attenuation, challenge unaddressed by recent techniques. A key distinguishing feature of SWD its orthogonality to existing methods: whereas prior approaches modify network structures or plasticity injection patterns, SWD acts at the data distribution level via intelligent experience reweighting, ensuring compatibility with existing plasticity-preseriing techniques and enabling synergistic performance improvements.
PRELIMINARIES
We consider an episodic Markov Decision Process (MDP) (s, A. H, {r with hori- zon H E Z (Puterman, 2014). Here, S, A measurable state, action spaces; Px 8, s,a) is the transition kernel at step h ; TA : S x A + [0, 1] is the reward at step h. At each episode, an initial state æ is drawn. At step
[H]. the agent observes TA S. chooses an E A. receives ry (Irk, ax),
and transits to Th+1 ~ (- Kh. ax). A policy is π = { J=1 with T (- (-)
H
For policy π, the value and action-value functions are defined as:
H
(x) = E
Σ
(1)
Zh x, a T((- xz1
Vr E S, h E [H],
(x rz a) Ex' V (e'))
V(r E A h E [H)
with terminal condition V 0. It convenient to write the transition expectation operator IPr and policy expectation operator J
(z, = B. (V(')) (I)Q)(x) R R A (1w)) 1Q(, a)).
Then the policy Bellman equations compactly read,
Qī a) - rn(2, a) + (PA VI (x,) a),
(x) = (J,)))(s,, V#+ 0.
3

---

Published as a conference paper at ICLR 2026
For any function g g:8x. A R, define the value maximization operator V and the step-h optimality Bellman operator Th by
(g(x) max g(, a)
(Thg) == (x) + (PnVa)(x,a).
THEORY ANALYSIS: THE RANK LOSS AND GRADIENT ATTENUATION
In this section, our primary objective is to establish a rigorous connection between the optimization process and plasticity loss. To this end, we first utilize Equation 3 to derive formal bound on the model's performance. We then simplify the dynamic optimization process by reducing it to an initialization problem-a key step that streamlines subsequent analyses. Finally, elaborate the derivation of our core results, with the full details presented in Section 4. and Section 4.2.
For the sake of clarity and analytical tractability, we focus our discussion on the simplest variant of Fitted Q-Iteration (FQI) (Emsst et al., 2005). Importantly, the theoretical framework proposed herein is not limited to this specific algorithm; it can be readily extended to accommodate wider class of value-based reinforcement learning methods. Of note, analogous analytical findings hold for entropy-regulariae Markov Decision Processes (MDPs). A comprehensive treatment of this extension, including detailed proof's and supplementary analyses, is provided in Appendix B.4.
Let Dj denote the replay buffer at step h following & episodes, and let f represent the estimated Q-value at step after episodes. The loss function is then defined as follows:
1
Σ
[s(on) (r(sh, un) max fhtt (5t1)))),
D
(s), Sh 1) DD
fht arg min JEF
fH41=0
Define the empirical distribution 2 of the replay buffer over (s, a) and the empirical state-action visitation frequency of the behavior policy nhi at time h in episode k:
( a) = iDA
Σ (E),
I1s = Si a = aas.
d +1 (8, = 11s =====0 =3 5/ a k+1)
(sh+t,a a.-,, (.t1)) ~ P (s,
To establish mathematical formulation for distribution shift and thereby quantify its impact on the loss function, we rely on Proposition to characterize such distributional non-stationarity. Further- more, to facilitate the subsequent gradient decomposition, we express the loss function in the form specified in Theorem 1. Finally, to connect the agent's 's performance the loss function, we leverage Theorem to provide a bound on the agent's 's final performance.
Proposition 1 (Empirical distribution recursion). The empirical distribution satisfies
k
K+1 H
k
Proof (sketch). By construction, |D ++ and D ( k+1 *t1)). Expand- ing the definition of 1r 1++1 and regrouping terms yields the stated convex combination.
Theorem 1 (Population loss limit). Let F be a measurable function class. As the cardinality (or appropriate size measure) of Dj, tends to infinity (i.e., [D]] + sc), the following probabilistic convergence holds:
Ch.f. Pkr1) Fg Eennaaweee (f(sh.
(Tn - (Tn aw))²) +
where Cj., is a constant independent of f. Henceforth, we do not rigorously distinguish between the empirical risk and the expected loss, focusing instead the underlying optimization problem.
4

---

Published as a conference paper at ICLR 2026
Takeaway The non-stationarity in training process
The population loss limit established in Theorem identifies two key sources of non- stationarity in the training process of Fitted Q-Iteration: the non-stationary distribution j and the non-stationary target Th +- Both of these sources drive variations in the target population risk across training episodes k.
Theorem 2 (Suboptimality bound via squared bellman residuals). Fix horizon H. Let { (i denote the final value estimates from the K-th iteration; write fn = j(K)), Define the greedy policy
" Ff (8) arg max fu(s,a), aEA
1=1,..., H.
For functions f,9: 3 A R, define the step-h squared Bellman residual
An(J,))(, = -((os)))(Th)))s))))))
Then for any start state x.
H An(fn. fn+1)
(1(c) - S
E
81 =
H
E
An (fr, fnt1)
(3)
Takeaway Suboptimality bound
Equation 3 links model performance the loss function for optimization. This means the agent's 's performance depends on Bellman residuals from the current and optimal policy tra- jectories, not historical ones.
Building on the foundational framework established above, we can observe that the loss function evolves at each iteration k; this phenomenon is analogous to initiating an entirely new round of training. A key distinction from supervised learning lies the initialization process: whereas su- pervised learning relies random initialization, reinforcement learning (RL) commences opti- mization from the arg min of the loss function obtained in the previous iteration. Consequently. investigating the properties of initialization points under such non-steady-state conditions becomes particularly crucial-an insight that further underscores the necessity of the theoretical exploration presented in our work.
4.1 NEURAL TANGENT KERNEL (NTK) DEGENERATION
A key advantage of random initialization is that it ensures the Neural Tangent Kernel (NTK) ma- trix of an overparameterized neural network full-rank with probability 1. This comes from the property that low-dimensional manifolds have zero measure in high-dimensional spaces, making the NTK matrix rank-deficient extremely unlikely. However, this random initialization is violated in Reinforcement Learning (RL). so the initial NTK matrix's 's structural properties (e.g., rank, condi- tioning) are no longer guaranteed, adding uncertainty to learning.
Prior research (Du al., 2019; Allen-Zhu et al., 2019) shows overparameterized networks achieve global convergence to zero training error via Gradient Descent (GD) or Stochastie Gradient Descent (SGD) if two conditions hold: (i) The initial NTK matrix Kg is well-conditioned:: its eigenvalues bounded (no extremely large/small values to distort optimization). (ii) The NTK matrix K remains stable during training, so its structural properties do not degrade and harm convergence.
4.2 GRADIENT ATTENUATION
Another advantage of proper random initialization lies in preserving an appropriate initial gradient magnitude a factor whose critical role in the optimization process has been well validated through
5

---

Published as a conference paper at ICLR 2026
both experimental observations and theoretical analyses. As illustrated in (Dixit et al., 2023), the time an optimizer needs to escape a saddle point is dictated by the magnitude of the initial state's projection onto the unstable nregative-curvaturg subspace of the saddle point. In consequence, excessively small initial gradients prolong the optimizer's stagnation near saddle points, resulting in significant deterioration in optimization performance. Unfortunately, however, gradient decay is an inherent and unavoidable phenomenon in reinforcement learning (RL) training, as evidenced by the theorem presented below.
Theorem 3 (Gradient Dynamics at Initialization). For the optimization objective defined in Equa- tion 1, the initial gradient of the loss function (evaluated at the parameter values that minimized the loss of the previous iteration, fk-1) satisfies:
(s- ThShh (
(I-Thhitttt )
OF k
Distributional skiftt
(4)
+ E
(Tnshitt Tn Tnihitz)
ph-
Torget drift
By setting lu+1 = O. This eliminates the target-drift term entirely, leaving only the distributional- shift component-wwe -here the ((1/k) scaling factor becomes the dominant driver of gradient decay. As the number of training iterations grows large, the magnitude of the imitial gradient will tend to approach zero. This near-zero gradient signal risks trapping the optimization process at saddle points, as the model lacks sufficient directional information to escape these suboptimal regions.
5 SAMPLE WEIGHT DECAY (SWD)
Algorithm Sample Weight Decay (SWD)
Require: Linear decay steps T, minimum weight Wmin, Current time I, timestamps (4)]
1: for = 1 to |D| do
2: age;
t for 1,...: 5: P: Σب wy
3: w; max (Wmin+
T
4: end for
6: I ~ Categorical ((p (p:) B)
7: return B {(s,, 0, Fz, d:)ee
SWD is principled algorithmic intervention to mitigate gradient signal degradation in non-stationary reinforcement learning environments. As in Algorithm 1, it addresses the core challenge in Theo- rem 3: the harmful & decay of gradient contributions from new data. It uses a linear decay mecha- nism, assigning each sample a weight w, = max max( 1 - age, ), where age; = f - 4 (t= current T""
training step, t, = sample collection step). The key insight of Algorithm is its rigorous sample weighting. It identifies the Y/ coefficient-overt attenuating gradients from the current policy distri- bution d=² a scheme: each sample gets probability p. Σ with p. proportional to sample recency. This
as as the root of gradient degradation. To counter this, SWD introduces a linear weighting
neutralizes the attenuation, restoring gradient magnitude and sustaining model plasticity during training.
EXPERIMENTS
The objective of the experiment is to validate the efficacy of the proposed SWD method in mitigating plasticity loss during morizon training and quantify its performance advantages with multifaceted analyses. Specifically, focus the following five key research questions:

---

Published as a conference paper at ICLR 2026
10300
T33
5000
4020
6200
5020
2020
2002
2000
1020
Stops
Stops
=10
Stess
x12
(a) Ant
(b) Halfcheetah
cc) Humanoid
Eptoodic Naturn
4200
Eptedde Buturn
3320
Staps
Staps
(d) Walker2d
(e) Hopper
Figure 2: Empirical validation of SWD across TD3 MuloCo environments (mean + std over runs). SWD consistently improves sample efficiency and performance.
Q1: Does the proposed method SWD consistently improve the training performance of mainstream reinforcement learning (RL) algorithms across different continuous and dis- crete control tasks?
Q2: Does the temporal weighting strategy of the proposed method SWD play a critical role in alleviating plasticity loss?
Q3: Can SWD adapt the training scenarios with increased Update-to-Data (UTD) tio configurations, where more severe plasticity loss should be addressed for better data efficiency?
Q4: How does SWD compare with other methods designed to address plasticity issues? And it feasible to combine SWD with these other methods?
Q5: How sensitive is the proposed SWD to the hyperparametens? How do different choices of heuristics influence the results?
To address Q1, we conduct experiments using the Double DQN, TD3, and SAC algorithms within the SimBa architecture (Lee et al., 2025a), evaluating their performance across the Arcade Learning Environment (Bellemare al., 2013), the MuJoCo environments (Brockman, 2016), and the DMC suite (Tassa et al., 2018). We also include the canonical method Prioritized Experience Replay (PER) (Schaul et al., 2016) direct baseline method. Furthermore, to provide reverse validation of SWD's effectiveness, we use variant called Sample Weight Augmentation (SWA), i.e., counterpart designed to produce the opposite effect by assigning higher weights to older samples. For Q2, we adopt GraMa (Liu et al., 2025) the metric for plasticity, using it to empirically demonstrate the superiority of our proposed method alleviating plasticity loss. To answer Q3, we evaluate performance of SWD based Simba-SAC under different UTD ratios, with specific focus the Humanoid Run environment. To address Q4, we compare SWD against other representative methods designed address plasticity issues. For Q5, conduct extensive experiments to analyze the hyperparameter sensitivity of SWD and the effects of different decay strategies, such as exponential decay and polynomial decay.
6.1 PERFORMANCE EVALUATION
Experimental Setup. We evaluate methods on three benchmark suites: (i) For the five MuJoCo environments (Ant, HalfCheetah, Hopper, Humanoid, Walker2d), we use TD3 (Fujimoto et al., 2018) as the base algorithm with conventional MLP networks. (ii) For the three ALE environments (DemonAttack, Phoenix, and Breakout), we use Double Deep Q-Network (Hasselt et al., 2016) as the base algorithm with the typical CNN-MLP networks. (iii) For the four difficult DMC tasks (Humanoid- Run, Humanoid- Walk, Dug-Run, Dog- Walk), use SAC (Haarnoja et al., 2018) as the base algorithm with the SimBa network architecture (Lee et al., 2025a). In this subsection,

---

Published as a conference paper at ICLR 2026
Double Double
50000
Spleode Reburn
Spleade Rueuur
Epieode Return t0c
12000:
03000
2540
20000
10000
Stap
Stop
Stap
(a) Phoenix
(b) DemonAttack
(c) Breakout
Figure 3: Empirical validation of SWD across Double DQN ALE environments (mean + std over
5 runs). SWD consistently improves sample efficiency and performance.
Median
IQM
Mean
Optimality Gap
SACIPER
RACY6WO
600
540
860
720
800
540
860
720
630
625
650
200
220
240
Figure 4: Performance comparison between SWD and PER based SAC. Aggregate Reliable met- rics (Agarwal et al., 2021) with 95% Stratified Bootstrap CIS in DMC tasks.
we include PER as a canonical baseline for comparison. Detailed hyperparameters and details are provided in Appendix C.
Results As illustrated in Figure 2, Figure 3 and Figure 4, SWD demonstrates remarkable ability to enhance the algorithm's performance. Specifically, it facilitates accelerated learning during the early phases of training and attains superior final policy quality upon convergence- an advantage that is particularly prominent in the Ant and Humanoid environments. In sharp contrast, PER (Prior- itized Experience Replay) demands nearly several times more training time, while the performance improvements it yields remain extremely limited. This observation aligns well with our theoretical framework, and Equation further confirms that performance enhancement can only be achieved by optimizing the TD errors along both the optimal policy path and the current policy path.
6.2 ABLATION STUDY
To provide reverse validation of SWD's effectiveness, we develop contrasting method calledSample Weight Augmentation (SWA), which implements the opposite weighting strategy by assigning higher weights to older data samples. This design allows empirically verify theoretical hypothesis that prioritizing recent experiences is crucial for maintaining neural plasticity.Mi More details are shown in Appendix where employed GraMa (Liu et al., 2025) as measure of neural plasticity.
1002
0.0
Epitodt netarr
200
1002
150
8002
2002
0.
Stass
Steps
Stapa
KI2
(a) Performance Comparison
(b) Gradient Norm Evolution
(c) GraMa
Figure 5: Experiments conducted in the humanoid-rum environment demonstrate that SWA exhibits lower gradient magnitude, GraMa, and inferior performance, which validates our hypothesis.
Results The reverse validation experiment yields key insights: (i) As shown in Figure 5(a), SWA consistently underperforms SWD and uniform sampling, validating that prioritizing recent experi- ences is critical for non-stationary RL learning; (ii) Figure 5(b) shows SWA reduces gradient norms during training (weakened learning signals), aligning with gradient attenuation analysis and confirming older data exacerbates plasticity loss; (iil) GraMa analysis in Figure 5(c) reveals SWA causes sparser gradients and greater plasticity loss than SWD (reduced neural activation/adaqpatioon
8

---

Published as a conference paper at ICLR 2026
2.4
12
0.7
0.4)
64
0.
Steps
Staps
Staps
x12
(a) Humanoid Run
(b) Humanoid Walk
(c) Humamoid Stand
Figure 6: GraMa Metric in Humanoid Locomotion: Run, Walk, and Stand: The results clearly demonstrate that SWD effectively mitigates the loss of plasticity in humanoid robots across these key locomotor states.
capacity), providing direct empirical support for our theoretical framework's s plasticity degradation prediction.
6.3 THE EFFECT IN ALLEVIATING PLASTICITY Loss
To verify whether SWD can mitigate plasticity, we employed GraMa as the evaluation metric to quantify the degree of plasticity during the model training process. Notably, larger GraMa value indicates weaker learning capability of the neural network.
The corresponding results are illustrated in Figure 6. As depicted in this figure, our pro- posed SWD effectively alleviates the gradient sparsity that arises during the training process. Notably, the most pronounced effects are observed in the Humanoid Run environment and the Humanoid Stand environment. can be clearly seen from the figure that SWD exerts its function in the mid- dle and late stages of training gradient attenuation is not severe in the early stage and this observation is consistent with our theoretical predictions.
Results
6.4 COMPATIBILITY AGAINST HIGHER SOATTE-O--DAAA RATIOS
SAC
SAC+SWD
350
+37.3%
285
1QM Perforance
300
9225.4%
286
200
100
UTD-1
VTD-2
UTD-5
Update-to-Data Ratio
Figure 7: Performance comparison across different UTD ratios (1, 2, 5) in Humanoid Run. SWD consistently outperforms uniform sampling across all UTD settings, with improvements ranging from 17.3% to 30.1%.
The Update-to Data Ratio (UTD) is a critical metric for measuring an algorithm's data utilization efficiency. Intuitively, uniform sampling assigns equal weight to each sample; after multiple updates, the gradient signals that can effectively guide the update of network parameters become very weak. In contrast, SWD method assigns greater weight strong gradient signals are maintained even after multiple updates.
recent samples, ensuring that sufficiently
As shown in Figure 7, SWD demonstrates consistent effectiveness across UTD ratios of 1, 2, and 5. Notably, the method shows the largest improvement (+30.1%) at UTD=5, suggesting that SWD particularly beneficial when gradient updates frequent. This robustness indicates that our approach is broadly applicable across different algorithmic configurations without requiring UTD- specific tuning.

---

Published as a conference paper at ICLR 2026
Median
IQM
Mean
Optimality Gap
SWD+S&P
Plasticity Injection
S&P
ReGraMa
SWD
SAC
200
240
280
200
240
280
200
240
280
40
120
Figure 8: Performance comparison between SWD+S&P and other methods designed to address plas- ticity issues. Aggregate Reliable metrics (Agarwal et al., 2021) with 95% Stratified Bootstrap CIS in Humanoid run.
6.5 COMPARISON WITH OTHER METHOSS DESIGNED TO ADDRESS PLASTICITY LOSS
To further evaluate the effectiveness of SWD, compare it in the Humanoid Run environment with three representative methods that are designed to address plasticity issues: ReGraMa (Liu et al., 2025), S&P (Ash & Adams, 2020), and Plasticity Injection (Nikishin al., 2023b). Moreover, explore SWD's 's synergistic potential with S&P, i..., SWD+S&P, which demonstrates the orthogonality.
Results As Figure SWD outperforms other NTK-based methods on the SimBa (Lee et al., 2025a) network. Moreover, SWD combined with S&P yields the best result, validating its orthogo- nality to NTK-based methods. We provide a detailed discussion on the relationship between SWD and prior works for plasticity loss in Appendix C.2.
6.6 OTHER RESULTS
Hyperparameter Choices and Decay Strategies To analyze the hyperparameter sensitivity of SWD, we conduct a grid-search test for two core hyperparameters, i.c., linear decay steps T and min- imum weight threshold Wmin- In Table 12 of Appendix F, SWD exhibits low sensitivity to different choices, demonstrating its stability. Moreover, we compare the linear decay strategy SWD with the other two commonly adopted strategies, i.e., exponential decay and polynomial decay. Table 13 shows that the linear decay strategy outperforms the other two strategies.
Compute-efficien Approximation of SWD To further reduce the computational overhead of per- sample weight, we propose a nucket-based approximation method. As in Table 2 of Appendix D, this approximation significantly reduces the training time no compromise policy performance.
CONCLUSION
In this paper, we identified and addressed the critical issue of plasticity loss long-horizon rein- forcement learning through both theoretical analysis and algorithmic innovation. Our theoretical framework reveals that gradient attenuation follows O(1/k) decay pattern, fundamentally limit- ing the agent's 's ability to adapt to new experiences over extended training periods. To counteract this degradation, we proposed Sample Weight Decay (SWD), a simple yet effective method that ap- plies age-based weighting replay buffer sampling. Through comprehensive experiments across MuloCo, ALE and DMC environments with TD3, DDQN and SAC algorithms, we demonstrated consistent performance improvements ranging from 13.7% 30.1% 1% in IQM scores. Our ablation studies and reverse validation experiments confirm that temporal weighting direction is crucial for maintaining neural plasticity. The broad applicability of SWD across different algorithms, environ- ments, and training configurations, combined with its minimal computational overhead, makes it practical solution for enhancing long-horize RL performance. This work opens new avenues for understanding and mitigating plasticity loss in deep reinforcement learning.
Limitations Owing to computational constraints, our evaluation is restricted to tasks within the MuJoCo, ALE and DeepMind Control Suite (DMC). Additionally, our exploration and practical ap- plication of the proposed theoretical framework remain at a preliminary the "tip of the iceberg." Moving forward, future research will extend SWD more complex scenar- ios, real-world environments. Ultimately, our goal is to develop SWD into a practical, robust tool that effectively preserves the learning capacity of deep reinforcement learning (RI.) agents.
bonge-representiiin merely
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 62533021, 62541610, 92370132), the Fundamental Research Program of Shanxi Province (No. 202503021212091), and the National Key Research and Development Program of China (Grant No. 2024YFE0210000).
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 13, 2019.
Zeyuan Allen-Zhu, Yuanzhi Li. and Zhao Song. A convergence theory for deep learning via over- parameterization. ICML, 2019.
Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep RL: an empiri- cal study. In NeurIPS, 2023.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. NeurIPS, 2020.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: /912. 2,066000 2019.
Celeste Biever. Chatgpt broke the turing test-the race is on for new ways to assess ai. Nature, 2023.
G Brockman. Openai gym. arXiv preprint arXiv:/606.011400. 2016.
Johan Samir Obando Ceron, Mare G Bellemare, and Pablo Samuel Castro. Small batch deep rein- forcement learning. In NeurlPS, 2023.
Johan Samir Obando Ceron, Aaron Courville, and Pablo Samuel Castro. In value-based deep rein- forcement learning, pruned network is good network. In ICML. PMLR, 2024.
Wesley Chung, Lynn Cherif, Doina Precup, and David Meger. Parseval regularization for continual reinforcement learning. In NeurIPS, 2024.
Rishabh Dixit, Mert Gurbuzbalaban, and Waheed U. Bajwa. Exit time analysis for approximations of gradient descent trajectories around saddle points. Information and inference, 2023.
Shibhansh Dohare, Fernando Herrandez-Garria, Qingfeng Lan, Parash Rahman, A. Rupam Mah- mood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of over-parameterized neural networks. In ICLR, 2019.
Benjamin Ellis, Matthew T. Jackson, Andrei Lupu, Alexander D. Goldie, Mattie Fellows, Shimon Whiteson, and Jakob Foerster. Adam on local time: Addressing nonstationarity with relative adam timesteps. arXiv preprint, arXiv:2412.17113 2024.
Mohamed Elsayed and A Rupam Mahmood. Addressing loss of plasticity and catastrophic forget- ting in continual learning. arXiv preprint arXiv:2404.0078/ 2024.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. JMLR, 2005.
11

---

Published as a conference paper at ICLR 2026
Lapo Frati, Neil Traft, Jeff Clune, and Nick Cheney. Reset it and forget it: Relearning last-layer weights improves continual and transfer learning. In ECAI 2024, pp. 2998-999888999 1998-3005. 2024.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. ICLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, and Pieter Abbeel. Soft actor-critic algorithms and applica- tions. arXiv preprint arXiv:1812.05900 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. AAAI, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnel: Convergence and gen- eralization in neural networks. In NeurlPS, 2018.
Zilin Kang, Chenyuan Hu, Yu Luo, Zhecheng Yuan, Ruijie Zheng, and Huazhe Xu. A forget-and- grow strategy for deep reinforcement learning scaling in continuous control. ICML, 2025.
Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, and Ben- jamin Van Roy. Continual learning as computationally constrained reinforcement learning. arXiv preprint, 03X7:22307.04355, 2023.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. ICLR, 2025a.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspher- ical normalization for scalable deep reinforcement learning, 2025b.
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado. Directions of curvature explanation for loss of plasticity. volume arXiv:2312.00244 2023.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv 2015.
preprint
Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning. 2025.
arXiv preprint
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No repre- sentation, no trust: connecting representation, collapse, and trust issues in ppo. NewrIPS, 37: (1652-69699, 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In ICML, 2022.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. arXiv preprint arXiv:2305.15555 2023a.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski. Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. NeurIPS, 2023b.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In /CML., 2023.
12

---

Published as a conference paper at ICLR 2026
Hongyao Tang and Glen Berseth. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. In NeurlPS, 2024.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite, arXiv 2018.
preprint
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models (LLMs) were utilized to support the writing and refinement of this manuscript. Specifically, an LLM was employed to assist in enhancing language clarity, improv- ing readability, and ensuring coherent expression across different sections of the paper. It aided in tasks like rephrasing sentences, checking grammar, and optimizing the overall textual flow.
should be noted that the LLM played no role in the conception research ideas, the formulation of research methodologies, the design of experiments. All research concepts, ideas, and analyses were independently developed and carried out by the authors. The LLM'ss contributions were strictly limited to elevating the linguistic quality of the paper, without any involvement in the scientific content or data analysis.
The authors fully assume responsibility for the entire content of the manuscript, including any text generated or polished with the help of the LLM. We have verified that the text produced with the LLM complies with ethical guidelines and does lead plagiarism or any form of scientific misconduct.
B
PROOF
B. PROOF OF THEOREM 1
In this section, we prove Theoreml.
E Peasirpa(llnna (F(sH) -r(s,, max fhr (84+2, ']]
E, -Bionan)~at
(f(sn, - -Thhi
-n
Eontirpa (-len,a
litz (onti,)))
+ Pn max a'),,, a' a'
( an) Tn. Thnnhi an)
Bentimon (lea,aw)
Pn max fh (8h411 ('),,, (n) max Phtt (h+1(
(I(s), an) - (on,
x (Ph max 2 (") (sn.
max f (
Dn (())) -i and) )
Bennioma((onaa)) ( max (s (sh, max fhr a'))
(f(s))
(S -
B Vara, 1mpp (1)) max (
The loss function can be decomposed into two components:
Bellman Residual Term: (s (f(sh. an) -
Measures the
function approximation error.
14

---

Published as a conference paper at ICLR 2026
Environmental Stochasticity Term: (ss Var 1 (-1s.sm))
maxs fhi (9h+1, d]]]
Reflects the intrinsic randomness of state transitions.
B.2 PROOF OF THEOREM 2
First, we prove one lemma to help the proof.
Lemma Consider an episodic MDP with horizon H. Let nt' { [= denote any policy,
and let {Q denote any set of estimated Q-functions. Let π = {mm the greedy policy
induced by {Qu
For all h E [H]. define:
Value function: (s) 27) where (T.f(s) Eo (l
Bellman residual: (a(s, a) Qn(s,a) -(Th(h-))(ssa)
Then, for all elements x E fllouighllo
H
[x(x)) - VT (x)
Σ
Exy
(Jh - "K (((()) 11 =
h=1
Ey'
181
Proof.
(x)x)-1 - (x) = JKQ)(=) - J QI
(K))(r) - J.
(z)
JK (Ô, +J (()())
= =J (4, + - - - (2(())
=J (1, + P. Ph+t -PAV 1 (x))))
J. 4 (x) + D7 PA - (Ig J (x)
Using recurrence relations and the boundary condition VH+1 = V++1 = 0, we can derive that
H
(x)
=
J PR
J (x))
J PA
(I - (n))))
Which complete proof.
15

---

Published as a conference paper at ICLR 2026
let n' be the optimal policy n², be the greedy policy induced by 1Q and Wi be the corresponding value function. Then, the suboptimal bound is given by:
(x) VI (=) (2( (x()) V1(x)) (x)
H
Ex
TAQun+ (sn; an) - (sn, an) 81 = x
E
Qn(sn, an) - ThQu- an) | 81 x
7m 1
7m1
E
(JR
J
Qn
81
Since π is the greedy policy with respect to Q, we have 3 < O, and for (I) we can drive that:
[Th an) Qn an)
(ThQNmi(3R)aa) - (i(ssnaa)) 1
Bx
H
VH H
Σ En
(ThQntt(ssn ON) - Qn an) an))) s1 = x
The last step makes use of the Cauchy- Sehwwr- inequality, and the second step employs Jensen's inequality. Similarly, we can derive that ② also satisfies:
H
VSH
Σ E
(TrQutI((s)aa) and Qu (sh, ) 1 =
A=1
By combining the above results, we complete the proof of Theorem
B.3 PROOF OF THEOREM
In this section, we prove Theorem 3.
Proof.
VEn (T+Ttn
E 44
2 f
ThJh+1 h+1
Gk
Vf
8b
=B
2 (s
Thisk pk-1
Tn.
Th Th
vS
17-1
Eug
2 (Th.hh
2k-1 Thth+t) Vf
Bug
2 (
Tn Pi
Vf
pk-1
Recall the define of fi--! and the Proposition of 1
4x [s
VE
pt-1
t:
E Ki
4-1
(1 Tn pk--)
27
V
Tn (T-TT(tt))
2
pt
16

---

Published as a conference paper at ICLR 2026
By combining the above results, complete the proof of Theorem 3
B.4 ENTROPY REGULARIZED MDP
In this section, we present the theoretical analysis and error bounds for the Markov Decision Process (MDP). Specifically, the state value function with an entropy reward is defined as follows:
a (x) E
Σ (re( + a log (ae z1)) aa a, Te(112))
Vo E S, [H],
2% T (x, a) (x a) + PA W441 (x. a),
V(z, a) E x A, [
with terminal condition Vi = Then the policy Bellman equations compactly read
Qa (z, a) (z, a) Ps 1 (2,
Villl F (a) J (Q log (z),
#+1
For any function g : S x A R. define the soft value operator Soofft and the step-h soft optimality Bellman operator 7, by
V == max Buwn (9(8,a) a) or log (a)s))
(T ool (s. a) = r().a) + (P (s, a)
We define the Boltzmann policy of induced by the function f f x A + R, which is given by: arg max Ba- [f (8, a) - a log π [a]s)]].
π f
Similarly, we have the following lemma. Lemma 2. Consider an entropy-regularized episodic MDP with horizon H. Let x = { H denote any policy, and let Q {Qn) denote any set of estimated soft Q-functions. Let h=ll (m h=
be the Boltzmann policy induced by Q = (Qu) For all h E [H), define:
Value function: Vr. (Q, a log nn (s) where I = BBorr (1))((ss)))
Bellman residual: In(s, a) == Qn(s a) (T "T (s,))(saa)
Entropy: H((((11)) = - Ban (0ogg=(()) (-13)]
Then for all x E S, we have
H1
(x(
V. (x) =
Exd
(JK Jz' (n()) + (((7((((((s)) (((x(((ss))) 11
h=1
Ex (2)(s Qn.
Proof.
Vi(xx) - 1/o0i,i
(x) -] - a log JI aalog
=]īQx(z) ((a (
=J (Qn - (4
(T) J
JT - +
=J =J (I, + The @ Q++1 - r -r-Pa vontt -1000 (((()))))))))
= -JH (I, + Pa (V/+1 -
= Jy (x) + JK Pn (v - (zx O,, - H((x(((1z))
17

---

Published as a conference paper at ICLR 2026
Using recurrence relations and the boundary condition VH+i V
0, we can derive that
7h-1
(i(x) Vi
Pk
(x)
H
II Jk Pk
JJ) ( H(+)((1))))
This completes the proof.
Theorem 4 (Suboptimality bound for rntropy-regularized MDP via squared Bellman residuals). Fix horizon H. Let (Qn) be the soft value estimates. Define ( as the Boltzmann policy induced by {Q Let { be the optimal policy.
For functions fig : A define the step-h squared Bellman residual:
An(F.g)(s,a) (f(s, -Tr"" g(s. )())a))))
Then we have
H
V-008 (x) Vi 1
(x) S V
Ex
Σ An Qnt1) an) 81
E 4
Σ
An(an, Q1+)(nnn)))))
1=11
Proof.
(1:) 0 (x)) V.on. (x) - Vi(x)) Vi(x))
0 (x)
H1
Q (Sh. an) Qu an) |s = x
+ Bmg (u(()) an) Th Qu+ an)
E
1
H
E,
(J - 1he Qn + (M((T((+ssw) H(TTn()1s))) 81 =
h=1
Since oे is the Boltzmann policy induced by (Qn { can deduce that 3 < The remainder of the proof follows the same reasoning as that of Theorem 2.
h==1 - property that satisfies Equation B.4
C RELATED PRELIMINARIES
In this section, we present the detailed parameters and settings of the experiments.
C.1 ALGORITHM
TD3 In our paper, we utilize TD3 representative of deterministic policies. TD3, an Actor Critic algorithm, is widely adopted as a baseline in various decision making scenarios and has given rise to a multitude of variants, which have established new state of the (SOTA) results on numerous occasions. Different from the traditional policy gradient method DDPG (Lillicrap al., 2015), TD3 makes use of two heterogeneous critic networks, denoted as Q to alleviate the problem of over optimization in Q learning. Thus, the loss function of the critics is
(O(),) E.g.....' [(v (8, for Vi E (1,2).
18

---

Published as a conference paper at ICLR 2026
Where y = r y-m Qo (s', (s())) denotes the target network parameters. The actor is updated according to the Deterministic Policy Gradient:
VoJ(Ø) = E, Qo, (s, (
SAC We select SAC representative of stochastic policies and combine it with SWD in the main experiment. SAC devised to maximize expected cumulative rewards while also boosting exploration via the maximum entropy principle. The actor strives to learn stochastic policy that outputs a distribution over actions, where the critics estimate the value of taking a specific action in given state. This enables more diverse range of actions, facilitating better exploration of the action space. In traditional reinforcement learning, the objective to maximize the expected return. However, SAC introduces additional term that maximizes the entropy of the policy, encouraging exploration. The objective function for optimizing the policy is given by:
( = Es,,o [r(st. 0z) a<H(T(-(s:)))
where H (TT((s)) denotes the entropy of the policy, and a a temperature parameter that balances the trade-off between the immediate reward and the policy entropy. The training procedure of SAC involves two main updates: updating the value function and updating the policy. The value function is updated by minimizing the following loss:
[]
f(Q) = (aannd)
where y is the discount factor, dictating the weight assigned future rewards. V(s') denotes the value function of the next state, which is typically approximated using separate neural network. The policy is updated by maximizing the following objective:
( = Ea,DD [uurs [Q(st, at) - a log (ar[s:))]
Here, log (ael))) represents the entropy of the policy, which serves promote exploration.
SimBa We adopt SimBa (Lee et al., 2025a) our SAC network architecture, which is specifically designed for reinforcement learning (RL) scenarios. Distinctive for embedding "simplicity bias," SimBa not only mitigates overfitting but also enables parameter scaling in deep BL-addressing two key challenges in large-scale RI. model training. Concretely, SimBa comprises three core com- ponents: (i) an observation normalization layer that standardizes input data using running statistics, ensuring stable data distribution for subsequent layers; (ii) a residual feedforward block that estab- lishes direct linear pathway from input to output, facilitating gradient propagation and preserving low-complexity feature representations; and (iii) a layer normalization module that regulates feature magnitudes, preventing excessive value drift during training.
We adopt Prioritized Experience Replay (PER) (Schaul et al., 2016) to bias sampling toward transitions that are expected to yield larger learning progress. Instead of drawing mini-batches uniformly from the replay buffer, PER assigns each transition i a priority P: based on its temporal-differee difffrrecce (TD) error and samples proportionally:
Prioritized Experience Replay
8, = Ir + ((()) Q(s,, a;)]: p: = (6. + P(i)
P Σ,
where ε > 0 avoids zero priorities, E [0, 1] controls the degree of prioritization (a 0 recovers uniform sampling). To correct the sampling bias introduced by P()), PER uses importance-sampling (IS) weights
9
Wi
Wc;
N P()
max, wy
where N is the buffer size and [0, 1] annealed toward during training.
Gradient Magnitude-based neuron activity assessment We employ GraMa (Liu et al., 2025)
urchitecture-agnosicc metric- as our plasticity metric. Specifically. for each individual neuron (or predefined parameter group). GraMa calculates the mag- and maintains a normalized score for each layer,
nitude of gradients computed over crucially, higher scores correspond greater neural plasticity.
19

---

Published as a conference paper at ICLR 2026
Given input distribution D, let Vx) denote the gradient magnitude of neuron in layer é under an input x E D. and let Hy represent the number of neurons in layer L. The learning capacity score for each individual neuron by leveraging the normalized average of its corresponding layer L as formulated below:
EZED [V)
G H, EKEH, EEED [VhTL(x)
GraMa (Gradient Magnitude-basse based neuron activity assessment) identifies neuron in layer in- active if G S T, where denotes the predefined inactivity threshold.
Double DQN We adopt Double Deep Q-Network (DDQN) (Hasselt et al., 2016) as our reinforce- ment learning (RL) baseline, specifically chosen for both pixel-based input scenarios and tasks with long time horizons. By decoupling action selection from target value estimation, DDQN effectively mitigates the overestimation bias inherent in standard DQN, ensuring more stable and accurate value learning. Concretely, while standard DQN maximizes the estimated value using the same network, DDQN utilizes the online network with parameters to select the optimal action and the target network with parameters 0 to evaluate that action.
C.2 RELATIONSIIIP AND COMPLEMENTARITY WITH EXISTING WORK
Prior research plasticity loss has predominantly centered NTK-based methods, which classify into three core categories based on their underlying mechanisms:
(1) Reset-based methods (leveraging random initialization properties): These approaches capi- talize on a key characteristic of over-parameterized neural networks: randomly initialized networks exhibit full-rank Neural Tangent Kernel (NTK) matrices. To mitgate plasticity loss, they period- ically reset network parameters to refresh the NTK and restore the model's capacity for learning. Representative examples include:
ReDo (Sokar et al., 2023): Employs activation-driven reinitialization to reset critical net- work components
ReGraMa (Liu et al., 2025): Utilizes gradient information to guide parameter reinitializa- tion, targeting degraded NTK structures
S&P (Ash & Adams, 2020): Introduces controlled noise into network parameters to reac- tivate dormant plasticity
Plasticity Injection (Nikishin et al., 2023b): Under the premise of keeping the output unchanged, thoroughly refresh the final linear layer.
(2) Implicit NTK regularization methods: This category focuses on detecting early signs of NTK rank deficiency- -uch as unconstrained parameter norm growth-and implementing targeted con- straints to avert rank collapse. Key strategies within this framework are:
Reducing Churn (Tang Berseth, 2024): Suppresses off-diagonal elements of the NTK matrix to minimize gradient correlations, while dynamically adjusting step sizes in rein- forcement learning (RL) settings to preserve NTK integrity
Auxiliary-loss-baseee representation stabilization (Moalla et al., 2024): Integrates addi- tional loss terms to stabilize feature representations, indirectly safeguarding NTK rank
(3) Architecture-based methods: These approaches address plasticity loss at the network design level, either by constructing inherently larger and more robust architectures or by dynamically ex- panding parameter counts during training to prevent NTK rank collapse. Notable instances include:
Hyperspherical Normalization for Scalable Deep RL (Lee al., 2025b): Designs archi- tectures with built-in stability, leveraging hyperspherical normalization to maintain NTK full-rank properties
20

---

Published as a conference paper at ICLR 2026
Forget-and-Grve Strategy for Deep RL Scaling (Kang al., 2025): Implements dy- namic parameter expansion to sustain NTK rank and preserve plasticity
Our work fundamentally orthogonal to these NTKKbbaedd paradigms. Unlike existing methods- which tackle plasticity loss through architectural modifications, explicit NTK regularization, or pa- rameter resetting- we adopt a novel gradient dynamics perspective: our core objective is to mit- igate the temporal distribution shift in the replay buffer, primary driver of gradient magnitude decay and subsequent plasticity loss. Theoretically. our distribution- sampling strategy does overlap with NTK-based plasticity preservation techniques; instead, it offers complementary approach to addressing the root causes of plasticity loss in deep learning systems.
D APPROXIMATE BUCKET-BASEL SAMPLING
Efficient Approximation via Bucket Sampling To mitigate the computational overhead of re- calculating weights for the entire replay buffer, exploit the monotonic age property of our weighting scheme. Since the weights are strictly determined by the temporal age of transitions, we propose bucket-based approximation method:
1. Partitioning: We divide the N transitions in the buffer into B sequential buckets (where B < N).
2. Approximation: Leveraging the monotonicity, we estimate the total weight of each bucket using the weight of its median sample, significantly reducing calculation redundancy.
3. Hierarchical Sampling: We first sample a bucket according to the approximated proba- bility distribution, then uniformly sample transition within that bucket.
As shown in Table 1, this approach reduces the sampling complexity from O(N) to O(B). With B = 2000 and buffer size of = 100, this yields a theoretical 500x speedup in the weight computation phase, rendering the overhead negligible.
Table 1: Computational complexity comparison. N denotes the buffer size (10%), M the batch size,
and B the number of buckets (2000).
Method
Complexity
Scale Dependency
Uniform Sampling
((M)
Independent of Buffer
Exact SWD
O(N + M)
Linear w.r.t Buffer
Approximate SWD
O(B + M)
Linear w.rt Buckets
Empirical Validation We validate the efficiency and effectiveness of this approximation the iimaanod--rus task. As presented in Table 2, the Approximate SWD method matches the wall- clock training time of Uniform sampling (approx. 8.7 hours) while preserving the performance gains of the exact method, achieving high episode return of 224.9 + 17.5.
Table 2:
Runtime and performance comparison on Humanoid Run. The approximate method retains performance while significantly reducing training time.
Method
Wall-Clock Time
Episode Return
Uniform
8.65 h
190.46 + 7.99
Exact SWD
10.43 h
229.01 + 37.43
Approximate SWD
8.70 h
224.93 + 17.47
21

---

Published as a conference paper at ICLR 2026
E EXPERIMENTAL DETAILS
STRUCTURE
TD3 In this paper, we adopt the official network architecture Twin Delayed Deep Deterministic
Policy Gradient (TD3) for baseline comparison, with detailed layer-wise configurations provided in Table
Table 3: Network Structures of the Twin Delayed Deep Deterministic Policy Gradient (TD3)
Network Component
Actor Network
Critic Network
Fully Connected Layer
(state (256)
(state.dim aation_dimi + (256)
Activation
ReLU
ReLU
Fully Connected Layer
(256) - (128)
(256) (128)
Activation
ReLU
ReLU
Output Fully Connected Layer
(128) + (action dim)
(128) + (1)
Activation
Tanh
None
TD3 adopts two identical critic networks (Critic 1 Critic 2) for delayed Q-value update, both following
the above structure;
Tanh activation constrains the actor's output action to the range [-1, 1], consistent with standard continuous action space settings.
Double DQN
We adopt the Nature CNN network architecture, the detailed specifications of which are presented in Table 4 and Table 5. Our implementation refers to the official code repository to ensure consistency with the original design.
Table 4: Architecture of the Nature CNN Encoder used in Double DQN. The input consists of stacked frames of size 84 x 84.
Layer Input Channels
Kernel Size / Stride
Output Channels
Activation
Convl
8 /
32
ReLU
Conv2
32
4 x 4 2
64
ReLU
Conv3
64
3/1
64
ReLU
Table 5: Architecture of the Double DQN Q-Network. The input is the flattened feature vector from the Encoder.
Layer
Configuration
Activation
Input (Flatten)
3136 units (7 7 x 64)
FC1
Linear(3136 + 512)
ReLU
Output
Linear(512 + [A])
SAC In this paper, we adopt the same configuration of SimBa as used in the Soft Actor-Critic (SAC) algorithm, with detailed network structures provided Table 6, Table and Table 8. Our implementation refers to the official SimBa code repository' to ensure consistency with the original design.
http nd/ dgn httpps:/ thub. com httmm//oooeeeeeeeeeaaaaaaa simba
22

---

Published as a conference paper at ICLR 2026
Table 6: Architecture of the SimBa Residual Block
Layer/Operation
Input/Output Dimensions
Activation Function
Layer Normalization
(hidden, _dim) > (hidden_ding __im) (hidden_dimg + (4 hiddeendddiff _dim)
None
Fully Connected (Expansion)
ReLU
Fully Connected (Compression)
(4xhidden_dim) - (hidden_dim)
None
Residual Connection
Input + Block Output"
None
" denotes element-wise addition between the original input and the block output.
Table 7: Architecture of the SimBa Encoder
Component
Structure & Dimension Flow
Input Projection (Fully Connected) Residual Block Stack
(input_dim) (hidden_dim)
X num_blocks (each block follows Table 6) (hidden dim) (hidden dim)
Final Layer Normalization
shuuk__locs" denotes the number of stacked residual blocks, configurable based on task requirements.
Table 8: Network Structures of the SimBa-SAC Framework
Component
Actor Network
Critic Network
Input Dimension SimBa Encoder
(state_dimg dim)
(state.ding + action
hidden dim=128; num blockssc
hidden 1iaden_dim=5111 12; num blocks=2
Fully Connected
(128) - (action_ding
(512) + (1)
Output Activation
Tanh
None
Tanh activation is used constrain the action output within the range [-1, 1, 1]. consistent with standard SAC implementations.
E.2 IMPLEMENTATION DETAILS
Our codes implemented with Python 3.10 and JAX. All experiments run on NVIDIA GeForce GTX 3090 GPUs. Each single training trial ranges from 10 hours to 21 hours, depend- ing on the algorithms and environments.
TD3 Implementation Our TD3 implementation refers to CleanRL an efficient and reliable repository for reinforcement learning (RL) algorithm implementations.
Notably, for all OpenAl MuloCo experiments, we directly use the raw state and reward signals from the environment without any normalization or scaling. To facilitate exploration, an exploration noise sampled from (0, 0.1) is added to the action selection process of all baseline methods. The discount factor set to 0.99, and the Adam optimizer adopted for all algorithms.
Table 9 presents the complete hyperparameters of TD3 used in our experiments; to reproduce the learning curves reported in the main text, we recommend using random seeds to 5.
Double DQN Implementation Our Double DQN implementation builds upon the CleanRL repos- itorys, recognized for its high-fidelity and reproducible reference algorithms. To ensure experimen- tal fairness, we strictly align our configuration with standard Atari benchmarks.
The detailed hyperparameters presented in Table 10. Notably, for the Arcade Learning Environ- ment (ALE) tasks, we incorporate the bucket-based approximate sampling mechanism from SWD to enhance efficiency.
*https: thub. http://hwwwaaaaeaaccaa/////a//aa action. .py 'https:/ / thub. http://gtthhhh...oo/eee/eegaaaat httpp//wwworrrggh..caaaaa//aaa/e
23

---

Published as a conference paper at ICLR 2026
Table 9: Hyperparameters of the TD3 Algorithm
Hyperparameter
TD3 Configuration
Actor Learning Rate
10-4
Critic Learning Rate
10-3
Discount Factor
0.99
Batch Size
128
Replay Buffer Size
106
SWD-Specific Hyperparameters
Linear Decay Steps
100, 000
Minimum Weight (min weight)
0.1
Table 10: Hyperparameters of Our Double DQN Implementation
Hyperparameter
Value
General Training
Optimizer
Adam
Learning Rate
1 x 10-4
Discount Factor (7)
0.99
Buffer Size
1 x 106
Batch Size
Learning Starts
80, 000 steps 4 steps 10 M
Train Frequency
Total Timesteps
Exploration (Epsilon-Greedy)
Start Epsilon (Esun)
1.0 0.01
End Epsilon (Eend)
Exploration Fraction
0.10 (1 10 steps)
Target Network
Target Update Frequency
1000 steps
Target Update Rate (r)
1.0 (Hard Update)
SWD-Specific Hyperparameters
Linear Decay Steps
80, 000
Minimum Weight
Number of Buckets
2000
24

---

Published as a conference paper at ICLR 2026
SAC Implementation Our Soft Actor- Critic (SAC) implementation is also based on the CleanRL repository, specifically referencing the continuous action SAC implementation".
Table 11: Hyperparameters of Our SAC Implementation (with SimBa Encoder)
Hyperparameter
SAC (with SimBa Encoder)
Optimizer
AdamW (weight decay = 10 ?)
Policy (Actor) Learning Rate
1 10
Q-Network (Critic) Learning Rate
1 10 10-4
Discount Factor
0.99
Batch Size
256
Warmup Steps (for Policy Update)
5000
Target Q-Network Update Rate (r)
0.005
Target Q-Network Update Interval
1 (step)
Policy (Actor) Update Interval
2 (steps, policy_frree (eiiueeffreeee))
Entropy Target
-|44 (|A] = action space dimension)
SimBa Encoder (Actor): Hidden Dim Blocks
128 /
SimBa Encoder (Critic): Hidden Dim Blocks
512/2
SWD-Specitic Hyperparameters
Linear Decay Steps
80, 000
Minimum Weight (min (min-weight)
0.1
PER-Specific Hyperparameters
Prioritization Exponent (a)
0.6
Importance Sampling Exponent (3)
0.4 1 10-6
Beta Increment Rate
The hyperparameters for our SAC (equipped with the SimBa encoder) are detailed Table 11.
E.3
SAC LEANRING CURVE
1000
SAC+SWD
SAC+SWD SAC+PER
600
Episode Return
SAC
400
200
200
Steps
x10
Steps
(a) Dog Run
(b) Dog Walk
250
SAC+SSWD
SAC SSW
Episode Return
200
SAC
aiisdde Retura
SAC
600
400
50
200
Steps
Stops
x10*
(c) Humanoid Run
(d) Humanoid Walk
Figure 9: SAC leanmning curve on DMC tasks
thub. http:////hhtccccooooccccc///e//r
action.py
25

---

Published as a conference paper at ICLR 2026
F ADDITIONAL EXPERIMENTS
F.I SWA
In this section, provide detailed information about our ablation experiments. First, present the algorithmic details of SWA, which are summarized in Algorithm
We adopt the detailed parameter settings of Soft Actor-Critic (SAC), presented in Ta- ble s-spccifiallyy, we use the same Linear decay steps and minimum weight winn as specified therein.
Algorithm 2 SWA
Require: Linear decay steps T, minimum weight Wiin Current time t, timestamps {+ [2]
1: for = 1 to (D] do
2: age; f
3: w; min (1, W'in + age;
T
4: end for
5: p: ΣJ for i 1..... D
6: I ~ Categorical (p: (p:) | B)
7: return B {(si, ai Ti, 8 diEE
F.2 ABLATION STUDY OF TO-DAA
250
SAC+SWD
Epllodde Netur:
1.0
50
X30
Sieps
K10
(a) UTD=1
(b) UTD=2
SAC-5WiD
200
150
Steps
(c) UTD==
Figure 10: Sensitivity analysis regarding the UTD. Data represents the mean = std of five experi- mental runs conducted on the Humanoid Run.
We adopt SAC (Soft Actor-Critic) as the backbone algorithm and aim to optimize the Update-to- Data (UTD) ratio. This optimization enables faster policy iteration, thereby better leveraging the advantages of SWD. As illustrated in Figure 10, with the increase in the UTD ratio, SWD consistently outperforms the uniform sampling baseline.
F.3 PARAMETER SENSITIVITY ANALYSIS
To assess the robustness of our proposed method, conducted extensive grid search to evaluate the sensitivity SWD to its two primary hyperparameters: the linear decay steps (T) and the minimum weight threshold (wmin.)
26

---

Published as a conference paper at ICLR 2026
We constructed 5 hyperparameter grid, varying Tdeeay from 20, 000 to 100, 000 and Wiin from 0.02 to 0.10. Experiments were performed on the Humanoid Run task, with each the 25 con- figurations averaged over random seeds (totaling 125 independent runs). The results, summarized Table 12, indicate that SWD maintains stable performance across a wide range of hyperparameter settings. While optimal performance fluctuates slightly, the method does not exhibit drastic failure modes within the tested range, demonstrating its robustness to hyperparameter selection.
Table 12: Parameter Sensitivity Analysis. Grid search results on Humanoid Run (Mean + Std).
The best performance marked in bold.
Decay Steps
Minimum Weight Threshold (Wmin)
(Tdccay)
0.02
0.04
0.06
0.08
0.10
20,000
229.7 126.4
240.9 137.1
234.9
115.0
217.9 138.6
226.1 #:337
40,000
231.4 #444
224.5 #314
231.3
=30.8
227.0 #23.1
225.5 #22.5
60,000
217.4 +40.2
231.2
420.9
240.5
=55.0
215.7
127.9
240.7
135.3
80,000
233.6 4:4.9
231.8 1:3.7
225.2
42.6
220.9
1:7.6
231.3 154.4
100,000
224.0 #32.1
201.8
#31.9
217.0 488.5
241.6 +38.4
229.2 229.5
F.4 IMPACT OF DECAY STRATEGY
We further investigate the influence of the weight decay schedule on performance. To this end, we compare our default Linear Decay against Exponential and Polynomial variants. The specific formulations are defined as follows:
Linear (Ours): w(t) = maix 1- t/T), providing a constant rate importance reduction.
Exponential: w((l) = max(Wmin (xpmminn)p(((()))) where 1, modeling rapid initial for- getting.
Polynomial:
w(t) max (min: (1 (1-t/T)"). where p penalizing older samples more
aggressively than the linear approach.
The empirical results on Humanoid run are summarized in Table 13. Our proposed Linear Decay strategy significantly outperforms alternative schedules. Notably, both Exponential and Polynomial decay perform worse than the SAC baseline, suggesting that overly aggressive weight reduction disrupts the learning stability required for high-dimensional control tasks.
Table 13: Performance comparison of different decay strategies on Humanoid Run. The relative difference is calculated with respect to our Linear Decay method.
Decay Strategy
Episode Return
vs. Linear SWD
Linear Decay (Ours)
229.01 + 37.43
SAC (Baseline)
190.46 +'t 7.99
-16.8%
Exponential Decay (T 1)
187.04 +'t 29.85
-18.3%
Polynomial Decay (p = 2)
132.91 +-t 11.12
-42.0%
27


Reviews:
[{"review_id": "QVzmltipsN", "reviewer": "Reviewer_HaDB", "summary": "The paper studies plasticity loss in deep RL and attributes it to two mechanisms induced by non-stationarity: (i) NTK Gram rank collapse, and (ii) attenuation of gradient magnitudes that decays as $\\Theta(1/k)$ over training iterations $k$. Motivated by the second mechanism, the authors propose Sample Weight Decay (SWD): linearly down-weight older transitions in the replay buffer so that recent data counteracts the $\\Theta(1/k)$ decay and restores effective gradient scale. SWD is plug-and-play for replay-based algorithms.  \nEmpirically, SWD is added to TD3 and SimBa-SAC on MuJoCo and DMC tasks, with strong gains on DMC Humanoid. A “reverse” ablation (SWA, which up-weights old data) predictably reduces gradient norms and hurts returns, supporting the causal story.", "questions": "1. NTK evidence: Can you track NTK Gram eigenvalues (or condition number) on fixed probe batches across training, ideally under “sequential initialization,” to confirm or quantify rank collapse?  \n2. Comparisons or compositions: How does SWD compare to and combine with ReGraMa or Plasticity Injection on a small DMC subset (including an SWD+X variant) to establish additivity?  \n3. Sensitivity and heuristics: Please add curves for $T$ and $w_{\\min}$. Is a simple heuristic (for example, $T$ as a fraction of buffer capacity or tied to a gradient-norm half-life) a reasonable default for new domains?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Tight theory to method link: The $\\Theta(1/k)$ gradient attenuation analysis (Theorem 3) cleanly motivates SWD’s linearly decaying replay weights; the SWA “reverse” ablation strengthens causal plausibility.  \n2. Simple and orthogonal: SWD lives at the sampling layer, is easy to add to any replay-based RL algorithm, and should compose with model-level plasticity fixes.  \n3. Compelling results on hard control: Consistent improvements, with standout gains on DMC Humanoid, suggest the effect is meaningful, not a small-n artifact.", "weaknesses": "1. Unprobed NTK mechanism: The paper posits NTK rank collapse but provides no spectrum or conditioning measurements; half of the causal story remains speculative.  \n2. Missing SOTA positioning: No head-to-head (or composition) with recent plasticity remedies such as ReGraMa, Plasticity Injection, or ReDo, leaving SWD’s incremental or additive value unclear.  \n3. Practical knobs under-explored: No sensitivity for $T$ (linear decay steps) and $w_{\\min}$; interactions with PER are not clarified (do they stack or conflict?); runtime or throughput impact is unreported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BSIbMCRqsy", "reviewer": "Reviewer_mmd4", "summary": "This paper investigates plasticity loss in deep reinforcement learning from a theoretical optimization perspective. The authors identify two mechanisms causing plasticity loss: (1) rank collapse of the Neural Tangent Kernel (NTK) Gram matrix, and (2) Θ(1/k) decay of gradient magnitude during training. Based on this analysis, they propose Sample Weight Decay (SWD), a lightweight sampling strategy that assigns higher probabilities to more recent samples in the replay buffer to counteract gradient attenuation. Experiments on MuJoCo and DeepMind Control Suite tasks with TD3 and SAC algorithms demonstrate consistent performance improvements.", "questions": "1. Can you provide rigorous analysis showing how the gradient dynamics derived for FQI extend to deep RL algorithms with experience replay and bootstrapping? How does the 1/k decay manifest in TD3/SAC specifically?\n2. How sensitive is SWD to hyperparameters T and w_min? Tables 5-6 show different T values—how were these chosen? Have you systematically compared different decay schedules (exponential, polynomial, etc.)?\n3. Since you claim SWD is orthogonal to existing methods, have you tested combinations with network reset or neuron recycling? Can you provide direct experimental comparisons with ReDo, ReGraMa, and Plasticity Injection to substantiate the SOTA claim?\n\nI will consider increasing the score if the author responds to these questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- SWD is remarkably simple to implement with minimal computational overhead, making it easily applicable across different RL algorithms and architectures as a plug-and-play solution, which is practically valuable compared to more invasive methods.\n- The experimental validation is comprehensive, including evaluation on two benchmark suites, comparison with PER, reverse validation through SWA ablation, plasticity measurement using GraMa metrics, and robustness analysis across different UTD ratios, all showing consistent improvements.", "weaknesses": "1. The linear decay design of SWD (w_i = max(w_min, 1 - age_i/T)) appears somewhat arbitrary without clear theoretical justification for why this particular weighting scheme optimally compensates for the 1/k gradient attenuation. Sensitivity analysis on decay schedules and hyperparameters is insufficient.\n\n2. Recency-based sampling is not novel conceptually, and the paper lacks direct comparisons with recent plasticity-preserving methods (ReDo, ReGraMa, Plasticity Injection). The claimed \"SOTA performance\" is primarily against uniform sampling and PER, making it difficult to assess the true contribution relative to the current state of the art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xL0rDYZPtK", "reviewer": "Reviewer_8vUF", "summary": "The paper studies plasticity loss in deep RL through an optimization lens, identifying two mechanisms: (i) NTK rank degeneration and (ii) gradient attenuation that scales as Θ(1/k) under non-stationary data/targets with experience replay. Building on this analysis, the authors propose Sample Weight Decay (SWD)—a simple, plug-and-play, age-based sampling scheme for replay buffers that linearly down-weights older samples to counteract gradient decay and restore gradient magnitude. Experiments on MuJoCo (TD3) and DMC (SimBa-SAC) show consistent gains—including strong results on Humanoid—plus robustness across UTD ratios; ablations (including a reverse “SWA” variant) and GraMa measurements support the mechanism.", "questions": "1. Beyond sample-efficient regimes. \n\nAuthors primarily test in sample-efficient settings. What happens with very long training (e.g., 10M env steps with a 1M buffer) where fresh samples repeatedly overwrite older ones? Do you observe more catastrophic forgetting under high churn, and does SWD still mitigate it or saturate? (This is important because small buffers + long horizons can exacerbate plasticity.)\n\n\n\nI’m open to increasing the score if the authors adequately address the weaknesses—particularly by deepening related work discussion, clarifying the novelty relative to ER-decay, and detailing the efficiency and long-horizon robustness experiments.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear theoretical framing with actionable takeaways. The paper formalizes how distribution/target non-stationarity yields NTK rank issues and a Θ(1/k) gradient-magnitude decay, then links performance to Bellman-residual control via a suboptimality bound—cleanly motivating data-weighting interventions.\n\n2. Simple, general, and orthogonal method. SWD is an easy drop-in change to replay sampling, compatible with TD3/SAC (and, in principle, other replay-based methods) and positioned as orthogonal to architectural “plasticity-injection” tricks. The paper claims minimal overhead and plug-and-play practicality.\n\n3. Compelling empirical evidence. Consistent improvements across MuJoCo and DMC (including Humanoid), robustness to varying UTD, and reverse validation via SWA plus GraMa analysis strengthen the causal story beyond raw scores.", "weaknesses": "1. Missing related work / plasticity literature coverage.\n\nThe related work should more deeply connect to recent plasticity and replay-weighting literature. Please discuss and contrast with, e.g.:\n\n- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML’24).\n- Disentangling the causes of plasticity loss in neural networks (CoLLA’24).\n- Hyperspherical Normalization for Scalable Deep RL (ICML’25).\n- Mitigating Plasticity Loss in Continual RL by Reducing Churn (ICML’25).\n- A Forget-and-Grow Strategy for Deep RL Scaling in Continuous Control (ICML’25).\n\n2. Overlapping idea; contribution clarity vs ER-decay.\n\nConceptually, SWD (linear, age-based down-weighting) looks very close to prior ER-decay heuristics, with differences seemingly in coefficients/schedules. The paper does offer more formalism, but the delta in contribution should be crystal clear: which parts are novel theory, which are new algorithmic prescriptions beyond a tuned decay, and what guarantees (if any) distinguish SWD from ER-decay? I’m open to a high score even if the mechanism is similar—provided the theoretical backup and empirical analysis are meticulous and make the case for why this instantiation is principled/non-equivalent.\n\n3. Implementation and efficiency details are thin.\n\nA per-sample weight update naively done every step can be costly. Please:\n- Describe the efficient implementation (lazy updates? piecewise-linear buckets? periodic renormalization?).\n- Report end-to-end wall-clock and GPU-hour costs on standard hardware for SimBa vs SimBa+SWD;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SxoeH4TwpM", "reviewer": "Reviewer_2bzK", "summary": "Studies plasticity loss in deep RL under non-stationarity. The theory isolates two mechanisms: (i) NTK rank collapse across sequential warm-starts and (ii) a $\\Theta (1/k)$ decay of the initial gradient each round. Motivated by (ii), the paper proposes Sample Weight Decay (SWD)—a lightweight recency-weighted replay scheme—to restore gradient magnitude. Experiments on TD3 (MuJoCo) and SAC with SimBa (DMC Humanoids/Dog) show consistent gains with reliable aggregate metrics (IQM/median/mean, bootstrap CIs) and a reverse ablation (SWA) that up-weights old samples and underperforms, alongside GraMa analyses.", "questions": "- How sensitive are results to **linear vs. exponential** decay and to $((w_{\\min}, T))$? Any failure modes with small buffers or rapid behaviour-policy shifts? \n- Do SWD’s benefits persist with **pixel observations** or **sparse rewards**? (A small add-on task would suffice.)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Identifies a crisp cause of plasticity loss and links it to a tractable remedy (recency weighting)\n- Very simple algorithm (SWD) with negligible overhead, orthogonal to architectural methods (ReDo, Plasticity injection, etc.)\n- Consistent empirical improvements across TD3 and SimBa-SAC; reverse ablation (SWA) plus GraMa trends support the mechanism.\n- Uses reliable RL reporting (IQM/median/mean + stratified bootstrap CIs).", "weaknesses": "- Theory scope. Main results are derived for FQI-style/population losses; transfer to fully practical bootstrapped targets with representation drift is not fully established.\n- Breadth. Evaluation is confined to continuous control; adding a pixel-based or sparse-reward task (e.g., DMControl pixels, AntMaze) would test generality.\n- Over-edited text (LLM side-effects). While LLM assistance can improve flow/grammar, several sentences become awkward or semantically off and harm readability—for example, the abstract’s “How plasticity loss arises, dissipates and can be dissolved.” A careful human pass is needed to fix misuses and improve readability.\n- Related work is too narrowly framed (over-emphasis on resets). The section concentrates on reset-style approaches while under-representing other relevant families—particularly **churn-reduction methods** and **auxiliary-loss–based representation stabilisation**. Please discuss these lines of work and clarify how SWD differs or complements them (see, e.g., churn-reduction: [https://arxiv.org/abs/2506.00592](https://arxiv.org/abs/2506.00592); auxiliary losses: [https://arxiv.org/abs/2405.00662](https://arxiv.org/abs/2405.00662)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)