A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Authors: Zilin Kang
Chenyuan Hu
Yu Luo
Zhecheng Yuan
Ruijie Zheng
Huazhe Xu
Venue: ICML 2025 poster (2025)
Keywords: Deep Continuous Control
Infantile Amnesia
Primacy Bias
Experience Replay
Primary area: reinforcement_learning->deep_rl
TL;DR: We find a forget-and-grow approach for deep reinforcement learning scaling in continuous control

Abstract: Deep reinforcement learning for continuous control has recently achieved impressive progress. However, existing methods often suffer from primacy bias—a tendency to overfit early experiences stored in the replay buffer—which limits an RL agent’s sample efficiency and generalizability. A common existing approach to mitigate this issue is periodically resetting the agent during training. Yet, even after multiple resets, RL agents could still be impacted by early experiences. In contrast, humans are less susceptible to such bias, partly due to *infantile amnesia*, where the formation of new neurons disrupts early memory traces, leading to the forgetting of initial experiences. Inspired by this dual processes of forgetting and growing in neuroscience, in this paper, we propose *Forget and Grow* (**FoG**), a new deep RL algorithm with two mechanisms introduced. First, *Experience Replay Decay (ER Decay)*—"forgetting early experience''—which balances memory by gradually reducing the influence of early experiences. Second, *Network Expansion*—"growing neural capacity''—which enhances agents' capability to exploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control benchmarks with more than 40 tasks demonstrate the superior performance of **FoG** against SoTA existing deep RL algorithms, including BRO, SimBa and TD-MPC2.

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Zilin Kang" 12 Chenyuan Hu '3 Yu Luo* Zhecheng Yuan
Ruijie Zheng?
Huazhe Xu
Xu316 316
Abstract
ER Decay
Deep reinforcement learning for continuous con- trol has recently achieved impressive progress. However, existing methods often suffer from pri- macy bias-a tendency to overfit early experi- stored in the replay buffer- which limits RL agent's sample efficiency and generalizabil- ity. In contrast, humans are less susceptible to such bias, partly due to infantile amnesia, where the formation of new neurons disrupts early mem- ory traces, leading to the forgetting of initial ex- periences (Akers al., 2014). Inspired by this dual processes of forgetting and growing in neu- roscience, in this paper. propose Forget and Grow (FoG), new deep RL algorithm with two mechanisms introduced. First, Experience Re- play Decay (ER Decay)="forgetttnn early expe- -which balances memory by gradually reducing the influence of early experiences. Sec- ond, Network Expansion-"grrowin neural ca- pacity"-which enhances agents' capability to ex- ploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control bench- marks with more than tasks demonstrate the superior performance of FoG against SoTA exist- ing deep RL algorithms, including BRO, SimBa and TD-MPC2.
Sample weights
Sample times
ER Decay
Normal ER
timestamps
timestamps
Network Expansion
Blok
Block
(n-1) x expansion
expand
Block
Heck
Dense
(n-20
Dense
Total
Mujoco
HumanoidBench 14 Tasks
42 Tasks
Taskss
10-
1.0
1.0
Normalized Score
0.0
Nermalized Score
0.0
0.8
0.
0.6
TD MPC2
TD-MPC2
MD-MPC2
SimBa
SimBa
SimBa
FoG
BRO
FoG
BRO
FoG
BRO
MaMMMeusy&Meiumm
DMC-Hard
MetaWorld
Tasiss
5 Taskss
10 Tasks
1.0
10
100
Normalized Score
0.8
Nermalized Score
0.0
0.8
"Fqual contribution 'Shanghai Qi Zhi Institute Department of Computer Science and Technology, Tsinghua University 'Institute for Interdisciplinary Information Sciences, Tsinghua University "Huawei Noah's sss Ark Lab 'Computer Science, University of Maryland "Shanghai Artificial Intelligence Laboratory. Cor- respondence to: Zilin Kang <kzl22 Aaal22@mall..eee www.iii.sinnga..d....oo Chenyuan Hu <cy-huu2ee @ waw.ilsssinghu.ccuu..oo Huazhe Xu <huazhe xu mail. www.gguh.acuuuun........
06
0.5
TD MPCM
TD-MPC2
Normalized
ID-MPC2
SimBa
SimBa
SimBa
FoG
BRO
FoG
BRO
FoG
BRO
Figure I. Overview. Top: illustrate two key components our strategy: ER Decay and Network Expansion. Bottom: comparison of normalized FoG outperforms popular model-based and model-free methods including TD-MPC2, SimBa and BRO.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
1. Introduction
our question: can reinforcement learning agents follow similar process, combining forgetting and growth, to miti- gate primacy bias and improve performance?
Do humans remember learning how to speak or walk? For the majority, the answer is no. This phenomenon, known as infantile amnesia in neural science (Josselyn & Frank- land, 2012), occurs because the hippocampus generates large number of new neurons during infancy, which disrupts existing memory traces and leads to forgetting (Alberini & Travaglia, 2017). Observed in humans and other mammals, this phenomenon plays a critical role in the development of memory and learning abilities.
Our answer is affirmative, with two novel methods: Experi- ence Replay Decay (ER Decay) and Network Expansion, both of which are simple and efficient. ER Decay reduces the sampling probability of older data in the replay buffer, ef- fectively allowing the agent to "forget" outdated transitions in a way analogous to the memory disruption observed in infantile amnesia. Meanwhile, Network Expansion intro- duces new neurons to the model early in training, providing fresh capacity to adapt and reorganize, much like the growth of new neurons in infancy.
Sample Efficiency Compared
1000
OBAC
Together, these methods are directly inspired by the "forget and grow" mechanism observed in infantile amnesia, and they work in tandem to suppress primacy bias and enhance overall performance.
OBACescaling
Normalized Score
BRO
SimBa
TD-MPC2
FoG
500
In this work, we give theory-based intuition on how does such a "forgetting plus growth" mechanism work, and we provide a thorough empirical investigation of its ef- fectiveness. We also propose a new algorithm Forget- and-Grow (FoG), which integrates these methods into the OBAC algorithm (Luo et al., 2024) and further boosts per- formance using scaled networks and replay ratios. Our approach achieved highly competitive results across more than 40 environments in several benchmarks, including Mu- joco (Todorov et al., 2012), DMControl (Tassa et al., 2018), Meta-World (Yu al., 2020), and HumanoidBench (Sfer- razza et al., 2024) surpassing popular methods including SimBa (Lee et al., 2024), BRO (Hansen et al., 2024) and TD-MPC2 (Hansen et al., 2024) in multiple settings.
500k
1M
steps
Figure 2. Normalized scores of algorithms on DMC-Hard tasks (5 hardest Dog & Humanoid tasks). The performance of the OBAC+scaling method is comparable to that of SimBa and BRO, but when combined with FoG, it achieves superior results.
However, in the field of deep reinforcement learning, agents typically do not have a natural mechanism to forget their early training experiences. Instead, they often overfit to initial data, which is often described as primacy bias (Nik- ishin et al., 2022b; Qiao et al., 2023). In particular, deep RL methods that rely heavily on experience replay (Mnih et al., 2015; Fedus et al., 2020) with high replay ratios (D'Oro et al., 2022) tend to repeatedly revisit old transitions, rein- forcing patterns formed in the early stage of training.
To summarize, the contributions of this paper are three-fold:
1. We show empirically that reset mechanisms alone can- not fully resolve the primacy bias issue.
2. We introduce two strategies: ER Decay and Network Expansion, demonstrating their effectiveness in miti- gating primacy bias.
To address this issue, previous works (Nikishin et 2022b; Qiao et al., 2023) have introduced a reset mechanism that periodically resets part of the policy's network parameters. While resets partially reduce primacy bias, the older sam- ples are still replayed more frequently than the newer ones. Thus, this imbalance could still lead to overfitting on the old experience, harming the overall performance.
3. Develop new deep continuous control algorithm FoG, achieving fftte-of-the-ss art performance across several benchmarks.
2. Related Works
Off-policy RL. Off-policy reinforcement learning a fre- quently used paradigm where agents learn policies from data generated by previous policies (Mnih et al., 2015; Munos et al., 2016; Prudencio et al., 2023; Ma et al., 2024), al- lowing for more efficient use of prior experiences. Due to the advantage of improving sample efficiency, it is widely used in scenarios where collecting on-policy data is costly
To draw a parallel, infantile amnesia involves two key as- pects-forge and grow. During infancy, the brain generates a large number of new neurons, which not only disrupt ex- isting memory traces and lead to forgetting but also provide the capacity for reorganizing and forming new structures critical for memory and learning. This phenomenon inspires
2

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
or risky. Many approaches focus on real-world application (Delarue et al., 2020; Yang et al., 2022) and algorithmic improvements such as reducing bias in Q-value estimation (Fujimoto et al., 2018; Lan et al., 2021), better utilizing offline datasets (Fujimoto et al., 2019; Schaul et al., 2016), and integration with other paradigms (Luo et al., 2024; Tan et al., 2024).
et al., 2021). As for off-policy RL, model size scaling has exhibited advantages for both discrete action representa- tion (Schwarzer et al., 2023) and continuous control (Nau- man et al., 2024b).
Besides scaling, internal structural changes, such as acti- vation functions and normalization, also improve capacity, For example, TD-MPC2 (Hansen et al., 2024) enhanced model performance by incorporating LayerNorm stabi- lize gradients, Mish smoother activation function, and SimNorm to maintain stable updates across layers. Simi- larly, Nauman et al. (2024a) demonstrated that LayerNorm and residual connections significantly enhance performance, while SimBa (Lee et al., 2024) used running statistics nor- malization, residual feedforward blocks, and post-I.ayer normalization to address simplicity bias. These modifica- tions highlight that structural improvements, alongside care- ful scaling, are crucial for leveraging high-capacity models effectively in RL.
Primacy bias. The concept of primacy bias in deep rein- forcement learning (RL) refers to the overfitting of policies to earlier experiences when training on progressively grow- ing datasets, which can negatively impact the following learning process (Nikishin et al., 2022a). This phenomenon is particularly problematic under high replay ratios, where policies overfit to out-of-distributtin data from past experi- ences, as noted by Li et al. (2023); Lyu et al. (2023). One straightforward approach to mitigate primacy bias is re- initializing the network to restore plasticity, as explored by Nikishin al. (2022b); Ma al. (2023); Nauman al. (2024b). There are other methods to alleviate the problem in- cluding model ensembles (Chen et al., 2021), regularization (Kumar et al., 2023b), plasticity injection (Nikishin et al., 2023), and ReDo (Sokar et al., 2023). These approaches aim to balance stability and adaptability, reducing the impact of primacy bias and enhancing overall learning performance
3. Method
3.1. A Motivating Example
Primacy bias has been studied in previous works (Nikishin et al., 2022b; Qiao al., 2023), and common approach to mitigate it is to adopt reset strategy. However, even if an agent resets multiple times during training, experience replay remains imbalanced: older transitions dominate the sampling process. The following theorem formalizes this issuc:
Experience replay. To better utilize the previous experi- ence and improve sample efficiency, Lin (1992) propose the concept of experience replay, which revisits transitions in the replay buffer with a uniform sampling strategy to update the agent. Following Lin (1992), Prioritized Experience Re- play (Schaul et al., 2016) measures the priority of transitions according to the magnitude of their eemporal-difference (TD) error so that the agent can focus on transitions that are more important to improve sample efficiency. Andrychow- icz al. (2018) 8) introduces Hindsight Experience Replay (HER) which incorporates a set of additional goals into each trajectory to avoid complicated reward engineering. Zhang & Sutton (2018) proposes Combined Experience Replay (CER) that adds the latest transition to the batch and uses the corrected batch to train the agent. Corrected Uniform Experience Replay (CUER) (Yenicesu et al., 2024) also adopts the idea of balancing the sampling of the transitions in the replay buffer to make the sampling distribution more uniform considering the whole training process.
Theorem 3.1. Given uniformly sampled replay buffer D that stores N sequentially added transitions the earliest transition is sampled
{K1, "oog,..,,NN" 213 log N) times in expectation, where 3 (the product of replay ratio and batch size) is a constant.
More specifically. for any transition Kt (with >> 1), its expected number of samples Blme) satisfies:
N
1 N
E[me]
N
In
In
1
1
t
1
t
Proof. See Appendix A.
This result indicates that older transitions are sampled con- siderably more often, regardless of how frequently resets happen. In fact, shorter reset intervals often lead to higher replay ratios and can even exacerbate primacy bias. To verify this, we train four agents on humanoid-walk and HalfCheetah-v4 with a replay ratio of 10 and batch size of 256 (i.e., 3 = 2560). To ensure stability of training under such a high replay ratio, we add a layernorm after every dense layer.
Model capacity improvement in RL. The most straight- forward way to improve model capacity is model size scal- ing (Hestness et al., 2017). However, in RL, naive scaling can lead to instability or degraded performance (van Has- selt et al., 2018; Sinha et al., 2020; Bjorck et al., 2022). High-capacity models have shown effectiveness in offline RL (Kumar et al., 2023a; Lee et al., 2022) and model-based RL (Hafner et al., 2024; Hansen et al., 2024; Hamrick
3

---

A Forget-and- Strategy for Deep Reinforcement Learning Scaling in Continuous Control
humanoid-walk
Along
Hestmap
800
SAC
19
0.0130.000 02..0.0270.000.0 0520 082 0.11
0.18
0.4
SAC with reset
150
Return
PER-SAC
Balanced-SAC
Step 100k
26
125
100k
0.0610.07 073
0.11
0.15
0.25
0.3
100k Avg Loss Volue
Expanded SAC
10
66
100
1000 Aag Lop
0.14 0.17
0.22
0.33
400
13
86
0.25
0.27
0.38
50
18
116
0.37
0.37
170
0.43
1
2
3
5
300k
600k
Step 100k
Step 100K
steps
HalfCheetah-v4
Critic Loss Heatmap Along Time
Critic Loss Heatmap Along Time
-0.4
0.035
0,0630
0.099
00.3
10000-00 sug Loss Val value
100k Avg Loss Value
Return
Step 100k
0.4
0.2
0.3
10500
Step
Took
0.2
0.1
7000
300k steps
600k
i
2 Step
3
4
$
-0
Step -0
Figure 3. Learning Curves and Heat Maps of SAC Variants. Top Left: Return of various SAC variants in humanoid-walk, Top Center: Critic loss heatmap of Normal SAC in humanoid-valk: Top Right: Critic loss heatmap of SAC with reset in the humanoid-walt environment. Bottom Left:: Return curves of various SAC variants
HalfCheetah-ve Bottom Center: Critic loss heatmap of Balanced SAC in sumanoid-walk Bottom Right: Critic loss heatmap of Expanded SAC in humanoid walk. About Critic Loss Heatmaps: Every 100k steps, we measure critic loss over the entire huffer and average the loss every 100k steps to get critic loss heatmaps. The darker the color near the diagonal, the less influenced by primacy bias; conversely, the darker the color towards the top-right corner indicates greater influence from primacy bias.
SAC: A baseline Soft Actor-Critic agent (Mnih al., 2015) with uniform replay buffer and no resets.
We measure the critic loss across the buffer every 100k steps to generate critic loss heatmaps. As shown in Figure 3, the critic loss of Normal SAC is significantly higher than that of other SAC variants that incorporate resets, indicating its failure to fit the data after millions of updates. Even SAC with reset experiences spike in loss in the diagonal areas as training progresses. This suggests that, despite multiple resets, the SAC with reset agent still overfits to early data and fails to adapt effectively to newer transitions. This observation highlights that resetting alone is insufficient to mitigate primacy bias.
SAC with reset: SAC that resets at 15k, 50k, 100k, 200k, 400k, 600k, and 800k steps.
PER-SAC: Based on SAC with reset, plus PER (prior- itized experience replay) (Schaul al., 2016).
Balanced SAC: Based on SAC with reset, plus our ER decay mechanism to mitigate primacy bias, with le 4, le 1.
In contrast, Expanded SAC exhibits a darker diagonal area, indicating that the model less influenced by primacy bias, with little increase in loss over time. Not only does Expanded SAC reduce the loss recent transitions more effectively than SAC with reset, but it also achieves superior
Expanded SAC: Based on SAC with reset, plus both ER decay and network expansion, which expands critic networks from 3 dense layers to dense layers, at 50k and 200k network iterations after each reset, layers at time.

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
performance across both tasks. Specifically, it delivers 53% and 27% improvements in final scores on humanoid-walk and HalfCheetah-v4. respectively, compared to SAC with reset. Furthermore, Expanded SAC outperforms recent methods like SimBa (Lee et al., 2024) by 74% and 22%, while maintaining a simpler design.
Sample Times of Transitions
2000
Normal ER
ER decay
sample times
Although this scall-scale experiment is not exhaustive, it underscores the critical role of addressing primacy bias in
1000
experience replay and provides strong motivation for our proposed method.
3.2. Experience Replay Decay and Network Expansion
50k steps
100k
Experience replay decay. We incorporate a decay factor into experience replay so that the sampling probability of older transitions gradually decreases. This strategy partially "forgets" older samples and mitigates primacy bias.
Figure 4. Sample times of transitions in a normal buffer and a decayed buffer with 1e 4,7 = 0.01 over 100k steps.
Theorem 3.2. Let D be a replay buffer with ER decay c. For any transition Kú in D, the expected number of times it sampled, E[n.]. bounded by a constant C.
Cati Lood bestroo Arsoo Tiree
Catii Modomoo A Tano
00077 M8 0.661 b.sh
0.0830
Proof. See Appendix A.
6:32
0:30
0:05
02
0:33
0.33
This result implies that the sampling frequency of older transitions stays within a constant range. However, purely exponential decay quickly diminishes the sample weight of older transitions, causing them to virtually disappear from the replay buffer. This effectively reduces the buffer size and, in practice, can harm the final performance. Therefore, we set a lower bound for sampling weights:
c.25 0.24
0.35
700 Aog
0.23 0.26 0.27
0.18
0.3 0.27
0.20
05
0.05
Step
Step/
Figure 5. Changes in critic loss over time for PER(left) and ER decay(right) in humanoid-walk: The darker the color near the diagonal, the less influenced by primacy bias; conversely, the darker the color towards the top-right corner indicates greater influence from primacy hias.
((t,,)) max(T, (T, (1 - '-²)
where e is the decay rate, T is the minimum weight, and the sampling probability of transition H; at time is:
Puu E/= Wias
Network expansion. Early training relies heavily on ini- tial transitions in replay buffer, which often deviate signif- icantly from the final policy distribution. Yet during this stage, the neural network has the highest plasticity. To address this, propose network expansion, inspired by infantile amnesia in mammals. The method involves grad- ually adding new parameters to the critic network early in training (e.g., after each reset) through residual connections. These newly added parameters are not influenced by early transitions, enabling better adaptation to data shifts when combined with ER decay.
Simulation (see Figure 4) shows that ER decay can effec- tively suppress sample times of older transitions and balance the sample times of transitions across large range of steps.
During the experiments, we compared the performance of our ER decay method with Prioritized Experience Replay (PER). PER assigns higher replay weights to transitions with larger TD errors. In the previously discussed motivat- ing example, we analyzed the loss landscape of both PER and ER decay, As demonstrated in Figure 5, the value in the diagonal row of ER Decay is generally smaller than those of PER, which indicates that ER Decay may better alleviate primacy bias. Experimentally, ER decay achieved a significant advantage over PER in the motivating example. This result was further validated through ablation studies conducted on a broader range of scenarios.
Our network structure is based on BRO's design, which incorporates layer normalization after each dense layer and residual connections for parameter management. Building on this, we modularized the network into distinct blocks, facilitating the implementation of network expansion.
Although starting with a large network can provide better
5

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
ER decay and network expansion. We introduce ER decay into the replay buffer, where older transitions assigned lower sampling probabilities, and apply network expansion to the critic networks early in training. Together, these methods embody the concept of forget and grow, al- lowing the agent to better adapt to shifts in replay data. While simply scaling up OBAC yields performance on par with SimBa BRO, the forget-and annge--wn-groww grow approach is crucial for FoG's superior performance.
Block
Dense
Intelu
Block
t
Block
Dense In
Dense
xn
Dense
Figure 6. Network expansion illustration. We initialize networks with fewer parameters and progressively add a new block (in the frame) to residual connections cach expansion step.
4. Experiment
To evaluate the performance of FoG, we collect in total 41 tasks from 4 domains: Mujoco (Todorov et al., 2012), DM- Control (Tassa et al., 2018), Meta-Worde (Yu al., 2020), and HumanoidBench (Sferrazza al., 2024), covering a wide range of challenges, including high-dimensional states and actions, sparse rewards, multi-object and delicate manip- ulation, and complex locomotion. The implementation de- tails and environment settings are provided in Appendix B.
performance in the early stages of training, it tends to counter early convergence issues in the mid-to-late stages. In contrast, agents trained with network expansion adapt more effectively to changing training objectives. This observation is further validated in the FoG algorithm. Even when training begins with a network significantly larger than the one used in the motivating example, without network expansion, the agent is highly prone loss explosions under larger replay ratios. This directly impacts the agent's 's stabil- ity. Therefore, network expansion is necessary component in FoG. For more details, refer to Section 4.
Baselines. We compare FoG against 3 state-of- of-the-art off- policy RL algorithms, including model-free methods and I model- base- method. Our baselines contain: 1) BRO (Nau- man et al., 2024b), which scales the critic network of SAC while integrating distributional Q-learning, optimistic ex- ploration, and periodic resets. 2) SimBa (Lee et al., 2024), which adopts running statistics normalization, residual feed- forward blocks, and post-layer normalization to address simplicity bias. 3) TD-MPC2 (Hansen et al., 2024), high- efficient model-based RL method that combines model pre- dictive control and TD-learning.
A similar idea, called plasticity injection (Nikishin et al., 2023), also involves network expansion. However, com- pared to the complex parameter adjustments required by plasticity injection, our approach is simpler implement, imposing minimal constraints. Our experiments demon- strate its superior performance and efficiency.
3.3. The Forget and Grow Strategy (FoG)
4.1. Experimental Results
We combine ER decay and network expansion into uni- fied algorithm, FoG, built on the following key ideas:
Figure 7 presents the learning curves that demonstrate the performance of FoG alongside various baselines across di- verse task suites. Overall, we observe that FoG typically outperforms most model-free and model-based baselines across various environments in terms of exploration effi- ciency and asymptotic performance. In HumanoidBench, FoG also exhibits comparable capabilities to the best base- line TD-MPC2.
OBAC backhone. Our FoG is based on the Offline Boosted Actor-Critic (Luo et al., 2024). OBAC enhances online policies using offline data and demonstrates strong performance across various standard benchmarks.
Notably, with identical hyperparameters, FoG achieves con- sistently high performance across all benchmarks. Other baselines have certain weaknesses in some benchmarks. Due to the task-specific done signal setting, TD-MPC2 may perform poorly on Mujoco. SimBa gets lower scores in simple environments with small action dimensions, such as Mujoco and DMC-Easy while BRO performs worse in more complex environments with high action dimensions, such as DMC-Hard and HumanoidBench.
Scaled critic networks and replay ratio. For scaling up the networks, we utilize a larger network for the critic. To increase flexibility, we modularized the network structure, constructing the critic entirely from blocks connected via residual connections. Each block contains two dense layers, each followed by layer normalization. This modular archi- tecture proves highly compatible with our FoG mechanism.
Additionally, we increase the replay ratio to 10 and introduce a reset list to manage the agent's reset behavior effectively.
The key takeaway is that with very simple algorithmic
6

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Ant
Dog Run
Assembly
Balance Simple
1000
1000
7000
Success Rate
Return
Return
Return
3500
500
0.5
500
500k steps
1000k
steps HalfCheetah
steps
steps
Hopper Hop
Coffee pull
Maze
1.0
1000
Return
Retur
Return
8000
500
Success
0.5
500
300K
600
250k steps
500k
500K
1000k
500K steps Slide
1000k
steps
steps
Humanoid
Humanoid Run
Pick place wall
1000
1000
7000
Return
Return
Return
Success
3500
500
300k
BOoK
500k
1000k
500k
1000k
0
500k steps
1000k
steps
steps
steps
Walker2d
Humanoid Walk
Push back
Walk
1000
1.0
1000
Success Rate
Return
Return
Return
500
0.5
500
2500
steps
steps
steps
steps
FoG
TD-MPC2
SimBa
BRO
Figure 7. Main results. We provide performance comparisons for 16 of the 41 tasks, four for each task suite. Please refer to Appendix C for the comprehensive results. The solid lines are average return/success rate, while the shades indicate 95% confidence intervals. All algorithms are evaluated with 3 random seeds.
changes, including decay, network expansion, and sim- ple model structure modifications, FoG successfully over- comes primacy bias and achieves better performance. No- tably, previous studies found that model performance sat- urated when the model parameters reached 5M (Nauman et al., 2024b). However, with the help of network expan- sion, FoG unlocks new levels of favorable model size scal- ing up to 23M parameters. We will discuss in detail the performance improvements from each modification in the upcoming ablation section.
Choice of experience replay methods. One of the key designs of our algorithm is ER decay, which gradually de- creases the sampling weight of older transitions. To evalu- ate its effectiveness, implement other experience replay methods including PER and CUER. For the sake of fair- ness, we simply replace the experience replay method of FoG while keeping other parts unchanged for comparison. Additionally, we also test the results without using any ex- perience replay method. Results are shown in Figure 8. We observe that both using PER and ER decay significantly im- prove the final performance and convergence speed, with ER decay showing the best results. However, the improvement with CUER is marginal. In certain cases, CUER provides no improvement.
4.2. A blation Studies
We conduct several ablations demonstrate the effective- ness of the design choices of FoG in this section.

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
HalfCheetah-v4
Humanoid-v4
dog-run
humanoid-run
16000
CUER
8000
CUPR
PER Nome Decty
600
PER
PER
None Decay
None Decay
500
None Decay
Return
Return
Return
Return
8000
230
0
500x steps
50nk
500K Steps
1020k
1000k
steps
steps
Figure& 8. Choice of experience replay methods. We adopt 4 tasks from Mujoco and DMControl, two for each task suite, compare different experience replay methods. Mean runs; shaded areas are 95% confidence intervals.
Ant-v4
HalfCheetah-v4
dog-run
humanoid-run
8000
16000
500
Blockss
Blocks
Bllocks
Wllcks Blocks
Grow
Retur
Return
Retur
Return
6000
8000
250
300
0
0
500k
steps
steps
steps
steps
Figure Ablation network expansion. We adopt tasks from Mujaco and DMControl, for each task suite, showease the necessity of network expansion. Mean of 3 runs; shaded areas are 95% confidence intervals.
dog-run
humanoid-run
HalfCheetah-vd
humanoid walk
50
100
Critic Buffer Loss
2
Blocks
Critic Buffer Loss
Blocks
Dormant Ratio
0.5
w/o FoG FeG
Dormant Ratio
0.5
FsD
FoG
25
50
0.36
0.35
0.2
0.2
150kk steps
300k
155kk
3001
250k steps
500k
250k steps
500k
steps
Figure 10. Critic buffer loss with and without network expan- sion. We visualize the critic buffer loss of the first 300k steps for the tasks of DMControl suite. Mean 3 runs; shaded areas are 95% confidence intervals.
Figure 11. Dormant ratio during training. We the tio of dormant neurons during training on HalfCheetah-vé and humanoid-wee walk.. Red lines indicate time steps where network expansion happens.
signals, causing the loss to explode. However, network ex- pansion helps to suppress this excessive catastrophic growth in the loss.
Necessity of network expansion. To establish the impor- tance of network expansion in our Framework of Growth (FoG), we conducted a series of experiments. Firstly, we compared the standard FoG, which incorporates dynamic network expansion, against versions with a fixed model size, denoted FoG (fixed). These fixed models were configured with either additional blocks, and network expansion was disabled. As illustrated in Figure 9, enabling network expansion leads to discernible performance improvement over both fixed-size configurations. The critic buffer loss, presented in Figure 10, offers insight into this advantage: after resetting the network, the critic buffer network overfits early transitions and generate incorrect behavior cloning
Furthermore, network expansion significantly enhances model plasticity by reducing the prevalence of dormant neu- rons. Activated neurons, characterized by non-zero gradi- ents, can be updated by new data, whereas dormant neurons remain static during training. Consequently, higher ratio of dormant neurons indicates more severe loss of plasticity. This metric has been adopted in several recent studies as an indicator a model's 's representational capacity and plastic- ity degradation (Liu et al., 2024; Sokar et al., 2023; Xu et al., 2023). Our experiments, shown Figure 11, reveal that
8

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
tional complexity and longer training times may limit its practicality in scenarios that require rapid training. Addi- tionally, the effectiveness of our two strategies is primarily supported by empirical experiments and intuitive theoreti- cal insights, lacking a thorough and in-depth investigation into their mechanisms. Future works include seeking theo- retical guarantees for the FoG strategies and applying the Forget-and- -row strategy a broader range of algorithms.
2Blocks
4Blocks
grow
40
40
-40
-40
-40
-80
-80
-80
80
-40
0
40
80
80
-40
0
40
-80
-40
80
Figure 12. T-SNE visualization of representations. We visual- ize the representations via t-SNE after training 0.2M steps HalfCheetah-v4. From left to right are the t-SNE results of 2 blocks, 4 blocks and expansion from to blocks.
Impact Statement
This work contributes to advancing the field of Reinforce- ment Learning (RL), particularly in the domain of off off-policy RL algorithms. The proposed algorithm holds potential im- plications for real-world applications, especially in areas such as robotics. It's worth noting that exploration of an RL agent in real-world environments may require several safety considerations to avoid unsafe behavior during the process.
FoG effectively reduces the number of dormant neurons. Notably, this reduction is achieved even when compared to baseline models that, despite possessing a larger overall parameter count than a fully expanded FoG agent, do not employ network expansion. This highlights the efficacy of the expansion mechanism itself. Moreover, steep drops in the dormant neuron ratio are consistently observed immedi- ately following network expansion events, underscoring its direct role in reactivating parts of the network.
References
Akers, K. G., Martinez-Canabal, A., Restivo, L.,, Yiu, A. P., De Cristofaro, A., Hsiang, H.-L.,, Wheeler, A. L., Guskjolen, A., Niibori, Y., Shoji. H., et al. Hippocampal neurogenesis regulates forgetting during adulthood and infancy. Science, (44(6184))599 00088866022222 2014.
To further understand how the expanded network adapts to new memories and improves representations, we con- ducted representation analysis on the HalfCheetah-v4, We sampled experiences from the replay buffer, passed them through the critic network, and extracted features from the final layer for I-SNE visualization in 2D space. We com- pared three settings: FoG with network expansion, FoG (fixed) with blocks, and FoG (fixed) with 4 blocks. As depicted in Figure 12, network expansion facilitates the formation of clearer clusters in the feature space. This ob- servation suggests that network expansion enables the learn- ing of contributing to the overall performance gains.
Alberini, C. M. and Travaglia, A. Infantile amnesia: critical period of learning to learn and remember. Journal of Neuroscience, 37(24))5783-579) 2017.
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay, 2018. URL pttpss:/aaxxiv.orgge http:///aaii...rraa///////////// 01495.
and more structured representations,
Bjorck, J., Gomes, C. P., and Weinberger, K. Q. Towards deeper deep reinforcement learning with spectral nor- malization, 2022. URL http::::/aaxxi.or///// c1151.
5. Conclusion
In this work, we propose Borget-and-Grow (FoG), which effectively addresses the primacy bias problem in deep con- tinuous control through two simple yet effective methods: ER decay and network expansion. By incorporating forget and grow, FoG enables agents to mitigate the overfitting to early experiences and boost their performance across vari- ous continuous control tasks. Abundant experiment results show the superiority of FoG compared with existing state- of-the-art off-policy RL and model-based RL algorithms, including BRO, SimBa and TD-MPC2. Our findings reveal new perspective to alleviate the primacy bias, highlighting the potential of integrating inspired cognitive mechanisms into reinforcement learning frameworks. While FoG outper- forms existing algorithms across various continuous control tasks, it is important to note that the increased computa-
Chen, X. Wang. C., Zhou, Z., and Ross, K. Random- ized ensembled double qelamning: Learning fast without a model, 2021. URL https: http///axvvvvvo//e wwaeiiivoorg/aaaoo /bs//
Delarue, A., Anderson, R., and Tjandraatmadja, C. Re- inforcement learning with combinatorial actions: An application vehicle routing, 2020. URL https:
//arxiv.org/aateeeee.. aa/////as//000..0 02200.1120000
D'Oro, P., Schwarzer, M., Nikishin, E.,, Bacon, P.-L.,, Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W. Revisit- ing fundamentals of experience replay. In International conference on machine learning, pp. 3061-3071, PMLR, 2020.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Lee, H., Hwang. D., Kim, D., Kim, H., Tai, J. J., Subrama- nian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning, arXiv preprint arXiv:2410.09754, 2024.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional conference on machine learning, pp. 1587-1596. PMLR. 2018.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration, 2019, URL pittps://arxee org/ abs 1obs/1812 111/11111.. 02900.
Lee, K.-H., Nachum, O., Yang. M., Lee, L., Freeman, D., Xu, W., Guadarrama, S., Fischer, I., Jang. E., Michalewski, H., and Mordatch, I. Multi-game decision transformers 2022. URL https://arxiv.cr abs 0bs/2205.11244
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018. URL https: //arxiv. www/abg/////ooooooooeo....o...at 111111111111 01290.
Li, Q. Kumar, A., Kostrikov, I., and Levine, S. Efficient deep reinforcement learning requires regulating overfit- ting, 2023. URL https: //axxvi.vv//.//////////////aaav/ org/ abs/2304, 1 10466.
Hafner, D., Pasukonis, J.,
Ba, J., and Lillicrap, T. Master- ing diverse domains through world models, 2024. URL https: / //axxvvivv///////////a////aaaav/ www.i.orgg/...............ooo.rr wwt/abs/////eooooooo...oo....sae 23011. 04104.
Lin, L.J.. Self-improving reactive agents based on reinforce- ment learning, planning and teaching. Machine learning, 8:293-321, 1992.
Hamrick, J. B., Friesen, A. L., Behbahani, F., Guez, A., Viola, F,, Witherspoon, S., Anthony, T., Buesing, L.., Veličković, P. and Weber, T. On the role of planning in model-based deep reinforcement learning, 2021. URL https: / arxiv. org/ abs 2011. 04021.
Liu, J., Obando Ceroon, J., Courville, A., and Pan, L. Neuro- plastic expansion deep reinforcement learning. arXiv preprint arXiv:2410.07994, 2024.
Hansen, N., Su, H., and Wang, X. Td-mpe2: Scalable, robust world models for continuous control, 2024. URL https:// /tts:///axxvv org/ abs/ 2200. 6828.
Luo, Y.. Ji, T., Sun, F., Zhang, J., Xu. H., and Zhan, X. Offfine-boosse actor-critie: Adaptively blending optimal historical behaviors in deep off off-policy rl. arXiv preprint arXiv:2405. 200.44558880 18520, 2024.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, 2017. URL https::///// /otes///axxiiee wawaaaiivv.... org/ abs 00409.
Lyu, J., Wan, L., Lu, Z., and Li, X. Off Off-policy rl algorithms can be sample-efficient for continuous control via sample multiple reuse, 2023. URL https: //aaaaxxvvooo/.//aaae.a.......aa abs/2305.1844
Josselyn, S. A. and Frankland, P. W. Infantile amnesia: neurogenic hypothesis. Learning & Memory, 19(9): 423 433, 2012.
Ma, G., Li, L.,, Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data. Modules and Training Stages, 2023.
Kumar, A.. Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q q-learning on diverse multi-task data both scales and generalizes, 2023a. URL htg//bb/// 2211. 15144.
Ma, G., Zhang, L., Wang, H., Li, L., Wang, Z., Wang. Z., Shen, L., Wang. X., and Tao, D. Learning better with less: effective augmentation for sample-efficient visual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Kumar, S., Marklund, H., and Roy, B. V. Maintaining plasticity in continual learning via regenerative regular- ization, 2023b. URL https: wttpaaivvvvooo... org. abs/ 2308. 11958.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M.. Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Lan, Q., Pan, Y., Fyshe, A., and White, M. Maxmin q- learning: Controlling the estimation bias of q-learning, 2021. URL https:/ :// arxiv. www.ii..rggbb////eeeeeeeeese// 2002.. 06487.
10

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M. I. Tactical optimism and pessimism for deep reinforcement learning, 2022. URL https:// arxiv.org org/ abs 221.2.1122200 22t.b///...00.555 03765.
Schwarzer, M., Obando-Ceron, J., Courville, A., Bellemare, M., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency, 2023. URL https: httpvrxivvv.o... org abs /2005 19452.
Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient ooolicy reinforcement learning. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http:://proceeddngg..eeee cc cc/ sc/paper__e /t httpeeeffieeeeapprr//////////// c3992e 2a572999a88aaa112222184448888883 Pape.. pdf.
Sferrazza, C., Huang, D.-M., Lin, X., Lee, Y., and Abbeel, P. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation, 2024. URL https://arxiv www./oorg/aao.eo abs 1000...0066.
Sinha, S., Bharadhwaj. H., Srinivas, A., and Garg, D2rl:
Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09/63 2020.
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp. 32145-32168. PMLR, 2023.
Nauman, M., Bortkiewicz, M., Mitos, P., Trzciński, T., Os- taszewski, M., and Cygan, M. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforce- ment learning, 2024a. URL https: http///axvvvv..///e//eeeeeaae.v/ org/ abs/2403 00514.
Tan, X., Qu, C.,, Xiong. J., Zhang, J., Qiu, X., and Jin, Y. Model-based off-policy deep reinforcement learning with model-embedding. IEEE Transactions on Emerging Topics in Computational Intelligence, 8(4):2974-2986, 2024. doi: 10.1 10.1109/1EECC1.2244.366022
Nauman, M., Ostaszewski. M., Jankowski, K., Miloš, P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16158, 2024b.
Nikishin, E., Schwarzer, M., D'Oro, P. Bacon, P.-L.,, and Courville, A. The primacy bias in deep reinforce- ment learning. In Chaudhuri, K., Jegelka, S., Song, L.. Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Ma- chine Learning Research, pp. 16828-16844 PMLR, 17- 23 Jul 2022a. URL https: http::///occceedieeoeo .ml.. press/vll https///11/nnnkiihihhe
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. I.., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint 1oXiv:/801.00 2018.
Todorov, E.,, Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
html.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learn- ing, pp. (8828-16847, PMLR, 2022b.
van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad, 2018. URL https:/ :ttpps:///rraae arxiv..rrg abs 1812.111.00 02648.
Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection, 2023. URL attps:://rxiis org/abs abs// 2305. 15555.
Xu, G., Zheng, R., Liang, Y.. Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., et al. Drm: Master- ing visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:23 10. 19668, 2023.
Prudencio, R. F., Maximo, M. R., and Colombini, E. L. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Yang, Y., Ding, Z., Wang, R., Modares, H., and Wunsch, D. C. Data-driven human-robot interaction without veloc- ity measurement using off ool-policy reinforcement Icarning. IEEE/CAA Journal of Automatica Sinica, 91):47-63, 2022. doi: 10.1109/JAS. W.w.IAA...22..000445
Qiao, Z., Lyu, J., and Li, X. The primacy bias in model- based rl. arXiv preprint arXiv:2310.150011 2023.
Yenicesu, A. S., Mutlu, F. B., Kozat, S. S., and Oguz, O. S. Cuer: Corrected uniform experience replay for off ool-po continuous deep reinforcement learning algorithms, 2024. URL httptts:/// /tatttps://rxivvv /wtwxv..vvv/v/////////..//.//avl
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Priori- tized experience replay, 2016. URL https:/ //rxiv.. org/abs/1511 05952.
U 09030.

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1 100. PMLR, 2020.
Zhang, S. and Sutton, R. A deeper look experience play, 2018. URL https: / 01275.
12

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
A. Proofs
A.1. Proof of Theorem 3.1
Proof. Given a uniformly sampled replay buffer D that stores N sequentially added transitions {Ki. Ki..... AN J. transition Be (with > 1) sampled with probability ! where > f is the number of transitions at sample time. the expectation of sample times E[ng] can be calculated
N
N
[-1
B[me] = 3
31
Σ
)
3(HN - H
where N the N-th harmonic number, and H is the (t - 1- harmonic number. And we have the following inequality:
In + n
Hn < Inn +
So we can get:
N 1
(In )) 1 1)< <
E(ms]
1
N
In t N
-1=
(In (Int 1+
In
B
t t-1
For the earliest transition Ki, its expected sample times B/ni] is bounded by:
1 Eim1]
In N
< N
N
3
The earliest transition is at least sampled (2 log N) times in expectation.
A.2. Proof of Theorem 3.2
Proof. Let D be a replay buffer with ER decay e. For any transition Ki in D, its expected sample times E[me] can be calculated as:
(1 - e)r
m2 B
(1- ... (1-ttitt-1)
have:
E Emn.] E[n]] E 21, ,N
So the expected sample times of any transition D is bounded by the earliest transition.
- e)¹
c1 c) e)
B|m]] 30
(1 (1-c)² ) 3
1=0
Asc=1- -1 -1-c) c)++1
we have:
8 2 (1 =) 3
Emm
E
1=
So find a constant C' such that:
Eln:] < C, Vi E (1, ... .1, NN
The expected sample times of any transition in D is bounded by constant.
B. Implementation Details
B.1. Hyperparameters
In this section, we delve into the specific implementation details of FoG. Our Hyperparameters are listed in Table
13

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Table 1. The hyperparameters of the proposed method
Hyperparameter
Value
Hyperparameters
Optimizer (Critic)
Adam W
Critic learning rate
3e-4
Critic initial depth
2
Critic maximal depth
4
Critic expansion iters
50k, 200k
Optimizer (Actor)
Adam
Actor dense layers
3
Actor learning rate
3e-4
Actor log std clipping
(-20,2)
Discount factor
0.99
Batch size
256
Replay buffer size
le6
ER Decay E
le-5
Minimal buffer weight 7 Behavior clone weight
0.1
le-3
Network hidden dim
512
Network Architecture
Network activation function
elu
Critic depth
2-4
For all tasks, we use max-entropy framework (Haarnoja et al., 2018) for the online learning policy x with automatic temperature tuning. Besides, we set the pessimism of the online policy (Moskovitz et al., 2022) to MetaWorld tasks to further encourage exploration. In other benchmarks set to
We use reset lists for FoG. For relatively simple locomotion tasks (hl-stand, hl-walk, h1-stair and h1-slide) in HumanoidBench,
reset list of 15k, 50k, 250k, 500k, 750k (The same BRO's reset list) further improve exploitation. We use reset list of 15k, 50k, 100k, 200k. 400k, 600k, 800k for other benchmarks and other tasks in HumanoidBench.
B.2. Details of Network Expansion
The expansion of the critic network is key component of FoG. Each expansion step adds new block composed of two dense layers with 512 hidden dims and ELU activation functions. Surprisingly, we find that there is no restriction on the initialization of the new block, and we initialize the new block with the same initialization as the original network (which orthogonal init with scale of 22).
At each expansion step, we reinitialize the optimizer and decay the learning rate by the number of parameters in the network. paratps
Namely, use init_rr current params decay the learning rate, where init_Ir is the initial learning rate, init_params approximated by the number of dense layers in the initial network, and current params is the number of dense layers in the current network.
Each time the network is reset, we reinitialize the depth of the critic network back to 2, so the network can expand again.
B.3. Other Implementation Details
Small actor network Previous work (Lee et al., 2024; Nauman et al., 2024b) has shown that scaling up the actor network only provides marginal improvements in performance. To simplify our framework, we did not impose any special characteristics or constraints on the actor network or optimizer beyond what is typically done in standard SAC implementations. Specifically, we used the simplest MLP architecture and the Adam optimizer, which also provides a solid foundation for deploying the actor.
14

---

A Forwet-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
About OBAC implementation In the OBAC algorithm, an offline agent is used to improve the online agent. Specifically, the offline agent adds constraint the online actor, encouraging it to learn from the offline agent when the Q-value estimate of the online actor is lower than that of the offline agent. In our experiments, we found that after reset during training, the offline agent could quickly gain an advantage over the online actor by better utilizing the information in the buffer. Allowing the actor to immediately learn from the offline agent could lead to early convergence. To address this, we introduced "protection period"' for the online actor, during which the OBAC algorithm is temporarily disabled after a reset. This period, which we call OBAC wail, was found to yield fine results when set 250k network iterations all cases. Thus, we use 250k the default in all experiments.
B.4. Is Forget and Grow a Universal Technique?
In the FoG algorithm, we use OBAC as the foundational framework, combined with scaling the network size and replay ratio. A natural question arises: is the Forget and Grow technique universally applicable?
We tested SAC in the MuJoCo environment and observed the following: even with the standard SAC algorithm a replay ratio of 1, using ER decay "forget" technique led to stable performance improvements. However, network expansion required a larger replay ratio to achieve relatively significant effects. This may be due to the fact that the newly introduced parameters in network expansion require more intense updates before they become effective.
Ant-v4
HallCheetah-v4
SAC
SAC with butte decyy
Return
Return
6000
500k steps
1000K
1000K
steps
Humanoid-v4
Walker2d-v4
5000
Return
Retur
2500
0
1000k
BOOK steps
1000k
steps
Figure 13. The results of SAC and SAC with ER decay 4 tasks in Mujoco.
We also evaluated the data efficiency of the FoG-enhanced SAC (FoG-SAC) algorithm in both DMC-Hard and MuJoCo environments. We found that FoG-SAC achieved data efficiency close that of BRO, showing significant improvement in the later stages of training. However, it did not outperform BRO, which had undergone other algorithmic adjustments.
We also tested FoG with BRO, but FoG-BRO yielded performance very similar to that of the original BRO. This suggests that some of the adjustments in BRO may conflict with the FoG mechanism.
Overall, our experiments indicate that FoG integrates well with the native SAC and OBAC algorithms. However, its effectiveness when combined with other existing algorithms warrants further investigation.
B.5. Baselines and Environments
We compare FoG with BRO, SimBa and TD-MPC2, we use their official implementations, hyperparameters and results to ensure fair comparison.
15

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Sample Efficiency Compared
dog-walk
humanoid-wall
1000
1000
Normalized Score
Fud-BAC
Return
Return
500
500
500
500k Steps
1M
500K steps
1000K
500K steps
1000k
Figure 14. The results of FoG-SAC
1. BRO: We use the official implementation from http://iiitcccoaaamm///ggggeeele
2. SimBa: We use SimBa-SAC from official implementation and results from http://giiialbcoommSSyyyyeeeeeee
3. TD-MPC2: We use the official implementation and results from http:://gghiub..oommiiiciaaaaaaa
4. SAC: We use implementation from http://ghhhhb.ccmm/ooooouuaaaahh replay ratio .ootinnuuuu ratio_contiuuuss
control..
We use the official setting of each task domain, including the reward setting, the task horizon, the done signal and there original state-action spaces.
B.6. Official Implementation of FoG
Please check http://gghubbbooomooottnnbbbbbbu for official implementation of FoG.
C. Complete Experimental Results
To show the superiority of FoG, we list all the experimental results in this section.
C.1. Numerical Results
Table Normalized Score benchmarks
Environment
FoG
TD-MPC2
SimBa
BRO
Mujoco
0.96 + 0.02
0.07 + 0.00
0.58 + 0.03
0.85 + 0.04
DMC-Fasy
0.99 + 0.00
0.73 + 0.01
0.74 + 0.04
0.90 + 0.11
DMC-Medium
0.85 + 0.05
0.65 0.03
0.63 0.02
0.79 + 0.04
DMC-Hard
0.99 + 0.01
0.50 + 0.02
0.75 0.01
0.76 0.02
MetaWorid
0.99 +-2 0.00
0.90 + 0.08
0.64
0.96 + 0.04
HumanoidBench
0.82 + 0.02
0.81 + 0.02
0.73 0.04
0.54 0.01
Total
0.92 + 0.01
0.70 + 0.02
0.69 0.02
0.76 + 0.02
16

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Table Returns 600k steps from Mujoco tasks
Method
FoG
TD-MPC2
SimBa
BRO
Ant HalfCheetah
6979.19 + 98.06
563.04 + 54.12
4827.24 + 148.80
6798.11 + 237.61
15119.91 + 583,41
2445.42 + 48.43
10393.10 656.98
13765.24 712.18
Humanoid
7170.79 + 24.58
282.40 + 25.21
2803.31 + 290.75
6038.88 + 918.07
Walker2d
5310.89 + 320.60
93.38 + 66.43
3365.35 + 695.81
4223.97 + 457.07
Table 4. Returns of 150k steps from DMC-Easy tasks
Method
FoG
TD-MPC2
SimBa
BRO
Cartpole Balance
999.62 0.08
997.77 = 0.58
999.64 + 0.17
999.70 + 0.22
Cartpole Swingup
879.58 0.43
839.40 = 35.20
872.01 + 8.37
878.20 + 0.88
finger-spin
983.07 = 4.49
947.50 = 27.89
703.57 + 124.98
941.23 + 20.54
Hopper Stand
922.97 + 14.54
28.17 + 28.21
237.10 = 106.32
604.57 416.54
Table 5.
Returns of 500k steps from DMC-Medium tasks
Method
FoG
TD-MPC2
SimBa
BRO
Acrobot Swingup
423.88 + 21.81
368.27 + 39.63
377.23 + 28.29
509.34 = 38.23
Hopper Hop
409.42 119.77
285.43 + 56.43
278.30 + 2.66
288.40 +:: 9.97
Humanoid Stand
866.60 + 27.14
401.87 = 40.85
402.85 = 46.63
769.53 = 131.23
Walker Run
820.02 + 2.45
818.07 + 7.18
753.80 + 10.10
760.53 + 18.42
Table 6. Returns of 1000k steps from DMC-Hard tasks
Method
FoG
TD-MPC2
SimBa
BRO
Dog Run
652.78 + 7.02
183.13 + 14.75
534.02 + 16.77
479.68 + 12.26
Dog Trot
911.62 + 7.98
423.13 +: 42.88
856.32 +: 26.84
842.09 + 51.18
Dog Walk
954.09 + 6.14
719.83 + 55.70
924.67 + 8.60
948.46 + 4.95
Humanoid Run
436.34 + 10.87
178.17 = 2.93
177.10 + 9.81
235.07 + 48.55
Humanoid Walk
932.03 = 6.96
572.10 = 14.33
609.73 36.53
609.56 + 3.05
17

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Table Success rates of 1000k steps from Metaworld tasks
Method
FoG
TD-MPC2
SimBa
BRO
Assembly
1.00 = 0.00
0.67 0.47
0,70 = 0.42
1.00 = 0.00
Coffee pull
1.00 0.00
1.00
0.00
1.00 0.00
0.93 = 0.09
Coffee push
0.93 + 0.05
1.00 0.00
0.97 = 0.05
0.87 + 0.19
Disassemble
1.00 0.00
0.67 0.47
1.00 + 0.00
1.00 + 0.00
Pick out of hole
1.00 = 0.00
1.00
0.00
1.00 = 0.00
1.00 = 0.00
Pick place
1.00 = 0.00
1.00 = 0.00
1.00 = 0.00
1.00 = 0.00
Pick place wall
1.00 + 0.00
1.00 0.00
0.00 + 0.00
0.80 + 0.28
Push back
1.00 - 0.00
0.67 0.47
0.67 = 0.47
1.00 = 0.00
Shelf place
1.00 0.00
1.00
0.00
0.10 + 0.14
1.00 1 0.00
Stick push
1.00 0.00
1.00 0.00
0.00 0.00
1.00 0.00
Table 8. Returns of 1000k steps from HumanoidBench tasks
Method
FoG
TD-MPC2
SimBa
BRO
Balance Hard
72.84 + 1.82
61.23 + 2.83
79.87 9.15
62.10 2.488
Balance Simple
546.66 + 66.01
52.88 = 8.01
256.46 + 135.38
68.77 5.11
Crawl
971.02 +: 0.65
963.36 +: 2.48
939.58 + 18.65
897.50 = 31.20
Hurdle
86.31 + 7.19
363.48 + 34.69
208.65 = 7.76
48.50 0.65
Maze
380.13 5.53
323.64 4.12
389.39 19.61
273.00 = =6.73
Pole
817.70 = 73.53
647.96 = 172.64
754.56 4.06
340.60 = 24.14
Reach
3963.71 + 289.43
3913.07 740.94
4418.50 395.17
3984.43 236.95
Run
437.49 + 49.97
780.80 3.18
262.65 73.32
49.33 + 12.19
Sit Hard
814.95 = 12.93
749.94 = 14.30
667.38 = 206.50
829.10 + 4.34
Sit Simple
843.99 + 8.91
801.01 + 1.15
860.78 + 5.00
850.03 + 3.88
Slide
396.74 + 25.44
329.21 + 26.01
270.43 17.06
250.10 1 7.94
Stair
386.91 + 117.06
562.50 + 24.62
226.79 + 162.50
77.57 0.70
Stand
812.80 = 2.95
845.22 = 9.60
799.73 = 16.74
Walk
842.72 8.53
813.88 + 1.38
619.36 200.68
186.17 27.30
C.2. Learning Curves
One thing to notice about DMC-easy tasks Figure 16 is that the starting point of the learning curve is 25k steps, which because the first evaluation step of BRO at 25k steps. At this time, FoGhas already achieved convergence in some environments, like cartpole-swingup.
Ant
HalfCheetah
Humanoid
Walker2d
16000
7000
7000
5000
Return
Return
Return
Return
3500
8000
3500
2500
300k steps
600k
300K steps
6001
300k steps
600k
300k steps
600k
FoG
TD-MPC2
SimBa
BRO
Figure 15. The results of tasks in Mujoco.
18

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Cartpole Balance
Cartpole Swingup
Finger Spin
Hopper Stand
Return
Return
Return
Return
500
500
500
500
25k
150k
25k
150k
25k
150k
25k
150k
steps
steps
steps
steps
FoG
TD-MPC2
SimBa
BRO
Figure 16. The results of 4 tasks in DM Control Easy.
Acrobot Swingup
Hopper Hop
Humanoid Stand
Walker Run
1000
1000
Return
Return
Return
Return
500
500
500
500
250k steps
500k
250k steps
500k
250k
500K
0
250k steps
500k
FoG
TD-MPC2
SimBa
BRO
Figure 17. The results of 4 tasks in DM Control Medium.
Dog Run
Dog Trot
Dog Walk
Humanoid Run
Return
Return
Return
Return
500
500
500
500
0
steps
steps
steps
steps
Humanoid Walk
1000
Return
500
0
500k steps
1000k
FoG
TD-MPC2
SimBa
BRO
Figure 18. The results of 5 tasks in DM Control Hard.
19

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Assembly
Coffee pull
Coffee push
Disassemble
1.0
1.0
Success Rate
Success Rate
Success Rate
Success Rate
500k
1000k
500K
1000k
500k
1000k
500k
1000k
steps
steps
steps
steps
Pick out of hole
Pick place
Pick place wall
Push back
1.0
1.0
Success Rate
Success Rate
Success Rate
Success Rate
0.5
0.5
0.5
500k steps
1000k
500k
1000k
500k steps
1000k
500k steps
1000k
steps
Shelf place
Stick push
Success Rate
Success Rate
0.5
0.5
500k steps
1000k
steps
FoG
TD-MPC2
SimBa
BRO
Figure results of tasks Meta-World.
20

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Balance Hard
Balance Simple
Crawl
Hurdle
Return
Return
Retum
Return
500
500
500
500
0
steps
steps
steps
steps
Maze
Pole
Reach
Run
1000
1000
.500
1000
Return
Return
Return
Return
500
500
2500
500
0
S00k
1000k
500k steps
1000k
500k steps Slide
1000k
500k steps Stair
1000k
steps
Sit Hard
Sit Simple
1000
1000
1000
Return
Return
Return
Return
500
500
500
1000k
1000k
steps
steps
steps
steps
Stand
Walk
1000
1000
Return
Return
500
500
500k steps
1000k
500k steps
1000k
TD-MPC2
SimBa
Figure 20. The results of 14 tasks HumanoidBench.
D. Performance comparison under similar computation cost
To demonstrate that the performance gains of FoG not merely due increased computation, we provide performance comparison under similar computational costs.
We evaluate SimBa (depth=10, approximately 42M parameters), TD-MPC2 (19M version), and BRO (depth=10, approxi- mately 42M parameters) on the humanoid benchmark to compare their performance against FoG, despite all of them having more parameters than FoG (at most 21M). Our experiments demonstrate that FoG outperforms competitive baselines even when compared to larger models, achieving superior performance while using less computation in this setup.
21

---

A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
h1-run
h1-walk
1000
TD-MPG2
TD MD-MPC2
Retum
FoG
Retum
FeG
500K steps
1M
500K steps
1M
Figure 21. The results under similar computation cost
E. Comparison to similar methods
We conducted a comparative analysis between FoG and the Neuroplastic Expansion (NE) algorithm(Liuu et al., 2024), utilizing the official NE implementation to ensure faimesss. The evaluation was performed across four diverse tasks: HalfCheetah-v4, dog-run, dog-walk, and humanoid-walk. Both the original NE model and variant with expanded capacity were tested to examine the impact model scaling on performance.
The results consistently demonstrate that FoG outperforms NE across all tasks, even when operating at relatively low update-to-data ratios. Notably, increasing the capacity of the NE model yields only marginal performance gains, whereas FoG maintains robust improvements without requiring significant model enlargement. These findings underscore FoG's superior scalability and adaptability, establishing it as a more effective approach for handling complex reinforcement learning environments.
dag-run
HalfCheetah-v4
1000
Larg))))
14000
NE/Smal0
Return
Return
500
7000
1M
500k steps
steps
dog-walk
humanoid-walk
1000
1000
Retur
Return
500
0
500k steps
1M
500K steps
1M
Figure 22, Comparison to Neuroplastic Expansion method
22


Reviews:
[{"review_id": "wT3CmIpbCB", "reviewer": "Reviewer_TdFu", "summary": "This paper focuses on the problem of sample efficiency in deep reinforcement learning. The authors introduce the Forget and Grow (FoG) method, which relies on three ideas: 1) reducing the sampling probability of older samples, 2) expanding the network size, and 3) resetting the network with a certain frequency. The results suggest that FoG outperforms multiple performant methods, namely Simba, BroNet, and TDMPC2 in a number of environments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort you have dedicated to reviewing our work. We deeply appreciate your careful and thorough review. In the following, we seek to address each of your concerns.\n___\n**Q1:** *\"The comparison against other methods may be unfair since FoG uses increased computation while the other does not.\"*\n\n**A:** Our experiments on the humanoid benchmark (h1-walk & h1-run) demonstrate that FoG outperforms competitive baselines even when compared to larger models. We tested SimBa (depth=10, ~42M params), TD-MPC2 (19M version), and BRO (depth=10, ~42M params)—all of which are larger than FoG (at most 21M params). Despite this, FoG achieves superior performance while using less computation in this setup.\n\nMoreover, TD-MPC2 and BRO fail to improve over their default sizes, and only SimBa shows an improvement in h1-walk but not in h1-run, indicating that comparing FoG to these baselines at their default sizes is fair. This further highlights FoG’s ability to efficiently manage a larger number of parameters.\n\nFor additional insights, please refer to the training curve figure:https://anonymous.4open.science/r/ICML-full_size-25B6/README.md.\n___\n**Q2:** *\"It’s not clear what defines old and new experience\"*\n\n**A:** The distinction between old and new experience is based on the time at which the data was collected and added to the replay buffer. The earlier the data was collected and stored in the buffer, the older it is. ER Decay allows agents to focus on ~100k transitions that's newly collected to buffer. We will provide a clearer explanation in the paper.\n___\n**Q3:** *\" An experiment is needed to show how FoG performs without resets.\"*\n\n**A:** In the scenario without resets, FoG still outperforms SimBa and BRO with the maximum network size (~42M) on humanoid-run, humanoid-walk, and HalfCheetah-v4, while using less computation. However, removing resets hinders FoG’s performance, showing a clear drop compared to our original version. The reset mechanism was introduced to increase the replay ratio, allowing better utilization of data collected, and hence maximizing data efficiency.\n\nFor details, refer to the training curve figure:  https://anonymous.4open.science/r/ICML-no_resets-76A1/README.md.\n___\n**Q4:** *\"the paper misses comparison against similar methods.\"*\n\n**A:** We currently do not have access to the code from the referenced paper. However, as demonstrated in their work, we also observe that naively expanding network capacity significantly improves performance in many Gym environments, especially HalfCheetah. Yet, this success does not easily extend to more challenging environments like DMC, MetaWorld, or the humanoid benchmark. While increasing the replay ratio enhances sample efficiency, this approach alone is not sufficient to prevent early convergence.\n___\n**Q5:** *\"using the critic loss for comparison in Figure 3 and Figure 5 is problematic\"*\n\n**A:** Your point is valid, relying solely on critic loss is insufficient. However, SAC's critic loss in this experiment obviously exceeds reasonable thresholds. Such abnormally large errors can destabilize actor gradients and impede training. \n\nTo further substantiate our claim, we tracked the ratio of activated neurons[1]. FoG does indeed increase the ratio of activated neurons. This indicates that the model can effectively utilizes more of its capacity during training, achieving better learning efficiency[2].\n\nFor the dormant ratio curves, please refer to the following link: https://anonymous.4open.science/r/ICML-dormant-E41C/README.md.\n\n[1]:Liu, J., Obando-Ceron, J., Courville, A., & Pan, L. (2024). Neuroplastic Expansion in Deep Reinforcement Learning. arXiv preprint arXiv:2410.07994.\n[2]:Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., ... & Xu, H. (2023). Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668.\n___\n**Q6:** *\"the experiments are conducted with only three independent runs (seeds), which is a very small number of runs. \"*\n\n**A:** We are currently running 16 tasks for the main result in Figure 7, with each task using 10 seeds. However, due to time constraints, we have not yet obtained the results. We will release the results within one week.\n\nCurrently we have finished with 4 tasks, please check https://drive.google.com/drive/folders/14u4Ag3MrD9GVl4pib5RneSo6n5WGeQO_ for training curves\n___\n**Q7:** *\"the parallel with infantile amnesia needs to be de-emphasized in the paper\"*\n\n**A:** Your point is valid. FoG lacks neural pruning, unlike the brain's dual process. However, the analogy with infantile amnesia stemmed from its link to neurogenesis-disrupted memory connections, mirroring network expansion (supported by our neuron activation experiments). We will de-emphasize this parallel in revisions and clarify to prevent misinterpretation.", "claims_and_evidence": "While some of the claims are supported with evidence, many are problematic. Here I focus on a few:\n- The comparison against other methods may be unfair since FoG uses increased computation while the other does not. One approach to improve the empirical evaluation is to use other methods that use the maximum network size used by FoG.\n- It’s not clear what defines old and new experience, so it is hard to just make claims about deemphasizing the probability of old data in favor of new data.\n- The authors focus on the two mechanisms they propose: experiencing replay decay and growing the network. However, there is a missing mechanism which is reset. Resetting is part of FoG, but it is not emphasized as the other new mechanisms, which might be confusing. An experiment is needed to show how FoG performs without resets.", "methods_and_evaluation_criteria": "- The paper uses standard benchmarking tasks that are well-accepted to study the problem of sample efficiency in deep RL methods.\n- However, the paper misses comparison against similar methods. For example, the paper “Neuroplastic Expansion in Deep Reinforcement Learning” by Jiashun Liu, Johan Obando-Ceron, Aaron Courville, and Ling Pan is based on growing the network along with some experience replay techniques.\n- Additionally,  it’s not clear that the evaluation is fair if FoG uses much more computation than other methods. \n- Additionally, using the critic loss for comparison in Figure 3 and Figure 5 is problematic since the loss doesn’t necessarily represent any meaningful results. For example, a method that gives zero-value function prediction everywhere would achieve the maximum score based on the metric you’re proposing, which shows that it’s problematic.\n- Finally, the experiments are conducted with only three independent runs (seeds), which is a very small number of runs. Many of the figures have overlapping confidence intervals, so their statistical significance may be compromised.", "theoretical_claims": "No theoretical claims are presented in the paper apart from the ones used to motivate intuition.", "experimental_designs_or_analyses": "Check the Methods And Evaluation Criteria section.", "relation_to_broader_scientific_literature": "The paper considers an important research question about sample-efficient deep RL methods, which is of interest to many people in the research community.", "essential_references_not_discussed": "The paper “Neuroplastic Expansion in Deep Reinforcement Learning” by Jiashun Liu, Johan Obando-Ceron, Aaron Courville, and Ling Pan is based on growing the network along with some experience replay techniques.", "other_strengths_and_weaknesses": "The paper considers a fundamental problem in deep reinforcement learning. The idea seems exciting but evaluation is lacking rigor. Overall, I would like the paper to be published, but I think it’s not ready in its current format to be published in ICLR 2025, and thus my recommendation is to reject it.\n\nAdditionally, the parallel with infantile amnesia needs to be de-emphasized in the paper since it doesn’t match the natural phenomenon. The brain doesn’t only contain neurogenesis but also neural pruning, which breaks the parallel since FoG does not prune its network.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How does FoG perform without resetting? \n- How does FoG perform to the Neuroplastic Expansion (NE) method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mWUsYZGFYq", "reviewer": "Reviewer_ttxg", "summary": "The paper addresses the challenge of Deep Reinforcement Learning (DRL) in continuous control tasks, where models suffer from primacy bias, overfitting to older memories in the replay buffer. Drawing inspiration from infantile amnesia in humans, the authors propose two modifications: (1) a decaying replay buffer to mitigate overfitting to older memories, and (2) an increase in network size to enhance continuous learning. These modifications are empirically validated across 41 control tasks, demonstrating their effectiveness in improving learning performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort you have dedicated to reviewing our work. We deeply appreciate your careful and thorough review. In the following, we seek to address each of your concerns.\n___\n**Q1:** *\"The authors should analyze the learned representations and their evolution under the proposed modifications (decaying replay buffer and network expansion)..\"*\n\n**A:** We tracked the ratio of activated neurons during training, and FoG does indeed increase the ratio of activated neurons. This indicates that the model effectively utilizes more of its capacity during training.\n\nFor the training curve, please refer to the following link: https://anonymous.4open.science/r/ICML-dormant-E41C/README.md.\n\nDue to time constraints, we cannot complete the comparison across all 41 tasks immediately, but we will conduct it in the future and add the results to the appendix.\n___\n**Q2:** *\" The authors do not provide a representation analysis to explain how the expanded network adapts to new memories without interfering with older ones.\"*\n\n**A:** We analyze representations via replay buffer samples processed by the critic's last layer for t-SNE visualization, Fixed 2/4 blocks vs. expanded networks. Network expansion shows clearer clustering and better feature separation/structure.\n\nFor the visualization results, please refer to: https://anonymous.4open.science/r/ICML-t_SNE-D076/README.md.\n\nDue to time constraints, we currently provide results for HalfCheetah-v4 at 200k steps, but will release more within a week.\n___\n**Q3:** *\"An inconsistency arises in Figure 3, where the critic loss for Normal SAC is significantly higher than for other models, yet it still achieves decent task performance.\"*\n\n**A:** We sincerely apologize for the color mislabeling of SAC and SAC with reset in the left-bottom plot of Figure 3. However, this does not affect the validity of our experimental results.\n\nEven though SAC performs reasonably well, it still performs the worst among all variants, achieving less than half the performance of Expanded-SAC. Its decent early-stage performance stems from the effective use of initial data, but its reward growth stagnates later due to poor utilization of new data.\n\nAdditionally, SAC shows low loss for old data but significantly higher loss for new data, supporting our claim that standard experience buffers overfit to early memories, hindering new learning.\n___\n**Q4:** *\"However, it does not discuss recent advancements in noise-based regularization techniques.\"*\n\n**A:** These techniques are indeed effective in mitigating plasticity loss. However, they are orthogonal to the ER Decay and network expansion proposed in our work, and their combined effects remain an open question for future exploration. While FoG does not incorporate these techniques, we will still discuss them in the related works section to provide a more comprehensive perspective.\n___\n**Q5:** *\"The number of random seeds used for experiments should be explicitly stated in all figure captions, and error bars should be included in bar graphs e.g. Fig. 1.\"*\n\n**A:** We are currently running 16 tasks for the main result in Figure 7, with each task using 10 seeds. However, due to time constraints, we have not yet obtained the complete results. We will release the results within one week.\n\nCurrently we have finished with 4 tasks, please check https://drive.google.com/drive/folders/14u4Ag3MrD9GVl4pib5RneSo6n5WGeQO_ for training curves.\n___\n**Q6:** *\"The title's use of \"continuous control\" may be misleading\"*\n\n**A:** Thank you for the suggestion. However, we have only evaluated FoG on continuous control benchmarks and have not yet explored its performance on discrete control tasks. We primarily followed BRO and SimBa, which also focus on continuous control.\n___\n**Q7:** *\"The term \"scaling\" in the title might be misinterpreted\"*\n\n**A:** Thank you for raising this important point. We acknowledge that the term \"scaling\" in the title could indeed be conflated with scaling laws in machine learning (ML), and we will revise the title in subsequent versions to use clearer phrasing, such as \"Scaling Up Parameters\" or \"Bigger Models,\" to explicitly distinguish our focus.\n___\n**Q8:** *\"The consistent dips in reward curves in Figure 8 require further explanation\"*\n\n**A:** The dips in Figure 8 occur when the network is reset, requiring it to relearn from the replay buffer, causing a temporary drop before recovery.\n___\n**Q9:** *\"Clarify the difference between Expanded SAC (Figure 3) and FoG.\"*\n\n**A:** The difference between Expanded SAC and FoG lies primarily in the use of the OBAC backbone and some slight modifications to the network structure in FoG. We will provide a clearer and more detailed explanation of these differences in the paper and will also modify the names of these SAC variants to make them easier to understand.", "claims_and_evidence": "- The authors claim that standard experience buffers cause models to overfit to older memories, hindering the learning of new information. This is supported by a critic loss heat map, which visualizes the loss in plasticity over time. However, a more comprehensive comparison across all 41 tasks would strengthen this claim. The analysis can be placed in the appendix.\n\n- The claim that increasing the critic network size improves learning is supported by empirical ablation studies. However, the authors do not provide a representation analysis to explain how the expanded network adapts to new memories without interfering with older ones. For instance, does adding new neural blocks increase interference within the network and noise, hence supporting new learning? Such an analysis would offer deeper insights into the mechanism behind this improvement.", "methods_and_evaluation_criteria": "The critic loss heat map effectively illustrates the loss in plasticity as training progresses. However, additional metrics, such as tracking parameter changes across learning steps (e.g., Kumar et al. 2024, bioRxiv 2024.12.12.627755 Supp. Fig. 4), could provide a more robust evaluation of plasticity loss and the efficacy of the proposed strategies.", "theoretical_claims": "The theoretical claims regarding oversampling older memories appear correct and are well-supported.", "experimental_designs_or_analyses": "- An inconsistency arises in Figure 3, where the critic loss for Normal SAC is significantly higher than for other models, yet it still achieves decent task performance. This discrepancy warrants further explanation.\n\n- The authors should analyze the learned representations and their evolution under the proposed modifications (decaying replay buffer and network expansion). Empirical results alone are insufficient to fully validate the mechanisms in improving performance.", "relation_to_broader_scientific_literature": "The paper draws an interesting analogy between infantile amnesia in humans and primacy bias in DRL. However, it does not discuss recent advancements in noise-based regularization techniques, such as those proposed by Dohare et al. (2024, Nature) and Kumar et al. (2024, bioRxiv), which could offer additional insights into continual learning and flexibility in RL models.", "essential_references_not_discussed": "- Noise-based regularization techniques, such as reinitializing unused parameters with noise (Dohare et al. 2024, Nature), could enhance continual learning in RL models.\n- The hypothesis that noise regularization allows networks to explore degenerate solution spaces and escape local minima (Kumar et al. 2024, bioRxiv) is relevant but not discussed.", "other_strengths_and_weaknesses": "- Strengths: The paper presents a novel approach to addressing primacy bias in DRL, with empirical validation across a wide range of tasks.\n- Weaknesses: The number of random seeds used for experiments should be explicitly stated in all figure captions, and error bars should be included in bar graphs e.g. Fig. 1. Increasing the number of random seeds to 10, particularly for Figure 7, would improve the robustness of the results.", "other_comments_or_suggestions": "- The title's use of \"continuous control\" may be misleading, as the framework could potentially apply to discrete control tasks as well.\n- The term \"scaling\" in the title might be misinterpreted as referring to scaling laws in machine learning.\n- The consistent dips in reward curves in Figure 8 require further explanation.\n- Clarify the difference between Expanded SAC (Figure 3) and FoG.", "questions_for_authors": "- Can the authors perform a representation analysis to elucidate how block expansion facilitates learning without interfering with older memories?\n0 Could the authors discuss the potential benefits of injecting noise into network parameters, as suggested by recent literature?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8a33U8FkHJ", "reviewer": "Reviewer_LRoZ", "summary": "The paper draws inspiration from the phenomenon of infantile amnesia in neuroscience, proposing a \"forget and grow\" mechanism to mitigate primacy bias in deep reinforcement learning. The authors identify limitations in existing reset mechanisms and introduce two novel strategies: Experience Replay Decay (ER Decay), which gradually reduces the influence of older experiences, and Network Expansion, which introduces new neurons early in training to facilitate adaptation. These methods are integrated into a new algorithm, Forget-and-Grow (FoG). Empirical results across multiple benchmarks demonstrate FoG's superiority over existing methods, establishing it as a competitive approach in sample-efficient deep reinforcement learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort you have dedicated to reviewing our work. We deeply appreciate your careful and thorough review. In the following, we seek to address each of your concerns.\n___\n**Q1** *\"the evidence provided—particularly Figure 10—seems inadequate\"*\n\n**A:** We tracked the ratio of activated neurons during training, and FoG does indeed increase the ratio of activated neurons. This indicates that the model effectively utilizes more of its capacity during training.\n\nTraining curves available here: https://anonymous.4open.science/r/ICML-dormant-E41C/README.md.\n\nMoreover, we analyze representations via replay buffer samples processed by the critic's last layer for t-SNE visualization, Fixed 2/4 blocks vs. expanded networks. Network expansion shows clearer clustering and better feature separation/structure.\n\nVisualization results available here: https://anonymous.4open.science/r/ICML-t_SNE-D076/README.md.\n___\n**Q2:** *\"A natural baseline to consider would be a smaller replay buffer—does it lead to similar on-policyness effects and performance improvements? \"*\n\n**A:** We appreciate the suggestion to compare ER Decay with a smaller replay buffer. We conducted experiments on SAC with only modifications of replay buffer on 3 tasks. The results show that SAC with ER Decay significantly outperforms those with a smaller buffer size ranging from 5e4 to 5e5.\n\nWhile ER Decay introduces more \"on-policy\" characteristics, it also retains older transitions, preventing forgetting and improving final performance.\n\nTraining curves available here: https://anonymous.4open.science/r/ICML-buffer_size-5FB0/README.md.\n___\n**Q3:** *\"the comparison with plasticity injection appears somewhat misleading\"* \n\n**A:** Your concern is valid—we acknowledge potential imprecision. By \"complex,\" we referred to implementation complexity (parameter freezing, residual construction) versus FoG's direct parameter addition without tricks. We will clarify this distinction and refine descriptions in the revised paper.\n___\n**Q4:** *\"The current discussion does not sufficiently situate the proposed network expansion mechanism within this broader landscape. \"*\n\n**A:** We have compared our network expansion mechanism with three related works:\n\n1) **Neuroplastic Expansion in Deep Reinforcement Learning**  \n     \n  This paper presents a clever and effective approach by utilizing sparse networks and expanding through network topology adjustments. Our method, better suited for high replay ratios and larger networks, is easier to implement. While we don't have their code, we observe that naively expanding network capacity boosts performance in Gym environments, especially HalfCheetah. However, this success doesn't easily extend to more complex environments.\n\n2) **Progressive Neural Networks & Dynamically Expandable Networks**  \n\n   Both methods are designed for multitask learning, expanding network width when new tasks are introduced, whereas our approach increases depth. Since the settings differ, direct performance comparison is not feasible, but we will discuss these methods in the related works section.\n___\n**Q5:** *\"the computational cost of increasing network depth needs further discussion\"*\n\n**A:** FoG's computation is about 4× that of BRO. However, our goal is to explore the limits of effective network scaling. On humanoid tasks, FoG outperforms larger models, including SimBa (10-layer, ~42M params), TD-MPC2 (19M params), and BRO (10-layer, ~42M params), despite having at most 21M params and lower computational cost. We will add a discussion on computational cost in the paper.\n\nTraining curves available here: https://anonymous.4open.science/r/ICML-full_size-25B6/README.md.\n___\n**Q6:** *\" the current training setup is complex\"*\n\n**A:** Both FoG's network structure and OBAC backbone play significant roles in improving performance. We conducted an ablation study to test FoG without OBAC(training curves: https://anonymous.4open.science/r/ICML-structure-B1DF/README.md). While FoG with SAC still gives strong performance, using OBAC backbone is helpful for both performance and parameter scaling.\n\nRegarding the setup, we already use a consistent OBAC wait period across experiments. The plasticity loss differs for each task, and we haven't yet explored adaptive reset or other methods like partial resetting or noise injection. However, we’ve minimized the number of reset lists to simplify the setup.\n___\n**Q7:** *\"It would be beneficial to incorporate relevant prior work\" \" I suggest introducing expansion earlier for better readability.\" \"The OBAC backbone is not widely adopted in the community\"  \"Figure resolution is too high\" \"In Related Work, SimBa should be described accurately\"*\n\n**A:** Thank you for the suggestions. We will add a dedicated section on network expansion in related works and make corresponding adjustments throughout the paper.", "claims_and_evidence": "The empirical results demonstrate that FoG performs well, but the underlying reasons behind the effectiveness of the forget-and-grow mechanism remain insufficiently explored. While I am not requesting a full theoretical analysis, the evidence provided—particularly Figure 10—seems inadequate. A deeper investigation into the effects of network expansion would be valuable for the community. For example, measuring various weight and gradient norms or incorporating plasticity metrics (e.g., dormant neurons) could provide better insights into what happens when new neurons are introduced.\n\nRegarding the forgetting mechanism, my interpretation is that ER Decay shifts learning dynamics toward on-policyness rather than purely off-policy. However, I am curious whether ER Decay is truly the key factor in achieving this effect. A natural baseline to consider would be a smaller replay buffer—does it lead to similar on-policyness effects and performance improvements? While I find the motivation for on-policyness convincing, as a practitioner, I would like to understand whether ER Decay is the best approach or if there are simpler alternatives with comparable benefits.\n\nFor the growth mechanism, the comparison with plasticity injection appears somewhat misleading. Plasticity injection was originally designed to diagnose plasticity loss, not to improve performance, so stating that it is \"complex\" seems unfair. Instead, this paper can be viewed as an extension of plasticity injection, applying it to intermediate layers rather than just a specific subset of the network. I recommend revising the framing of this comparison to reflect the relationship between these methods better. Additionally, after reviewing the appendix, it is unclear whether the authors expanded the network in depth or width—clarifying whether new layers were appended or if existing layers were widened would improve transparency.", "methods_and_evaluation_criteria": "The evaluation and comparisons in the paper are limited and require further depth.\n\nA crucial missing comparison is with the Plasticity Injection paper. Given its relevance, a direct empirical and conceptual comparison would strengthen the evaluation. Additionally, there are multiple established methods for expanding neural networks, such as Progressive Neural Networks (PNN) [1], Dynamically Expandable Networks (DEN) [2], and Neuroplastic Expansion (NE) [3] in reinforcement learning. The current discussion does not sufficiently situate the proposed network expansion mechanism within this broader landscape. A more comprehensive analysis is necessary to highlight the novelty and advantages of the approach compared to these existing methods.\n\nIn particular, the paper should address how FoG's expansion strategy differs from or improves upon these prior works. Does it offer advantages in terms of computational efficiency, stability, or sample efficiency? \n\nAdditionally, the computational cost of increasing network depth needs further discussion. Since deeper networks require more sequential computation, they may introduce significant overhead. It is important to compare the computational cost of FoG against BRO, SimBa, and TD-MPC, as these methods are widely used baselines.", "theoretical_claims": "I've checked the proof in the Appendix.", "experimental_designs_or_analyses": "- The experimental design could be improved by exploring different network growth strategies, including existing methods (e.g., PNN, DEN, NE), and analyzing learning dynamics using plasticity metrics (e.g., dormant neurons, gradient norms).\n\n- Additionally, the current training setup is complex, making it harder for the community to adopt. I suggest two possible improvements:\n   - Simplify the experimental framework: Instead of integrating forgetting and growing mechanisms with multiple training protocols, evaluating them within a simpler baseline (e.g., SimBa or TD-MPC2) would better isolate their contributions and improve clarity.\n   - Unify and streamline the training setup: The current reset protocol is overly detailed, using different reset lists for specific benchmarks and tasks (e.g., locomotion tasks in HumanoidBench vs. other environments). A more integrated and standardized reset mechanism, along with a consistent OBAC wait period (e.g., 250k iterations) across all experiments, would improve reproducibility and ease of adoption while maintaining strong performance.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "It would be beneficial to incorporate relevant prior work on progressive neural network architectures in continual learning and reinforcement learning, ensuring a more comprehensive discussion.\n\n- Progressive Neural Networks, Rusu et al, arXiv'16.\n- Lifelong learning with dynamically expandable networks, Yoon et al, ICLR'18.\n- PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning, Lee et al, NeurIPS'23.\n- Mixtures of Experts Unlock Parameter Scaling for Deep RL, Johan et al, ICML’24.\n- Neuroplastic Expansion in Deep Reinforcement Learning, Liu et al, ICLR'25.\n- Towards General-Purpose Model-Free Reinforcement Learning, Fujimoto et al, ICLR’25.", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "- In Section 3.1, the concept of expansion appears after the preliminary experiments, making it hard to follow. I suggest introducing expansion earlier for better readability.\n- The OBAC backbone is not widely adopted in the community, so a more self-contained explanation would help readers unfamiliar with it.\n- Figure resolution is too high, causing scrolling lag—reducing DPI would improve readability.\n- In Related Work, SimBa should be described accurately—it promotes simplicity bias rather than alleviating it to enable network scaling.\n\n\nI appreciate the idea and direction, but the paper needs more completeness. Strengthening connections to related work, providing a deeper analysis of growing effects, and offering a more thorough comparison with existing methods would improve its clarity and impact. I am open to increasing my score if these concerns are well addressed.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "442fhCq6k0", "reviewer": "Reviewer_6rRf", "summary": "Deep reinforcement learning suffers from primacy bias, a tendency to overfit early experiences stored in the replay buffer. This paper proposes Forget and Grow (FoG), a novel method with two new components: Experience Replay Decay (ER Decay) and Network Expansion. ER Decay gradually reduces the influence of early experiences, while Network Expansion dynamically adds new parameters during training. Comprehensive experiments on all kinds of continuous control tasks show that FoG outperforms the state-of-the-art methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort you have dedicated to reviewing our work. We deeply appreciate your careful and thorough review. In the following, we seek to address each of your concerns.\n___\n**Q1:** *\"The ER Decay in Figure 1 is not very clear. It looks like ER Decay will sample a transition more times than Normal ER, making me confused why it is called 'Decay'.\"*\n\n**A:** The \"sample times\" in Figure 1 represent relative values, not absolute ones, so they should only be used for horizontal comparison. The term \"Decay\" actually refers to the fact that the probability of sampling transitions gradually decreases as training progresses. When a transition first enters the replay buffer, it is considered new and has a higher sampling probability. However, as training continues and the data ages, the probability of sampling that transition decreases over time.\n___\n**Q2:** *\"There are large space in page 1 and 2, which can be reorganized to make the paper more compact.\"*\n**A:** Thank you for the suggestion. We will make the necessary adjustments in the revised version to make the paper more compact.\n___\n**Q3:** *\"ER decay gradually decreases the sampling weight of older transitions, how to understand Figure 4 that ER decay is lower than Normal ER at the beginning and then higher than Normal ER? What does the x-axis 'steps' mean, environment step? Does the curve represent the sample times of a fix transition?\"*\n\n**A:** The \"steps\" on the x-axis actually represent environment steps. The curve shows the total number of times that transitions collected at each environment step are sampled from the replay buffer during the entire training process. Since ER decay reduces the sampling weight of older data, the sample times for earlier transitions will be lower at first. However, as training continues, the sample times for later transitions will increase. This aligns with the goal of ER Decay: to ensure that transitions collected at different environment steps are sampled similarly, rather than favoring older transitions as in the case of normal ER. We will provide a clearer explanation of the axes in the paper.\n___\n**Q4:** *\"Any intuition for the design of the Network Expansion? Are there any alternative ways to expand the network?\"*\n\n**A:** The intuition behind the design of the Network Expansion is based on the concept of infantile amnesia, as mentioned in the introduction. We believe that as data increases, the network's capacity must be expanded to enhance its representation power. In our early experiments, we also found that naive network expansion could improve the agent's sample efficiency in certain environments. There are several alternative ways to expand the network, such as increasing the network's width (e.g., PNN[1], DEN[2]) or using sparse networks and adjusting the network topology (e.g., Neuroplastic Expansion in Deep Reinforcement Learning[3]). However, increasing the depth of the network is the most straightforward and simple approach. When designing our method, we aimed for an easy implementation that could be applied to other algorithms besides OBAC to boost performance, rather than being limited to our specific backbone.\n[1]: Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., ... & Hadsell, R. (2016). Progressive neural networks. arXiv preprint arXiv:1606.04671.\n[2]: Yoon, J., Yang, E., Lee, J., & Hwang, S. J. (2017). Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547.\n[3]: Liu, J., Obando-Ceron, J., Courville, A., & Pan, L. (2024). Neuroplastic Expansion in Deep Reinforcement Learning. arXiv preprint arXiv:2410.07994.\n___\n**Q5:** *\"line 37, Network Expansion introduces new neurons to the model early in training. Any implementation details about how to decide 'early in training'?\"*\n\n**A:** We expand our critic networks at the 50k-th and 250k-th iterations, as detailed in Appendix B.1. These points are considered early in the training process, especially given that the networks are updated up to 2 million times during training.\n___\n**Q6:** *\"What is the final size of the network after training compared to other methods?\"*\n\n**A:** The largest network in FoG has a depth of 4, resulting in 9 linear layers, each of size 512 x 512 (calculated as 4 * 2 + 1 = 9). With 9 such networks in OBAC, the total parameter count is approximately 21 million. This is about four times the size of both BRO and SimBa.", "claims_and_evidence": "Yes. The claims are supported by comprehensive experiments and ablation studies. The results are convincing.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The proofs are reasonable. I did not check them in detail.", "experimental_designs_or_analyses": "Yes. The experiments are comprehensive and well-designed. The ablation study is also convincing.", "relation_to_broader_scientific_literature": "The paper is related to the rl in continuous control tasks. The key contribution is to alleviate the primacy bias in deep reinforcement learning. The method is related to the experience replay and network expansion. The paper is well-positioned in the literature.", "essential_references_not_discussed": "No. The paper is well-positioned in the literature.", "other_strengths_and_weaknesses": "Strengths\n- The paper is well-written and well-organized. The method is simple and effective. The experiments are comprehensive and convincing.\n- Figures are beautiful and mostly informative. \n\nWeaknesses\n- The ER Decay in Figure 1 is not very clear. It looks like ER Decay will sample a transition more times than Normal ER, making me confused why it is called 'Decay'.", "other_comments_or_suggestions": "There are large space in page 1 and 2, which can be reorganized to make the paper more compact.", "questions_for_authors": "- ER decay gradually decreases the sampling weight of older transitions, how to understand Figure 4 that ER decay is lower than Normal ER at the beginning and then higher than Normal ER? What does the x-axis 'steps' mean, environment step? Does the curve represent the sample times of a fix transition?\n- Any intuition for the design of the Network Expansion? Are there any alternative ways to expand the network?\n- line 37, Network Expansion introduces new neurons to the model early in training. Any implementation details about how to decide 'early in training'?\n- What is the final size of the network after training compared to other methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an algorithm attacking the issue of primacy bias in RL inspired by neuroscience. The proposed techniques appear to be sound, but it is unclear exactly what the neuronal mechanism at play here is. Experiments show improvements over baselines, but with fewer random seeds than we would like. Additional desired baselines were provided during the rebuttal period. Therefore, I would recommend weak accept.

I would also encourage the authors to include an impact statement in their next draft.
Decision: Accept (poster)