Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
Authors: Roger Creus Castanyer
Johan Obando-Ceron
Lu Li
Pierre-Luc Bacon
Glen Berseth
Aaron Courville
Pablo Samuel Castro
Venue: NeurIPS 2025 spotlight ()
Keywords: Reinforcement learning
Deep reinforcement learning
Gradients
Residuals
Primary area: reinforcement_learning
TL;DR: We stabilize gradients for training increasingly deep reinforcement learning agents by using a second-order optimizer and residual connections

Abstract: Scaling deep reinforcement learning networks is challenging and often results in degraded performance, yet the root causes of this failure mode remain poorly understood. Several recent works have proposed mechanisms to address this, but they are often complex and fail to highlight the causes underlying this difficulty. In this work, we conduct a series of empirical analyses which suggest that the combination of non-stationarity with gradient pathologies, due to suboptimal architectural choices, underlie the challenges of scale. We propose a series of direct interventions that stabilize gradient flow, enabling robust performance across a range of network depths and widths. Our interventions are simple to implement and compatible with well-established algorithms, and result in an effective mechanism that enables strong performance even at large scales. We validate our findings on a variety of agents and suites of environments.

Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
Roger Creus Castanyer!2-
Johan Obando-Ceron
Lu Li
Pierre-Lue Bacon
Glen Berseth
Aaron Courville
Pablo Samuel Castro 1,2
Mila Québec Al Institute
Université de Montréal
Google DeepMind
Abstract
Scaling deep reinforcement learning networks is challenging and often results in degraded performance, yet the root causes of this failure mode remain poorly understood. Several recent works have proposed mechanisms to address this, but they are often complex and fail to highlight the causes underlying this difficulty. In this work, we conduct series of empirical analyses which suggest that the combination of non-stationarity with gradient pathologies, due to suboptimal architectural choices, underlie the challenges of seale. We propose a series of direct interventions that stabilize gradient flow, enabling robust performance across a range of network depths and widths. Our interventions are simple to implement and compatible with well-established algorithms, and result in an effective mechanism that enables strong performance even at large scales. We validate our findings on a variety of agents and suites of environments. Source code here.
We must be able to look at the world and see it as a dynamic process, not static picture." -David Bohm
Introduction
Recent advances in deep reinforcement learning (deep RL) have demonstrated the ability of deep neural networks to solve complex cecisionnmmaing tasks from robotics to game play and resource optimization [Mnih al., 2015, Vinyals et al., 2019, Bellemare al., 2020, Fawzi et al., 2022]. Motivated by successes in supervised and generative learning, recent works have explored scaling architectures in deep RL, showing gains in representation quality and generalization across tasks [Farebrother et al., 2023, Taiga et al., 2023]. However, scaling neural networks in deep RL remains fundamentally challenging [Ceron al., 2024b,a]. A central cause of this instability lies in the unique optimization challenges of RI. Unlike supervised learning, where data distributions are fixed, deep RL involves policy-dependen data that constantly change during training [Lyle et al., 2022]. Each update of the policy alters future states and rewards, making the training objective inherently non-stationary. Value-based methods exacerbate these issues bootstrapping, recursively using predicted values as targets.
time [Fujimoto et al., 2018], especially under sparse or delayed rewards [Zheng et al., 2018], leading to unstable updates, policy collapse, or value divergence [Van Hasselt et al., 2016. Lyle et al., 2023, 2024]. These dynamics are tightly coupled with archi- tectural vulnerabilities. Deep networks face well known pathologies such as vanishing/exploding gradients [Pascanu et al., 2013], ill-conditioned Jacobians [Pennington al., 2017], and activation saturation [Glorot and Bengio, 2010]. J. In deep RI,, these are magnified by the "deadly triad" [Sutton and Barto, 2018, Van Hasselt al., 2018), ool-py corrections, and changing targets. As networks
Estimation errors compound
Equaal contribution. Correspondence to: Roger Creus C <roger wrroger..reuueeeeeeee creuus-casttnnyerdmilaaa queeeeeee......uu.............. quebec>, Johan Obando-Ceron < jobando07300gmail. com>, Pablo Samuel Castro <psc@google.com com>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

scale, the risk of signal distortion and misalignment increases, resulting in underutilized capacity and brittle learning [Obando Ceron et al., 2023, Ceron et al., 2024a].
One overlooked source of these failures lies in how gradients propagate through the network. Specifi- cally, the gradient decomposition, the layer-wise structure of backpropagation and weights determine how information flows during learning [Lee et al., 2020]. While gradient signal preservation has been studied in supervised learning [Schoenholz et al., 2017, Jacot al., 2018]. its role in deep RL, where both inputs and targets shift continually, remains poorly understood.
chain of Jacobians
In this work, investigate how gradient decomposition interacts with non-stationarity and network scaling in deep RL. We demonstrate that in non-stationary settings like RL where targets are bootstrapped, policies evolve continually. and data distributions shift gradient signals progressively degrade across depth. This motivates the need for methods that explicitly preserve the structure of gradient information across layers. We explore this through series of controlled experiments and ablations across multiple algorithms and environments, demonstrating that actively encouraging gradient propagation significantly improves stability and performance, even with large networks. Our work offers a promising approach for scaling deep RL architectures, yielding substantial performance gains across variety of agents and training regimes.
2 Preliminaries
Deep Reinforcement Learning A deep reinforcement learning agent interacts with an environment through sequences of actions (a A)), which produce corresponding sequences of observations (s E S) and rewards (r E IR), resulting in trajectories of the form т := {so. Go. ro, $1, a1 ... The agent's behavior often represented by a neural network with parameters 0, composed of where is fed
convolutional layers (th1, ,5..,,,,)))) and dense (fully connected) layers 1/,,......... vrr,
Vr,,, has an output dimensionality of [A]. At every timestep 4, an observation 8, through the network to obtain an estimate of the long-term value of each action: Qo(se,) L. (1... (Ør, (... (01 (sr)) .).. The agent's 'ss policy Te(- st)) specifies the probability
of selecting each action, for instance by taking the softmax over the estimated values in Eq. 1. The training objective is typically defined as the maximization of expected cumulative reward as in Eq. 2,
Qa((ssa))
To (at 8t)
(1)
((0) = B,
A
(2)
EaEA Qu(or.a)
where y E [0, 1) is a discount factor and r denotes a trajectory generated by following policy wa- Optimization proceeds by minimizing surrogate loss L(0), which may be derived from temporal- difference (TD) errors, policy gradients, or actor-critic estimators [Sutton and Barto, 2018]. In TD-based methods, the TD error at timestep l is defined as:
8, = rt + (Vo(s++1) Vo(st).
where Vo(s) [al) Qo(s, a). The recurrent nature of introduces dependencies on both current estimates and future rewards, making L(0) inherently non-stationary. As the policy ma evolves, the data distribution used for training shifts, further complicating optimization. Training is performed by collecting trajectories, computing gradients VL(O), and updating parameters via 0 <- 0 - V/ L()), where 1j > 0 is the learning rate. Following conventions from supervised learning, deep RL algorithms often adaptive variants of stochastic gradient descent, such Adam [Kingma and Ba, 2014] or RMSprop [Hinton, 2012], which adjust learning rates based on running estimates of gradient statistics. The gradients with respect to each layer are denoted by;
V OL
VØZ = aL
Do,"
where di and ú represent the parameters (i.e., weight matrices or bias vectors) of layer and j respectively. The structure and magnitude of these gradients (Vé, and Vayy) are influenced by the loss function, data distribution collected from the environment, and the architecture itself. These per-layer gradients determine how effectively different parts of the network adapt during training.
While training large models in supervised learning settings present challenges, advances in initial- ization, normalization, and scaling strategies have enabled relatively stable optimization [[offe and

---

Szegedy, 2015, Ba et al,, 2016, Glorot and Bengio, 2010]. Scaling up model size has been a central driver of progress across domains, improving generalization, enhancing representation learning, and boosting downstream performance (Kaplan et al., 2020].
Deep RL differs substantially from supervised learning- First, the data distribution is non-stationary, continually shifting as те updates. Second, learning signals are often sparse, delayed, or noisy, which introduces variance in the estimated gradients [Han al., 2022, Fujimoto et al., 2018]. These factors destabilize optiiization and lead to loss surfaces with sharp curvature and complex local structure [llyas et al., 2020, Achiam al., 2019]. Moreover, increasing model capacity often degrades performance unless regularization or architectural interventions are applied [Gogianu et al., 2021, Bjorek et al., 2021, Schwarzer al., 2023, Wang al., 2025].
These challenges are further compounded by both architectural and environmental factors. Network depth, width, initialization, and nonlinearity affect how gradients are propagated across layers. Meanwhile, reward sparsity, exploration dificulty, and transition stochasticity impose additional structure on the optimization landscape. The resulting geometry reflects the joint dynamics of policy, environment, and architecture, making deep RL optimization uniquely complex.
Gradient Propagation Training deep networks poses fundamental challenges for effective gradient propagation [Glorot and Bengio, 2010]. As network depth increases, gradients may either vanish or explode they backpropagated through multiple layers, impeding the optimization of early layers and destabilizing learning dynamics [Ba et al., 2016]. These issues arise from repeated applications of the chain rule. For network with intermediate hidden representations {ho, hi,..., ... ht), where h& E Re the gradient of the loss L with respect hidden layer he is:
Oc Thg
II 1o
Ohk ahg-1
ac
ThL
where each The-1 1gde is the Jacobian. If the singular values of these Jacobians are not properly controlled, their repeated multiplication can cause the norm of the gradient to shrink or grow exponentially with L. This severely impairs convergence, as earlier layers receive little to no useful gradient signal or become numerically unstable [Ioffe and Szegedy, 2015, He et al., 2016].
Dh.
In addition to depth, the width of the network also influences gradient propagation. Consider a fully connected layer with weight matrix W E gmxa and input vector R". The output is Wh E RT and under the assumption that W and h have i.i.d. zero-mean entries with finite variance ow and The respectively, the variance of the output is given by Var[Wh] n nw oh- Thus, scaling the width n without adjusting ow and Ty leads to instability in forward and backward signal propagation affecting gradient norms and optimization trajectories.
Beyond depth and width, the choice of nonlinearity also plays a central role in determining how gra- dients propagate... In typical feedforward network, hidden activations evolve h2 <(Wihh 1). where (() is a nonlinear activation function (e.g., ReLU, tanh, sigmoid), and W₁ is the weight matrix at layer During backpropagation, the gradient with respect hidden layer includes the product of the Jacobian of the linear transformation and the derivative of the nonlinearity:
ac WiT (<'(Wahx-1) 2
OL
Thn-1
Dhk
where 5'(-) denotes the elementwise derivative of the activation function, and O represents elemen- twise multiplication. For ReLU, ('(x) = 1z20. so the gradient is entirely blocked wherever the neuron is inactive. This leads to the well-known dying ReLU problem, where significant portion of the network ceases to update and becomes untrainable [Lu et al., 2019, Shin and Karniadakis, 2020].
3 Diagnosis: Gradients Under Non-stationarity and Scale
A fundamental premise of modern deep learning is that scaling model capacity yields consistent gains in performance [Kaplan et al., 2020, Chowdhery et al., 2023]. This has held true large-scale supervised learning, where training data distributions are stationary and i.i.d., and gradient descent operates under relatively stable conditions. However, non-stationary settings, such as RI, gradient- based optimization faces severe challenges that scaling alone may exacerbate [Ceron et al., b0.
3

---

In this section, we diagnose how gradient pathologies emerge and intensify across different settings, with focus on architectural scaling in width and depth (network scales used specified in Table 1).
3.1 Gradient Pathologies
We train neural networks of varying depths and widths and analyze their training dynamics.
Supervised Learning (Stationary and Non-Stationary) We use the CIFAR-I0 image classification benchmark [Krizhevsky et al., 2009], where the input-output mapping remains fixed over time. Models consist of standard 6-layer convolutional neural networks (CNN) followed by multi-layer perceptron (MLP). We vary the depth and width of the MLP to explore how model scale influences learning behavior. To introduce non-stationarity. we periodically shuffle the training labels during training, following the setup by Sokar et al. [2023]. This creates a loss landscape that changes over time, echoing the challenges of deep RL. Fig. 1 illustrates the contrast in training behavior and gradient flow between stationary and non-stationary supervised learning. Under non-stationarity. deep networks fail to recover accuracy, which aligns with a marked degradation in gradient magnitudes.
MLP Depth: Small
1.0
MLP Depth Medium
Large
Nov Sorineary
2.2
0.5
Stationey
Train Accuracy
0.0
MLP Depth: Large
Meam Orad
1.0
0.5
Large
0.0
100
Training Steps
XLP
Largo
MLP Width
Lang
MLP Wisth
Figure 1: Training dynamics under stationary and non-stationary supervised learning. (Left) In the stationary setting, both shallow and deep models fit the data effectively across widths. Under non-stationarity only shallow networks partially recover during training, while deeper ones collapse. (Right) This collapse correlates with degraded gradient flow. In stationary settings, gradient norms remains stable across all network scales (shaded boxes) while in non-stationary settings (solid-colored boxes), gradient magnitudes diminish with depth and width, suggesting poor adaptability.
Reinforcement Learning As discussed in Sec. 2, RL introduces fundamentally different sources of non-stationarity due to the policy-dependers data distribution and moving target estimates. To study gradient dynamics, we use PQN [Gallici et al., 2025], recent value- based algorithm that achieves strong performance without relying on a target network or replay buffer. PQN ensures stability and convergence using Layer Normalization [Ba et al., 2016] and supports GPU-based training through vectorized environments for online parallel data collection. In Sec. C.I we extend our investigation to DQN [Mnih et al., 2015] and Rainbow [Hessel et al., 2018], demonstrating the generality of our observations. As shown in Fig. 2, deeper networks trained with PQN exhibit collapse in both episode returns and gradient norms?, highlighting the fragility of deep models under non-stationarity.
3.2 Training Degradation
In Fig. we evaluate diagnostic metrics capturing expressivity and training dynamies, revealing that deeper networks exhibit pronounced training pathologies and degraded performance. We first measure the fraction of dormant neurons, defined as units with near-zero activations over a batch of trajectories [Sokar et al., 2023], and find that dormant neurons grow with depth, signaling underutilized capacity, Next, we assess representational expressivity using SRank, the effective rank of penultimate-laye activations [Kumar et al., 2020], observing that deeper networks tend to collapse state representations into lover-dimensional, and less expressive (as evidenced by declining returns) subspaces. To study loss curvature, we compute the Hessian trace of the -emporal-difference loss. This metric serves as a proxy for sharpness or smoothness in optimization [Ghorbani et al., 2019], 9J, similarly to tracking the largest eigenvalue. Fig. 3 shows that only shallow networks exhibit high Hessian trace values,
Unless otherwise specified, all ALE results are averaged over three seeds.

---

MLP Depth
X10>
MLP Dopth
0.15
Pioerdt
Small
9.30
Episods Roturn
Mean Grad Norms
9.05
x30P
Phoenix
2:50
Qsert
Small
Mediu
Total Timestops in Millions
MIL Wath
14LP Weith
SLPP Watts
Figure Mean episode returns and gradient norms across increasing MLP depths and widths on two ALE games using PQN. (Left) Only shallow networks achieve high episode returns; performance collapses for deeper networks. (Right) The collapse correlates with vanishing gradient norms, suggesting that deeper models fail adapt non-stationarity in deep RL.
16 Dead Neurens MLP
1o2
SRack
3:3 Trace of the Hessinn
Q values
Episedo Beturs
25-4
bryth teall
20 Enviroement Stope lnn Millioss
Figure 3: Training pathologies emerge as MLP depth increases. Deeper networks exhibit a higher fraction of inactive neurons, reduced representation rank (SRank), vanishing Hessian trace (loss curvature), and degraded learning performance (mean Q-values and episode returns). These trends indicate that scaling depth limits expressivity and plasticity, impairing policy quality.
suggesting access to sharper regions of the loss surface with pronounced directions of improvement. In contrast, deeper architectures consistently show near-zero trace, indicating poorly conditioned geometry that hinders effective gradient- based updates. These findings suggest breakdown in representation, plasticity, and optimization as networks scale, ultimately impeding learning.
Key observations on gradients under non-stationarity and scale:
Non-stationarity amplifies gradient degradation in deeper and wider networks.
In deep RI., deeper models suffer from vanishing gradients, reduced activations, and loss of representational expressivity.
The flat loss curvature intensifies with depth, correlating with poor learning.
Stabilizing Gradients
Having identified the pathologies that emerge in non-stationary regimes, particularly under large-scale architectures, we investigate strategies mitigate these instabilities. We focus on two complementary interventions: skip connections [He al., 2016] and optimizers [Martens and Grosse, 2015), as these directly improve gradient flow. We continue to use PQN as our base RI. algorithm and evaluate on the Atari-10 10 suite [Aitchison et al., 2023]. In Sec. 5, we demonstrate that the effectiveness of our proposed gradient interventions generalize beyond this specific algorithm and environment suite.
4.1 Intervention 1: Multi-Skip Residuals for Gradient Stability
Gradient instability in deep networks often aggravated by increasing depth, non-linear activations, and misaligned curvature across layers. While standard residual connections offer some relief by introducing shorteut paths for gradient flow [He al., 2016]. they typically span only one or two layers, which can be insufficient in the presence of severe gradient disruption due to sonsttaiinn,rit.
5

---

Nonlinearities
Baseliie
Encoder
(ax,-)
+
Increase Depth
DenseNet
Encoder
(x.-))
Increase Width
MultiSkip
Encoder
(x.-))
Dense layers
Figure 4: (Left) MLP architectures and (right) scaling strategies studied.
introduce multi-skip residual connections, which the flattened convolutional features are broadcast directly to all subsequent MLP layers. This design ensures that gradients can propagate from any depth back to the shared encoder without obstruction.
We compare our network architecture against the standard fully connected baseline across varying depths. As shown Fig. 5 (left), performance collapses with increased depth in the baseline, while the multi-skip architecture maintains stable learning and continues to improve across widths. This improvement is accompanied by consistently higher gradient magnitudes. Complete results across all network depths and widths are presented in Sec. C.4.
MLP Depth: Large
MLP Dapth: Large
With:
Seal
0.5
WMdh::
Medac
1.0
Humaa Normaltzed Score
Mean Grad Norms
0.4
KZZ4 PON
Hummn Normalized Score
Mean Grad Norms
0.8
ize PQN
0.8
0.5
0.3
0.6
0.6
0.6
0.2
0.4
0.0
0.0
-022
Eustrosment Stagi Di. Millions
100
20
Small
MLP Width
Medium
Lange
-0.2
Bavievnnennt Stegs lin Millios)
10
20
30
Small
MLP Width
Mediuum Large
Figure 5: Gradient-stabiliann interventions improve scalability in deep RL. (Left) Standard fully connected networks trained with PQN collapse at greater depths due to vanishing gradients. In contrast, multi-skip architectures maintain gradient flow and scale effectively. (Right) The default RAdam optimizer leads to instability in deep networks, while switching to the Kron optimizer preserves gradient signal and enables stable learning without architectural changes.
4.2 Intervention 2: Second-Order Optimizers for Non-Stationarity
First-order optimizers such SGD and Adam rely on local gradient estimates and fixed heuristics (e.g., momentum, adaptive step sizes) [Kingma and Ba, 2014], which are agnostic to curvature and often brittle under shifting data distributions. In contrast, second-order methods adjust parameter updates using curvature information, enabling more informed and stable adaptation.
Let C(0) denote the loss function, and g VL()) its gradient. A second-order update takes the form 0++1 0, n/l n g. where H the curvature matrix, typically the Hessian or the Fisher Information Matrix (FIM) [Martens, 2020]. Directly inverting H is computationally infeasible in deep neural networks so Kronecker-fie factored approximations, such as K-FAC [Martens and Grosse, 2015], address this challenge by approximating H using low-rank Kronecker products.
Kronecker-factoree optimizer (Kron for short) approximates the FIM and applies structured precon- ditioning that captures inter-parameter dependencies, unlike Adam's diagonal scaling. This yields directionally aware preconditioning that better aligns with the curvature of the loss surface [Martens, 2020]. In non non-stationary settings, such as deep RL, where both the data distribution and curvature

---

evolve over time, curvature aware updates can help preserve gradient signal by maintaining stable update magnitudes and directions. As shown in Fig. (right), replacing RAdam with Kron prevents performance collapse at greater depths, even in standard MIP architectures. Complete results across all network depths and widths are presented in See. C.6.
4.3 Combining Gradient Interventions
We combine both gradient interventions to PQN and evaluate it on the full ALE suite (57 games), across 3 seeds and 200M frames. Fig. 6 shows that augmented agent outperforms the baseline in 90% of the environments, achieving a median relative improvement of 83.27%. Notably, the baseline PQN is itself competitive with strong agents such as Rainbow [Gallici et al., 2025], highlighting the effectiveness of our interventions. Detailed per-environment learning curves can be found in Sec. F.
In Fig. 7 we validate the effectiveness of the combined gradient interventions in the non-stationary SL setting we used motivation in Sec. The results verify that these interventions enable high accuracy and sustained adaptability across depths and widths, even under dynamic label reshuffling.
wus::
MLP Depth:
Depth: Large
Normalied Exore
Voon Cood Norms
PQN
9.4
Larpo
Figure 6: Gradient-stabilized PQN achieves superior scalability. (Left) On Atari-10, the combined interventions lead high HNS even greater depths, outperforming either intervention alone (see Fig. 5) and increased gradient gradient flow. (Right) On the full ALE suite, our agent outperforms the baseline in 90% of the games with a median performance improvement 83.27%.
Baralina
Large
MLP Depth
MLP Depth: Small
Small
Large
1.0
MW
2.0
Train Accuracy
0.5
Mean Grad Norms
CANDAOY
Baseline
1.5
Ours
0.0
Depth: Large
1.0
1.0
W
0.5
0.5
0.0
100
Small
Mediam
Large
Small
Mediuum
Large
Training Steps
MLP Width
MLP Width
Figure 7: Gradient interventions enable rapid recovery in non-stationary SL. (Left) Models with combined gradient interventions rapidly recover accuracy after label reshuffling, demonstrating robust adaptation non-stationary settings. (Right) This is supported by stable gradient flow across depth. Dashed curves and shaded boxes indicate MLP baselines.
4.4 Alternative
Methods
To ensure that findings were not specific to narrow choice of interventions, conducted broader exploration of alternative strategies for improving gradient stability in deep RL. We tested variety of approaches inspired by prior work on optimization and representation stability in both supervised and RL settings (see Sec. C.7 for more details).
As summarized in Tab. 3, none of these methods consistently improved stability performance compared to our proposed combination of multi-skip residuals and Kronecker-factored optimization. In many cases, the alternatives yielded either negligible gains or degraded performance as network depth increased, reinforeing that architectural and curvature-aware interventions are key to preserving gradient flow scale.
7

---

Beyond the ALE and PQN
To evaluate the generality of our findings, extend our analyses. Specifically, (i) apply proposed methods to PPO [Schulman et al., 2017 7] on the full ALE and on continuous control tasks in Isaac Gym [Makoviychuk et al., 2021]; (ii) assess the impact of richer convolutional encoders by replacing the standard CNN backbone used in the ALE with the Impala CNN architecture [Espeholt al., 2018]; (iii) augment Simba [Lee et al., 2025] with our proposed techniques and evaluate performance on the DeepMind Control Suite (DMC) [Tassa et al., 2018]; and (iv) investigate whether our interventions can stabilize and scale wide range of Q-learning algorithms challenging offline RL benchmarks [Park et al., 2025].
Cartpote
10" ton 1o1 1or
1I Curd angervenstt ona ITO Suntaw
PPO
PPO (+ Oursi
Anymal
80
60
Chelinde
Small
Medium
Large
MLP Degth
Figure 8: PPO with gradient interventions. Left: On the full ALE suite, applying the combined gradient interventions to PPO yields median performance improvement of 31.40% and outperforms the baseline 83.64% of the games. Right: In the Cartpole and Anymal tasks from IsaacGym, only the augmented PPO maintains stable performance across depths and widths.
PPO with Gradient Interventions. Fig. 8 (left) shows that augmenting PPO with the same strategies as in PQN (Layer Normalization by default on PQN, multi-skip residual connections, and connecke--factore optimization) significantly boots performance. On the ALE benchmark, the augmented PPO outperforms the baseline 83.64% of the environments, achieving median relative improvement of 31 11.40% In Isaac Gym's 's continuous control tasks, including Cartpole and Anymal (Fig. 8, right), the baseline PPO collapses as model size increases, while the augmented variant remains stable and achieves superior performance at all depths and widths.
Gradient Interventions in Scaled Encoder Variants The Impala CNN is a scalable convolutional architecture that has demonstrated strong performance gains in agents such as Impala [Espeholt al., 2018] and Rainbow [Hessel al., 2018]. We investigate whether, given its capacity to extract richer representations from visual input, combining Impala CNN with our gradient flow interventions enables effective scaling the MLP component. As shown in Fig. 9, PPO and PQN benefit significantly from replacing the standard CNN with the Impala CNN. For PQN, the Impala encoder enables successful scaling of the MLP, in contrast to the performance collapse seen without our interventions. These results suggest that the expressivity of richer visual encoders is more effectively leveraged by deeper networks when gradient flow is preserved.
Impolia CNX
Impola CNN
TAutun Narmatar Zecre
Ouz
2.0
KLP Dapro
SLEP Depts
NLF Dopris
XLP Depta
Figure 9: Scaling performance with standard vs. Impala CNN encoders on PQN (left) and PPO (right). Each agent evaluated using both the Atari CNN (left sub-panels) and the Impala CNN (right sub-panels) as the encoder. Gradient interventions enable successful scaling in both cases.
Simba with Kron Optimizer. Simba eee et al., 2025] a scalable actor-critic framework that integrates normalization, residual connections, and LayerNorm. We augment Simba by replacing its
8

---

Soaling Depti
Sculing
Soilin Depdi
Sealing Duyth
Scaling Winth
Scaling und Dupc
62
Hamanced Walk
1c2
Humandd Run
162
Dog Trii
1c2
Dog Ran
9.0
3.0
6.0
4.5
6.0
Eioioda Roturn
4.5
5AC
SAC
2
SAC
1.5
5AC
1ma
162
7.5
6.0
1.5
4.5
3.0
1.0
1.5
4.5
9
17
35
70
120
4.5
17
35
70
120
4.5 9
17
35
70
120
4.5
9
17
35
70
120
Paraameee (M)
Figure 10: Performance comparison between Adam (dashed lines) and Kron (solid lines) optimizers using the SimBa architecture with SAC and DDPG, averaged over 5 random seeds. As model size increases, Adam leads to consistent performance degradation, while Kron enables stable and improved learning with larger networks.
default AdamW optimizer with Kron while keeping all other hyperparameters fixed. We evaluate SAC [Haarnoja et al., 2018] and DDPG [Lillicrap al., 2015] on challenging DMC tasks, using the Simba architectures of varying depth and width. Despite its design for scalability, default Simba collapses across all tasks networks grow shown in Fig. 10 (additional results in Sec. E.2). In contrast, the Kron-augmented version successfully scales in both depth and width, achieving consistent and stable performance gains. These findings underscore the generality of approach as effectively enabling parameter scaling in deep RI. agents.
Gradient Interventions for Scalable Offline Q-Learning. Park al. [2025] highlight significant challenges in scaling Q-learning algorithms for complex offline RL tasks, demonstrating that many standard offline RL baselines fail to learn effective policies, key finding was that performance improvements were primarily driven by techniques that shorten the effective credit assignment horizon, such n-step returns and hierarchical methods. This led their proposal of two new high-performing algorithms, SHARSA and DSHARSA, which are designed to operate with shorter effective horizons. This finding motivates parallel investigation: can our proposed gradient interventions, which are designed to stabilize and accelerate deep network training, also address the scaling limitations of offline Q-learning? To test this, we augment the full suite of baselines and novel algorithms from Park et al. [2025] with our proposed gradient interventions. The results, presented in Fig. 11, show that our methods provide a complementary path scalability. Applying our interventions generally improves the performance of the baselines across all tasks.
large, high-quality datasets. Their
The performance gains humanoi wwwze-aazzzaaeea -aapaae-giantt-aiigtttee where our gradient interventions enable multiple methods to achieve near-optimal performance, whereas their baseline counterparts largely fail. Furthermore, this stability extends to generalization across task difficulty. When moving from puzzle e 1--5--x-lll to the harder puzzle-4zz -ar-pplay task, many baselines exhibit a sharp performance degradation. In contrast, the performance of several algorithms with our interventions remains consistent and high, demonstrating improved robustness. Finally, we note that while the primary focus of this paper is to address gradient pathologies arising from scaling and non non-stationarity. these results highlight that our interventions are also highly beneficial in offline deep RL, where inputs are stationary.
particularly pronounced in the most sparse-reward task,
Related Work
A central challenge in scaling deep RL lies in the inefficient use of model capacity. Increasing param- eter counts often fails to yield proportional gains due under-utilization. Sokar al. [2023] and Liu et al. [2025b] show that online RL induces a growing fraction of inactive neurons. a phenomenon also observed offline settings. Ceron et al. [2024a] report that up to 95% of parameters can be pruned post-training with negligible performance drop, underscoring substantial redundancy. These findings have motivated techniques such as weight resetting [Schwarzer et al., 2023], tokenized computation [Sokar al., 2025], and sparse architectures [Ceron et al., 2024b, Willi et al., 2024, Liu

---

Cube Octuple Play
Humanoidmaze Giant Navigate
Puzzle 4X5 Play
Puzzle 4X6 Play
1.0
Baseline
0.5
0.4
0.2
oy
Figure 11: Performance of offline Q-learning algorithms with and without our gradient inter- ventions. We compare the original algorithms from Park et al. [2025] against our augmented versions.
The results, averaged over 3 seeds demonstrate a general improvement in scalability.
et al., 2025a, Ma et al., 2025], along with auxiliary objectives to promote capacity utilization [Fare- brother et al., 2023]. While scaling model size offers greater expressivity, its benefits depend on appropriate training strategies [Ota et al., 2021]. Architectural interventions such as SimBa [Lee et al., 2025] improve robustness by regularizing signal propagation through components such as observation normalization, residual feedforward blocks, and layer normalization. Complementarily, BRO [Nauman et al., 2024] shows that scaling the critic network yields substantial gains in sample and compute efficiency, provided it is paired with strong regularization and optimistic exploration strategies.
Gradient flow, however, remains a central bottleneck. We complement prior efforts by explicitly targeting vanishing gradients as a mechanism for improving scalability. Our approach builds on the role of LayerNorm in stabilizing training and enhaneing plasticity [Lyle et al., 2024], and leverages its theoretical effect gradient preservation as formalized in PQN [Gallici et al., 2025]. Optimization- level interventions such as sccond-order methods [Martens and Grosse, 2015, Muppidi et al., 2024] and adaptive optimizers [Ellis et al., 2024, Bengio et al., 2021, Wu et al., 2017] also address instability under stusstationnrtyy Our approach integrates architectural and optimizer-level interventions to enable stable gradient flow and unlock parameter scaling in deep RL agents.
7 Discussion
Our analyses in Sec. 3 suggest that the difficulty in scaling networks in deep RL stems from the interaction between inherent non-stationarity and gradient pathologies that worsen with network size. In Sec. 4, we introduced targeted interventions to address these challenges, and in Sec. 4.3, we demonstrated their effectiveness. We validated the generality of our approach across agents and environment suites, consistently observing similar trends. These findings reaffirm the critical role of network design and optimization dynamics training scalable RL agents. While our proposed solutions may not be optimal, they establish a strong bascline and provide a foundation for future work on gradient stabilization in deep RL. More broadly, our findings suggest that scaling limitations in deep RL are not solely attributable to algorithmic instability or insufficient exploration, but also stem from gradient pathologies amplified by architectural and optimization choices. Addressing these issues directly, without altering the learning algorithm, yields substantial gains in scalability and performance. This suggests that ensuring stable gradient flow is a necessary precondition for effective parameter scaling in deep RL.
Limitations, Our study is constrained by computational resources, which limited our ability to explore architectures beyond certain size. While our interventions show consistent improvements across agents and environments, further scaling remains an open question. While using second order optimizers introduced additional computational overhead (see Tab. 12), this cost is mitigated by leveraging vectorized environments and efficient deep RL algorithms, narrowing the gap relative to standard methods. These limitations highlight promising directions for future work, including the development of more computationally efficient gradient stabilization strategies and scalable optimization techniques.
10

---

Acknowledgment The authors would like to thank João Guilherme Madeira Araújo, Evan Walters, Olya Mastikhina, Dhruv Sreenivas, Ali Saheb Pasand, Ayoub Echchahed and Gandhary Patil for valuable discussions during the preparation of this work. João Araújo deserves a special mention for providing us valuable feed-back on an early draft of the paper. We want to acknowledge funding support from Natural Sciences and Engineering Research Council (NSERC) of Canada, Google Research, Fonds de recherche du Québec (FRQNT) and The Canadian Institute for Advanced Research (CIFAR) and compute support from Digital Research Alliance of Canada, Mila IDT, and NVidia. We would also like to thank the Python community Van Rossum and Drake Jr [1995], Oliphant [2007] for developing tools that enabled this work, including NumPy Harris et al. [2020], Matplotlib Hunter [2007]. Jupyter Kluyver et al. [2016]. and Pandas McKinney [2013].
Broader Impact This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on lifelong learning agents, pages 620-636. PMLR, 2023.
Joshua Achiam, Ethan Knight, and Picter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXiv:
wweXiv:/903..0m /903 0809, 2019.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 34:29304 29320, 2021.
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-S: Distilling the arcade learning environment down to five games. In International Conference Machine Learning, pages 421-438. PMLR, 2023.
Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep rl: An empirical study. Advances in Neural Information Processing Systems, 01472284-72324 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyun Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588:77 - -2, 2020.
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Correcting momentum in temporal difference learning. arXiv preprint arXiv:2106.088 03955, 2021.
Nils Bjorck, Carla Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems, 34:8242-8255, 2021.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, pages 1373-1383. PMLR, 2021.
Johan Samir Obando Ceron, Aaron Courville, and Pablo Samuel Castro. In value-based deep reinforcement learning, pruned network a good network. In International Conference on Machine Learning, pages 38495-38519. PMLR, 2024a.

---

Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup. and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. In International Conference Machine Learning, pages 38520-38540. PMLR, 2024b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Reseurch, 24(240):1-113, 2023.
Benjamin Ellis, Matthew T Jackson, Andrei Lupu, Alexander D Goldie, Mattie Fellows, Shimon Whiteson, and Jakob Foerster. Adam on local time: Addressing nonstationarity with relative adam timesteps. Advances in Neural Information Processing Systems, 37: 37:134667-13499, 2024.
Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A. Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. In Reinforcement Learning Conference, 2024. URL https://openreviet www.:eee//eeeee formmrrdmoo wiw.miiii--iVvPPPP.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. In The Eleventh International Conference on Learning Representations, 2023. URL https: / /ttp:///ppeeeeeeeeeee .ew. weet/forummid==GDSSsttt
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Moham- madamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schritwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: https:///ppnttte /ttp:////pp///eeeeeeee net/ forum?id==IzellOktu
Behrooz Ghorbani, Shankar Krishman, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232- 2241. PMLR, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pages 3734-3744. PMLR, 2021.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion. In International Conference on Machine Learning, pages 1842-1850. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
Beining Han, Zhizhou Ren, Zuofan Wu, Yuan Zhou, and Jian Peng. Off-policy reinforcement learning with delayed rewards. In International conference on machine learning, pages 8280-8303. PMI.R, 2022.
12

---

Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Couraapeau, Eric Wieser, Julian Taylor, Sebastian Berg. Nathaniel Smith, al. Array program- ming with numpy. Nature, 585(7885):3357-366 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
D Hendrycks. Gaussian error linear units (gelus). arXiv preprint arXiv: 16066666661411 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence. volume 32, 2018.
Geoffrey Hinton. Lecture nmp-roppop::
Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26, 2012.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(03): 90-95, 2007.
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In International Conference on Learning Representations, 2020. URL https: http:::///ppeeevieee
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Arthur Juliani and Jordan T. Ash. A study of plasticity loss in on-policy deep reinforcement learning. In The Tiihth-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htttps://openrviiee net/ wwwtfooummiiiiii wwwoorrddmmuuuoooooeiiio 8kpKTF.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Thomas Kluyver, Benjain Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic. Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter Development Team. Jupyter Note- books-a publishing format for reproducible computational workflows. In IOS Press, pages 87-90.
2016. doi: 10.3233/978- 1-778----1999-644- 110--649-----
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under parameeparamttriizaatio inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. 14498, 2020.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline Q- learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, 2023. URI.https wttphtts://oppereeieeeee net/ formm?dd= 4- k-KkUavjj.
13

---

Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim. Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone. and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In The Thirleenth International Conference on Learning Representations, 2025. URL URL. https::/p/// fttaps://openreveee
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. A signal propagation perspective for pruning neural networks initialization. In International Conference on Learning Representations, 2020. URL
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (ttp:::0999022700 2015.
Jiashun Liu, Johan Samir Obando Ceron, Aaron Courville, and Ling Pan. Neuroplastic expan- sion in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025a. URL htttts://openreeieww..e.. wtt.:e/ooormmmmddoo22TT......
Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning. arXiv preprint arXiv:2505.2400/ 2025b.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint 1933006333 2019.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreeviet wwtpeeewwwwwwweeeeeeeeeeeeeeeeee http:/orummiiddzccstttee
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning. pages 22190-23211. PMLR, 2023.
Clare Lyle, Zeyu Zheng. Khimya Khetarpal, James Martens, Hado P van Hasselt, Razvan Pascanu, and Will Dabney, Normalization and effective learning rates in reinforcement learning. Advances in Neural Information Processing Systems, 37: 37:106440-106473, 2024.
Guozheng Ma, Lu Li, Zilin Wang. Li Shen, Pierre-Luc Bacon, and Dacheng Tao. Network sparsity unlocks the scaling potential of deep reinforcement learning. In Forty-second International Confer- ence on Machine Learning, 2025. URL /ttpps://ppenrviiii wwt.:/ewwwweeeeeeeeeeeeeeeeeeeee forum ?idmm Imtsmmmmmmsaaaaa kaa.
Xuezhe Ma. Apollo: An adaptive parameter-wise diagonal quasi-newton method for nonconvex stochastic optimization. arXiv preprint arXiv:2009. 1re00.335550 13586, 2020.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprini arXiv:21086 10470, 2021.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 21(146):1-76. 2020.
James Martens and Roger Grosse. Optimizing neural networks with factoree approximate curvature. In International conference machine learning, pages 2408-2417. PMLR, 2015.
Wes McKinney. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. URL http:://www http::////wwaaaooon.mmmmm amazon. O' O'Reilly Media, 1 edition, February 2013. ISBN -Dt.://paaaaAASS11/1414111111
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015.
14

---

Skander Moalla, Andrea Miele. Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No represen- tation, no trust: connecting representation, collapse, and trust issues in ppo. Advances in Neural Information Processing Systems, 37:69652-669668 2024.
Aneesh Muppidi, Zhiyu Zhang, and Heng Yang. Fast trac: parameter-free optimizer for lifelong reinforcement learning. Advances in Neural Information Processing Systems, 37:51169-51199 2024.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitoš, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Johan Obando Ceron, Mare Bellemare, and Pablo Samuel Castro. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36:26003-26024, 2023.
Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9()): 10-20, 2007. doi: 10.1 10.1109//CSE.200..555
Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. arXiv preprint urXiv:2102.07920, 2021.
Kei Ota, Devesh K Jha, and Asako Kanezaki. A framework for training larger networks for deep reinforcement learning. Machine Learning. 113(9):6115-6133) 2024.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv preprint arXiv:2506.04168, 2025.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. Pmlr, 2013.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.v:7707 2017.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger. better, faster: Human-level Atari with human-level efficiency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 30365-30380. PMLR, 23-29 Jul 2023. UtLpttts: /tapttss:///rreeeeeeee http:///rrreeeennmmmececs
wiiggs.lr..rres. walr.press//v00 20:0022//cwaaaae swwwwrzer23a....m html.
Yeonjong Shin and George Em Karniadakis. Trainability of relu networks and data-dependent initialization. Journal of Machine Learning for Modeling and Computing, 1(1), 2020.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning, In International Conference on Machine Learning, pages 32145-32168. PMLR, 2023.
Ghada Sokar, Johan Samir Obando Ceron, Aaron Courville, Hugo Larochelle, and Pablo Samuel Castro. Don't flatten, tokenize! unlocking the key to softmoe's 's efficacy in deep RL. In The Thirteenth International Conference Learning Representations, 2025. URL https:
Richard S. Sutton and Andrew Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
15

---

Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Mare G Bellemare. Investigating multi-task pretraining and generalization in reinforcement learning. The Eleventh International Conference on Learning Representations, 2023. URL. htw.::ttp:::/eettm httpp:://pennreview net/ forum2id==St9FROSZZ0
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint 1rX1v:1812.02648, 2018.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-33) 2019.
Kevin Wang, Ishaan Javali, Michaf Bortkiewicz, Benjamin Eysenbach, et al. 1000 layer networks for self-supervised rl: Scaling depth enable new goal-reaching capabilities. arXiv preprint WwXiv:2503.44855 2025.
Timon Willi, Johan Obando-Ceron Jakob Foerster, Karolina Dziugaite, and Pablo Samuel Castro. Mixture of experts mixture of rl settings. arXiv preprint vrXiv:2406.18422 2024.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using fronecker-factored approximation. Advances in neural information processing systems, 30, 2017.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in neural information processing systems, 31, 2018.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikkonda, Nicha Dvoreek, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806 2020.
16

---

A Environment Details
Throughout the paper, we evaluate the deep reinforcement learning agents' performance on the Atari- 10 suite [Aitchison et al., 2023], curated subset of games from the Arcade Learning Environment (ALE) [Bellemare et al., 2013]. Atari-10 consists 10 games selected to capture the maximum variance in algorithm performance, achieving over 90% correlation with results the full ALE benchmark. This makes it a computationally efficient yet representative testbed for deep reinforcement learning. We follow the experimental protocol of Obando Ceron al. [2023], Ceron et al. [2024b], Agarwal et al. [2021], running each experiment with three random seeds and reporting the aggregate numan-normalinn score across games.
The games in Atari-10 are:
Amidar, Battle Zone, Bowling, Double Dunk, Frostbite, Kung Fu Master, Name This Game, Phoenix, Q*Bert and River Raid.
Additionally, further support the generality of our findings, we evaluate the proposed combined gradient interventions on the full ALE benchmark. We also assess their effectiveness on continuous control tasks from the IsaacGym simulator [Makoviychuk et al., 2021] and the DeepMind Control Suite (DMC) [Tassa et al., 2018], extending analysis to robotics-based environments, We conduct experiments on the challenging tasks DMC:
Humanoid Walk, Humanoid Run, Dog Trot and Dog Run.
B Network Sizes
Throughout the paper, we experiment with models of varying depths and widths. Unless stated otherwise (e.g, in Sec. 5, where evaluate the Impala CNN), the convolutional feature extractors are kept fixed. Consequently, our experiments focus primarily on scaling strategies and architectural variations in the MLP components of the networks.
To enable meaningful comparisons across different learning regimes, the MLP architectures are kept consistent across supervised learning (SL), non-stationary SL, and reinforcement learning (RL) experiments. This consistency ensures that observed differences in gradient behavior arise from the learning setting itself, rather than confounding factors due to domain-speciff architectures.
Table provides detailed information on the number of parameters for each depth- wiith configuration, categorized as small, medium, or large, as used throughout the paper.
Table I: Number of parameters (in millions) for different MLP architectures.
Depth Width
Small
Medium
Large
Small
2.39
11.90
27.70
Medium
3.45
21.35
53.93
Large
4.50
30.79
80.15
Additional Experiments
C.1 Scaling with DQN and Rainbow
To further support our hypothesis the emergence of gradient pathologies in deep reinforcement learning, we investigate whether similar issues arise in algorithms beyond PQN and PPO, as discussed in the main paper. Specifically, we study the effects of architectural scaling two widely used value-based algorithms: DQN [Mnih et al., 2015] and Rainbow [Hessel et al., 2018].
DQN is foundational deep RL algorithm that learns action-value functions using temporal difference updates and experience replay, serving as a standard baseline for value-based methods. Rainbow extends DQN by integrating several enhancements, such as double Q-learning, prioritized experience
17

---

replay, dueling networks, multi-step learning. distributional value functions, and noisy exploration, to achieve improved sample efficiency and stability.
In Fig. 12, we report the performance DQN and Rainbow as we scale the depth and width their networks. As with PQN and PPO, observe consistent degradation in performance larger scales. In Fig. 13, we present the corresponding gradient behavior, which reveals the same vanishing and destabilization phenomena discussed this work. These findings reinforce the generality of the identified gradient pathologies across both policy-based and value-based deep RL algorithms.
Watth
Handum
Lapp
Starge
Midi Small
Nadter
Midhs Large
Degas
Depth
Smal
Degti: Nudtun
Deyt: Lange
Degtts Maagei
Depli Seat
Degth Media
Dopth Lorge
Depit. Xiorge
Rainhow by Size
Purformancs by
Sormatont Stven (1000)
Baaan Nortalted Scoee (HINSY
(a) DQN HNS
(b) Rainbow HNS
Figure 12: Median human normalized scores for DQN (left) and Rainbow (right) as a function of total network parameters. Lines of different colors denote varying network depths, while marker shapes indicate different widths. For both agents, performance consistently declines as network size increases, highlighting the adverse effects of scaling.
Depit
Deydi
Deyti Large
Deydi MLarge
Tranung Stope
Trahing Sogge
Truciig Stepe
Trandin Strge
Dryah
Boytk Carge
Depit Marg
Toaldiy Sigg
Tralding Stigs
Chilliy Sige
Figure 13: Gradient magnitudes during training for DQN (top) and Rainbow (bottom). As network depth increases, gradient flow systematically diminishes, ultimately collapsing near-zero values. This consistent decay mirrors the performance degradation observed at larger scales.
C.2 Combining Gradient Interventions in Non-stationary Supervised Learning
Building findings in Sec. 4.3, extend analysis by applying the proposed combined gradient interventions to the same image classification models used Sec. 3. Specifically, we train the models in the non-stationary supervised learning setup, where the CIFAR-10 labels are iteratively shuffled, following the experimental design from Sokar et al. [2023]. As demonstrated in Sec. 3, while models in standard supervised learning settings able to scale effectively and maintain high performance, introducing non- contstttionnaitt leads to failure adaptation for baselines that use fully connected layers and the Adam optimizer. This issue exacerbated model scale increases.
Our results, presented Fig. 7, show that combining the multi-skip architecture for the MI. component with the Kronecker-factores optimizer and Layer Normalization enables near-perfect
18

---

continuous adaptation. The models quickly adapt to the changing optimization problem following label reshuffling, with gradient magnitudes remaining stable throughout the process.
C.3 Architecture and Optimizer Ablations
In this work, we introduce the multi-skip architecture, an extension of the standard residual MLP design, and propose the use of the Kronecker- factored optimizer for online deep RL. While these techniques form the basis of our primary interventions, our broader goal is not to prescribe a fixed set of methods, but rather motivate a general class of architectural and optimization interventions that promote healthy gradient flow in deep networks. To this end, we expand the scope of our evaluation by incorporating a wider range of baselines. Specifically, we compare various optimizer choices, including Adam and AdaBelief [Zhuang et al., 2020]. alongside MLP architectures such as the standard residuapl MLP [He et al., 2016] and DenseNet [Huang al., 2017]. These architectures have been previously explored in the context of scaling networks in online deep RL [Lee et al., 2025, Ota et al., 2024], providing a relevant basis for comparison.
We also evaluated state-of-the-nn optimizers that have demonstrated success in training large-scale models such as transformers in supervised learning. Specifically, tested Shampoo [Gupta et al., 2018], a second-order optimizer that maintains and preconditions gradients using full-matrix statistics per layer, and Apollo [Ma, 2020], adaptive optimizer that leverages curvature information without explicitly computing or storing second-order matrices.
Despite extensive hyperparameter tuning for both methods, we were unable to achieve strong performance in the online deep RL setting. This suggests that further investigation is needed to understand the key properties required for these optimizers to be effective in this regime. Asadi et al. [2023], Ceron and Castro [2021] demonstrate that optimizer behavior plays a critical role in the training dynamics of online deep RL methods, with Asadi et al. [2023] showing that stale optimizer states can hinder learning, and Ceron and Castro [2021] ] revealing that optimizer sensitivity interacts with the choice of loss function, particularly when comparing Huber and MSE losses.
We present the results for PPO and PQN across all tested optimizers in Fig. 14.
MLP Depth: Small
MLP Depth: Medium
MLP Depth: Large
Hummn Normaltzed Score
2.0
RAdam
AdaBelief RAdam
AdaBellef
AdaBelief
Apollo Shampoo
Apollo
RAdam Apollo
2.0
Shampoo
Shampoo
-03
24
08
10
step
Step
MLP Depth: Small
MLP Depth: Medium
MLP Depth: Large
Human Non-Noamalledd Score
AdaBelief
0.5
0.4
RAdam
AdaBelief
AdaBelief
Apollo
Apollo
RAdam
Apollo
02-
RAdam
-0.2
Shampoo
Shampoo
Shampoo
-04
Global
Global Step
Global
Figure 14: Median human normalized scores on Atari-10 for PPO (top row) and PQN (bottom row), comparing range of optimizers including RAdam, AdaBelief. Shampoo, Apollo, and Kron (shown in the main curves). While adaptive optimizers like AdaBelief show some robustness, only Kron consistently enables stable and performant training models scale. Each curve represents the mean performance across three random seeds per algorithm, with shaded areas indicating 95% bootstrap confidence intervals.
19

---

C.4 Results with the Multi-Skip Architecture.
We present the full learning curves comparing the proposed multi-skip architecture to the baseline fully connected architecture across all depths and widths studied in the paper. We follow the experimental protocol of Obando Ccron et al. [2023], Ceron al. [2024b], Agarwal et al. [2021], running each experiment with three random seeds.
Width:
Small
Medium
Large
Baseline
Depth: Small
Depth: Medium
Depth: Large
Human Normalized Score
0.6
0.4
0.2
0.0
0M
10M
20M
30M
40M
0M
10M
20M
30M
40M
0M
10M
20M
30M
40M
Steps
Steps
Environment
Figure 15: Median human-normalized scores with PQN on the Atari-10 benchmark, comparing the baseline agent and the proposed multi-skip architecture across varying depths and widths. The multi-skip architecture not only improves performance at shallow depths, but also enables PQN to remain trainable across all scales considered, whereas the baseline MLP rapidly collapses as depth and width increase. Each curve represents the mean performance across three random seeds per algorithm, with shaded areas indicating 95% bootstrap confidence intervals.
C.5 Ablation on the number of skip connections.
To isolate the effect of skip length on performance, we fix the main network of our proposed MultiSkip architecture (in large size, which includes 5 residual blocks) and vary how many of these blocks receive skip connections from the encoder. When Skip= = k, we apply the encoder features as skip connections to the first & residual blocks immediately following the encoder, while the remaining (5 - k) blocks operate without direct encoder input. The table below reports human-normalized scores on the Atari- benchmark Agarwal et al. [2021].
Table 2: fluman-normalinn scores on the Atari-10 benchmark, varying the number of residual
blocks (k) that receive skip connections from the encoder. Performance generally improves more connections are added.
Environment
Skip=1
Skip=2
Skip=3
Skip=4
Skip=5
Amidar-v5
0.20
0.17
0.19
0.20
0.36
BattleZone-v5
0.01
0.67
0.62
0.60
0.69
Bowling-v5
0.07
0.04
0.04
0.08
0.23
DoubleDunk-v5
-2.09
-2.00
-1.55
-1.36
-1.32
Frostbite-v5
0.67
0.70
0.79
0.92
0.88
KungFuMaster-v5
0.93
0.95
0.93
1.12
0.98
N5
0.79
0.65
0.72
0.85
1.24
Phoenix-v5
0.69
0.68
0.68
0.66
0.66
Qbert-v5
0.84
1.01
1.06
0.91
1.09
Riverraid-v5
0.42
0.44
0.65
0.69
0.99
Aggregate (mean)
0.25
0.33
0.41
0.47
0.58
Performance steadily improves as more skip connections are added, peaking when all blocks are connected. This supports original design decision to broadcast features to all MultiSkip blocks.
20

---

C.6 Results with the Kron Optimizer.
We present the full learning eurves comparing the Kron optimizer to the baseline RAdam optimizer originally used in PQN [Gallici et al., 2025], across all depths and widths studied in the paper. We follow the experimental protocol of Obando Ceron al. [2023], Ceron al. [2024b], Agarwal et al. [2021], running each experiment with three random seeds.
Width:
Small
Medium
Baseline
Depth: Small
Depth: Medium
Depth: Large
Human-Normall Normalzzed Score
0.8
0.6
0.4
0.2
0.0
20M 30M
40M
0M
10M
40M
20M 30M
Environment Steps
Environment Steps
Environment Steps
Figure 16: Median human- mormalzee scores with PQN on the Atari-10 benchmark, comparing the Kron optimizer to the baseline RAdam optimizer across varying depths and widths. Similar to the multi-skip architecture, Kron not only improves performance at shallow depths, but also enables PQN to remain trainable across all scales considered. In contrast, performance with RAdam rapidly collapses as depth and width increase. Each curve represents the mean performance across thee random seeds per algorithm, with shaded areas indicating 95% bootstrap confidence intervals.
C.7 Justification for Selected Interventions.
The choice of fronecker-factored optimization and the MultiSkip architecture was the result of systematic exploration of candidate interventions aimed at improving gradient flow, plasticity, and stability in deep RL. We evaluated wide range of alternative methods from prior work known to mitigate optimization pathologies [Moalla et al., 2024, Juliani and Ash, 2024].
Our evaluated methods, summarized in Tab. 3, included:
Second-order and adaptive optimizers: Apollo [Ma, 2020], Shampoo [Gupta al., 2018), AdaBelief [Zhuang et al., 2020].
Regularization: L2 norm penalties [Kumar et al., 2023], weight clipping, and weight decay [Elsayed al., 2024].
Activation functions: GELU (Hendrycks, 2016] and CReLU [Abbas et al., 2023].
Learning rate schedules: Cosine annealing and cyclic schedulers.
Learning rate scaling: Multiplying and dividing the default learning rate (2.5e-4) by 10 to compensate for increased network scale.
As shown in the table, none of these interventions consistently improved performance compared to our proposed combination. This motivated our decision to focus on the combination of Kronecker- factored optimization and the multi-skip architecture.
21

---

Table Comparison mean human-normaliied scores on Atari-10 for alternative interven- tions. Our proposed method (Ours) is compared against ablations using only a single alternative intervention. The results (3 seeds per experiment) show that no single alternative consistently matches the performance of our combined approach.
Environment
Cusine I.R
Cyclic
GELI
CRdli
Wright Clip
Wright Deray
T.R=3.006-1
Amidae-v5
0.355897
0.002276
0.015143
0.016514
0.029293
0.012283
0.013013
0:018848
0.078660
BanlcZone
0,694566
0.812997
0.800287
0,015505
0.004594
0.06599
0.035600
0.061446
0.0198122
0,575407
Bowling-yS
0,251K31
0050145
-0.066221
0,025706
.0008811
-0000722
0.8900533
0.051235
-0.086483
0.367587
DaubleDank-w5
-1.318182
2.318182
-2.318182
-2363636
-2.31188
-2,656636
2. 2.454555
-2,454545
-2454545
-2 2,000909
Troathite-05
0,881087
-0001921
0.027706
0.005590
00.010936
0.004755
0.075372
0.021736
0.008034
0.110036
0,9752251
00110055
0,003181
0.00022
0.011500
Q.011278
0.011500
0,011005
00011055
1.362077
Name ThisCism--55
1.3421666
-U630191
-0.132767
40.261400
40.195430
-0.681477
4,197795
148227
-0.154417
0.747251
Phoenix
0.655141
0.026877
0,087257
0.090014
0.090166
0.068722
0,093940
0.093949
-0.102358
0,611968
(bert-5
1009894
0013814
0.0102522
0.001235
0,008359
0,006102
0.016259
0:001305
0.112282
Riveraid-u5
0,993568
0.345100
0.30436
0.056646
-0056404
0,293514
-0,044456
0.237365
D Hyper-parameters
Below, we provide details of the hyperparameters used throughout the paper for each algorithm. In general, they match those proposed in the corresponding original papers.
Table 4: PQN Hyperparameters
Hyperparameter
Value / Description
Learning rate
2.5e-4
Anneal ir
False (no learning rate annealing)
Num envs
128 (parallel environments)
Num steps
(steps per rollout per environment)
Gamma
0.99 (discount factor)
Num minibatches
32
Update epochs
2 (policy update epochs)
Max grad norm
10.0 (gradient clipping)
Start
1.0 (initial exploration rate)
End e
0.005 (final exploration rate)
Exploration fraction
0.10 (exploration annealing fraction)
Q lambda
0.65 (Q(A) parameter)
Use In
True (use layer normalization)
Activation fn
relu (activation function)
22

---

Table 5: PPO Hyperparameters
Hyperparameter
Value Description
Learning rate
2.5e-4
Num envs
8
Num steps
128 (steps per rollout per environment)
Anneal Ir
True (learning rate annealing enabled)
Gamma
0.99 (discount factor)
Gae lambda
0.95 (GAE parameter)
Num minibatches
Update epochs
4
Norm adv
True (normalize advantages)
Clip coef
0.1 (PPO clipping coefficient)
Clip vloss
True (clip value loss)
Ent coef
0.01 (entropy regularization coefficient)
Vf coef
0.5 (value function loss coefficient)
Max grad norm
0.5 (gradient clipping threshold)
Use In
False (no layer normalization) relu (activation function)
Activation fn
Shared cnn
True (shared CNN between policy and value networks)
Table 6: PPO Hyperparameters for IsaacGym
Hyperparameter
Value Description
Total timesteps
30,000,000
Learning rate
0.0026
Num envs
4096 (parallel environments)
Num steps
16 (steps per rollout)
Anneal Ir
False (disable learning rate annealing)
Gamma
0.99 (discount factor)
Gae lambda
0.95 (GAE lambda)
Num minibatches
2
Update epochs
4 (update epochs per PPO iteration)
Norm adv
True (normalize advantages)
Clip coef
0.2 (policy clipping coefficient)
Clip vloss
False (disable value function clipping)
Ent coef
0.0 (entropy coefficient)
Vf coef
2.0 (value function loss coefficient)
Max grad norm
1.0 (max gradient norm)
Use In
False (no layer normalization)
Activation fn
relu (activation function)
23

---

Table 7: DQN Hyperparameters
Hyperparameter
Value Description
Learning rate
le-4
Num envs
Buffer size
1,000,000 (replay memory size)
Gamma
0.99 (discount factor)
Tau
1.0 (target network update rate)
Target network frequency
1000 (timesteps per target update)
Batch size
32
Start
1.0 (initial exploration epsilon)
End e
0,01 (final exploration epsilon)
Exploration fraction
0.10 (fraction of total timesteps for decay)
Learning starts
80,000 (timesteps before training starts)
Train frequency
(training frequency)
Use In
False (no layer normalization)
Activation fn
relu (activation function)
Table 8: Rainbow Hyperparameters
Hyperparameter
Value Description
Learning rate
6.25e-5
Num envs
Buffer size
1,000,000 (replay memory size)
Gamma
0.99 (discount factor)
Tau
1.0 (target network update rate)
Target network frequency
8000 (timesteps per target update)
Batch size
32
Start
1.0 (initial exploration epsilon)
End
0.01 (final exploration epsilon)
Exploration fraction
0.10 (fraction of total timesteps for decay)
Learning starts
80,000 (timesteps before training starts)
Train frequency
(training frequency)
N step
(n-step Q-learning horizon)
Prioritized replay alpha
0.5
Prioritized replay beta
0.4
Prioritized replay eps
le-6
N atoms V min
51 (number of atoms in distributional RL)
10 (value distribution lower bound)
V max
10 (value distribution upper bound)
Use In
False (no layer normalization)
Activation fn
relu (activation function)
Table 9: Image Classification Hyperparameters (CIFAR-10)
Hyperparameter
Value
Batch size
256
Epochs
100
Learning rate
0.00025
24

---

Table 10: SAC Hyperparameters
Hyperparameter
Value / Description
Critic block type
SimBa
Critic num blocks
{2, 4, 6, 8)
Critic hidden dim
[512, 1024, 1536, 2048)
Target critic momentum (r)
5e-3
Actor block type
SimBa
Actor num blocks
[1. 3,
Actor hidden dim
{128, 256, 384, 512)
Initial temperature (oo)
le-2
Temperature learning rate
le-4
Target entropy (?(*)
LA|/2
Batch size
256
Optimizer
(AdamW, Kron)
AdamW's learning rate
le-4
Kron's learning rate
5e-5
Optimizer momentum (81, 32)
(0.9, 0.999)
Weight decay (A)
le-2
Discount (7)
Heuristic
Replay ratio
2
Clipped Double Q
False
Table 11: DDPG Hyperparameters
Hyperparameter
Value Description
Critie block type
SimBa
Critic num blocks
{2, 4. 6, 8)
Critic hidden dim
[512, 1024, 1536, 2048)
Critic learning rate
le-4
Target critic momentum (7)
5e-3
Actor block type
SimBa
Actor num blocks
2, 4]
Actor hidden dim
[128, 256, 384, 512)
Actor learning rate
le-4
Exploration noise
V(000013)
Batch size
256
Optimizer
[AdamW, Kron]
AdamW's learning rate
le-4
Kron's 's learning rate
5e-5
Optimizer momentum (81, (2))
(0.9, 0.999)
Weight decay (A)
le-2
Discount (7)
Heuristic
Replay ratio
2
Clipped Double Q
False
25

---

E
Compute Details
All experiments were conducted on single-GPU setup using an NVIDIA RTX 8000, 12 CPU workers, and 50GB of RAM.
Table 12: Training times across model scales for two optimizers K-FAC shows increased cost as depth and width grow.
Depth
Width
Optimizer
Time
RAdam
Small
Small
Adam
51m
Small
Medium
Adam
53m
Small
Large
Adam
57m
Medium
Small
Adam
Ih 4m
Medium
Medium
Adam
Ih 10m
Medium
Large
Adam
Ih 11m
Large
Small
Adam
Ih 18m
Large
Medium
Adam
1h 18m
Large
Large
Adam
1h 27m
Kron
Small
Small
Kron
Ih 59m
Small
Medium
Kron
2h 27m
Small
Large
Kron
3h 38m
Medium
Small
Kron
2h 44m
Medium
Medium
Kron
3h 32m
Medium
Large
Kron
5h 59m
Large
Small
Kron
3h 27m
Large
Medium
Kron
4h 36m
Large
Large
Kron
7h 42m
E.1 Results on the Full ALE
In this section, we provide the full training curves corresponding to the aggregated results shown in Sec. 4.3, where we evaluate the performance of the PQN and PPO agents on the full set of environments from the ALE after applying our two proposed gradient interventions. The per- environment learning curves are presented in Fig. 17 for PQN and Fig. for PPO. We follow the experimental protocol of Obando Ceron et al. [2023], Ceron et al. [2024b], Agarwal et al. [2021], running each experiment with three random seeds.
26

---

Ours
Baseline
A
Condopda
CraperCarmnnddd
Cracyll
0
Poo
Copfisnn3
1010
3030
MEW Phoentouu
1
Genqueeus
Spacttondereed
Staniam sarud
UpMCCommi
0.2
2016
WiardOtrie-
Techhouegg
1000
Figure 17: Mean human-normalized score the full ALE suite, comparing the baseline PQN agent (light curves) with the augmented agent using our combined gradient interventions (dark curves).
27

---

Ours
Baszline
Artertord
Adtrradees
Duamfiiderch
Seraetudd
Bowlingud
Soe ngnd
2001
2014
1000
Drooloedin
Ocrtpodoro
CapporCammant
Vide
W0
Frckenu5
Copharas
vide
2011
1000
1006
1966
60.6
1420
Kingf slaiterd
Phoerboo
Ad8
J:M
J0NN
100m
moom
Seaagest.1
Spocch Fo
StarCCen vor
Revistan
-22
600
236
2004
Weder
3000
2004
1400
Figure 18: Mean human- human-normalized score on the full ALE suite, comparing the baseline PPO agent (light curves) with the augmented agent using our combined gradient interventions (dark curves).
28

---

E.2 Simba on DMC
In this section, we present the full results accompanying the experiments combining Simba [Lee et al., 2025] with our proposed gradient interventions, as introduced in Sec. 4.3. For these experiments, we retain Simba's original architectural choices but replace the AdamW optimizer with Kron.
We compare Simba using both SAC and DDPG as the underlying RI. algorithmss While SAC generally outperforms DDPG, consistently observe that scaling depth and width, either independently or jointly, leads to a degradation in performance with Simba. However, this degradation is mitigated, and in many cases reversed, when using the Kron optimizer, resulting in improved performance as model capacity increases.
The following figures illustrate these findings:
Humana
Humannodd
Dag But
Figure 19: SAC scaling depth
A
Figure 20: SAC scaling width
Ehrooned Walk
Tret
Dog
Figure 21: SAC scaling both depth and width
Humaro Mllk
Hlamarodd
Tot
Dog
Figure 22: DDPG scaling depth
29

---

Figure 23: DDPG scaling width
Figure 24: DDPG scaling both depth and width
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction are grounded in empirical evidence gathered under controlled settings, spanning both supervised and reinforcement learning regimes. We identify and characterize gradient pathologies that emerge with increasing model scale. The proposed gradient-based interventions are evaluated in a modular fashion throughout the paper. Furthermore, the scope of our conclusions is clearly delineated, with detailed descriptions of the specific algorithms and environments studied, ensuring alignment between claims and contributions. While claims are carefully scoped, our extensive experimentation provides preliminary evidence suggesting that the uncovered phenomena may generalize broader applications in deep learning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
31

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper does not include novel theoretical results, but presents well- established claims Section 2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The experimental setup thoroughly documented in each section presenting results. We specify the number of independent runs, the methods used for aggregating results, and provide detailed descriptions of the algorithms, hyperparameters, and tasks. This level of disclosure ensures that the main experimental findings and conclusions can be independently reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
32

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Although the data and code will not accompany the submission of this work, they will be released upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ public/gen /iwidde/CoeeSuunnne
oniolicyyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: www.de/s////eeeeeee/oeoc.eo..e / /Coo//oooeeeeeeeeeeeee ssionnoll
/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide all essential details of the experimental setup in the main paper, including key training configurations. Additional specifics, such as hyperparameter values and environment settings, are included in the Appendix (Appendix D, Appendix A), ensuring that readers have access all necessary information to understand and interpret the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
33

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Throughout the paper, report statistical variability using confidence intervals and standard deviations in training curves, error bars in bar plots, and percentiles in boxplots. We also clearly state the aggregation methods used for each plot (c.g.. mean, median. inter-quantil mean), ensuring transparency and appropriate interpretation of the statistical significance of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is provided in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https ///:://nneee http///eeuurr.eeeeee/eee/e https.cc/pubble hetee/ wtt.hhhhiiiiiiiieenee""
Answer: [Yes]
Justification:
The authors preserved anonymity during the submission of this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
34

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix ??.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
35

---

Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's ' creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
36

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: /ttp///eerpp../e/ cc/Connerrences//000 /LLM) for what should or should not be described.
37


Reviews:
[{"review_id": "P3J2Scz0Oy", "reviewer": "Reviewer_5mBc", "summary": "This paper looks to address a major challenge in reinforcement learning. Namely, how to scale deep RL algorithms to larger architectures and datasets. As shown in prior work, RL does not scale in the same respects that generative modeling nor supervised learning have scaled. There are many theories as to why and yet this problem remains poorly understood. The authors highlight the core challenge is the inherent non-stationarity of the training objective in RL.\n\nThe authors proposed looking at how gradients propagate throughout the network during training, rather than modifying a specific RL algorithm or aspects of RL training. The authors note that this approach is well-studied in supervised learning, but unexplored in RL. (I am unaware of anyone addressing this issue in RL, but my expertise is also outside RL. It would surprise me if no one studied this before.)\n\nThe contribution of the paper is an extensive study of how gradient decompositions interact with non-stationary problems. Their proposed approach is to actively control gradient propagation in the network, through both second-order optimization methods and multi-skip residual connections. They show that this approach to stable gradients can greatly improve a variety of RL algorithms across a variety of environments. This is a promising line of research that could unlock the potential of scaling RL algorithms to larger architectures.\n\nOverall, this paper seems thorough and well-presented. It introduces a highly relevant problem, namely that of understanding why RL algorithms fail when networks are scaled up. It provides excellent evidence for this problem in (Section 3), reasonable and understandable solutions (Section 4), and comprehensive experiments (Section 4-5, Appendix).", "questions": "- Is it true that only MLP’s (aside from convnet image encoders for ALE) were studied in this paper? How would you apply these approaches to different architectures (transformers, diffusion policy, etc)? Or is more research needed?\n\n- Can the approach be augmented for different rates of change in stationarity? How much non-stationarity can the new approach tolerate before failure?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "- The paper is well-written and clear. \n- The paper could use a front figure on the second page to highlight the key mechanisms of the approach.\n- The authors give a good review of deep reinforcement learning as well as gradient propagation and stability.\n- The paper is friendly to readers from both a supervised learning and reinforcement learning background.\n- The experiments to study this problem are extensive. Ranging from the full ALE and several RL algorithms.\n- The improvements due to gradient stabilization are large on several of the environments. Though I’m someone surprised that the approach doesn’t lead to reduced variance in outcome.\n- As the authors note, the study is computationally expensive. Due to this, the networks sizes were not scaled beyond 120 million parameters. Given the popularity of RLHF on large language models, it would be useful to see a single run of the approach on a network of similar scale to see if performance improves.\n- A question not addressed in this paper is the rate of change in stationarity. Small and large changes in stationarity may affect the underlying training algorithm differently. On different RL problems or algorithms, the data collected from the policy can change at different rates even during the same run of RL training. It’s not clear to me how this approach could detect and address this aspect of the problem.", "was_revised": true, "final_justification": "I think this is a great paper taking insights from the generative modeling and supervised learning community to tackle an essential problem for getting RL to scale to larger networks. While I'm not from the RL field, I've known many groups trying to address this difficult problem themselves and have commented on the need to having something that scales to larger networks in RL. Other groups have shared a dissatisfaction with current approaches of getting RL to scale to larger networks, which makes me excited to see a work scale RL to large networks so robustly. The experiments in the paper were thorough and well-motivated and improved during the discussion period. Other reviewers brought up concerns for the degree of novelty given many of the ideas being discovered in these other fields. I personally do not think that's too important in this context. The paper does a great job building off ideas from disparate fields and getting them to work in more challenging non-stationary domains. Reading the paper gives me confidence the ideas work and generalize.", "rebuttal": "We thank the reviewer for their feedback! We are happy that the reviewer found that “the paper is well-written and clear” and that “the experiments to study this problem are extensive”. We respond to their main concerns below.\n\n# **Q1) Is it true that only MLP’s (aside from convnet image encoders for ALE) were studied in this paper?**\n\nYes. Our experiments focus on combinations of convolutional backbones (e.g. to process ALE frames) and fully‑connected heads i.e., MLP architectures, mirroring the vast majority of prior model‑free deep RL work [1,2,3]. \n\n# **Q2) How would you apply these approaches to different architectures (transformers, diffusion policy, etc)? Or is more research needed?**\n\nKron and standard normalization layers extend directly to any layer whose parameters or activations admit a preconditioning or normalization step: transformers, LSTMs, diffusion‑policy networks, and others.\n\nOur multi‑skip broadcast mechanism relies on propagating intermediate feature maps forward to all downstream layers. While the concept could extend to attention-based models (e.g. broadcasting early Transformer block outputs to later ones), deciding what to broadcast (keys, values, token embeddings, etc.) and how to merge them meaningfully (residual, cross-attention, etc.) is non-trivial. We expect this to require additional research to identify the right injection and aggregation strategies in attention‑based or recurrent settings. We view this as promising future work, and we will add a discussion around this in the final version.\n\nWe also ran PQN with LSTM policies on the full ALE benchmark (57 games, 200M frames) to evaluate whether our gradient interventions generalize to recurrent architectures. The table below shows improvements of Kron + **MultiSkip** + LSTM  over Adam + **MLP** + LSTM across all environments.\n\n| **Statistic**                     | **Value**    |\n|-----------------------------------|-------------:|\n| Average Improvement (%)           | 1558.23      |\n| Median Improvement (%)            | 86.93        |\n| Games with Improvement            | 48 / 55      |\n| Percentage of Games Improved (%)  | 87.27        |\n| Maximum Improvement (%)           | 43,844.44    |\n| Minimum Improvement (%)           | -219.38      |\n\nDue to space constraints, we cannot include the full per-game results in the main text or in the rebuttal, but we will add them to the appendix, along with:\n\n- Per-game learning curves (similar to Figs. 16 and 17)\n\n- Per-game % improvements (similar to Fig. 6, right).\n\n# **Q3) Can the approach be augmented for different rates of change in stationarity? How much non-stationarity can the new approach tolerate before failure?**\n\nYes, our approach generalizes across a range of non-stationarity regimes, and our results suggest it performs robustly under varying degrees of stationarity. \n\nPreviously, we demonstrated this in online RL experiments, where non-stationarity arises primarily from  policy-induced distribution shifts and target bootstrapping. To further evaluate the generality of our method across different degrees of non-stationarity, we introduced new **offline RL experiments** where the only form of non-stationarity comes from bootstrap targets (i.e., no policy-induced drift, but still changes in the network targets for actions after Bellman backups propagate), using a challenging benchmark from a recent study [4]. Park, Seohong, et al. (2025) showed that standard offline Q-learning baselines fail even with 100M samples and near-perfect coverage.\n\nWe retrained those same baselines with and without our Kron + Multi-Skip interventions. Across nearly all environments and algorithms, our method consistently improves performance, even in these highly brittle offline settings (see table below). Notably, in difficult environments like humanoidmaze-giant-navigate, gains were substantial (in many cases even over 90%).\n\n| **Algorithm**                 | **cube-octuple-play** | **humanoidmaze-giant-navigate** | **puzzle-4x5-play**  | **puzzle-4x6-play**  |\n|-------------------------------|-----------------------|---------------------------------|----------------------|----------------------|\n| **contrastive-rl (Baseline)** | 0.000 ± 0.000         | -                               | 0.060 ± 0.085        | 0.047 ± 0.066        |\n| **contrastive-rl (Ours)**     | 0.000 ± 0.000         | -                               | 0.047 ± 0.047        | 0.013 ± 0.019        |\n| ---                           |                       |                                 |                      |                      |\n| **dsharsa (Baseline)**        | **0.260 ± 0.028**     | 0.713 ± 0.009                   | 0.973 ± 0.019        | 0.727 ± 0.028        |\n| **dsharsa (Ours)**            | 0.100 ± 0.009         | **0.953 ± 0.028**               | **0.993 ± 0.009**    | **0.853 ± 0.019**    |\n| ---                           |                       |                                 |                      |                      |\n| **gc-fbc (Baseline)**         | 0.000 ± 0.000         | 0.040 ± 0.019                   | **0.020 ± 0.009**    | 0.007 ± 0.009        |\n| **gc-fbc (Ours)**             | 0.000 ± 0.000         | **0.267 ± 0.038**               | 0.000 ± 0.000        | 0.000 ± 0.000        |\n| ---                           |                       |                                 |                      |                      |\n| **gc-fql (Baseline)**         | 0.000 ± 0.000         | 0.053 ± 0.000                   | 0.160 ± 0.057        | 0.187 ± 0.019        |\n| **gc-fql (Ours)**             | 0.000 ± 0.000         | **0.920 ± 0.057**               | **0.600 ± 0.566**        | **0.647 ± 0.405**    |\n| ---                           |                       |                                 |                      |                      |\n| **gc-iql (Baseline)**         | 0.000 ± 0.000         | -                               | 0.200 ± 0.000        | 0.100 ± 0.047        |\n| **gc-iql (Ours)**             | 0.000 ± 0.000         | -                               | 0.200 ± 0.000        | **0.213 ± 0.094**    |\n| ---                           |                       |                                 |                      |                      |\n| **gc-sac+bc (Baseline)**      | 0.000 ± 0.000         | 0.347 ± 0.207                   | 0.187 ± 0.019        | 0.100 ± 0.047        |\n| **gc-sac+bc (Ours)**          | 0.000 ± 0.000         | **0.880 ± 0.019**               | 0.193 ± 0.009        | **0.333 ± 0.000**    |\n| ---                           |                       |                                 |                      |                      |\n| **h-fbc (Baseline)**          | 0.253 ± 0.057         | 0.260 ± 0.009                   | 0.053 ± 0.019        | 0.047 ± 0.028        |\n| **h-fbc (Ours)**              | 0.280 ± 0.038         | **0.360 ± 0.000**               | **0.173 ± 0.019**    | **0.180 ± 0.104**    |\n| ---                           |                       |                                 |                      |                      |\n| **h-iql (Baseline)**          | 0.067 ± 0.000         | 0.673 ± 0.009                   | 0.093 ± 0.019        | 0.120 ± 0.000        |\n| **h-iql (Ours)**              | **0.367 ± 0.009**     | **0.807 ± 0.047**               | **0.227 ± 0.019**    | 0.133 ± 0.038        |\n| ---                           |                       |                                 |                      |                      |\n| **n-gc-sac+bc (Baseline)**    | 0.000 ± 0.000         | 0.847 ± 0.009                   | **0.907 ± 0.000**    | 0.300 ± 0.066        |\n| **n-gc-sac+bc (Ours)**        | 0.000 ± 0.000         | **0.920 ± 0.019**               | 0.547 ± 0.038        | **0.780 ± 0.028**    |\n| ---                           |                       |                                 |                      |                      |\n| **sharsa (Baseline)**         | **0.427 ± 0.038**     | 0.533 ± 0.057                   | 0.893 ± 0.075        | 0.653 ± 0.075        |\n| **sharsa (Ours)**             | 0.280 ± 0.019         | **0.813 ± 0.038**               | **1.000 ± 0.000**    | **0.887 ± 0.047**    |\n\n*Baseline algorithms are implemented in [4]. Ours corresponds to adding Kron + MultiSkip networks. gc- means “goal-conditioned” and h- means “hierarchical”*\n\nCombined with the online RL results in the main paper, which involve compounded non-stationarity from both policy and value updates, these new results show that our approach:\nIs effective across different rates and sources of non-stationarity.\nDelays or prevents failure even under conditions where standard methods collapse.\nOur method not only tolerates significant non-stationarity, but also improves stability under a broad spectrum of dynamics.\n\n\n# **Paper Formatting Concerns**\n\n*Thanks a lot for bringing these points to our attention. We will incorporate all suggested corrections into the final version of the paper.*\n\n\n# **References**\n\n[1]. Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J. N., Dziugaite, G., Precup, D., and Castro, P. S.. Mixtures of experts unlock parameter scaling for deep rl. ICML’24\n\n[2]. Lyle et al. \"Understanding plasticity in neural networks.\" ICML’23\n\n[3]. V Mnih et al. Human-level control through deep reinforcement learning, Nature’ 15\n\n[4]. Park, Seohong, et al. Horizon Reduction Makes RL Scalable, arXiv’25."}, {"review_id": "NFNGCd4Prr", "reviewer": "Reviewer_rJWh", "summary": "This work considers how to make Deep RL scale better with network size. They highlight that RL objectives are non-stationary due to both a drift in the TD-error target as well as a changing policy. Then, due to poor plasticity effects of some neural networks, learning can get stalled. They examine this effect also on supervised learning tasks that also show this effect. They claim that the issue is partially due to vanishing gradient magnitudes, and show that gradient magnitudes becomes smaller when increasing network depth or width, and learning performance also degrades. To tackle these problems they use skip connections and also a Kronecker factored optimizer. These techniques remedy the issue and greatly improve performance in both RL tasks and the supervised learning task that they considered. Experimentally, they were mainly working with the PQN algorithm (a recent scalable and strong parallelized Q-learning based algorithm), and their interventions lead to around 80% improvement in performance. They also combined their method with the IMPALA CNN architecture, and showed an improvement there as well, moreover, they considered combining the Kron optimizer with the SIMBA architecture (that already has skip connections) and SIMBA also achieved better scaling with the network size and significantly improved performance. Moreover, combining their methods with PPO also lead to better scaling and improved performance.", "questions": "In Figure 5 (right), is the gradient magnitude for +Kron the magnitude of the update vector (i.e., inverse curvature * gradient), or is it just the magnitude of the gradient? To clarify, does the optimizer lead to different regions in the optimization space where the gradient does not become small, or does it just add a preconditioner to the gradient that adjusts the magnitude?\n\n“significantly boots performance” → significantly boosts performance", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n* They show strong performance improvements.\n\n* The paper is clearly written.\n\n* The results are consistent across many different architecture types and algorithms.\n\nWeaknesses\n\n* The novelty is not that high as skip connections are well-known, the Kron optimizer is an existing one, and gradient/plasticity issues are discussed in other works as well.\n\n* Using the Kron optimizer introduces a significant computational overhead (e.g., for the large network size setting, the computation time goes from 1h27min to 7h 42min in Table 1). They have discussed this in their limitation section.\n\n* The gradient magnitudes becoming smaller may be coincidental to the performance dropping; it did not seem proven that performance drops because the gradient magnitudes are small.\n\n\nOverall, this is a strong well-written paper that shows how to improve performance via better scalability in Deep RL.", "was_revised": true, "final_justification": "The authors responded to my questions, and I believe that review remains adequate, and the paper is a clear accept. The authors argued about the novelty, but I believe my view regarding the novelty remains adequate as the novelty appears marginal without any groundbreaking ideas. The response regarding the gradient magnitudes was interesting.", "rebuttal": "We thank the reviewer for their feedback! We are happy that the reviewer found that “the paper is clearly written” and that “the results are consistent across many different architecture types and algorithms”. We respond to their main concerns below.\n\n# The novelty is not that high as skip connections are well-known, the Kron optimizer is an existing one, and gradient/plasticity issues are discussed in other works as well.\n\nWhile the individual components (skip connections, Kronecker-factored optimization, and plasticity analysis) are based on existing ideas, our contributions lie in how we integrate and apply them in a deep RL context to address specific failure modes, particularly those involving gradient degradation.\n\nWhile Kronecker-factored methods (e.g., [1]) have been explored before, they are not commonly adopted in deep RL. Our contribution includes a careful and extensive empirical study of using a Kronecker-factored optimizer within modern deep RL. We position this as a step toward broadening the toolkit for deep RL practitioners.\n\nFor architectures, we extend prior work on skip connections with our multi-skip network, which broadcasts CNN-extracted features to all subsequent layers. To our knowledge, this form of broadcasting has not been explored in deep RL, making the design novel. Figure 4 illustrates its structure relative to standard ResNet-style MLPs and DenseNets. Importantly, these multi-skip connections emerged as an effective mechanism for mitigating the vanishing gradient pathologies identified in our analysis. We have updated the paper to make the novelty clear.\n\n# The gradient magnitudes becoming smaller may be coincidental to the performance dropping; it did not seem proven that performance drops because the gradient magnitudes are small. \n\nWhile asserting a direct causal relationship is difficult, we do observe a strong correlation between vanishing gradients and performance drop. The fact that performance increases when we mitigate vanishing gradients strengthens this connection. Nonetheless, we will clarify in the final version that this correlation does not explicitly imply causality.\n\n# In Figure 5 (right), is the gradient magnitude for +Kron the magnitude of the update vector (i.e., inverse curvature * gradient), or is it just the magnitude of the gradient? \n\nIt represents the magnitude of the preconditioned gradient. That is, the update vector resulting from the multiplication of the gradient by the preconditioning matrix.\n\n# To clarify, does the optimizer lead to different regions in the optimization space where the gradient does not become small, or does it just add a preconditioner to the gradient that adjusts the magnitude?\n\nThis is a great question, but quite difficult to answer assertively, particularly given the high levels of non-stationarity in deep RL (which result in shifting optimization landscapes). We hypothesize that the improvements we observe are primarily due to better pre-conditioning, though a deeper understanding remains an open question. We agree this is an important direction for future investigation and we will add a note in the discussion accordingly.\n\n---\n*Thank you for the thorough review. Please let us know if any points remain unclear.*\n\n# References\n\n[1]. Firouzi, Mohammad. KF-LAX: Kronecker-factored curvature estimation for control variate optimization in reinforcement learning. arXiv’18."}, {"review_id": "Qnn0mwheQc", "reviewer": "Reviewer_vtbr", "summary": "In this work, the authors study the impact of scaling model parameters on the ability of models to learn and adapt to the nonstationary training rewards typically encountered during reinforcement learning. As the number of model parameters increases and the depth of the network increases, several well-known stability issues specifically hinder RL networks' ability to learn effectively. The address these issues the authors propose and study the impact of adding more long-range connections to propagate gradient information over long depths and incorporating higher-order optimizers to utilize better informed gradient updates during training. Empirically, the authors show that in the ALE learning environment that these updates result in improved gradient stability for large networks and generate networks is higher rewards in several ALE environments.", "questions": "How does the change in the skip length impact the training dynamics?", "limitations": "The limitations of the proposed method were outlined by the authors adequately in section 7 of the paper. Specifically, the study is conducted over a relatively small range of network sizes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n--------------\n1. This work is well-written with figures that illustrate visually the author's intentions and complement the writing seamlessly. I particularly enjoyed the figures in Section 4 that displayed the mean gradient norms and normalized scores.\n2. The topic of scaling RL to larger sizes is an interesting line of questioning, and the authors offer a systematic study of the impact of the 2 changes on the ability of these large networks to propagate gradient information effectively.\n3. Results are provided on several environments from the ALE environment and Issac Gym for a number of network architectures and using different training algorithms.\n4. Although the techniques are not novel, the application of them to stabilize the gradient flow during RL training appears to be a great application of which I was not previously aware. Overall, this study appears to be an incremental improvement on existing training strategies and provides interesting examples to improve the performance of RL models for networks and a large number of parameters.\n\nWeaknesses:\n------------------\n1. Neither of the proposed methods to stabilize the performance is novel and has indeed been studied extensively in several other contexts where the training dynamics were stable.\n2. Of the 2 proposals, I believe usage of higher-order optimizers to be the least novel innovation. I would assume in most contexts, integrating higher-order information would improve training and would be the default if it were not considerably more computationally expensive. This aspect is exacerbated by the fact that the authors are interested in scaling up the model size, which would make the application of higher-order optimization methods even more precarious.\n3. Skip connections were studied extensively in the context of CNNs, and it's not clear from the study what the minimum the skip connections should be to ensure gradient information is passed effectively.", "was_revised": true, "final_justification": "Based on the author's thorough rebuttal and comments regarding the input from other reviewers, I will raise my score.", "rebuttal": "We sincerely appreciate the reviewer’s valuable comments. We are happy that the reviewer found that “the paper is well-written and clear”. We're also pleased that the reviewer recognized “the systematic nature of our study” and that we present results “on several environments from the ALE environment and Issac Gym for a number of network architectures and different RL algorithms”. We respond to their main concerns below.\n\n# Neither of the proposed methods to stabilize the performance is novel and has indeed been studied extensively in several other contexts where the training dynamics were stable.\n\nWe agree that the individual techniques we employ, such as architectural modifications and second-order optimization, have been explored in other domains with more stable training dynamics. Our contribution lies in demonstrating how these techniques, when adapted thoughtfully, can address fundamental challenges in deep RL: the implicit non-stationarity that undermines plasticity and learning stability; a problem that is further exacerbated as model scale increases [1,2,3]. We approach this through targeted gradient interventions, which we show to be highly effective across diverse tasks and rl algorithms.\n\nWhile some components are well-known individually, our multi-skip architecture, which broadcasts CNN features to all downstream layers (see Figure 4), proved particularly effective for improving gradient flow under non-stationarity conditions. We believe the thoroughness of our experimentation and analysis makes these findings a valuable foundation for future research in deep RL at scale.\n\n# Of the 2 proposals, I believe usage of higher-order optimizers to be the least novel innovation. I would assume in most contexts, integrating higher-order information would improve training  ...\n\nWe agree that incorporating higher-order information should improve training in principle, and that computational cost remains a key barrier. However, in deep RL, simply introducing second-order methods does not guarantee improvements. In fact, our experiments with other second-order optimizers such as Shampoo [4] and Apollo [5], despite extensive hyperparameter tuning, did not yield gains (see Figure 13 in the appendix). This suggests that applying second-order optimization in deep RL is non-trivial.\n\nOur analyses and systematic experimentation were key in identifying Kronecker-factored optimization as particularly effective for improving trainability under non-stationary settings. We believe this careful empirical investigation offers valuable insights for future work aiming to move beyond the standard reliance on Adam in deep RL and towards scaling deep RL.\n\n# Skip connections were studied extensively in the context of CNNs, and it's not clear from the study what the minimum the skip connections should be to ensure gradient information is passed effectively. How does the change in the skip length impact the training dynamics?\n\t\nWe appreciate the reviewer’s insightful comment regarding the importance of understanding the minimum number and placement of skip connections required to ensure effective gradient propagation. While our original submission did not include a detailed ablation on this point, we have now conducted additional experiments to specifically probe this question.\n\nTo isolate the effect of skip length on performance, we fix the main network of our proposed MultiSkip architecture (in *large* size, which includes 5 residual blocks) and vary how many of these blocks receive skip connections from the encoder. When Skip = k, we apply the encoder features as skip connections to the first k residual blocks immediately following the encoder, while the remaining (5 - k) blocks operate without direct encoder input. The table below reports human-normalized scores on the Atari-10 benchmark [6]:\n\n| Environment         | Skip=1 | Skip=2 | Skip=3 | Skip=4 | Skip=5 |\n|---------------------|--------|--------|--------|--------|--------|\n| Amidar‑v5           | 0.20   | 0.17   | 0.19   | 0.20   | **0.36** |\n| BattleZone‑v5       | 0.01   | 0.67   | 0.62   | 0.60   | **0.69** |\n| Bowling‑v5          | 0.07   | 0.04   | 0.04   | 0.08   | **0.23** |\n| DoubleDunk‑v5       | -2.09  | -2.00  | -1.55  | -1.36  | **-1.32** |\n| Frostbite‑v5        | 0.67   | 0.70   | 0.79   | **0.92** | 0.88   |\n| KungFuMaster‑v5     | 0.93   | 0.95   | 0.93   | **1.12** | 0.98   |\n| NameThisGame‑v5     | 0.79   | 0.65   | 0.72   | 0.85   | **1.24** |\n| Phoenix‑v5          | **0.69** | 0.68   | 0.68   | 0.66   | 0.66   |\n| Qbert‑v5            | 0.84   | 1.01   | **1.06** | 0.91   | 1.09   |\n| Riverraid‑v5        | 0.42   | 0.44   | 0.65   | 0.69   | **0.99** |\n| **Aggregate (mean)**| 0.25   | 0.33   | 0.41   | 0.47   | **0.58** |\n\nPerformance steadily improves as more skip connections are added, peaking when all 5 blocks are connected. This supports our original design decision to broadcast features to all MultiSkip blocks.\n\nFurthermore, our main paper already demonstrates consistent gains when skip connections are present in the CNN network. In particular, Figure 9 reports large improvements when using the Impala CNN architecture (which includes residual connections) compared to a plain CNN. \n\n---\n*Thank you for the thoughtful review. We hope these additions address the main concerns regarding novelty and applicability. Please let us know if any points remain unclear.*\n\n# References\n\n[1]. Guozheng Ma et at, Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning, ICML’25\n\n[2]. JSO Ceron et al, In value-based deep reinforcement learning, a pruned network is a good network, ICML’24\n\n[3]. JSO Ceron et al, Mixtures of Experts Unlock Parameter Scaling for Deep RL, ICML’24\n\n[4]. Vineet Gupta et al, Shampoo: Preconditioned Stochastic Tensor Optimization, arXiv’18\n\n[5]. Xuezhe,  Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization, arXiv’21\n\n[6]. Aitchison, Matthew, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. ICML’23."}, {"review_id": "iQyGPtSIv9", "reviewer": "Reviewer_Uxog", "summary": "This paper investigates the scaling of network architectures in deep reinforcement learning, which often leads to degrading performance, contrary to trends observed in supervised learning. The authors provide empirical evidence on a continual learning toy task based on CIFAR10, as well as 2 Atari environments, that scaling networks sizes lead to decreasing performance because of gradient pathologies. They propose two interventions 1. multi-skip connections and 2. using the KFAC optimizer which is a second-order optimizer. Their interventions are evaluated across mutliple environments and algorithms successfully showing scalability.", "questions": "- Why did the authors specifically choose these two interventions? Did they also try other interventions to improve the gradient pathologies?\n- Could the authors clarify which confidence intervals were used for the plots in the main part of the paper?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper has a clear motivation and is well written and easy to follow\n- The authors provide extensive empirical evaluations both for the motivation as well as their interventions\n- The proposed interventions are simple and easy to implement, making it easy for others to adapt\n\n **Weaknesses**\n- The paper is purely empirical and does not provide a theoretical contribution\n- As a nitpick, while overall the plots are very nice, the readability should be improved by properly scaling the text to the template. I find the text to be relatively small in many of the plots.\n- I believe the authors are missing a few references in their related work:\n\nCrossQ [1] showed that scaling the critic to 2048 neurons, in combination with their batch normalization, results in significant sample efficiency improvements. As such it should be cited among the other approaches for scaling parameters in deep RL.\n\nAnd both [2] and [3] investigate scaling deep RL (mostly in terms of compute); however, similarly to Lyle et al. 2024, they identify growing parameter norms and gradient problems to be the bottleneck and introduce architectural interventions to stabilize the gradients. As such, these works should be discussed as well.\n\n[1] Bhatt et al. CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity. ICLR 2024\n\n[2] Palenicek et al. Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization. 2025\n\n[3] Lee et al. Hyperspherical Normalization for Scalable Deep Reinforcement Learning. ICML 2025", "was_revised": true, "final_justification": "The authors provided a good rebuttal answering my open questions in great detail. I increased my score to accept.", "rebuttal": "We thank the reviewer for their feedback! We are happy that the reviewer found that “the paper is well written and easy to follow”, and that it “provides extensive empirical evaluations”. We are  also glad the reviewer recognized that the \"interventions are simple and easy to implement, making it easy for others to adapt”. We respond to their main concerns below.\n\n# As a nitpick, while overall the plots are very nice, the readability should be improved by properly scaling the text to the template. I find the text to be relatively small in many of the plots.\n\nThank you for pointing this out. We agree that readability is important and will ensure that all figures are properly scaled to meet the template guidelines in the final version. \n\n# I believe the authors are missing a few references in their related work ..\n\nThank you for pointing us to these relevant works. We agree that they are highly relevant to the discussion on scaling in deep RL and will include them in the final version. While these works focus on batch/weight normalization and architectural constraints, our method offers a complementary and distinct approach by identifying multi-skip connectivity and Kronecker-factored optimization as synergistic mechanisms that improve both gradient propagation and parameter efficiency. In the revised manuscript, we will discuss these papers as approaches that analyze optimization pathologies in the context of scaling and propose architectural interventions to stabilize deep RL training.\n\n# Why did the authors specifically choose these two interventions? Did they also try other interventions to improve the gradient pathologies?\n\nThank you for raising this important point. We selected Kronecker-factored optimization (Kron) and multi-skip connectivity based on the outcome of a systematic exploration of several candidate interventions aimed at improving gradient flow, plasticity, and stability in deep RL.\n\nAs part of our investigation, we evaluated a wide range of alternative methods from prior work that aim to mitigate optimization pathologies in deep RL [2,10,11]. The methods were chosen because they have been used in other domains (e.g., supervised learning) and also deep RL to alleviate similar gradient-related issues. We excluded ReDo [1], Pruning [2], and Reset [3] as they require extensive hyperparameter tuning and, based on [2], their performance collapsed or stagnated when increasing network capacity. The evaluated methods include:\n\n- Second-order and adaptive optimizers: Apollo [4], Shampoo [5], AdaBelief [6]\n\n- Regularizers: L2 norm penalties [7], weight clipping [8], weight decay\n\n- Activation functions: GELU, CReLU [9]\n\n- Learning rate schedules: Cosine annealing and cyclic schedulers.\n\n- Learning rates at different scales: we try both multiplying and dividing the default learning rate (2.5e-4) by 12 to potentially compensate for the increased networks scale.\n\nAs summarized in the table below (3 seeds, mean performance), none of these interventions consistently showed improved performance compared to our proposed combination.\n\n| Environment         | Ours    | Cosine LR Scheduler | Cyclic LR Scheduler | GELU activation | CReLU activation | L2 Norm  | Weight Clip | Weight Decay | lr=0.003 | lr=2.083e-05 |\n|---------------------|---------|---------------------|---------------------|-----------------|------------------|----------|-------------|--------------|---------------------|-------------------------|\n| Amidar‑v5           | **0.355897** | -0.001663          | 0.002276            | 0.015143        | 0.016514         | 0.029293 | 0.012283    | 0.013013     | 0.018848            | 0.078660                |\n| BattleZone‑v5       | **0.694566** | 0.031297           | -0.000287           | 0.015505        | -0.004594        | -0.004594| 0.035604    | 0.061446     | 0.019812            | 0.575407                |\n| Bowling‑v5          | **0.231831** | 0.050145           | -0.008721           | 0.028706        | -0.009811        | -0.000727| 0.030523    | 0.051235     | -0.086483           | 0.067587                |\n| DoubleDunk‑v5       | **-1.318182**| -2.318182          | -2.318182           | -2.363636       | -2.318182        | -2.363636| -2.454545   | -2.454545    | -2.454545           | -2.090909               |\n| Frostbite‑v5        | **0.881087** | -0.001921          | 0.027708            | 0.000890        | -0.010938        | 0.004755 | 0.075372    | 0.021736     | 0.008034            | 0.110036                |\n| KungFuMaster‑v5     | 0.975251 | -0.011055          | 0.003181            | -0.000823       | -0.011500        | -0.011278| -0.011500   | -0.011055    | -0.011055           | **1.362077**             |\n| NameThisGame‑v5     | **1.242066** | -0.030191          | -0.132767           | -0.261400       | -0.198430        | -0.068147| -0.197995   | -0.148227    | -0.134417           | 0.747251                |\n| Phoenix‑v5          | **0.655141** | 0.072687           | -0.087237           | -0.090014       | -0.090168        | -0.068722| -0.090940   | -0.093949    | -0.102358           | 0.541968                |\n| Qbert‑v5            | **1.094894** | 0.013814           | 0.010052            | -0.002833       | 0.008359         | 0.006102 | 0.016259    | 0.000177     | 0.001305            | 0.112282                |\n| Riverraid‑v5        | **0.993568** | -0.000127          | 0.345100            | 0.304636        | -0.056846        | -0.058494| 0.293514    | -0.044456    | -0.057765           | 0.237365                |\n\nParticularly for second order optimizers, one of our main interventions, we explored Apollo, Shampoo, and AdaBelief, as well as different architectural variants like ResNets and DenseNets. These results are included in the appendix (see Figure 13 for optimizer comparisons and Section D.3). Despite extensive hyperparameter tuning across both architectures and optimizer, these alternatives did not consistently improve trainability in deep RL.\n\nIn contrast, our proposed combination of Kronecker-factored optimization (Kron) and multi-skip connections consistently outperformed these baselines. To strengthen this claim, we ran additional experiments for the rebuttal on the Atari-10 benchmark [12], comparing skip connectivity patterns when paired with Kron. Results show that richer skip connectivity is generally beneficial, with our multi-skip design performing best:\n\n| Method | ResNet + Kron | Multi-skip + Kron | DenseNet + Kron |\n|--------|---------------|--------------------|------------------|\n| **PPO** | 0.66          | **1.22**          | 0.69             |\n| **PQN** | 0.87          | **0.92**          | 0.90             |\n\nTable: Human-normalized scores on Atari-10 [12]. More skip connections generally improve performance, with multi-skip connections yielding the best results.\n\nWe will add a brief discussion in the paper clarifying that these interventions were selected based on systematic experimentation, as documented in the appendix.\n\n# Could the authors clarify which confidence intervals were used for the plots in the main part of the paper?\n\nThank you for the question. All plots in the main paper for deep RL experiments report 95% confidence intervals based on 3 independent seeds, following standard practice in recent deep RL literature [13]. We will clarify this in the figure captions and main text in the final version to avoid ambiguity. \n\nIn addition to per-task learning curves, we report aggregate performance metrics across tasks, including interquartile mean (IQM), median [13]. This aggregate result provides a more reliable estimate of overall performance and demonstrates that our improvements are statistically significant and consistent across environments.\n\n---\n*Thank you for the thorough review. Please let us know if any points remain unclear.*\n\n# References\n\n[1]. JSO Ceron et al, In value-based deep reinforcement learning, a pruned network is a good network , ICML’24\n\n[2]. G Sokar et al, The dormant neuron phenomenon in deep reinforcement learning, ICML’23\n\n[3]. E Nikishin et al, The primacy bias in deep reinforcement learning, ICML’22\n\n[4]. Xuezhe,  Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization, arXiv’21\n\n[5]. Vineet Gupta et al, Shampoo: Preconditioned Stochastic Tensor Optimization, arXiv’18\n\n[6]. Juntang Zhuang et al, Adabelief optimizer: Adapting stepsizes by the belief in observed gradients, NeurIPS’20\n\n[7]. A Kumar et al, Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes, ICLR’23\n\n[8]. Mohamed Elsaye et al, Weight clipping for deep continual and reinforcement learning, RLC’24\n\n[9]. Zaheer et al, Loss of Plasticity in Continual Deep Reinforcement Learning, CoLLAs’23\n\n[10]. Skander et al, No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO, NeurIPS’24\n\n[11]. Arthur et al, A Study of Plasticity Loss in On-Policy Deep Reinforcement Learnin, NeurIPS’24\n\n[12]. Aitchison, Matthew, Penny Sweetser, and Marcus Hutter.  Atari-5: Distilling the arcade learning environment down to five games. ICML’23.\n\n[13]. R Agarwal, M Schwarzer, PS Castro, A Courville, MG Bellemare, Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS’21"}]

Author rebuttal:

Meta-review: This paper aims to tackle the challenges involved with scaling RL algorithms. Reviewers uniformly praised the writing, presentation, and thoroughness of the paper, and were all enthusiastic about seeing it at the conference.
Decision: Accept (spotlight)