Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
Authors: Daniel Palenicek
Florian Vogt
Joe Watson
Jan Peters
Venue: NeurIPS 2025 poster ()
Keywords: deep reinforcement learning
batch normalization
weight normalization
update-to-data ratio
Primary area: reinforcement_learning

Abstract: Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications.
Recently, CrossQ has demonstrated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1.
In this work, we explore CrossQ's scaling behavior with higher UTD ratios.
We identify challenges in the training dynamics, which are emphasized by higher UTD ratios.
To address these, we integrate weight normalization into the CrossQ framework, a solution that stabilizes training, has been shown to prevent potential loss of plasticity, and keeps the effective learning rate constant.
Our proposed approach reliably scales with increasing UTD ratios, achieving competitive performance across 25 challenging continuous control tasks on the DeepMind Control Suite and Myosuite benchmarks, notably the complex dog and humanoid environments.
This work eliminates the need for drastic interventions, such as network resets, and offers a simple yet robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.

Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
Daniel Palenicek 1.2 Florian Vogt Joe Watson' Jan Peters 1.2.5,6
Technical University of Darmstadt Phessian.Al University of Freiburg 'University of Oxford German Research Center for Al (DFKI) Robotics Institute Germany (RIG)
daniel. pan.eel.p.lenicekkku-aaraaraa.
Abstract
Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications. Recently, CrossQ has demon- strated state-of-the-ard sample efficiency with low update-to-data (UTD) ratio of 1. this work, we explore CrossQ's scaling behavior with higher UTD ratios. We identify challenges in the training dynamics, which are emphasized by higher UTD ratios. To address these, we integrate weight normalization into the CrossQ frame- work, solution that stabilizes training, has been shown prevent potential loss of plasticity and keeps the effective learning rate constant. Our proposed approach reliably scales with increasing UTD ratios, achieving competitive performance across 25 challenging continuous control tasks on the DeepMind Control Suite and MyoSuite benchmarks, notably the complex dog and humanoid d environments. This work eliminates the need for drastic interventions, such as network resets, and offers simple yet robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.
1 Introduction
Reinforcement Learning (RL) has shown great successes in recent years, achieving breakthroughs in diverse areas. Despite these advancements, a fundamental challenge that remains in RL is enhancing the sample efficiency of algorithms. Indeed, real-world applications, such as robotics, collecting large amounts of data be time-consuming, costly, and sometimes impractical due to physical constraints or safety concerns. Thus, addressing this is crucial to make RI. methods more accessible and scalable.
Different approaches have been explored to address the problem of low sample efficiency in RL. Model-based RL, the one hand, attempts to increase sample efficiency by learning dynamic models that reduce the need for collecting real data, a process often expensive and time-consuming [39, the other hand, have explored increasing the number
22, 12. 17]. Model-free RL approaches, of gradient updates on the available data, referred to as the update-to- to---ta (UTD) ratio [33, 10], modifying network architectures [4], or both [9, 18, 20, 32]. A central tension in these research directions is balancing the sample efficiency of the agent against the computational complexity, i.e,, wall-clock time, of the underlying algorithm. Algorithmic adjustments such model-based rollouts, computing auxiliary exploration rewards and higher UTDS can all significantly increase the wall-clock time of the algorithm. Likewise, architectural changes such as larger models [32] and more ensemble members [9] also increase the wallclock time of the method. Ideally, we desire architectural and algorithmic changes that balance sample efficiency and simplicity, such as CrossQ [4], which showed that careful use of batch normalization unlocks significantly greater sampler efficiency of deep actor-critic methods without significantly impacting the wallclock time.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Dog & Humanoid
DMC Med+Easy (8 tasks at 200k)
Myo Hard (5 tasks at IM)
Myo Easy (5 tasks at 200k)
(7 tasks at IM)
0.8
0.6
0.2
0.0
CrossQ WN UTD=2
Simba UTD-2
BRO UTD-2
TD-MPC2
CrossQ
Figure 1: CrossQ + WN UTD=2 outperforms UTD=2oupprforms SIMBA UTD=2 and BRO 2. In comparison, our proposed CrossQ + WN a simple algorithm that, unlike BRO, does not require extra exploration policies or full parameter resets. We present results for 25 complex continuous control tasks from the DMC and MyoSuite benchmarking suites. 1.0 marks the maximum score achievable on the respective benchmarks (DMC return up 1000 / MyoSuite up to 100% success rate). We present IQM and 90% stratified bootstrap confidence intervals aggregated over multiple environments and 10 seeds each.
In this work, we build upon CrossQ [4], the model-free RI. algorithm that showed state-of-the-art sample efficiency on the MuJoCo [42] continuous control benchmarking tasks, and also enabled learning onni-directional locomotion policies
minutes of real-world experience [6]. Notably, the authors achieved this by carefully utilizing batch normalization (BN, loffe [21]) within the actor-critic architecture. A technique previously thought not to work in RL, as reported by Hiraoka et al. [18] and others. The insight that Bhatt et al. [4] offered is that one needs to carefully consider the different state-action distributions within the Bellman equation and handle them correctly to succeed. This novelty allowed CrossQ at a low UTD of 1 to outperform the then state-of the-art algorithms that scaled their UTD ratios up to 20. Even though higher UTD ratios are more computationally expensive, they allow for larger policy improvements using the same amount of data. This naturally raises the question: How can we extend the sample efficiency benefits of CrossQ and BN to the high UTD training regime? Which we address in this manuscript.
Contributions. this work, we show that the vanilla CrossQ algorithm is brittle to tune on DeepMind Control (DMC) and MyoSuite environments and can fail to scale reliably with increased compute. To address these limitations, we propose the addition of weight normalization (WN),
Dog Humanoid (7 tasks at IM)
DMC Med+Easy tasks 200k)
Myo Hard
Myo Easy (5 tasks at 200k)
(5 tasks 1M)
Return/ Success
0.5
0.0
150
150
200
150
200
150
200
Steps per second
Steps second
Steps per second
Steps per second
Cross() WN UTD=2
Simha UTT=>
BRO tTTD=2
Figure 2: Comparing performance against wall clock time, measured in environment steps per second (so larger is better) on a single RTX 4090 workstation, we observe the CrossQ WN outperforms SIMBA and BRO across all environments. We present results for 25 complex continuous control tasks from the DMC and MyoSuite benchmarking suites. 1.0 marks the maximum score achievable on the respective benchmarks (DMC return up to 1000 MyoSuite up 100% success rate).

---

which we show to be a simple yet effective enhancement that stabilizes CrossQ. We motivate the combined use of WN and BN on insights from the continual learning and loss of plasticity literature and connections to the effective learning rate. Our experiments show that incorporating WN not only improves the stability of CrossQ but also allows us to scale its UTD, thereby significantly enhancing sample efficiency.
2 Preliminaries
This section briefly outlines the required background knowledge for this paper.
Reinforcement learning. A Markov decision process (MDP) [35] is a tuple M (S.A, P, r, Ho. 7). with state space S S R", action space A S R". transition probability P: x A A- A(S). the reward function : S x A - R. initial state distribution μ4 and discount factor y We define the RL problem according to Sutton and Barto [40]. A policy π : S + ((A) is behavior plan, which maps a state s to a distribution over actions a. The discounted cumulative 8, ag and $t+1 ~ Pl |st, az) and a T( (sx) otherwise. The Q-function of policy π is the expected discounted return ("(ss, a)= E[R (s, a)]. The goal of an RL agent is to find an optimal policy x²" that maximizes the expected return from the initial state distribution x = arg max, Es-no (s"(
return is defined as RD (s, a) Ei= t= 7'r(s,, at), where
Actor-critic methods address this optimization problem by jointly learning neural network representa- tions for the Q-function and the policy. The policy network is optimized to maximize the Q-values, while the Q-function is optimized to minimize the squared Bellman error, where the value target is computed by taking an expectation over the learned Q function
V(s++1) Ea, (1)1)) (Q((L-1,
(1)
To stabilize the Q-function learning. Haarnoja et al. [14] found it necessary to use target Q-network in the computation of the value function instead of the regular Q-network. The target Q-network is structurally equal to the regular Q-network, and its parameters 0 are obtained via Polyak Averaging over the learned parameters 0. While this scheme ensures stability during training by explicitly delaying value function updates, it also arguably slows down online learning [34, 23, 31].
Instead of relying on target networks, CrossQ [4] addresses training stability issues by introducing batch normalization (BN, Ioffe [21]) in its Q-function and achieves substantial improvements in sample and computational efficiency over Soft actor-critic (SAC). A central challenge when using BN in Q networks is distribution mismatch: during training, the Q-function is optimized with samples 82, ag from the replay buffer. However, when the Q-function evaluated to compute the target values (Equation (I)), it receives actions sampled from the current policy ax+1 TT( |s(+1). Those samples have no guarantee of lying within the training distribution of the Q-function. BN is known to struggle with out-of-distributi
samples, as such, training can become unstable if the distribution mismatch is not correctly accounted for [4]. To deal with this issue, CrossQ removes the separate target Q- funcciinn and evaluates both Q values during the critic update in single forward pass, which causes the BN layers to compute shared statistics over the samples from the replay buffer and the current policy. This scheme effectively tackles distribution mismatch problems, ensuring that all inputs and intermediate activations are effectively forced to lie within the training distribution.
Normalization techniques in RL. Normalization techniques are widely recognized for improving the training of neural networks, as they generally accelerate training and improve generalization [19]. There are many ways of introducing different types of normalizations into the RL framework. Most commonly, authors have used layer normalization (LN) within the network architectures to stabilize training [18, 30. 27J. Recently, CrossQ has been the first algorithm to successfully use BN layers in RL [4]. The addition of BN leads to substantial gains in sample efficiency. In contrast to LN, however, one needs to carefully consider the different ftate-action distributions within the critic loss when integrating BN. In different line of work, Hussing et al. [20) proposed the integration of unit ball normalization and projected the output features of the penultimate layer onto the unit ball in order to reduce Q- Q-function overestimation.
Increasing update-to-data ratios. Although scaling up the UTD ratio is an intuitive approach to increase the sample efficiency, in practice, it comes with several challenges. Nikishin et al. [33]
3

---

Return/ Success
Parameter Norm
Gradient Norm
ELR
Dead Neurons [%]
Dog & &omonneld
10a
0.4
10
0.5
0.002
10-3
0.0
0.000
1.0
104
Myo Hard
104
0.5
0.5
10a
200
10a
10a
0.0
0.0
x10*
x10
X10
x10*
x10*
Env Steps
Env Steps
Env Steps
Env Steps
Env Steps
Cross WN UTD=5
CrossQ+ WN UTD=1
CrossQ UTD=5
Cross UTD=1
Figure 3: Growing parameter norms hinder learning. The performance benefits of CrossQ fail to scale to more complex, higher dimensional tasks such as humanoid locomotion and muscular manipulation. Investigating this, we find that the critic parameter norms increase significantly with increasing UTD ratios. As a result, the effective learning rate (ELR) drops and the number of dead neurons increases. Regularizing the critic parameters with weight norm (WN), successfully mitigate the parameter norm growth and therefore maintain a more consistent ELR, leading to better performance these complex tasks. Uncertainty quantification depicts the 90% stratified bootstrap confidence intervals.
demonstrated that overfitting on early training data can inhibit the agent from learning anything later in the training. The authors dub this phenomenon the primacy bias. To address the primacy bias, they suggest to periodically reset the network parameters while retraining the replay buffer. Many works that followed have adapted this intervention [10, 32j. While often effective, regularly resetting is very drastic intervention and by design induces regular drops performance. Since the agent has to start learning from scratch repeatedly, it is also not very computing efficient. Finally, the exact reasons why parameter resets work well practice are not yet well understood [28]. Instead of resetting there have also been other types of regularization that allowed practitioners to train stably with high UTD ratios. Janner et al. [22] learn a world model and use it to generate synthetic data which is used in high UTD setting to stabilize training. REDQ [9] leverages sampling from ensemble of Q-functions, while Hiraoka et al. [18] use dropout and LN to effectively scale to higher UTD ratios.
3 Batch normalization alone fails to scale with respect to task complexity
Bhatt et al. [4] demonstrated CrossQ's s state-of of-the-art sample efficiency on the MuJoCo task suite [42], while at the same time also being very computationally efficient. However, on the more extensive DMC and MyoSuite task suites, we find that CrossQ requires tuning. We further find that it works on some, but not all, environments stably and reliably.
Figure shows CrossQ training performance on the DMC dog and humanoid and the Myo Hard tasks aggregated by environment suite. We plot the IQM and 90% confidence intervals for each metric across 10 seeds. The figure compares a CrossQ with UTDS and 5, where the hyperparameters were identified through grid search over learning rates and network sizes, detailed in Table 1. The first shows the agent's 's training performance, while the other columns show network parameter norms, gradient norms, the effective learning rate [44], and the fraction of dead neurons. Here, identify three different training behaviors. We notice that CrossQ does not benefit from higher UTD ratios, but performance remains similar the provided tasks. Overall, for all CrossQ runs notice large confidence intervals.
Growing network parameter norms. The second column of Figure 3 displays the sum over the 1.2 norms of the dense layers in the critic network. This includes three dense layers, each hidden layer with a dimension of 512. On both task suites, CrossQ exhibits growing network weights over the course of training. We find that the effect particularly pronounced for CrossQ

---

with increasing UTD ratios. Growing network weights have been linked to loss of plasticity, phenomenon where networks become increasingly resistant to parameter update, which can lead to premature convergence [1 I. Additionally, the growing magnitudes pose challenge for optimization, connected to the issue of growing activations, which has recently been analyzed by Hussing et [20]. Further, growing network weights decrease the effective learning rate when the networks contain normalization layers [43, 30].
In summary, the scaling results for vanilla CrossQ are mixed. While increasing UTD ratios is known to yield increased sample efficiency, if careful regularization used [22, 9, 33], CrossQ alone with BN cannot benefit from it. We notice that with increasing UTD ratios, CrossQ's 's weight layer norms grow significantly faster and overall larger. This observation motivates us to further study the weight norms in CrossQ to increase UTD ratios.
4 Combining batch normalization and weight normalization enables scaling
Inspired by the combined insights of Van Hasselt et al. [43] and Lyle et al. [30], we propose to integrate CrossQ with weight normalization (Salimans and Kingma [36], WN) as a means of counteracting the rapid growth of weight norms we observe with increasing update-to-data (UTD) ratios. weight normalized parameter vö is constrained to have an L2 norm of c, additional hyperparameter,
i = c w ||
(2)
Our approach is based on the following reasoning: Due to the use of BN in CrossQ, the critic network exhibits scale invariance, as previously noted by Van Laarhoven [44].
Theorem (Van Laarhoven [44]) Let s(X; u. b, 7, 3) be a function, with inputs X and param- eters ur and b and and 3 butch normalization parameters. When f is normalized with batch with respect to its parameters, i.e.,
normalization, f becomes f(X;cw, cb, 7, B) f(X; b,,
with scaling factor e > O.
The proof is provided in Appendix
This property allows us to introduce WN as a mechanism to regulate the growth of weight norms in CrossQ without affecting the critics outputs. Further, it can be shown, that for such scale invariant function, the gradient scales inversely proportionally to the scaling factor e >> 0.
Theorem (Van Laarhoven [44]) Let f(X; w. 6, 7, be a scale-invariant function. Its gradient
ff cb, y, 3) Vf(X: w, b,/.
scales inversely proportional to the scaling factor e. E IR of its parameters w.
The proof is provided in Appendix B.
Recently, Lyle et al. [30] demonstrated that the combination of LN and WN can help mitigate loss of plasticity. Since the gradient scale is inversely proportional to c, keeping norms constant helps to maintain a stable effective learning rate (ELR, Van Hasselt et al. [43]), further enhancing training stability. We conjecture that maintaining a stable ELR could also be beneficial when increasing the UTD ratios in continuous control RL. As the UTD ratio increases, the networks are updated more frequently with each environment interaction. Empirically, we find that the network norms tend to grow quicker with increased UTD ratios (Figure 3), which in turn decreases the ELR even quicker and could be the reason for lower training performance. From this observation, we hypothesize that the training phenomena that affect plasticity also appear when attempting sample-efficient learning with higher UTDS. This hypothesis suggests that regularization techniques for plasticity could also be used to achieve more sample-efficient RL. As result, we empirically investigate the effectiveness of combining CrossQ with WN with increasing UTD ratios.
Implementation details. We apply WN to all hidden linear layers of the network, ensuring that their weights remain unit norm after each gradient step by projecting them onto the unit ball, similar to Lyle et al. [30]. The reason for not placing WN on the output layer is allow scaling of predicted
5

---

Q values to arbitrary magnitudes and crucially avoid the risk of underfitting. Additionally, layers that are not followed by BN are not scale invariant w.r.t. re-scaling their weights. While we could employ a learning rate schedule [30] we did not investigate this here as this would add additional complexity. Additionally, we impose weight decay on all parameters that remain final dense output layer. In practice, we use AdamW [29] with decay of 0 (which falls back to vanilla Adam [24)) for the normalized intermediate dense layers and 1e 1 otherwise.
speciffilllllllllllll. the
Target networks. CrossQ removes the target networks from the actor-critic framework and showed that using BN training remains stable even without them [4]. While we find this to be true in many cases, we find that especially in DMC, the re-integration of target networks can help stabilize training overall (see Section 5.4). However, not surprisingly, we find that the integration of target networks with BN requires careful consideration of the different state-action distributions between the s, a and s', a' ~ π (s') exactly as proposed by Bhatt et al. [4]. To satisfy this, we keep the joined forward pass through both the critic network as well as the target critic network. We evaluate both networks in training mode, i.e., they calculate the joined state-action batch statistics on the current batches. As is common, we use Polyak-averaging with a r = 0.005 from the critic network to the target network.
5 Experimental results
To evaluate the effectiveness of our proposed CrossQ + WN method, we conduct a comprehensive set of experiments the DeepMind Control Suite [41] and MyoSuite [7] benchmarks. For DMC report individual results for the hard (dog and humanoid), d), as well as Medium+Easy (cheetah-run, walker-run, hopper-stande finger-turn hard quadruped-run, fish- hopper-hop. gandulum-swinguw due to their varying difficulties. Equally, we split MyoSuite hard and easy environments. Our primary goal is to investigate the scalability of CrossQ WN with increasing UTD ratios and to assess the stabilizing effects of combining CrossQ with wN. We compare our approach to several baselines, including the recent BRO [32], CrossQ [4]. TD-MPC2 [16]. and SIMBA [27], a concurrent approach utilizing layer norm. Figure 6 in the appendix further provides SR-SAC [10] baseline, version of SAC [14] with high UTD ratios and network resets.
5.1 Experimental setup
Our implementation is based on the SAC implementation of jaxrl codebase [25]. We implement CrossQ following the author's s original codebase and add the architectural modifications introduced by [4], incorporating batch normalization in the actor and critic networks. We extend this approach by introducing WN to regulate the growth of weight norms and prevent loss of plasticity and add target networks. We perform a grid search to focus on learning rate selection and layer width.
We evaluate 25 diverse continuous control tasks, 15 from DMC and 10 from MyoSuite. These tasks vary significantly in complexity, requiring different levels of fine motor control and policy adaptation with nigh-dimensional state spaces up to 223 Each experiment is run for I million environment steps and across 10 random seeds to ensure statistical robustness. We evaluate agents every 25, 000 environment steps for 5 trajectories. For the DMC Medium&Far Easy and Myo Easy we plot the first 200k steps, as all methods learn much faster than the 1 million steps. As proposed by Agarwal et al. [1], we report the interquartile mean (IQM) and 90% stratified bootstrap confidence intervals (CIs) of the return (or success rate, respectively), if not otherwise stated. For the BRO and SIMBA baseline results, for computational reasons, we take the official evaluation data that the authors provide. The official BRO codebase is also based on jaxil, and the authors followed the same evaluation protocol. All experiments were run on a compute cluster with RTX 3090 and A5000 GPUs, where all 10 seeds run in parallel on a single GPU via jax. vmap.
5.2 Weight normalization allows CrossQ to scale to harder tasks effectively
We provide empirical evidence for our hypothesis that controlling the weight norm and, thereby, the ELR can stabilize training (Figure 3). We show that through the addition of wN, CrossQ + WN shows stable training and can stably scale with increasing UTD ratios.
Figure 7 shows per environment results of our experiments encompassing all 25 tasks evaluated across 10 seeds each. Based on that, Figure 1 shows aggregated performance over all environments

---

Dog & Humanoid (7 Lasks at 1M)
DMC Med+Easy (8 tasks at 200k)
Myo Hard
Myo Easy (5 tasks 200k)
(5 tasks at IM)
Return / Success
0.8
1.0
0.8
0.75
0.6
0.50
0.5
0.4
0.25
2
5
10
20
2
5
10
20
5
10
20
5
10
20
UTD
UTD
UTD
UTD
CrossQ
TD3 BN
BRO
Cross()
ID-MPC2
TD3 BN WN
Simba
Figure 4: Cross WN UTD scaling behavior We plot the IQM return and 90% confidence intervals for different UTD ratios. Results are aggregated over 15 DMC environments and 10 random seeds each according to Agarwal et al. [1]. The sample efficiency scales reliably with increasing UTD ratios and remains stable even when there are no more performance gains, which is a crucial property.
from Figure per task suite, with separate aggregation for the most complex dog and humanoid environments.
Scaling results in Figure 4 show that CrossQ WN is competitive to the BRO and SIMBA baselines on both DMC and MyoSuite, especially on the more complex dog and humanoid tasks already on lower UTD ratios. Notably, CrossQ+ + UTD=5 uses only half the any parameter resets and no additional exploration policy, Further, it uses ~ 90% fewer network parameters-BRO reports 5M, while our proposed CrossQ WN uses only 600k (these numbers vary slightly per environment, depending on the state and action dimensionalities).
of BRO and does require
In contrast, vanilla CrossQ UTD=1 exhibits much slower learning most tasks and, in some environments, fails to learn performant policies. Moreover, the instability of vanilla CrossQ at UTD=5 is particularly notable, as it does not reliably converge across environments (Figure 7).
These findings highlight the necessity of incorporating additional normalization techniques sustain effective training at higher UTD ratios. This leads us conclude that CrossQ benefits from the addition of wN, which results in stable training and scales well with higher UTD ratios. The resulting algorithm can match or outperform state-of-the-at baselines on the continuous control DMC and MyoSuite benchmarks while being much simpler algorithmically.
5.3 Stable scaling of CrossQ + WN with UTD ratios
To visualize the stable scaling behavior of CrossQ + WN we ablate across UTD ratios. Figure shows training performances aggregated over multiple environments for 10 seeds each 1M steps and 200k steps respectively. We confirm that CrossQ + WN shows reliable scaling behavior. With increasing compute, the performance increases or stays constant which is desirable. We see, that for the same UTD ratio, CrossQ WN nearly always beats both the BRO and SIMBA baselines. Further, we demonstrate, that the sample-efficiency and stable UTD scaling benefits of BN+WN extend beyond CrossQ and SAC to other base algorithms, such as TD3.
Hyperparameter ablation studies
We also ablate the different hyperparameters of CrossQ + WN, by changing each one time. Figure 5 shows aggregated results of the final performances of each ablation. We will briefly discuss each ablation individually.
Removing weight normalization. Not performing weight normalization results the biggest drop in performance across all our ablations. This loss is most drastic on the MyoSuite tasks and often results in no meaningful learning. Showing that, as hypothesized, the inclusion of into the CrossQ framework yields great improvements in terms of sample efficiency and training stability,

---

Dog & Humanoid
Dog & Humanoid
Myo Hard UTD=1 (5 tasks)
Myo Hard
UTD=1 (7 tasks)
UTD=5 (7 tasks)
UTD=5 (5 tasks)
0.8
0.6
0.4
0.2
0.0
CrossQ WN
Cross L2(0.))
CrossQ L2(0.001)
CrossQ
WN wio target network CrosQQ
Cross++ WX Reset
CrossQ L2(0.01)
SAC LN
Figure 5: An ablation study comparing CrossQ WN against soft L2 penalty on the weights, as well as other design decisions such as target networks. All ablations use critic network with two hidden layers of dimension 512. The results show that the hard constraint outperforms the soft approach across range of regularization scales and tasks. Uncertainty quantification depicts the 90% stratified bootstrap confidence intervals.
especially for larger UTD ratios. In general, lower UTD ratios are already reasonably competitive in overall performance.
Target networks. Ablating the target networks shows that on MyoSuite, there is a significant difference between using a target network and not using a target network, Results on DMC show large drop in performance. There, removing target networks leads to significant drop in performance, nearly as large as removing weight normalization. This finding is interesting, as suggests that CrossQ WN without target networks is not inherently unstable. But there are situations where the inclusion of target networks is required. Further investigating the role and necessity of target networks in RL is an interesting direction for future research.
L2 regularization. Figure 5 investigates the performance of soft 1.2 penalty on the weights compared to the weight normalization proposed in this paper. Across all tasks, and sweeping across soft regularization scalings, weight normalization outperforms the soft L2 penalty. Our hypothesis is that, in principle, soft L2 regularization could work in similar way to the proposed WN; however, would require per-task tuning and potentially even scheduling to result in stable ELR. In comparison, the hard constraint via WN guarantees a stable weight norm by design and as such is easier to employ.
Parameter resets. Our experiments show that CrossQ + WN is able to scale without requiring drastic interventions such as parameter resets. However, it is still interesting to investigate whether CrossQ + WN's performance could benefit from parameter resets. CrossQ WN Reset in Figure investigates this question. We see, that there is slight improvement on the dog and humanoid tasks, on all other 18 tasks, performance remains the same. The main takeaway is, that CrossQ + wN scales stably without requiring parameter resets.
Related work
RL has demonstrated remarkable success across various domains, yet sample efficiency remains a significant challenge, especially in real-world applications where data collection is expensive or impractical. Various approaches have been explored address this issue, including model-based RL, UTD ratio scaling, and architectural modifications.
Update-to-data ratio scaling. Model-free RL methods, including those utilizing higher UTD ratios, focus on increasing the number of gradient updates per collected sample to maximize learning from available data. High UTD training introduces several challenges, such as overfitting to early training data, a phenomenon known primacy bias [33]. This can be counteracted by periodically resetting

---

the network parameters [33, 10]. However, network resets introduce abrupt performance drops. Nauman et al. [32] demonstrated that full parameter resets of the critic can effectively preserve learning capacity using a UTD ratio up to 10. However, such resets inevitably impact the wall-clock time due to relearning function approximation several times during learning. Alternative approaches use an ensemble of Q-functions to reduce overestimation bias that occurs under high UTD ratio training regimes [9]. Due to the decreased computational efficiency of using a large number of Q-functions, Hiraoka et al. [18] propose to replace the ensemble of critics with dropout and layer normalization. Both methods utilize a UTD ratio of 20, which is highly inefficient.
Normalization techniques in RL. Normalization techniques have long been recognized for their impact on neural network training. LN [3] and other architectural modifications have been used to stabilize learning in RL [18, 32]. Yet BN has only recently been successfully applied in this context [4], challenging previous findings, where BN in critics caused training to fail [18]. WN has been shown to keep ELRS stable and prevent loss of plasticity [30], when combined with LNN, making it a promising candidate for integration into existing RL frameworks. Bjorck et al. [5] show that using spectral normalization (SN) enables training with large scale neural networks in deep RL. SN divides layers weight matrix by its largest singular value, regularizing the Lipschitz continuity of the function approximation and therefore stabilizing the gradients. BN has been used by Bhatt et al. [4], where it achieves impressive results, but requires slight adjustments when the UTD ratio is scaled. Concurrent work [27] injected 'simplicity bias' into their actor and critic architecture, encouraging the model to favor 'simpler' features for its predictions. Simba incorporates a residual feedforward block and layer normalization in both the actor and critic networks and achieved state of the art results. WN has been shown to keep ELRS stable and prevent loss of plasticity [30], when combined with I.N, making it a promising candidate for integration into existing RT. frameworks. Ceron et al. [8] recently analyzed the impact of different normalization on the performance of RL algorithms. Hafner et al. [15] designed a vision-based MBRL algorithm Dreamer that leverages world models master wide range of diverse tasks, and also relies on several normalization techniques. They utilize root mean square layer normalization (RMSNORM) [46] before the activation function and normalize the returns. To successfully scale deep RL on the Atari 100k benchmark and achieve human-level sample efficiency. Schwarzer et al. [37] rely on regularization techniques. Rather than use normalization methods, they use the shrink-and aan--nnddrrrbrre method [2] in shallow layers and full parameter resets in deeper layers to preserve network plasticity. To scale to higher UTDs, they introduce AdamW [29] and gradually decrease the number of steps for the computation of the TD error for faster convergence. Lee et al. [26] argue that the loss of plasticity observed in deep RI. when the UTD ratio is increased can be mitigated by using LN sharpness- aware minimization (SAM, Foret et al. [13]), incorporating parameter resets in the last layers and replacing the ReLU activation function with a 'concetenated ReLU function [38]. Voelcker et al. [45] demonstrated that parameter resets are not strictly necessary when the UTD ratio is increased to improve sample efficiency. They identify the generalization ability of the critic as the main source of training instabilities under high UTD regimes. They demonstrate empirically that architectural regularization can mitigate overestimation and divergence, but it does not guarantee proper generalization. On the other hand, leveraging synthetic data generated by learned world model can help mitigate the effects of distribution shift, thereby enhancing generalization.
7 Conclusion, limitations & future work
In this work, we have addressed the instability and scalability limitations of CrossQ in RL by integrat- ing weight normalization. Our empirical results demonstrate that WN effectively stabilizes training and allows CrossQ to scale reliably with higher UTD ratios. The proposed CrossQ + WN approach achieves competitive or superior performance compared to state-of-the-art baselines across a diverse set of 25 complex continuous control tasks from the DMC and MyoSuite benchmarks. These tasks include complex and tigh--iimesiioall humanoid and dog environments. This extension preserves simplicity while enhancing robustness and scalability by eliminating the need for drastic interventions such as network resets. In this work, we only consider continuous state-action benchmarking tasks. While our proposed CrossQ WN performs competitively on these tasks, its performance on discrete state-action spaces or vision-based tasks remains unexplored. We plan to investigate this future work. Moreover, the theoretical basis of our work does not directly connect to the convergence rates or sample efficiency of the underlying RI. algorithm, but rather to mitigate observed empirical phenomena regarding the function approximation alone.

---

Acknowledgments and Disclosure of Funding
We would also like to thank Tim Schneider, Cristiana de Farias, João Carvalho and Theo Gruner for proofreading and constructive criticism on the manuscript. This research was funded by the research cluster Third Wave of AI, funded by the excellence program of the Hessian Ministry of Higher Education, Science, Research and the Arts, Aessian.Al and has benefited from the early stages of the funding by the Deutsche Forschungsgemeinccaatt (DFG, German Research Foundation) under Germanys Excellence Strategy EXC-3057; funding will begin in 2026. This work was also supported by a UKRI/EPSCC Programme Grant [EP/V000748/1).
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 2021.
[2] Jordan T. Ash and Ryan Adams. On warm-starting neural network training, 2020. URL https: / /ttp/vaxxvvv.//////////////aaav/ org/ 7 08475.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607: 06450, 2016.
[4] Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In International conference on learning representations, 2024.
[5] Johan Bjorck, Carla P. Gomes, and Kilian Q. Weinberger. Towards deeper deep reinforcement learning with spectral normalization, 2022. URL https::://///ttee orgooogg/aa....oooo..o /tbb///6.... 01151.
[6] Nico Bohlinger, Jonathan Kinzel, Daniel Palenicek, Lukasz Antezak, and Jan Peters. Gait in eight: Efficient on-robot learning for omnidirectional quadruped locomotion. arXiv preprint arXiv:2503.08::75 2025.
[7] Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite-a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600 2022.
[8] Johan Samir Obando Ceron, João Guilherme Madeira Araújo, Aaron Courville, and Pablo Samuel Castro. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. In Reinforcement Learning Conference, 2024.
[9] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double Q- learning: Learning fast without a model. In International conference on learning representations, 2021.
[10] Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International conference on learning representations, 2022.
[11] Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and Rupam Mahmood. Weight clipping for deep continual and reinforeement learning. In Reinforcement Learning Conference, 2024.
[12] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. In International Conference on Machine Learning, 2018.
[13) Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021.
[14] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900 2018.
10

---

[15] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- lions; 2020.
[16] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024. URL http:: http://wwwwwv..r.aaa.///oo att.aa/....//a.eaeee/.aaaeeeeeet 16828.
[17] Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap. Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in neural information processing systems, 2015.
[18) Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout q-functions for doubly efficient reinforcement learning. In International conference on learning representations, 2021.
[19] Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology. analysis and application. IEEE transactions on pattern analysis and machine intelligence, 2023.
[20] Marcel Hussing, Claas Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, and Eric Eaton. Dissecting deep rl with high update ratios: Combatting value overestimation and divergence. arXiv preprint arXiv:2403.055: 05599, 2024.
[21] Sergey Ioffe. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
[22] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in neural information processing systems, 2019.
[23] Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: Remov- ing the need for a target network in deep q-learning. In Proceedings of the Twenty-Eighth Inter- national Joint Conference on Artificial Intelligence, IJCAI-19, pages 2733-2739, International Joint Conferences on Artificial Intelligence Organization, 2019. doi: 10. 10.24963/iccii22119/777
URL https://doi. org/ 10. 24963/ijcai 2019/379.
[24] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 2014.
[25] Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 2021. https: thub. com/ http//koosrtkoo/j/e/oe
[26] Hojoon Lee, Hanseul Cho, Hyunseung Kim, Dachoon Gwak, Joonkee Kim, Jaegul Choo, Se-Yone Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning, 2023. URL bttps://arzir org/ abs 2206/2366222 10711.
[27] Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subrama- nian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. International Conference on Learning Representations, 2025.
[28] Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. In International conference on learning representations, 2023.
[29] I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 1. 2017.
[30] Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and s Will Dabney. Normalization and effective learning rates in reinforcement learning. In Neural information processing systems, 2024.
[31] Miguel Morales. Grokking deep reinforcement learning. 2020.
[32] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mio, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in neural information processing systems, 2024.
11

---

[33] Evgenii Nikishin, Max Schwarzer, Pierluca DOro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning. 2022.
[34] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-gos goal reinforcement learning: Challenging robotics environments and request 09464.
for research, 2018. URL http:////axxxi.../aabbbab/e
[35] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[36] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[37] Max Schwarzer, Johan Obando-Ceron Aaron Courville, Mare Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency, 2023. URL https: //arxiv..iiia/a/aa orggoor/ 19452.
[38] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units, 2016. URL https:
/ //arxiv.org/ aass///00 05201.
[39] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Machine learning, 1990.
[40] Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018
[41] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint
2018.
[42] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International conference on intelligent robots and systems, 2012.
[43] Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? In Advances Neural Information Processing Systems, 2019.
[44] Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv: 1706.05366000 2017.
[45] Claas A Voelcker, Marcel Hussing, Eric Eaton, Amir massoud Farahmand, and Igor Gilitschen- ski. Mad-td: augmened data stabilizes high update ratio rl, 2025. URL https: //arxiv. org/ awwpgaab//411.... 08896.
[46] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems (NeurIPS), 2019.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have made sure that our claims reflect our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have dedicated conclusion and limitations discussion at the end of the
paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Our theory is adopted from prior work and the proofs are provided in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have included information about implementation details and hyperparam- eters. To aid reproducibility, we plan to release the code together with the camera-ready version the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: At the current time we do not provide code, however, we already provide all implementation details in the paper. We plan to release the code together with the publication the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
ctaiss.ccpubblie gww.//guggs...//ee CooeeSummiss wwwiiissssnnniiioiooiooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have parts explaining implementation details and choice of hyperparameters in detail.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes:
Justification: We provide IQM and 90$ stratified bootstrap confidence intervals all results (if not stated otherwise). Results are aggregated over multiple environments and 10 seeds each.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reference the type of GPUs used for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //euripsu...e// wwweurps..ccpubllllee wttpcccccpcccceeeeccceccccccccet ctt.iiiiiiiiiieiinneeeee
Answer: [Yes]
Justification: As this paper did not involve human participants or real-world datasets or experiments, there were no ethical concerns with the research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper examines performance improvements of existing actor-critic meth- ods. As actor-critic methods already enjoy a long history, there is no additional societal impact with this research contribution.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA],
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite and link to the codebase which our code based upon.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswith thode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not provide any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines: [NA]
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //terr/ees.//////e cc/Conferrecces/2255/ for what should or should be described.
19

---

A Proof Scale Invariance
Proof of Theorem
f(x:ww cw, cb,,, g(cXw + cb) + cb)) 7
(5)
o(g(cXw + cb))
cg( X w + - cu(g( X w + bb)
+
(6)
Ic u + b))
g Xw + b) 1gg( + b))
7 + B = ((;;,,,,73))
(gg( X w + b))
B Proof Inverse Proportional Gradients
To show that the gradients scale inversely proportional to the parameter norm, we can first write
g(eX + p(g() +c)) 7
(8)
dg(cCW +cb) cb))
g(cXw+c)) M + cb)) ogg(eX
+ ( cb)) B.
(9)
o(g(
(10)
As the gradient of the weights backpropagated through the mean and standard deviation, we have
(w, 7,8) g'ce + ob) [clo(g( Xw +
(11)
(12)
The gradient of the bias can be computed analogously
g'(cXw + cb) Vo VbJ(X;ew, 0,) lcl(g((Xu Xx b))
(13)
(14)
20

---

C Hyperparameters
Table gives an overview of the hyperparameters that were used for each algorithm that was considered in this work.
Table 1: Hyperparameters
Hyperparameter
CrossQ
CrossQ WN
Simba
SR-SAC
BRO
Criiic learning
0.0003
0,0003
0.0001
0.0003
0.0003
Critic hidden dim
512
512
512
256
512
Actor learning rate
0.0003
0,0003
0.0001
0.0003
0.0003
Actor hidden dim
256
256
128
256
256
Initial temperature
1.0
1.0
0.01
1.0
1.0
Temperature learning rate
0.0001
0.0001
0.0001
0.0003
0.0003
Target entropy
/2|
/
|A]/2
|A|
|
Target network momentum
0.005
0.005
0.005
0.005
0.005
UTD
1,2,5,10,20
1,5
2,8
32
2,10
Number of critics
2
Action repeat
0.99 (DMC)
0.99 (DMC)
0.99 (DMC)
0.99 (DMC)
0.99 (DMC)
Discount
0.95 (Myo)
0.95 (Myo)
0.95 (Myo)
0.95 (Myo)
0.99 (Myo)
Optimizer
Adam
Adam
Adam
Adam
Adam
Optimizer momentum (f. Bg)
(0.9, 0.999)
0.999)
(0.9, 0.999)
0.999)
(0.9. 0.999)
Policy delay
Warmup transitions
5000
5000
5000
10000
10000
Adam W weight decay critic
0.01
0.01
0.0001
AdamW weight decay
0.0
0.01
0.01
0.0
0.0001
AdamW weight decay lemperature
0.0
0.0
0.0
0.0
0.0
Butch Normalization momentum
0.99
0.99
N/A
N/A
N/A
15k, S0K,
Reset Interval networks
N/A
N/A
N/A
every 80k steps
500k and 750k steps
Batch Size
256
256
256
256
128
21

---

D Aggregated learning curves
Dog & Humanoid
DMC Mod+Easy (8 tasks)
Myo Hard (5 tasks)
Myo Easy (5 tasks)
(7 tasks)
1.00
0.75
0.50
0.25
0.00
0.00
0.25
0,50
0.75
1.00
0
3 0.00
0.25
0.50
0.75
1.00
Env Steps x10
Env Steps x10
Env Steps x10
Env Steps x10
CrosQ UTD=5
SR-SAC UTD=32
CrossQ UTD=1
Simbu
UTD=10
CrossQ+ UTD=!
CrossQ UTD=5
Simba UTD-8
TD MPC2
BRO UTD-2
Figure 6: CrossQ WN UTD scaling behavior: We plot the IQM return and 90% stratified bootstrapped confidence intervals for different UTD ratios. The results aggregated over 15 DMC environments and 10 random seeds each according to Agarwal et al. [1]. The sample efficiency scales reliably with increasing UTD ratios.
22

---

E Per environment learning curves
dog-stand
dog-walk
dog-trot
dog-run
hopper-stand
Return
humanoid stand
numanoid-wale
humanoid -un
fish-swim
hopper-hop
1000
Retur
finger-turn hard
quadruped-run
walker-run
cheetah-run
pendulum-swingup
1000
Return
myo-key-tum
myo-pen-twirl
myo-obj-hold
myo-pose
myo-reach
Success Rate
myo-key-tune ttayy-tuuhrrdd
myo-pen-twirl-hard
myo-objjhold-harr
myo-pose-hard
myo-reach-hard
Success Rate
0.0
1.0
0.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
x10
x10*
x10"
x10"
x10"
Env Steps
Env Steps
Env Steps
Env Steps
Env Steps
CrusQ= - UTD=5
CrosQ UTD=5
CroeQQ UTD=1
BRO UTD=10
BRO UTD=2
CrossQ+ + WN UTD=1
Figure 7: CrossQ WN + UTD=5 against baselines. We compare our proposed CrossQ WN UTD=5 against two baselines, BRO [32] and SR-SAC UTD=32. Results are reported on all DMC and 10 MyoSuite tasks. We plot the TQM and 90% stratified bootstrapped confidence intervals over 10 random random seeds. Our proposed approach proves competitive to BRO and outperforms the CrossQ baseline. We want to note that our approach achieves this performance without requiring any parameter resetting or additional exploration policies.
23


Reviews:
[{"review_id": "QtQ5JF6mG4", "reviewer": "Reviewer_htkW", "summary": "* In this paper, the authors point out that the existing CrossQ framework shows high sample efficiency at low update-to-data ratio (UTD=1), but becomes unstable in a high UTD environment. To this end, by incorporating a weight normalization technique on the CrossQ framework (i.e., CrossQ + WN), the authors aim to suppress the explosive increase of parameter norm during training and improve the learning stability even at high UTD. By introducing a simple but effective method, the authors prove that it is more effective than existing works in various continuous state-action benchmarking tasks.", "questions": "* The authors applied weight normalization only to the first two linear layers. I wonder if there are any ablation studies or experimental results on where to apply WN. \n* In Fig. 5, what baseline should we interpret no target network, no WN based on? And is there any experimental result when LN/WN are excluded from SAC?", "limitations": "* I think it is a meaningful study in terms of practicality. However, the novelty would be ambiguous in terms of technical algorithms or theoretical advances of reinforcement learning. \n* It can be considered as close to fine-tuning through the introduction of WN in the existing algorithm CrossQ, so the technical novelty may be limited. \n* I wonder whether the proposed method can achieve the same effect when applied to other reinforcement learning frameworks (e.g., SAC, PPO, DDPG) rather than CrossQ.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**[Strengths]**\n* One of the advantages of the proposed CrossQ+WN is that training is stable even in high UTD environments and that it achieves high performance with minimal computational resources. \n* The authors propose a relatively simple but effective algorithm without additional exploration policies, parameter resets, etc. \n* In particular, the number of parameters of the proposed method is 600k, which is very parameter-efficient compared to the existing BRO with 5M parameters. \n* No per-task tuning is required. \n\n**[Weaknesses]**\n* The performance improvement is based on empirical observations, and it would be more valuable if mathematical analysis related to convergence or sample efficiency were included. \n* As the authors mentioned, the quality of the paper would be improved in terms of generalizability if the proposed method is applied and its performance is verified in a discrete state-action space or vision-based tasks.", "was_revised": true, "final_justification": "Based on my initial review and the author's feedback, I maintain a positive assessment of this paper.", "rebuttal": "We thank the reviewer for their positive feedback and their additional questions.\n\nWe were especially happy to read that the reviewer **acknowledges the simplicity and effectiveness** of our proposed approach, as well as its **parameter-efficiency** and that they mention it is a **meaningful study in terms of practicality**.\n\nRegarding the reviewer’s open questions, we address their points here.\n\n---\n\n### **Does the proposed method show the same effect when applied to other RL algorithms beyond CrossQ?**\n\nRegarding the requested SAC experiments, we want to clarify that all experiments in the paper are based on SAC as the base algorithm, since CrossQ+WN == SAC+BN+WN.\n\nFor the rebuttal, we ran TD3+BN+WN experiments on the DMC dog and humanoid environments and Myo Hard for different UTDs (a total of 12 different environments). We get results that are comparable to our CrossQ+WN experiments without any tuning of hyperparameters. As such, we can confirm that the **sample-efficiency and stable UTD scaling benefits of BN+WN extend to TD3 and beyond CrossQ**. As per the new NeurIPS rebuttal guidelines, we were not allowed to share a figure during the rebuttal. However, we will add the additional TD3 results to the final paper.\n\nWith respect to PPO, our paper focuses on off-policy RL, as such we believe that PPO is a bit out of scope. Of course it would be interesting to test, however, in on-policy deep RL the state-based critic acts as a baseline for the advantage estimation, and therefore plays a less critical role compared to off-policy RL where the state-action critic alone is used to update the policy. As such we hypothesise that in general it won't benefit as much from the proposed changes to the critic.\n\n---\n\n### **“The authors applied weight normalization only to the first two linear layers. I wonder if there are any ablation studies or experimental results on where to apply WN?”**\n\nWe believe there is a slight misunderstanding, for which we need to improve the formulation in the paper. We write that “We apply WN to the first two linear layers [in the network]”. Since we only use MLPs with two hidden layers, this means that we apply WN to all layers but the final linear output layer.\n\nThe reason for not placing WN on the output layer is to allow scaling of predicted Q values to arbitrary magnitudes and crucially avoid the risk of underfitting. Additionally, layers that are not followed by BN are not scale invariant w.r.t. re-scaling their weights.\n\nThe reason for applying WN to all hidden layers is to preserve a constant effective learning rate within the network, since the hidden layers are followed by BN. We hope this clarifies the open question and we will make sure to improve the formulation in the paper to remove the ambiguity. \n\nThat being said, we are more than happy to prepare additional ablation studies on the placement of WN within the network to the final paper.\n\n---\n\n### **“In Fig. 5, what baseline should we interpret no target network, no WN based on?”**\nWe apologize for the unclear caption and we will make sure to improve it as well as the main text to make it more clear. Both are ablations based on our proposed CrossQ + WN algorithm. As such, these two baselines refer to “CrossQ + WN without a target network” and “CrossQ + WN without WN” respectively. The latter would be the same as saying “CrossQ with a target network”. As such, we ablate the role of the target network and the role of WN in our proposed method.\n\n---\n\n### **“is there any experimental result when LN/WN are excluded from SAC”**\nWe assume that this question is referring to the SAC+LN+WN baseline? Since CrossQ+WN == SAC+BN+WN, our goal with that particular baseline was to ablate the effect of BN in our proposed algorithm over another normalization method that leads to scale invariance. We find BN to be advantageous. Should we have misunderstood the question, please let us know."}, {"review_id": "dF0WKBSpfb", "reviewer": "Reviewer_uACo", "summary": "This work extends CrossQ by integrating weight normalization (WN) to stabilize training at high update-to-data ratios. As a result, it achieves superior sample efficiency on continuous-control benchmarks and delivers competitive performance without requiring parameter resets.", "questions": "-  BN and WN are generic normalization techniques whose benefits may extend beyond CrossQ. Have you evaluated their effectiveness across other network architectures, design choices, or reinforcement learning algorithms to demonstrate their broader applicability?\n- I appreciate the comprehensive ablation study. Would you also consider evaluating the effect of applying WN at various layers within CrossQ, rather than only in the first two linear layers, to clarify how WN placement influences performance?", "limitations": "The authors have already noted several limitations. Additional limitations are discussed in the Questions section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths: \n\n- Offers clear theoretical explanations for why WN is effective.\n\n-  The presentation is clear and easy to follow.\n\n-  Includes extensive hyperparameter ablation studies.\n\n### Weaknesses:\n\n-  BN and WN are general-purpose normalization techniques that could be applied to other designs/algorithms, yet this study evaluates them only on CrossQ[1].\n\n[1]CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity, Bhatt et al, ICLR 2024.\n\n-  Figure 4 indicates that increasing the UTD ratio from 1 to 20 for CrossQ + WN yields no significant gains on the DMC medium + easy and Myo hard tasks, suggesting that  CrossQ + WN merely preserves rather than improves the performance when scaling up the UTD ratio, while increasing the compute cost by roughly 20 times.", "was_revised": true, "final_justification": "Since the authors have addressed all my questions and concerns, I have raised my score accordingly.", "rebuttal": "We want to thank the reviewer for the positive feedback and their valuable questions.\n\nWe were especially happy that they recognized the **superior sample efficiency** of our approach and positively mentioned the **clear theoretical explanations why WN is effective**, **clear and easy to follow** presentation and our **extensive hyperparameter ablations**.\n\nRegarding their questions, we believe we have been able to address their points here.\n\n---\n\n### **Does the proposed BN+WN solution show benefits beyond CrossQ (network architectures, design choices, or reinforcement learning algorithms)**\n\nFor the rebuttal, we ran TD3+BN+WN experiments on the DMC dog and humanoid environments and Myo Hard for different UTDs (a total of 12 different environments). We get results that are comparable to our CrossQ+WN experiments without any tuning of hyperparameters. As such, we can confirm that the **sample-efficiency and stable UTD scaling benefits of BN+WN extend to TD3 and beyond CrossQ**. As per the new NeurIPS rebuttal guidelines, we were not allowed to share a figure during the rebuttal. However, we will add the additional TD3 results to the final paper.\n\nFurther, our proposed BN+WN approach is agnostic to common design decisions (e.g., network width and depth). In the paper’s ablations, we test algorithmic design decisions like resetting and the removal of target networks. For some other design decisions like skip connections or different activation functions, one would need to ensure to preserve scale invariance w.r.t. parameter rescaling when applying those. Here we also want to mention that we did not specifically tune the architecture beyond our proposed BN+WN. For the architecture, we use a standard MLP with 2 hidden layers, which results in a very simple algorithm that has been positively pointed out by most reviewers. That being said, we will extend the final paper with some architectural ablations.\n\n---\n\n### **“Would you also consider evaluating the effect of applying WN at various layers within CrossQ, rather than only in the first two linear layers, to clarify how WN placement influences performance?”**\n\nWe believe there is a slight misunderstanding, for which we need to improve the formulation in the paper. We write that “We apply WN to the first two linear layers [in the network]”. Since we only use MLPs with two hidden layers, this means that we apply WN to all layers but the final linear output layer.\n\nThe reason for not placing WN on the output layer is to allow scaling of predicted Q values to arbitrary magnitudes and crucially avoid the risk of underfitting. Additionally, layers that are not followed by BN are not scale invariant w.r.t. re-scaling their weights.\n\nThe reason for applying WN to all hidden layers is to preserve a constant effective learning rate within the network, since the hidden layers are followed by BN. We hope this clarifies the open question and we will make sure to improve the formulation in the paper to remove the ambiguity. \n\nThat being said, we are more than happy to prepare additional ablation studies on the placement of WN within the network to the final paper.\n\n---\n\n### **UTD scaling on “DMC medium + easy and Myo hard tasks”**\nWe assume the reviewer meant to refer to Myo Easy and Myo Hard instead of DMC Med+Easy, if not we apologize.\n\nWe believe that the important contribution of BN+WN is that it **stabilizes training across scaling UTDs**. As such, UTD is not a brittle hyperparameter anymore, but performance stays at least constant or increases. While increasing the UTD hyperparameter may improve sample efficiency, there is no theoretical guarantee that increasing the UTD will always improve the downstream performance of the policy, as such stability is an important aspect. For instance on Myo Easy, the vanilla CrossQ baselines completely deteriorate in performance when increasing the UTD, which our method mitigates. \n\nOn Myo Easy, CrossQ+WN already reached the maximum score at UTD=1. So, on those tasks higher UTDs confirm the stability across UTDs in contrast to vanilla CrossQ.\nOn Myo Hard, our method also remains stable across all UTDs and outperforms every baselines method at every UTD.\nFinally, on DMC Med+Easy and the Dog & Humanoid environments CrossQ+WN also scales stably and its sample efficiency significantly benefits from increasing the UTD."}, {"review_id": "6Cig7KiBly", "reviewer": "Reviewer_Gun3", "summary": "The paper presents a modification to CrossQ, a strong RL algorithm based on SAC that cleverly uses batch normalization to imrprove performance while allowing one to remove the target network, where by adding weight normalization one can increase the update-to-data ratio, increasing performance while keeping the previous sample efficiency.\n\nFurthermore the authors show that it's possible to add back the target network to CrossQ + Weight Normalization, and that it gives them a further improvement in performance.", "questions": "Do you believe Weight Normalization could improve the scaling of UTD of other RL algorithms? Would it be possible to test a few of them?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is clear about its contribution and motivation, the results are compelling and the experiments are done over a good variety of environments, the presented method is simple while having a significant effect.\n\nThe main weakness is that the paper focuses on augmenting a single specific RL algorithm/method, and doesn't really evaluate whether Weight Normalization could be beneficial for other algorithms like a more basic SAC, TD3, PPO or others.", "was_revised": true, "final_justification": "The paper explores the impact of Weight Normalization on the performance of off-policy RL algorithms for continuous control, adding another result to our understanding of the interplay between Neural Network architecture and final performance in Reinforcement Learning tasks.", "rebuttal": "We thank the reviewer for their positive feedback and their additional questions.\n\nWe were especially happy to see that the reviewer called our results compelling and acknowledged the simplicity and significant effect of our proposed method.\n\nRegarding the reviewer’s question:\n\n---\n\n### **“Do you believe Weight Normalization could improve the scaling of UTD of other RL algorithms? Would it be possible to test a few of them?”**\n\nFor clarity, we want to answer this question in several steps:\n\n**Could WN alone be beneficial for scaling other more basic algorithms such as SAC, TD3, PPO?**\n\nOur paper focuses on off-policy RL, as such we believe that PPO is a bit out of scope. Of course it would be interesting to test, however, in on-policy deep RL the state-based critic acts as a baseline for the advantage estimation, and therefore plays a less critical role compared to off-policy RL where the state-action critic alone is used to update the policy. As such we will dismiss PPO here since on-policy scaling would deserve its own independent study.\n\nWe see the core contribution of our paper in showing that the combination of BN+WN leads to stable scaling across different UTDs. We provide analysis as to the underlying reason (exploding weight norms for the CrossQ baseline vs. stable effective learning rate for CrossQ+WN). The justification to apply WN is based on the scale invariance that BN brings to the network. As such, we need to differentiate between “just WN” and the combination of BN+WN.\n\n**Testing BN+WN with other more basic algorithms**\n\nWe want to clarify that all experiments in the paper are based on SAC as the base algorithm, since CrossQ+WN == SAC+BN+WN.\n\nFor the rebuttal we ran TD3+BN+WN experiments on the DMC dog and humanoid environments and Myo Hard for different UTDs (a total of 12 different environments). We get results that are comparable to our CrossQ+WN experiments  without any tuning of hyperparameters. As such, we can confirm that the **sample-efficiency and stable UTD scaling benefits of BN+WN extend to TD3 and beyond CrossQ**. As per the new NeurIPS rebuttal guidelines we were not allowed to share a figure during the rebuttal. However, we will add the additional TD3 results to the final paper."}, {"review_id": "MGjlB5CF0m", "reviewer": "Reviewer_z4vh", "summary": "The paper studies the updates-per-data (UTD) scaling properties of the recent CrossQ algorithm. The authors find that the vanilla CrossQ formulation does not scale reliably with UTD. They propose design changes that improve its scaling properties (i.e., using weight normalization and bringing back the target network). The authors test their approach on 25 simulated locomotion and manipulation tasks of varying complexity and find that incorporating these design choices improves not only the UTD scaling of CrossQ but also decreases the hyperparameter sensitivity of the algorithm.", "questions": "1. In Figure 5, what network size does SAC + LN + WN use? Do authors have any hypothesis as to why this version performs so poorly on MyoSuite given that Simba/BRO/CrossQ perform well (and all are driven by size+normalization)?\n2. Do authors have any insights into why target networks are important in their proposed version of CrossQ? Does the importance of the target network correlate with higher UTD?\n3. Do the authors have any opinion on the generality of the proposed architecture? For example, could the proposed architecture (BN + WN) be used in BRO instead of BRONet?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is very clearly written and delivers on the claims made in the introduction.\n2. Understanding compute scaling in off-policy RL is an active and promising research area\n3. The proposed method is simple and seems to be effective\n\nWeaknesses:\n\n1. Slightly limited novelty of the proposed approach. But still, I think the result is useful\n\n\nNitpicks/Editorial:\n\n1. (109) \"Janner et al. [21] generate additional modeled data, by virtually increasing the UTD. In REDQ, Chen et al. [8] leverage ensembles of Q-functions, while Hiraoka et al. [17] use dropout and LN to effectively scale to higher UTD ratios\" - the way MBPO is described sounds odd (it uses world model to simulate transitions that are used in high UTD training); description of RedQ also slightly misses the point (the sampling from the ensemble is important there)\n2. (165) \"which in turn decreases the ELR even quicker and could be the case for instabilities and low training performance\" - \"instabilities\" paired with very low ELR sounds a bit odd.", "was_revised": true, "final_justification": "Despite the hesitation of other reviewers and limited novelty, I believe that this work will be interesting and helpful for the community. As such, I recommend acceptance.", "rebuttal": "We thank the reviewer for their very positive feedback and their additional questions and comments to improve the paper further.\n\nWe were especially happy to see that they found the **paper clearly written** and acknowledged that we **deliver on the claims made in the paper**. Further, we were happy to read that they acknowledge the simplicity and effectiveness of our proposed solution and that we **contribute useful results to a promising research area**.\n\nWe thank the reviewer for pointing out the nitpicks, we will make sure to improve those points in the final version of the paper.\n\nRegarding the reviewer’s question:\n\n---\n\n### **“In Figure 5, what network size does SAC + LN + WN use?”**\n\nFor the ablation in Figure 5, all methods use the exact same critic network size (MLP with two hidden layers of size 512 neurons each), in order to isolate the respective changes as cleanly as possible.\n\n---\n\n### **“Do authors have any hypothesis as to why this version performs so poorly on MyoSuite?”**\n\nAs to why SAC+LN+WN performs worse is a very interesting question about which, we can only hypothesise for now.\n\nFirst, for SAC+LN+WN, we do see a performance improvement between UTD 1 and 5 in Figure 5 on Myo Hard from ~0.3 to ~0.6. For BRO we have two data points, UTD=2 and 10, if we assume a linear trend, then SAC+LN+WN would only be slightly worse than BRO on Myo Hard. Of course to confirm this we would need to run the BRO baseline for those UTDs, however, it shows that the performance gap for the SAC+LN+WN to the LN based baselines (BRO and Simba) might not be as large as it might look at first sight.\n\nSecond, the remaining performance delta of SAC+LN+WN towards BRO/Simba can potentially be attributed to the much simpler and smaller architecture compared to the BRO/Simba baselines (~600k vs. ~5M parameters) which include skip connections and additional implementation details. \n\nThird, the SAC+LN+WN baseline demonstrates higher sample- and parameter-efficiency of BN compared to LN in this context. As such, our results reproduce the findings from the original CrossQ paper but in the high UTD and WN setting. So far those findings are empirical and we believe further understanding this on a theoretical level would be a really interesting endeavour for future work.\n\n---\n\n### **Do authors have any insights into why target networks are important in their proposed version of CrossQ?**\nIn general the importance of the target network in off-policy RL is a really interesting question. In our specific case, Figure 5 shows that in general CrossQ+WN also manages to learn without a target network, however, the performance is significantly reduced. Considering the constant, but overall higher effective learning rate of CrossQ+WN compared to vanilla CrossQ (Figure 3), we think that the target network still adds a stabilizing effect to the training dynamics that ultimately increases the overall learning performance. At least in our experiments we did not find a meaningful correlation between UTD and the importance of the target network. However, it is a really interesting question that deserves a much deeper investigation in future work.\n\n---\n\n### **Generality of the proposed BN+WN solution**\nOf course, technically nothing speaks against replacing BroNet with a BN+WN architecture. We find that our proposed solution is competitive to BRO. Apart from the larger and more complicated network architecture, BRO is also based on SAC and has many additional implementation details like resetting and a second optimistic actor. Both are orthogonal to our proposed method. In our ablations we already show that our method neither requires, nor significantly benefits from additional network resets. As such, integration with BRO would investigate whether CrossQ+WN would additionally benefit from better exploration through BRO’s optimistic actor. Of course, this is an interesting question. Due to limited computational resources and time during the rebuttal, and since most reviewers asked about the generality of our approach w.r.t. \"more basic\" RL algorithms such as TD3, we chose to focus on extending our results to TD3.\n\nFor the rebuttal we ran TD3+BN+WN experiments on the DMC dog and humanoid environments and Myo Hard for different UTDs (a total of 12 different environments). We get results that are comparable to our CrossQ+WN experiments  without any tuning of hyperparameters. As such, we can confirm that the **sample-efficiency and stable UTD scaling benefits of BN+WN extend to TD3 and beyond CrossQ**. As per the new NeurIPS rebuttal guidelines we were not allowed to share a figure during the rebuttal. However, we will add the additional TD3 results to the final paper."}]

Author rebuttal:

Meta-review: The paper proposes adding Weight Normalization (WM) to the CrossQ RL algorithm, showing that it stabilizes the training and improves its performance, especially in the high update-to-data (UTD) ratios.

All reviewers are supportive of this work. We have four Accepts (5) and one Borderline Accept (4). Therefore, I also recommend acceptance of the paper.

Additional comments:
- Figures 1 and 2 appear without any description until much later in the paper. The first time a figure is explained is on page 4, and that is for Figure 3.

- Bellmann --> Bellman

- L67: It is not accurate to say that SAC solves the optimization problem indicated in the first few lines of Section 2. The optimization problem solved by SAC has an entropic term. And as a result, the state-value function (Eq. 1) is also defined differently in SAC. Please clarify.
Decision: Accept (poster)