Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control
Authors: Michal Nauman
Mateusz Ostaszewski
Krzysztof Jankowski
Piotr Miłoś
Marek Cygan
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Machine Learning
Reinforcement Learning
Scaling
Primary area: reinforcement_learning
TL;DR: We show that combining increased critic model capacity with certain RL-specific improvement leads to very efficient agents

Abstract: Sample efficiency in Reinforcement Learning (RL) has traditionally been driven by algorithmic enhancements. In this work, we demonstrate that scaling can also lead to substantial improvements.  We conduct a thorough investigation into the interplay of scaling model capacity and domain-specific RL enhancements. These empirical findings inform the design choices underlying our proposed BRO (Bigger, Regularized, Optimistic) algorithm. The key innovation behind BRO is that strong regularization allows for effective scaling of the critic networks, which, paired with optimistic exploration, leads to superior performance. BRO achieves state-of-the-art results, significantly outperforming the leading model-based and model-free algorithms across 40 complex tasks from the DeepMind Control, MetaWorld, and MyoSuite benchmarks. BRO is the first model-free algorithm to achieve near-optimal policies in the notoriously challenging Dog and Humanoid tasks.

Bigger, Regularized, Optimistic: scaling for compute and sample-efficien continuous control
Michal Nauman
Mateusz Ostaszewski
Krzysztof Jankowski?
Piotr Milog!.2. 1,24
Marek Cygan
Abstract
Sample efficiency in Reinforcement Learning (RL) has traditionally been driven by algorithmic enhancements. this work, we demonstrate that scaling can also lead to substantial improvements. We conduct thorough investigation into the interplay of scaling model capacity and domain-specinn RL enhancements. These empirical findings inform the design choices underlying our proposed BRO (Bigger, Regularized, Optimistic) algorithm. The key insight behind BRO is that strong regularization allows for effective scaling of the critic networks, which, paired with optimistic exploration, leads to superior performance. BRO achieves state-of-the- art results, significantly outperforming the leading model-based and model-free algorithms across 40 complex tasks from the DecpMind Control, MetaWorld, and MyoSuite benchmarks. BRO is the first model-free algorithm achieve near-optimal policies in the notoriously challenging Dog and Humanoid tasks.
1 Introduction
(MF)
(MB)
(MF)
(MF)
(MF)
(MF)
Dog Humanoid (7 tasks)
DeepMind Control (15 lasks)
MetaWoold lasks)
MyoSuite tasks)
1.0
1.4
0.4
0.
0.4
0.4
01
0.1
0.
0.0
Figure 1: BRO sets new state-of-the-an outperforming model-free (MF) and model-based (MB) algorithms on complex tasks covering benchmark suites. Y-axes report interquartile mean calculated 10 random seeds, with 1.0 representing the best possible performance in given benchmark. We use 1M environment steps.
Deep learning has seen remarkable advancements in recent years, driven primarily by the development of large neural network models (Devlin et al. 2019 Tan & Le 2019 Dosovitskiy et al. 2020). These advancements have significantly benefited fields like natural language processing and computer vision and have been percolating to RL as well (Padakkar et al. 2023 Zitkovich et al.) 2023). Interestingly, some recent work has shown that the model scaling can be repurposed in discrete control (Schwarzer et al. 20233] Obando-Ceron et al. 2024), but these approaches cannot be directly translated to continuous action RL. As such, they rely discrete action representation, whereas many physical control tasks have continuous, realevalued action spaces.
achieve sample efficiency
'Ieass NCBR; University of Warsaw, Warsaw University of Technology, *Polish Academy of Sciences, Nomagic. Correspondence to: Michal Nauman <nauman, mi c0gmail. com>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1.0
1.0
0.8
0.8
0.6
0.6
1QM
1QM
BRO
DRO (Fast
0.4
0.4
TD-MB MPCC
SR-SAC
0.2
0.2
TD3
0.0
0.0
CrossQ
0
0.25
0.50
0.75
1.00
10k
20k
30k
40k
50k
Environment Step (M)
Wallclock Time (Seconds)
Figure 2: We report sample efficiency (left) and wallclock time (right) for BRO and BRO (Fast) (BRO with reduced replay ratio for increased compute efficiency), well baseline algorithms averaged over 40 tasks listed in Table 4 BRO achieves the best sample efficiency, whereas BRO (Fast) matches the sample efficiency of model-based TD-MPC2. In terms of wall clock efficiency, BRO runs approximately 25% faster than TD-MPC2. Remarkably, BRO (Fast) matches the wallclock efficiency of a standard SAC agent while achieving 400% better performance. The Y-axis reports the interquartile mean, with 1.0 representing the maximal possible performance.
Conventional practice in continuous deep RL has relied on small network architectures (Haarnoja
al. 2018 Hiraoka et al., 202T; Raffin et al. [022 D Oro et al.) 2022), with the primary focus on algorithmic improvemenss. These enhancements aim to achieve better sample efficiency and address key challenges such as value overestimation (Fujimoto et al.] 2018 Moskovitz et al. 2021 Cetin & Celiktutan, 2023), exploration (Chen et al.] [2077|Ciosek et al.
& Cygan (0233)).
and increasing the number of gradient steps (Nikishin et al.] 2022 D'Oro et al. (002). Additionally, evidence suggests that naive model capacity scaling can degrade performance (Andrychowicz et al., 2021, Bjorek et al. 202T). We challenge this status quo by posing critical question: Can significant performance improvements in continuous control be achieved by combining parameter and replay scaling with existing algorithmic improvements?
In this work, we answer this question affirmatively, identifying components essential to successful scaling. Our findings are based on a thorough evaluation of a broad range of design choices, which include batch size (Obando Ceron et al. 2024), distributional Q-values techniques (Bellemare et al. 2017, Dabney et al. 2018), neural network regularizations (Bjorck et al. 202T Nauman et al. 2024). and optimistic exploration (Moskovitz et al. 2021 Nauman & Cygan 2023a). Moreover, we carefully investigate the benefits and computational costs stemming from scaling along two axes: the number of parameters and the number of gradient steps. Importantly, we find that the former can lead to more significant performance gains while being more computationally efficient in parallelized setups.
Our work culminates in developing the BRO (Bigger, Regularized, Optimistic) algorithm, novel sample-efficient model-free approuch. BRO sigmificantly outperforms existing model-free and model- based approaches on 40 demanding tasks from the DeepMind Control, Meta World, and MyoSuite benchmarks, as illustrated in Figures [ and Notably, BRO the first model-free algorithm to achieve near-optimal performance in challenging Dog and Humanoid tasks while being 2.5 times more sample-efficient than the leading model-based algorithm, TD-MPC2. The key BRO innovation is pairing strong regularization with critic model scaling, which, coupled with optimistic exploration, leads to superior performance. We summarize our contributions:
Extensive empirical analysis- conduct extensive empirical analysis focusing on critic model scaling in continuous deep RI. By training over 15, 000 agents, we explore the interplay between critic capacity, replay ratio, and comprehensive list of design choices.
BroNet architecture & BRO algorithm we introduce the BRO algorithm. a novel model-free ap- proach that combines BroNet architecture for critic scaling with domain-specite RL enhancements. BRO achieves the performance on 40 challenging tasks across diverse domains.
Scaling & regularization we offer several insights, with the most important being: regularized critic scaling outperforms replay ratio scaling in terms of performance and computational efficiency; the inductive biases introduced by domain-specific RL. improvements can be largely substituted by critic scaling, leading to simpler algorithms.

---

Bigger, Regularized, Optimistic (BRO) algorithm
This section presents our novel Big, Regularized, Optimistic (BRO) algorithm and its design prin- ciples. The model-free BRO is conclusion of extensive experimentation presented in Section 3 and significantly outperforms existing state-of-the-art methods on continuous control tasks from proprioceptive states (Figure
2.1 Experimental setup
We compare BRO against variety of baseline algorithms. Firstly, we consider TD-MPC2 (Hansen er al. (023). ) model-based state-of-the-art that was shown to reliably solve the complex dog domains. Secondly; we consider SR-SAC (D'Oro et al. 2022), a sample-efficient SAC implementation that uses large replay ratio of 32 and full-parameter resets. For completeness, we also consider CrossQ (Bhatt) jct al. 2023), a compute-cfficien method that was shown to outperform ensemble approaches, as well as standard SAC (Haarnoja et al. 2018) and TD3 (Fujimoto et al. 2018). We run all algorithms with 10 random seeds, except for TD-MPC2, for which we use the results provided by the original manuscript (Hansen al., 2023). We describe the process of hyperaarameter selection for all considered algorithms in Appendix D and share BRO pseudocode Appendix (Pseudocode [T]. We implement BRO based the JaxRL (Kostrikov) [2021) and make the code available under the following link:
Environments We consider a wide range control tasks, encompassing a total of 40 diverse, complex continuous control tasks spanning three simulation domains: DeepMind Control (Tassal et al. 2018). MetaWorld (Yu er al.) 2020), and MyoSuite (Caggiano et al. 2022) (a detailed list of environments can be found in Appendix Cp. These tasks include high-dimensional state and action spaces (with |S] and A| reaching 223 and 39 dimensions), sparse rewards, complex locomotion tasks, and physiologically accurate musculoskeletal motor control. We run the algorithms for LM environment steps and report the final performance unless explicitly stated otherwise. We calculate the interquartile means and confidence intervals using the RLiable package (Agarwal et 2021
(a) DeepMind Control (DMC)
(b) MetaWorld (MW)
(c) MyoSuite (MS)
Figure 3: We consider a total of 40 tasks from DeepMind Control (DMC), MetaWorld (MW), and MyoSuite (MS). In particular, we chose the tasks with the biggest optimality gap according to previous evaluations (Hansen al. 2023). We list all considered tasks in Table
2.2 BRO outline and design choices
The BRO algorithm is based on the well-established Soft Actor-Critic (SAC) Haarnoja et al. 2018 (see also Appendix A) and composed of the following key components:
Bigger BRO uses scaled critic network with the default of 5M parameters, which is approximately times larger than the average size of SAC models (Haarnoja al. 2018) as well as scaled training density with default replay ratio of RR 10, and RR for the BRO (Fast)
version.
Regularized the BroNet architecture, intrinsic the BRO approach, incorporates strategies for regularization and stability enhancement, including the utilization of Layer Normalization (Ba et al., 2016) after each dense layer, alongside weight decay (Loshchilov Hutter 2017) and parameeree resets (Nikishin et al. 2022).
'The replay ratio refers the number of gradient updates per environment step.

---

1.0
1.0
BroNec+ BRO
Vanilla + BRO
Vanilla + SAC
0.8
Special DRO
0.8
Spectral
0.6
0.6
1QM
1QM
0.4
0.4
0.2
0.2
0.0
0.0
0.55
1.05
2,83
4.92
26.31
0.34
0.79
2.47
4.56
25.74
Parameters (M)
Parameters (M)
Figure 4: Scaling the critic parameter count for vanilla dense (Fujimoto et al. 2018 spectral normalization ResNet (Bjorck et al., 2021). and our BroNet for BRO (left), and SAC (right). We conclude that achieve the best performance, we need both the right architecture (BroNet) and the correct algorithmic enhancements encapsulated in BRO. We report interquartile mean performance after 1M environment steps tasks listed Table 3 with error bars indicating 95% CI from 10 seeds. On the X-axis, we report the approximate parameter count of each configuration.
Optimistic-BRR uses dual policy optimistic exploration (Nuuman & Cygan 2023a) and non- pessimistic (Nauman et al. 2024) quantile Q- Q-value approximation (Dabney et al. 2018 Moskovitz et al. 2021 for balancing exploration and exploitation.
The full details of the algorithm, along with the pseudo-code, are provided Appendix B Figure summarizes the impact of removing components of BRO. We observe the biggest impact of scaling the critic capacity (scale) and replay ratio (RR), as well as using non-pessimistic Q-value, i.e. removing Clipped Double Q-learning (CDQ).
Scaling critic network and BroNet architec- ture The key contribution of this paper is showing how to enable scaling the critic network. We recall that naively increasing the critic ca- pacity does not necessarily lead to performance improvements and that successful scaling de- pends on carefully chosen suite of regular- ization techniques (Bjorck et al. 202T). Fig- ure shows BroNet architecture. which, up to our knowledge, did not exist previously in the literature. The architecture begins with dense layer followed by Layer Norm (Ba et al., 2016) and ReLU activation. Subsequently, the network comprises ResNet blocks, each consist- ing of dense layers regularized with Layer Norm. Consequently, the ResNet resembles the FFN sub-layer utilized in modern L.LM archi- tectures (Xiong et al., 2020), differing primarily in the placement of the Layer Norms. Crucially, we find that BroNet scales more effectively than other architectures (Figure 4 (left).. However, the right choice of architecture and scaling is silver bullet. Figure 4 (right) shows that when these are plugged into the standard SAC algorithm naively, the performance is weak. The important elements are additional regularization (weight decay and network resets) and optimistic exploration (see below). Interestingly, we did not find benefits from scaling the actor networks, further discussed in Section
BroNet
Layer Normalization
RelU
Dense
Layer Normallzation
ReLU
Nx
Dense
Layer Normallzation
Residual Addition
Figure S: BroNet architecture employed for actor and critic. Fach fully connected layer is augmented with Layer Norm, which is essential to unlocking scaling. We R 5M parameters and N 2 the default setting.

---

0.9
0.9
0.8
0.8
0.7
0.7
1QM
1QM
0.6
0.55
0.6
RX-1
RR-2
2.83
RR-5
0.5
4,92
0.5
RR-10
26.31
RR-15
0.4
0.4
2
10
15
5k
10k
20k
40k
80K
Replay Ratio
Wallelock Time (Seconds)
Figure 6: To account for sample efficiency, report the performance averaged 250k, 500k, 750k, and 1M environment steps across different 5 replay ratios and 5 critic model sizes. All agents evaluated in tasks listed in Table and 10 random seeds per variant. The left figure shows performance scaling with increasing replay ratios (shapes) and model sizes (colors). The right figure examines the tradeoff between performance and computational cost when scaling replay ratios versus critic model sizes. Increasing model size leads to substantial performance improvements at lower compute costs compared to increasing the replay ratio. We present more scaling results in Appendix including description of model sizes in Table
Scaling replay ratio and relation to model scaling Increasing replay ratio (D'Oro et al. 2022
another axis scaling. We investigate mutual interactions by measuring the performance across different model scales (from 0.55M to M) and RR settings (from RR to RR 15). Figure 6 reveals that the model scaling has a much stronger impact plateauing at RJ 5M parameters. Increasing the replay ratio also leads to noticeable benefits. For example, 26M model with RR = 1 achieves significantly better performance than a small model with RR = 15, even though the 26M model requires three times less wallclock time. Importantly, model scaling and increasing replay ratio work well in tandem and are interchangeable to some degree. We additionally note that the replay ratio has bigger impact on wallclock time than the model size. This stems from the fact that scaling replay ratio leads to inherently sequential calculations, whereas scaling model size leads to calculations that can be parallelized. For these reasons, BRO (Fast) with RR = 2 and 5. network offers attractive trade-off, being already very sample efficient and fast at the same time.
Optimistic exploration and Q-values BRO utilizes two mechanisms to increase optimism. We observe significant improvements stemming from these techniques in both BRO and BRO (Fast) agents (Figure 7). They are particularly pronounced in the early stages of the training and for smaller models (Figure 9a
The initial mechanism involves deactivating Clipped Double Q-learning (CDQ) (Fujimoto al. 2018). a commonly employed technique in reinforcement learning aimed at mitigating Q-value overestimation. For further clarification, refer to Appendix B.6 particularly Eq. where we take the ensemble mean instead of minimum for Q- Q-value calculation. This is surprising, perhaps, as it goes against conventional wisdom. However, some recent work has already suggested that regularization might effectively combat the overestimation (Nauman et al. 2024). We observe a much stronger effect. In Figures 7 Da] we compare performance of BRO with BRO that uses CDQ. This analysis indicates that using risk-neutral Q-value approximation in the presence of network regularization unlocks significant performance improvements without value overestimation (Table
The second mechanism is optimistic exploration. We implement the dual actor setup (Nauman & Cygan 2023a), which employs separate policies for exploration and temporal difference updates. The exploration policy follows an optimistic upper-bound Q-value approximation, which has been shown to improve the sample efficiency of SAC-based agents (Ciosek et al. [2099] Moskovitz al. 202T Nauman & Cygan 2023a). In particular, we optimize the optimistic actor towards a KL-regularized Q-value upper-bound (Nauman & Cygan| (202a)), with Q-value upper-bound calculated with respect to epistemic uncertainty calculated according to the methodology presented in Moskovitz et al. (2021). As shown in Figure using dual actor optimistic exploration yields around 10% performance improvement in the BRO model.
5

---

BRO
BRO (Fast)
Scale
+ CDQ
+ RR-1
Dual
Quantile
WD
BRO
40
50
60
70
80
90
100
40
50
60
70
80
90
100
% of Base Model Performance
Figure Impact of removing various BRO components on its performance. We report the percentage of the final performance for BRO (left) and BRO (Fast) (right). The y-axis shows the components that are ablated: -Scale denotes using standard-sizer network, +CDQ denotes using pessimistic Clipped Double Q-learning (which is removed by default in BRO). +RR=1 uses the standard replay ratio, -Dual removes optimistic exploration, and -Quantile and -WD stand for removing quantile Q-values and weight decay, respectively. We report the interquartile mean and 95% CIs for tasks in Table 3 with 10 random seeds. The results indicate that the Scale, CDQ, and RR=1 components are the most impactful for BRO. Since BRO (Fast) has RR-2 by default, reducing it to one does not significantly affect its performance.
Others We mention two other design choices. First, we use a smaller batch size of 128 than the typical one of 256. This is computationally beneficial while having marginal impact on performance, which we show in Figure 15) Secondly, we use quantile Q-values (Bellemare el al. 2017 Dabney al., 2018). We find that quantile critic representation improves performance (Figure 7). particularly for smaller networks. This inprovement, however, diminishes for over wver-paaameterized agents (Figure 9ap. On top of the performance improvements, the distribution setup allows us estimate epistemic uncertainties, which we leverage in the optimistic exploration according to the methodology presented Moskovitz al. (2021)
Analysis
This section summarizes the results of 15,000 experiments, detailed Table 2 develop the BRO algorithms. These experiments also provided numerous insights that believe will be of interest to the community. We adhered to the experimental setup described Section 2.1 We also present additional experimental results in Appendix
which led us to
Scaling model-free critic allows superior performance We recall that the most important finding that skillful critic model scaling combined with simple algorithmic improvements can lead to extremely sample-efficient performance and the ability to solve the most challenging environments. We deepen these observations in experiments depicted in Figure & Namely, we let the other algo- rithms, including state-of-the-an model-based TD-MPC2, run for 3M steps on the most challenging tasks in the DMC suite (Dog Stand, Dog Walk, Dog Trot, Dog Run. Humanoid Stand, Humanoid Walk, and Humanoid Run). TD-MPC2 eventually achieves BRO performance levels, but it requires approximately 2.5 more environment steps.
Algorithmic improvements matter less as the scale increases The impact of algorithmic im- provements varies with the size of the critic model. As shown in Figure Pa- while techniques like smaller batch sizes, quantile Q-values, and optimistic exploration enhance performance for 1.05. and 4.92M models, they do not improve performance for the largest 26.3M models. We hypoth- esize this reflects tradeoff between the inductive bias of domain-specinn RL techniques and the overparameterization of large neural networks. Despite this, these techniques still offer performance gains with lower computing costs. Notably, full-parameter resets (Nikishin et al. 2022 O'Oro et al.) 2022) are beneficial; the largest model without resets nearly matches the performance of the BRO with resets.

---

0.8
Approx. 2.5x more steps
0.6
BRO (MF)
SAC (MF)
10M
TD-MPC2 (MB)
TD3 (MF)
0.4
SR-SAC (MF)
CrossQ (MF)
0.2
0.0
0
20
50
100
150
250
300
Environment Step (x10*)
Figure 8: IQM return learning curves for four Dog and three Humanoid environments from the DMC benchmark, plotted against the number of environment steps. Notably, the model-based approach (TD-MPC2) requires approximately 2.5 times more steps to match BRO performance.
Scaling actor is not effective Previous works underscore the relative importance of critic and actor networks in off-policy algorithms like SAC (Fujimoto et al. 2018 D'Oro et al. 2022 Li et al.) 2022). For instance, Nikishin al. (2022) found that critic regularization is significantly than actor regularization. We confirm this result by showing that, for off-policy continuous control actor-critic algorithms, increasing critic capacity leads to much better results than increasing the actor model size, which in some cases might even detrimental (Figure Day. As such, practitioners can achieve performance improvements by prioritizing critic capacity over actor capacity while adhering to memory limitations.
important
Target networks yield small but noticeable performance benefits Using target networks doubles the memory costs (Schwarzer et al., 2020; Bhatt et al.) 2023 Lee et al. 2024). which can be a significant burden for large models. In Figure 9b] compare the performance of standard BRO and BRO (Fast) agents against their versions without target networks. Consistent with established understanding, we find that using target networks yields small but significant performance improve- ment. However, we observe substantial variation in these effects among benchmarks and specific environments (Figure 9bl & Figure 16). For example, the majority of performance improvements in DMC and MS environments are attributable to specific tasks.
Architecture matters (especially in complex environments) By break- ing down the results from Figure into individual environments, the BroNet architecture achieves better performance in all of them, but the differences are most pronounced in the Dog environments. Therefore, we deepened our analysis with extra metrics to understand these discrepan- cies better. Taeeonneeesssssttes that BroNet outperforms the other archi- tectures regarding final performance. The Vanilla MLP exhibits instabilities across all measured metrics, including gradient norm, overestimation, and TD error. While using the Spectral architecture maintains moderate gradient norms and overestimation, struggles significantly with minimizing the TD error.
Table 1:
Comparison of BroNet, Spectral (Bjorck et al. 202T), and Vanilla MLP architectures in notriously hard Dog environments. All metrics except return are averaged over time steps. All architectures are combined with BRO.
BroNet
Spectral
Vanilla
Final return
763.5
73.5
167.
|V||a
35.5
88.
9.61E+04
Mean Q-values
58.06
153.85
1.20E+05
TD-error
0.38
4.31E+04
6.03E+07
In Nauman et al. 2024), the authors indicate that the gradient norm and overestimation are strong indicators of poor performance in Dog environments. However, these results suggest that identifying a single cause for the challenges in training a reinforcement learning agent is difficult, highlighting the complexity of these systems and the multifaceted nature of their performance issues.
7

---

1.0
100
% BRO Performance
0.8
80
60
0.6
40
0.4
1,05 M
20
4,92 M
0.2
Target Not Target Net
26.31 M
0
+000
+000-1
-Keset
+Saaled
QQunale
+Barch
0.0
DMCC-Fae
DMM
MS
MWW-Fast
MW
(a) Design Choices
(b) Target Networks
Figure 9: (Left) We analyze the importance of BRO components dependent the critic model size. Interestingly, most components become less important as the critic capacity grows. (Right) We report the performance of BRO variants with and without a target network. All algorithm variants run with 10 random seeds.
What did not work? While researching BRO, we tested a variety of techniques that were found improve the performance of different RL. agents; however, they did work in our evaluations. Firstly, we found that using Nsstep returns (Sutton & Barto 2018 Schwarzer et al., 2023) does not improve the performance in the tested environments, We conjecture that the difference between N step effectiveness in Atari and continuous control benchmarks stems from the sparser reward density in the former. Furthermore, we evaluated categorical RL (Bellemare al. 2017) and HLGauss (Imani & White 2018 Farebrother et al. 2024) Q-value representations, but found that these techniques are not directly transferable
deterministic policy gradient setup and introduce training instabilities when applied naively, resulting in significant amount of seeds not finishing their training. Finally, we tested a variety of scheduling mechanisms considered by SSchwarzer et al. (2023) but found that the performance benefits are marginal and highly lask-dependent while introducing much more complexity associated with hyperparameter tuning. A complete list of tested techniques presented in Appendix B.8
Are current benchmarks enough? As illustrated Figure 10 even complex tasks like Dog Walk or Dog Trot can be reliably solved by combining existing algorithmic improvements with critic model scaling within I million environment steps. However, some tasks remain unsolved within this limit (c.g., Humanoid Run Acrobot Swingup). Tailoring algorithms to single tasks risks overfitting to specific issues. Therefore, we advocate for standardized benchmarks that reflect the sample efficiency of modern algorithums. This standardization would facilitate consistent comparison of approaches, accelerate advancements by focusing on a common set of challenging tasks, and promote the development of more robust and generalizable RL algorithms. On that note, in Appendix we report BRO performance at earlier stages of the training.
DeepMind Coarrol
MetoWorld
MyoSuite
1.0
0.8
0.4
BRD
SR. SAC
0.0
Figgrr Tom
Countrgpe Rum
Humannll Stamd
Posebtou toyop
Wolker Reo
Human Henurod Satk
Swiin
Hemoomane from
Hoo
Dul stand busc Hatk
Can Des Run
Back
Fishh
Pick Cort
Lever Pull
West
Catka Faal Inurrt
fleed Slick Pull
Basiiiual Phski
Coffee Auunddy
Object Hold
Ruset
Key Tum
Per Trest
Ronel Hard
Por Toist stard
Tum Hert
ORject Hard
Figure 10: Our experiments cover 40 of the hardest tasks from DMC (locomotion), MW (manip- ulation), and MS (physiologically accurate musculoskeletal control) considered in previous work (Hansen et al, 2023). In those tasks, the tfat--of-the-a model-free SR-SAC O'o et al. 2022) achieves more than 80% of maximal performance in 18 out of 40 tasks, whereas our proposed BRO in 33 out of 40 tasks. BRO makes significant progress in the most complex tasks of the benchmarks.
8

---

BroNet architecture is useful beyond continuous control We design additional experiments to
test the effectiveness of the naive application of BroNet to popular offline reinforcement learning problems in two offline RI. benchmarks: AntMaze (6 tasks); and Adroit (9 tasks) (Fu et al. 2020). We run Behavioral Cloning (BC) in pure offline (Sutton & Barto 2018), Implicit Q-Learning (IQL) offline fine-tuning (Kostrikov et al. 2022), as well as online reinforcement learning with offline data (Ball et al. 2023). We run all these algorithms with the default MLP network, as well as BroNet backbone. As shown in Figure 11 we find that the naive application of BroNet leads to performance improvements across all tested algorithms.
Aggregate (15 tasks)
AniMaze (6 tusks)
Adroit (9 tasks)
1.0
0.5
1.0
0.8
0.8
0.6
1044
0.0
0.2
0.50
00.5
0.25
0,5
41.50
02.25
0.25
0.5
17.00
40.55
0.25
0.5
Enstronment Step (M)
Enstronment Step (M)
Enstronment Step (M)
Figure 11: We test three scenarios: offline (comparing vanilla BC to BroNet-based BC), offline fine- tuning (comparing vanilla IQL. to BroNet-based IQL), and online with offline data (comparing vanilla SAC to BroNet-based SAC). The solid line represents BRO-based and the dashed line represents vanilla variants. Negative values on the X-axis refer to offline training. 10 seeds per task.
4 Related Work
4.1 Sample efficiency through algorithmic improvements
A significant effort in RL has focused algorithmic improvements. One recurring theme is controlling value overestimation (Fujimoto et al., 2018 Moskovitz et al. 2021 Cetin Celiktutan 2023). For instance, Fujimoto et al. (2018) proposed Clipped Double Q-learning (CDQ), which updates policy and value networks using a lower- bound Q-value approximation. However, since a pessimistic lower-bound can slow down learning, Moskovitz et al. (202T) introduced an approach that tunes pessimism online. Recently, Nauman et al. (2024) showed that layer normalization can improve performance without value overestimation, eliminating the need for pessimistic Q-learning.
notable effort has also focused on optimistic exploration (Wang et al., 2020 Moskovitz et al.) 2021). Various methods have been developed to increase sample efficiency via exploration that greedy with respect to a Q-value upper bound. These include closed- form transformations of the pessimistic policy (Ciosek et al. 2019) or using a dual actor network dedicated to exploration (Nauman & Cygan 2023a).
4.2 Sample efficiency through scaling
Recent studies demonstrated the benefits of model scaling when pre-training on large datasets (Driess
al. 2023) Schubert et al.] 2023: Taiga al. (023) in pure offline RL setups (Kumar al.) 2023).
Additionally, model scaling has proven advantageous for model-based online RL (Hafner et al. 2023
Hansen et al. 2023 Wang et al. 2024 However, in these approaches, most of the model scale
dedicated to world models, leaving the value network small. (Schare Schwwrzer et al. (202)) found that increasing the scale of the encoder network improves performance for DQN agents, but did not study increasing the capacity of the value network. Various studies indicate that naive scaling of the value model leads to performance deterioration (Bjorck et al.) 202T]] Obando-Cenrr Farebrother et al. 2024). For example, Bjorck et al. (202 demonstrated that spectral normalization enables stable training with relatively large ResNets with performance improvements.
et al. 2024
In addition to model size scaling, the community has investigated the effectiveness of replay ratio scaling (i.e., increasing the number gradient steps for every environment step) (Hiraoka et al. 202T Nikishin et al. 2022 Li al. 2022). Recent works have shown that a high replay ratio can improve performance across various algorithms in both continuous and discrete MDPs, provided the neural
networks are regularized (Li et al. 2022 D'Oro et al. 2022). In this context, layer normalization

---

and full-parameer parameter resets have been particularly effective (Schwarzer et al. 2023 Lyle et al. 2024 Nauman al. 2024).
Limitations and Future Work
BRO's larger model size compared traditional baselines like SAC TD3 results in higher memory requirements, potentially posing challenges for real-time inference in fighffreunnccy control tasks. Future research could explore techniques such quantization distillation to improve inference speed. While BRO designed for continuous control problems, its effectiveness in discrete settings remains unexplored. Further investigation is needed to assess the applicability and performance of BRO's components in discrete action MDPs. Additionally, our experimentation primarily focuses on continuous control tasks using proprioceptive state representations. Future research is needed to investigate the tradeoff between scaling the critic and the state encoder in image-based RL.
Conclusions
Our study underscores the efficacy of scaling a regularized critic model in conjunction with existing algorithmic enhancements, resulting in sample-efficient methods for continuous-action RL. The proposed BRO algorithm achieves markedly superior performance within million environment steps compared to the state-of-the-art model-based based TD-MPC2 and other model-free baselines. Notably, it achieves over 90% success rates World and MyoSuite benchmarks, as well as over 85% of maximal returns in the DeepMind Control Suite, and near-optimal policies in the challenging Dog and Humanoid locomotion tasks. While some tasks remain unsolved within million environment steps, our findings underscore the need for new standardized benchmarks focusing on sample efficiency to drive consistent progress in the field. The BRO algorithm establishes a new standard for sample efficiency, providing solid foundation for future research to build upon and develop more robust RL. algorithms.
Acknowledgements
This research was partially supported by the National Science Centre, Poland, under grant nos. 2020/39/BSST6/1511 and 2023/51/D/ST6/01 as well as by the Warsaw University of Technology and the University of Warsaw through the Excellence Initiative: Research University (IDUB) program. We also gratefully acknowledge the Polish high-performance computing infrastructure, Pl.Grid (HPC Center: ACK Cyfronet AGH), for providing computational resources and support under grant no. PLG/2024/017155 Marek Cygan was partially supported by an NCBiR grant POIR.0 A.......0... 11.1..0.. 01.01- -01-00- 0433/20. Piotr Mitoć research was supported by the National Science Center (Poland) grant number 2019/35/0/ST6/03464
References
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances neural information processing systems, 34:29304-29320, 2021.
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L.,, Geist, M., Pietquin, O., Michalski, M., et al. What matters in on-policy reinforcement learning? a large- scale empirical study. In ICLR 202t-Ninth International Conference on Learning Representations, 2021.
Ba, J. L.,, Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv: arXiv:1607.0644 1607.06400, 2016.
Ball, P. Smith, L., Kostrikov, L., and Levine, Efficient online reinforcement learning with offline data. Proceedings of the 40th International Conference on Machine Learning, 2023.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
10

---

Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In International conference on machine learning, pp. 449 458. PMLR, 2017.
Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., and Peters, J. Cross Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2023.
Bjorek, N., Gomes, C. P., and Weinberger, Q. Towards deeper deep reinforcement learning with spectral normalization, Advances in neural information processing systems, 34:8242-8255 2021.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.
Caggiano, V., Wang, H., Durandau, G., Sartori, M., and Kumar, V. Myosuite-a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600, 2022.
Cetin, E. and Celiktutan, O. Learning pessimism for reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 6971-6979, 2023.
Chen, R. Y., Sidor, S., Abbeel, P., and Schulman, J. Uch exploration via q-ensembles. arXiv preprint 17Xivv/706.0/11 502, 2017.
Ciosek, K. and Whiteson, S. Expected policy gradients for reinforcement learning. Journal of Machine Learning Research, 21(2020), 2020.
Ciosek, K., Vuong, Q., Loftin, R., and Hofmann, K. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Dabney, W., Ostrovski, G., Silver, D., and Munos, R. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pp. 1096-1105. PMLR, 2018.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, Bert: Pre-training of deep bidirectional trans- formers for language understanding. North American Chapter of the Association for Computational Linguistics, 2019, doi: 10. 10.18653/11/111-44222
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. Sample- efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference Learning Representations, 2022.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. Yu, T., Huang. W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. Palm-e: An embodied multimodal language model. arXiv preprint arXiv: 2303.03378, 2023.
Farebrother, J.. Orbay, J., Vuong, Q., Taïga, A. A.. Chebotar, Y., Xiao, T., Irpan, A.. Levine, S., Castro, S., Faust, A., et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2443..08. 03950, 2024.
François-Lavet, V., Fonteneau, R., and Ernst, D. How to discount deep reinforcement learning: Towards new dynamic strategies. arXiv preprint arXiv: 1,
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint trXiv:2004.072199 2020.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR. 2018.
11

---

Haarnoja, T., Zhou, A., Hartikainen, K. Tucker, G., Ha, S., Tan, J.. Kumar, V., Zhu, H., Gupta, A..
Abbeel, P. al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905. 2018.
Hafner, D., Pasukonis, J., Ba. J., and Lillicrap, T. Mastering diverse domains through world models.
arXiv preprint wtx.iii3311.00000 2023.
Hansen, N., Su, H., and Wang, X. Td-mpe2: Scalable, robust world models for continuous control.
arXiv preprini arviv: 2310.16828, 2023.
Hiraoka, T., Imagawa, T., Hashimoto, T., Onishi, T., and Tsuruoka. Y. Dropout q-functions for doubly efficient reinforcement learning. In International Conference on Learning Representations, 2021.
Hussing, M., Voelcker, C., Gilitschenski, I., Farahmand, A.-m., and Faton, E. Dissecting deep with high update ratios: Combatting value overestimation and divergence. arXiv preprint arXiv:2403.0599 2024.
Imani, E. and White, M. Improving regression performance with distributional losses. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 2157-2166. PMLR, 10-15 Jul 2018. URL htt.hhttts//prcceediinn .mr...ss........m...m......mmm.m mlr wwwps/wvsssssee.p 1110880/mmm/mm 18a. hhmml
Kearns, M. and Singh, S. Bias-variance error bounds for temporal difference updates. In Annual Conference Computational Learning Theory, 2000. URL https::/api. http://ppp..emaniccccccooooll org//orrpusI::5053875
Kostrikov, I. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021. URL http://ggttuuu...momeeeeee
Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2022.
Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning diverse multi- task data both scales and generalizes, In The Eleventh International Conference on Learning Representations, 2023.
Lee, H., Cho, H., Kim, H., Kim, D., Min, D., Choo, J., and Lyle, C. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In Forty-first International Conference on Machine Learning. 2024.
Li, Q., Kumar, A., Kostrikov, I., and Levine, S. Efficient deep reinforcement learning requires regulating overfitting. In The Eleventh International Conference Learning Representations, 2022.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. International Conference on Learning Representations, 2017.
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv: 2402.18762. 2024.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. International Conference on Learning Representatiinss, 2018.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Ried- miller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-5333 2015.
Moskovitz, T., Parker-Holdes J.. Pacchiano, A., Arbel, M., and Jordan, M. Tactical optimism and pessimism for deep reinforcement learning, Advances in Neural Information Processing Systems, 2021.
Nauman, and Cygan, M. On the theory of awske agents: Bridging actor-critic and economics. In JCML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2023a.
12

---

Nauman, M. and Cygan, M. Decoupled actor-critic. arXiv preprint arXiv:2310.1955222 2023b. URL https://arxiv.. wwwpxxi..orgppa//23332 /2310 19527v3
Nauman, M., Bortkiewicz, M., Milos, P., Trzcinski, T., Ostaszewski, M., and Cygan, M. Over- estimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learn- ing. In Proceedings of the st International Conference on Machine Learning, 2024. URL org/ påf 2203. 00514 PMLR 235:37342-37364.
Nikishin, E., Schwarzer, M., D'Oro, P.. Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Obando Ceron, J., Bellemare, M., and Castro, S. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Obando- J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J., Dziugaite, G. K., Precup, D., and Castro, P. S. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprini arXiv:2402.08669, 2024.
Padalkar, A., Pooley, A., Jain, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Singh, A., Brohan, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv 2023.
preprint ar Xrvi:233000000
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. URL htR:http://// http::/jmmrr ptg/ppapers/v222200 20- htm4.
Schubert, I., Zhang, J., Bruce, J., Bechtle, S., Parisotto, E., Riedmiller, M., Springenberg, J. T., Byravan, A., Hasenclever, L., and Heess, N. A generalist dynamics model for control. arXiv preprint arXiv: 2305. 10912, 2023.
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. Data-efficient rein- forcement learning with self- ceff-predictive representations. In International Conference on Learning Representations, 2020.
Schwarzer, M., Ceron, S. O.. Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. Bigger, better, faster: Human-level atari with hevan-level efficiency. In International Conference on Machine Learning, pp. 30365-30380 PMLR, 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018
Tai, J. J., Towers, M., and Tower, Shimmy: Gymnasium and PettingZoo Wrappers for Commonly wwwimmm/ssmmmmmmmmmmmmmmmmmmmmmm
Used Environments. URL
Taiga, A. A., Agarwal, R., Farebrother, J., Courville, A., and Bellemare, M. G. Investigating multi-task pretraining and generalization in reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. tttt:tttt:///peereee net/forumm?id sSt9fROSZRO
Tan, M. and Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, 2019.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki. A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint 1rXiv: 0009900 2018.
Van Seijen, H., Van Hasselt, H.. Whiteson, S., and Wiering, M. A theoretical and empirical analysis of expected sarsa. In 2009 ieee symposium on adaptive dynamic programming and reinforcement learning, pp. 177-184. IEEE, 2009.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30. 2017.
13

---

Wang, S., Liu, S., Ye, W. You, J., and Gao, Y. Efficientzero v2: Mastering discrete and continuous control with limited data. arXiv preprint arXiv:2403.00564, 2024.
Wang, Y. Wang, R., Du, S. S., and Krishnamurthy, A. Optimism in reinforcement learning with gen- eralized linear function approximation. In International Conference Learning Representations, 2020.
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang. H., Lan, Y., Wang, L.,, and Liu, T.-Y. On layer normalization in the transformer architecture. International Conference on Machine Learning, 2020.
Yu, T. Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1 100. PMLR, 2020.
Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, Wohlhart, P., Welker, S., Wahid, A., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165-2183, PMLR, 2023.
14

---

Broader Impact
The work presented in this study, while academic and based simulated benchmarks, advances the development of more capable autonomous agents. Although our contributions do not directly cause any negative societal impacts, we encourage the community to remain mindful of such potential consequences when extending our research.
A Background
We consider an infinite-horizon Markov Decision Process (MDP) (Puterman 2014) which described with tuple (S, A,r, rr,,, p, 7), where states S and actions A are continuous, r(s', 8, 8, is the transition reward, p(8 a) is the transition kernel and E (0, 1] is the discount factor.
The policy π (a)s) is state-conditioned action distribution with its entropy denoted H(((s)), Soft Value (Haarnoj et al., 2018) is the sum of expected discounted return and state entropies from given state
following the policy
V" (s) s' [r(s', s, a) + aH((T())) + +s)]
(1)
with a denoting the entropy temperature parameter. Q-value is the expected discounted return from performing an action and following the policy thereafter.
Q a) Es-p [r(s', s, + >VT(ss))
(2)
A policy is said to be optimal if it maximizes the expected value of the possible starting states so- such that i arg ma Exprop V (so).. with क denoting the optimal policy and II denoting the considered set of policies (c.g., Gaussian). Soft values and soft Q-values are related through the following equation:
(3) This relation is often approximated via single action sampled according to the policy ~ π (s).. In off-policy actor-critic, there continuous gradient-based learning of both Q-values (critic) and the policy (actor). The critic parameters & are updated by minimizing SARSA temporal-difference transitions 7 = (s, a, r, s'), with T being sampled from a replay buffer transitions (Fujimoto et al. Haarnoja et al. 2018)) according to:
V*(s) Eana (Q" a) log [a]ss)
2018
0 arg min ET~D (Qo(s, s, ~V((s),
0
(5) with a' ~ Tg. In this setup, Qo is the critic network. Q8 is the value target network, and D is the replay buffer (Mnih et al. 20115). Qp is trained to approximate the Q-value under the policy from which the bootstrap is sampled (Van Seijen et al. 2009 Sutton & Barto 2018). The policy parameters are updated to seek locally optimal values approximated by the critic (Ciosek & Whiteson 2020) according to:
V*(s) = Qá(s', a') - a log (a'ls)))
arg max EAED (Qo(s, a) - a log (alss)) with a a Tg.
=
B
BRO additional details
B.1
Base agent
BRO uses the well-established Soft Actor-Critic (SAC) (Haarnoja et al.) 2018) as its base. SAC is stochastic policy, maximum entropy algorithm (see Eq. that employs online entropy temperature adjustment and an ensemble of two critic networks. SAC models the policy via a Tanh-transformed Gaussian distribution whose parameters modeled by the actor network.
B.2 Architecture details
In the proposed architecture, we adopt the transformer feedforward blueprint from Vaswani al.) (2017) with novel layer normalization configuration, Dense layers in the BRO have a default width of 512 units, with a linear layer at both the input and output stages. To increase the model's 's depth, we add new residual blocks exclusively. While similar transformer backbone has been used previous work (Bjorck et al.) (02T), our design choices, detailed in Section led us to use layer normalization instead of spectral normalization.
shown in Figure [ Dropout is omitted.
15

---

B.3 Scaling
In Figure we examine the scaling capabiliitiss of SAC and BRO agents using a vanilla dense network (Fujimoto et al. [0188] Haarnoja et al. 2018 Moskovitz et al. 2021). ) a spectral normalization ResNet (Bjorek et al. [2021] Cetin & Celiktutan 2023). and layer normalization ResNet inspired by previous work (Nauman et al. 2024). As shown in Figure 4 increasing the model capacity of a vanilla-dense agent can lead to performance degradation beyond a certain model size. However, for the regularized architectures, we observe behavior similar to the empirically observed scaling properties of supervised models, where increasing model size leads to diminishing returns in performance improvements. Furthermore, we find that the layer normalization ResNet achieves better scaling properties than the spectral normalization architecture. Interestingly, the BRO agent consistently outperforms the baseline SAC across all architectures and network sizes, suggesting an interaction between parameter scaling and other algorithmic design choices. The highest performing SAC agent achieves around 25% of maximal performance, whereas our proposed BRO agent achieves more than 90%. Given that the BRO agent performs similarly at 4.92 million and 26.31 million parameters, we use the smaller model reduce the computational burden.
B.4 Scaling replay ratio
Replay Ratio (RR) scaling in small models leads diminishing performance increases as RR values rise, eventually plateauing at higher RRs (Nikishin et al. [022 D et al. (022). Unfortunately, increasing RR also results linear increases in computing costs, as each gradient step must be calculated sequentially. This naturally becomes burden as the model sizes increase. In Figure 6 we investigate the performance of the BRO agent across different model scales (from 0.55 million to 26.31 million parameters) and RR settings (from RR=1 to RR=15), measuring both performance and wall-clock efficiency. We find that with the BRO regularized critic architecture, critic scaling leads to performance and sample efficiency gains that match those of scaled RR. Scaling both RR and model size produces the best- best-performing agents. Interestingly, scaling the model size can lead to if RR is low while being more computationally efficient
significant performance improvements due to parallelized workloads (Figure (). For example, a million parameter BRO model with RR=1 outperforms 1 million parameter BRO agent with RR=15 despite being five times faster in terms of wall-clock time. This observation challenges the notion that a sample-effciien RI. algorithm must use high replay settings.
B.5 Batch Size
Inspired by recent findings that reducing the batch size can result significant performance gains for discrete action RL (Obando Ceron et al., 2024), we reduce the number of transitions used for gradient calculation from 256 to 128. As shown in Figures Pa & 15] this batch size reduction leads to marginal improvement in aggregate performance and decreased memory requirements of the algorithm. Interestingly, we find that batch size significantly affects performance, with no single value performing best across all tasks.
B.6 Risk-Neutral Temporal Difference
Using a pessimistic lower-bound Q-value approximation for actor-critic updates, known as Clipped Double Q-learning (CDQ) (Fujimoto et al., 2018 Haarnoja et al. 2018), is popular method to counteract Q-value overestimation, though it introduces bias. Formally, ly, modifies the value estimate in Eq. 5 to lower-bound estimation
(s) RJ Q% a log (als), (s).
where Q'*((s, a) is a lower-bound Q-value estimation derived from a critic ensemble, often using two networks (Fujimoto al. 2018 Haarnoja et al. 2018
Q% = min(Qu a), (s..
(8)
Recent studies have shown that techniques like layer normalization or full-parameter resets can be more effective at combating overestimation than pessimistic Q-value approximation (Nauman al. 2024). Since our critic architecture leverages multiple regularization techniques, we disable CDQ and use the ensemble mean instead of the minimum to calculate the targets for actor and critic
16

---

updates. In Figures & 9a we compare the performance of the baseline BRO to BRO agent that uses CDQ. Our hindins indicate that using risk-neutral Q-value upproximation in the presence of network regularization unlocks significant performance improvements without increasing value overestimation.
B.7 Optimistic Exploration
Optimism is algorithmic design principle that balances exploration and exploitation (Ciosek et al. 2019 Moskovitz et al. (02T). The dual actor setup (Nauman & Cygan 2023alb) employs separate policies for exploration and temporal difference updates, with the exploration policy pursuing an optimistic upper-boure Q-value approximation. This approach has been shown to improve the sample efficiency of SAC-based agents (Nauman & Cygan 2023a). We implement the optimistic policy such that the Q-value upper bound is calculated based on the epistemic uncertainty estimated via quantile critic ensemble (Moskovitz et al. (021). Figure 7 shows that using dual policy setup leads to performance improvements. We observe that these results are particularly pronounced the early training stages and for smaller networks (Figure 9a).
Algorithm 2 BRO training step with changes with respect to standard SAC colored red.
1: Input: T pessimistic actor; n. optimistic actor; Q² - kth quantile critic; Q kth quantile of ith target critic; a temperature; 30 optimism; r - KI. weight;
2: Hyperparameters: K.L target KI; K number of quantiles
3: Sample action from the optimistic actor
,',-= (TV.STEP(a) a e Ty
4: Add transition the replay buffer
BUFFER.ADD(Ss a, r')
5: for 1 to ReplayRatio do
6: Sample batch of transitions
8, s, BUFFER .SAMPEE
7: Calculate critic target value without CDQ
(5')s = 2 ( 0 (s',a')
2 with d' d')))((5))
8: Update critic using pessimistic actor
0 9 - Hbe Qo(ss a), (r + (s'.a')) a log lo''s(('(ss))
9: Calculate pessimistic actor value without CDQ
( 0, 7 ZR EK= (8, ())(),a) with az))
10:
Update pessimistic actor
Th²) - a log (a((s))) Calculate optimistic actor value
@ + & + &
11:
(%, (9) IK (t (s) (1 -/0
(b,(s,))), a =((s)
12:
Update optimistic actor
Ti 17 V,,Q
3 -
13:
Update eniropy lemperature
a ata a vV-Va((7 H(s))
14:
Update optimism
8 39+B* 30 -- 3P) KL KC
15:
Update KL weight
T V + K -KC")
Update target network
B - POOLYK((0
17: end for
B.8 Approaches Examined During the Development of BRO
Examined approaches are listed in Table 2 Methods incorporated into BRO include regulariza- tion techniques (LayerNorm, Weight Decay, removing CDQL), optimistic exploration, quantile distributional RL, resets and increased replay ratio.
17

---

Table 2: Approaches examined during BRO development. Methods incorporated into BRO are
highlighted in bold.
Methods Group
Specific Method
Source
Exploration
DAC
Nauman & Cygan
2023a
CDQL (removed)
Fujimoto al.
2018
Value Regularization
N-Step Returns
Sutton & Barto
2018
LayerNorm
Ba ct al.
2016)
Network Regularization
Weight Decay
Loshchilov & Hutter 2017)
Spectral Norm
Miyato et al.
2018
N-Step Schedule
Kearns & Singh 2000)
Discount Schedule
François Lavet et al.
2015
Scheduling
Pessimism Schedule
Entropy Schedule
Learning Rate Schedule
Andrychowicz et al.
2021
HI. Gauss
Imani & White
2018
Distributional RL
Categorical
Bellemare et al. 2017
Quantile
Dabney et al.
2018)
Plasticity Regularization
Resets
Nikishin et al.
2022
D'Oro et al.
2022
Learning
Replay Ratio
Nikishin et al.
2022
D D'Oro et al.
2022
C Tested Environments
We tested BRO on variety of 40 tasks from DeepMind Control Suite TTssa et al. 2018), MyoSuite (Caggiano et al. 2022) and MetaW (Yu et al. 2020). Selected tasks cover various challenges, from simple to hard, in locomotion and manipulation. Table presents the environments with specified dimensions of states and actions. BRO is a versatile agent that can successfully perform tasks of different difficulty and various action and state spaces. Our selection of 40 tasks focuses on the most challenging tasks from the Deep.Mind Control Suite (DMC), Meta World (MW), and MyoSuite (MS) benchmarks, identified in previous studies (Hansen et al. (023). We chose these hard tasks because many easy tasks from these benchmarks can be solved by modern algorithms within 100k environment steps (Hansen al. 2023 Wang et al. (0024). In the Meta World environment, we follow the TD-MPC2 evaluation protocol (Hansen et al. 2023). As such, the environment issues truncate signal after 200 environment steps, after which we assess if the agent achieved goal success within the 200th step. We do not implement any changes to how goals are defined in the original MetaWorld and we use V2 environments.
Table 3: List of tasks from DeepMind Control and MetaWorld on which the agents were ablated. The table also contains the dimensions of action and observation space.
Task
Observation dimension
Action dimension
DEEPMIND CONTROL
Acrobot-Swingup
6
Dog-Trot Trot
223
38
Hopper-Hop Hop
15
4
Humanoid-Run
67
24
Humanoid- Walk
67
24
METAWORLD
Assembly
39
4
Coffee-Push
39
4
Hand-Insert
39
4
Push
39
4
Stick-Pull
39
4
18

---

Table List of tasks from Deep Control, MetaWorld, and MyoSuite on which the agents were tested. The table also contains the dimensions of action and observation space.
Task
Observation dimension
Action dimension
DEEPMIND CONTROL
Acrobot-Swinguw
6
1
Cheetah-Run
17
6
Dog-Run
223
38
Dog-Trot
223
38
Dog-Stand
223
38
Dog- Walk
223
38
Finger- Turn-Hard
12
2
Fish-Swim
24
5
Hopper-Hop
15
4
Humanoid- Run
67
24
Humanoid-Staen Stand
67
24
Humanoid- -alk
67
24
Sondulum-Swingee
3
1
Quadruped-Run
78
12
Walker-Rum
24
6
METAWORLD
Basketball
39
4
Assembly
39
4
Button-Press
39
4
Coffee-Pull
39
4
Coffee-Push
39
4
Disassemble
39
4
Hammer
39
4
Hand-Insert
39
4
Push
39
4
Reach
39
4
Stick-Pull
39
4
Sweep
39
4
Lever-Pull
39
4
Pick-Place
39
4
Push-Back
39
4
MYOSUITE
Key-Turn
93
39
Key-Turn-Hard
93
39
Obj-Hold
91
39
Obj-Hold-Hart
91
39
Pen-Twirl
83
39
Pen-Twiri-Hard
83
39
Pose
108
39
Hard
108
39
Reach
115
39
Reach-Hard
115
39
D Hyperparameters
BRO (Fast) shares the same parameters BRO except replay ratio 2 which significantly speeds the algorithm without sacrificing performance that much. BRO features the BRONet architecture and resets all parameters done every 250k steps until 1M steps with additional resets steps 15k and 50k. The selection of
Hyperparameters BRO and other baselines are listed Table 5
19

---

hyperparameters for BRO was based on the values reported in the main building blocks of BRO and extensive experimentation coupled with ablations studies.
Table 5: Hyperparameter values for actor-critic agents used in the experiments.
Parameter
BRO
SAC
TD3
SR-SAC
CrossQ
Batch size (B)
128
256
Replay ratio
10
2
32
Critic hidden depth
RESIDUAL
2
2
2
Critic hidden size
512
256
2048
Actor depth
RESIDUAL
2
2
2
Actor size
256
Num quantiles
100
N/A
KL target
0.05
N/A
Initial optimism
1.0
N/A
Std multiplier
0.75
N/A
Actor learning rate
3e-4
1e-3
Critic learning rate
3e-4
le-3
Temperature learning rate
3e-4
N/A
3e-4
Optimizer
ADAMW
ADAM
Discount (7)
0.99
Initial temperature (oo)
1.0
N/A
1.0
Exploratory steps
2,500
10,000
25,000
10,000
5,000
Target entropy (H-"
|AI/2
N/A
|A]/2
[AI
Polyak weight (T)
0.005
N/A
We compare BRO against official and widely used implementations of CrossQ, SAC, SR-SAC, TD3 and TD-MPC2 with open source repositories listed in Table As the official results do not cover all 40 benchmarking tasks, we ran the baselines independently (except TD-MPC2, where all official results were available). SAC and TD3 are commonly used baselines; therefore, their hyperparameters vary across different implementations. To account for this fact, we ran 2 versions of these baselines: tuned and original. If not specified otherwise,
report the results of the tuned versions with hyperparameters in Table [] The original versions of SAC and TD3 both feature a replay ratio of 1 and in the case of SAC, target entropy (H") equal to the action space dimension (A]. The performance of both variants the implementations can be observed in Figure 24
Table 6: Links to the repositories of the used baselines. All are distributed under MIT license.
Baseline
Source code link
CrossQ (Bhatt et al. 2023
github. wwtp//ittta///aaa/cccc
Cross
SAC (Haarnoja et al. 2118
github. CcChhu/..coo/deeeeeeaaaaa/tooooo
SAC (Tuned Version)
github.com Btchub.ccom/kkksttikko///rsrt
SR-SAC ODoOro al.
2022
TD3 (Fujimoto et al.
2018
github.com/ /ttmub..com//ffuim/1333
TD3 (tuned version)
"tchub.com/1kostrikkor//
TD-MPC2 (Hansen et al. 2023
sttabb.com/niccllaccaaaaaa tdmmp22
As other baselines were developed and tested on only subset of our 40 selected tasks, we observed that achieving similar performance on new tasks was challenging. This can especially observed in the case of CrossQ, which is state-of-the-art algorithm selected tasks from OpenAl Gym
20

---

(Brockman et al. 2016), but as it was tested only on fraction of DeepMind Control Suite tasks, its performance does not transfer to our selection of tasks. Originally, CrossQ authors tested their agent on DeepMind Control Suite using Shimmy (Tai et al.) contrary to other agents that use the original codebase (Tassa et al. 2018). We run CrossQ using the DMC wrappers (D'Oro et al. 2022). Comparison between 2 variants of SAC and TD3 (Original and Tuned) is presented in Figure 24
Tuned versions feature higher value replay ratio (2 instead of 1) than the original and lower target entropy in the case of SAC (|A41/ instead of [A].
Table 7: Description of the considered model sizes.
Size
Number of block
Hidden Size
0.55M
1 BroNet block
hidden size of 128 hidden size of 256
1.05.M
1 BroNet block
2.83M
BroNet block
hidden size of 512
4.92M
2 BroNet blocks
hidden size of 512
26.31
BroNet blocks
hidden size of 1024
E
Additional Experiments
E.1
Scaling and Time Experiments
The exccution time was measured for all agents for each of the 40 tasks averaged over random seeds. We ran each agent for 105k steps with initial 5k exploration steps, 100k training steps, and 1 evaluation. We benehmark all 25 variants of BRO with 5 different model sizes and 5 values of replay ratio. Figure 12 different algorithms performance compared to execution time. Experiments were conducted on an NVIDIA A00 GPU with 10GB of RAM and 8 CPU cores of AMD EPYC 7742 processor. All tasks were run separately so the agents could use all resources independently.
Wallclock time and performance of agents
Patto
1004 performance
TT-MEC2
0.8
0.4
1B0
0.0
0.0
02
5.4
0.6
0.5
1.0
Normalized execution time
Figure 12: Scatterplot of the performance of agents plotted against normalized execution time.
Increasing the model size and replay ratio both improve the performance. However, the former is more efficient in terms of execution time due to GPU parallelism. For example, the largest BRO variant (26.31 M parameters) with replay ratio has similar execution time as the smallest one (0.55M parameters) with replay ratio 15, but the performance is much greater (Figures (3).
E.2 Additional BroNet Analysis
We examine various architectural blueprints on DMC and 5 MetaWorld environments (see Table 3 each with over 10 seeds per task. Our starting point was the transformer-bast based design by Bjorck et al. (202I), termed Spectral. This architecture incorporates recent transformer advancements, moving Layer Norm to the beginning of the residual block to prevent vanishing gradients in deep networks. While Spectral performs better than the vanilla MLP, its performance on the DMC benchmark, particularly the Dog environment, is weaker. This aligns with findings from Nauman et al. (2024); Hussing et al. (2024), indicating that Layer Norm is crucial for stability in such complex tasks. To analyze the importance of layer normalization in the BroNet architecture, we replaced spectral norms with Layer Norms in the residual blocks, resulting the BRO wo first LN architecture (Figure (5) This modification improves performance but still lags behind the full BRO architecture. Furthermore.
21

---

Mean oxcecioos lise scross 40 tasks
10M Performaanio acoous looko
24.1
29 30M
402M
4.0MM
Notalasa
283M
2.60M
1,9M
1.00
233M
2.000
10
Raplay radoo
Raplay ma3l
Figure 13: Heatmaps of execution time and IQM performance across 40 tasks of 25 BRO variants with various model sizes and replay ratio values. Black lines connect the same interpolated values.
examine simple MLP architecture with Layer Norm before each activation function. Since BRO consists two residual blocks, we compare it with 5-layer model, (Dense + LN) x 5. Figure [4] shows that Layer Norm after each Dense layer is effective, and in aggregated IQM, this model comparable to BRO. However, skip connections in BRO are beneficial for managing complex environments like Dog. In conclusion, BroNet architecture uses Layer Norm and residual blocks for superior robustness and performance in challenging tasks.
0.8
0.6
1QM
0.4
0.2
0.0 BRO
Spectral
BRO wo fust LN
(Dense XN)
Figure 14: Comparison of five architecture designs across different environments: The top plot shows results DMC and 5 MetaWorld environments, the middle plot focuses on the 5 DMC environ- ments, and the bottom plot highlights the Dog Trot environment. BRO and Spectral architectures each consist of residual blocks. (Dense LN) hidden layers, each incorporating Layer Norm before activation. Lastly, BRO wo first LN refers to the BRO architecture without Layer Norm in the first Dense block, before the residual connection.
represents standard MLP networks with
E.3
Additional analysis
We ablate of the minibatch size impact on BRO and BRO (Fast) performance across different benchmarks is depicted in Figure 15 The figure shows that using half or even a quarter of the original minibatch size (256) does not significantly hurt BRO's performance.
Batch sizes
BRO
BRO-Fast
1.0
1.0
0.8
0.8
10M
0.4
0.4
64
64
128
128
0.2
0.2
256
256
Total
D&H
DMC
MW
MS
Total
D&H
DMC
MW
MS
Figure 15: Performance of BRO and BRO (Fast) with different minibatch sizes for: D&H (Dogs and Humanoid), DMC (DeepMind Control), MW (MetaWorld), and MS (MyoSuite).
22

---

Target network We investigate the performance benefits stemming from using a target network with the BRO agent. We present these results in Figure 16] Interestingly, we observe that the target network yields performance improvements only in specific environments.
1.0
0.8
0.6
0.4
0.2
BRD
No Target Bro
Fisper Tom
Quatroped pustropet Rim
Cleveih Roo
Himardd Stand
Walker Rast
Falh So
Hepper Heep
Htundde Shropey
Dug Stane Dee Walk Dup frot Dhig Aioo
Pash Back
Post
Dution Hascovrr
Pisk Place Nill
Purl
Culfe Pael
Pull fins finkeahul! Sutt
Asumadl
Ofjecct Hold
Raiei
Key Turn
Paus Piet Tpvee
OPjor Haadi Food Pase Hard
Hard Foolt Hard
Sortige
Pooddao Humamed
Hes
Figure 16: We compare BRO against BRO without target network. 10 seeds per task, 1M steps.
More baselines To evaluate BRO performance beyond maximum entropy objectives, we tested BRO and BRO (Fast) with TD3 backbone. BRO with SAC backbone slightly outperformed TD3, though TD3 remains a viable option. Furthermore, we compare BRO performance to three additional baselines. These results are presented in Figure 17
15 DeepMind Control Tasks
Dog and Humanoid
1.0
1.0
BRU
0.8
0.8
BR0 (Faet)
BROTDT
0.6
0.6
(Fis)
SMI
0.4
0.4
TD?
0.2
0.2
01.25
0.50
0.75
1.00
0
02.25
0.50
01225
1.00
Environmennt Step (M)
Environment Step (M)
Figure 17: We rum BRO and BRO (Fast) with TD3 backbone (BROTD3). 10 seeds.
Longer training We expanded BRO training beyond IM environment steps, although in single- task setup. We trained BRO and BRO (Fast) for 3M and 5M steps respectively on 7 Dog and Humanoid tasks and compared them to TD-MPC2 and SR-SAC. BRO significantly outperforms these baselines and notably almost solves the Dog Run tasks at 5M steps (achieving over 80% of possible returns). We show the 3M results in Figure 18
Dog Rus
Dog Stard
Dog Troe
Dog Walk
2..
1903 150
NES
358
300
1300
150
140
103
100 178
1900 1000 303 100 305
Hunazzid
Huranold Stndd
Hunasoid Walk.
Humanood
1.5
1.5
11.0
300 100 2000
200 2006 Skp
200
130 1500
280
300 200 20
3000
200 180 1200 200 2000 200 Stip (24)
2005
Figure 18: We run BRO on complex tasks for 3M steps. 5 seeds.
Image-based tasks To analyze the impact of BroNet on image-based RL tasks, we experiment with 3 tasks from the Atari 100k (Bellemare et al. 2013) benchmark. Here, we changed the regular
23

---

Q-neworrk of the SR-SPR (RR=2) model (D et al. 2022) to BroNet, and considered changing the reset schedules to better fit the plasticity of the BroNet model. As depicted in the Table below, applying BroNet to discrete, image-hased tasks is promising avenue for future research.
Table 8: We replace the Q-network in SR-SPR with BroNet, while keeping the standard convolutional encoder. We test two values of reset interval (RI) and shrink-and-perturd (SP) and find that these hyperparameters impact the performance of the BroNet agent. 5 seeds.
Task
SR-SPR
SR-SPR-BroNet (RI=20k;SP=0.8)
SR-SPR-BroNet (R==20k;SP=0.5)
SR-SPR-BroNet (RI=5k;SP=0.8)
Pong
-10.5
4.8
10.2
-12.0
Seaquest
714.7
399.0
420.7
782.0
Breakout
24.5
13.5
13.7
33.3
Multi-Task RI Finally, we evaluate whether the increased model size improves the agent's ca- pability in multi-task setup Yuu et al., 2020, Hansen et al. 2023). To this end, we compare the BRO (Fast) SAC on the MT550 benchmark from MetaWorld which comprises 50 different tasks. To accommodate the task diversity, vector representing the task identifier in the network input. Otherwise, both algorithms are run with the same hyperparameters as the main experiments. We present the results in Figure 19
increase the width of all models twofold and pass one-hot
MetaWorld MT50
0.5
0.4
0.3
SAC (Fest)
0.0
125
250
375
500
Environment Step (Thousand)
Figure 19: We compare BRO (Fast) with SAC on the multi-task benchmark. 3 seeds
F Training Curves
We present the aggregated performance of BRO compared to other baselines at Dog & Humanoid tasks, DeepMind Control Suite, Metaworld and MyoSuite in Figure togoteeerr with summarized performance results at 100k, 200k, 500k and 1M steps in Table The performance on each of the
40 individual tasks in shown in Figures 21
23 22
BRO
BRO (Fast)
TD-MPC2
SR-SAC
SAC
TD3
CrossQ
Humaroid
DeepMind Contol
MenaWodk
MycSutte
204
122
1000
2.15
170
12:35
231
1.11
170
0.5
335
12.55
180
1400
Figure 20: BRO aggregated performance over 1M steps on 40 tasks from DeepMind Control Suite, World and MyoSuite. Y-axis represents the IQM of normalized rewards.
24

---

BRO (MF)
TD-MPC2 (MB)
SR-SAC (MF)
SAC (MF)
TD3 (MF)
CrossQ (MF)
36B
Eeigne Stat) Soand
Euppert Aup
M
1.9
160
3.09
Figure 21: Results tasks from DeepMind Control Suite for BRO and other baselines run for 1M steps. We present the IQM of rewards and 95% confidence intervals.
BRO (MF)
TD MPC2 (MB)
SR SAC (MF)
SAC (MF)
TD3 (MF)
CrossQ (MF)
0.2
140
1.99
Grazauz2der22
140
Figure 22: Results of 15 tasks from Meta World for BRO and other baselines run 1M steps. We present the IQM of success rate and 95% confidence intervals.
25

---

BRO (MF)
TD-MPC2 (MB)
SR-SAC (MF)
SAC (MP)
TD3 (MF)
CrossQ (MF)
145
205
100
2e6
260
10' 2001
1007
Spp.ou Eare
166
106.
Figure 23: Results of 10 tasks from MyoSuite for BRO and other baselines run for 1M steps. We present the IQM of success rate and 95% confidence intervals.
Table 9: Summary of IQM results of BRO and other agents evaluated 40 tasks from DeepMind Control Suite, Metaworld and MyoSuite achieved at 100k, 200k, 500k and 1M steps. BRO achieves better results than other saate-of-theaart agents (both model-based and model-free) while featuring great sample efficiency.
Step
BRO
BRO (Fast)
TD-MPC2
SR-SAC
SAC
TD3
Cross
AGGREGATED 40 TASKS
100k:
0.254
0.113
0.204
0.046
0.007
0.008
0.004
200k
0.560
0.384
0.519
0.083
0.043
0.054
0.009
500k
0.862
0.772
0.745
0.373
0.167
0.157
0.037
1M
0.945
0.878
0.842
0.595
0.316
0.294
0.042
DEEPMIND CONTROL SUITE
100k:
0.230
0.123
0.128
0.089
0.030
0.046
0.031
200%
0.442
0.282
0.332
0.195
0.088
0.091
0.038
500k:
0.726
0.613
0.532
0.412
0.259
0.261
0.092
1M
0.836
0.794
0.673
0.487
0.391
0.381
0.106
METAWORLD
100k:
0.247
0.113
0.452
0.046
0.000
0.000
0.000
200k
0.642
0.571
0.835
0.062
0.018
0.047
0.000
500k
0.984
0.929
0.952
0.421
0.108
0.084
0.000
1M
1.000
0.976
0.978
0.769
0.303
0.250
0.000
MYOSUITE
100k
0.392
0.124
0.088
0.000
0.000
0.000
0.020
200k
0.748
0.400
0.394
0.015
0.024
0.008
0.140
500k
0.888
0.776
0.688
0.285
0.140
0.120
0.354
1M
0.980
0.876
0.775
0.538
0.276
0.256
0.474
DOG & HUMANOID
100k:
0.038
0.008
0.014
0.007
0.006
0.013
0.011
200k
0.238
0.056
0.058
0.024
0.015
0.040
0.013
500k
0.661
0.469
0.302
0.107
0.080
0.133
0.025
1M
0.864
0.772
0.527
0.099
0.155
0.221
0.040
26

---

(Original)
SAC (Tuned)
(Original)
TDS (Tuned)
1100
1.6
direvennberr
MM
2415
165
146
100
240
143
165
340
146
145
165
160
2070-2004
140
142
165
160
Figure 24: Results of different variants of SAC and TD3 on 40 tasks.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction clearly state the claims, contributions, assumptions and limitations made in the text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the main text have Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper includes description of the experiments (Sec- and B.8), as well as code used to generate the results
tions B D
(https:/ /el///anonyy anooymouss 4open
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
share the code used generate the results beetps://anonynnnn 4open www.4..pp.ee........... wwt.eceee//eurrTTTTTeee IPS Bar_BurrrS-_-BBB--BSSE 4
We
md).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
(hollt//guidde//ddeettbnassssss casssiinnllyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttp:/cccpblccggiuiieeeee
wtteeee//ooSSSeesss..ee.. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are in the Appendix sections B Guidelines:
D and B.8
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are performed for multiple tasks, with 10 random seeds per task. We calculate 95% bootstrapped confidence intervals using RLiable package (Agarwal
et al. 2021)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report wallclock time required to run all algorithms, on a uniform compute setup described in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::t//////
Answer: [Yes]
Justification:
Our research conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include Broader Impact Section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Research presented in this paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We discuss acknowledgements in Section ?? and provide the sources of used code in alongsieee information about the license. Repositories used for benchmarking are distributed under MIT (MetaWorld) and Apache 2.0 (DeepMind Control Suite, MyoSuite) licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
32

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We discuss the reproducibility of our experiments in Sections B D
and
B.8
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do have human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "n6nTiuX6bW", "reviewer": "Reviewer_zd6E", "summary": "The paper studies how to scale up RL algorithms in the continuous domain and introduces the BRO (Bigger, Regularized, Optimistic) algorithm, designed to enhance sample efficiency with (relatively) large models. The authors conduct extensive experiments to verify the effectiveness of factors like replay ratio, regularizations, optimistic exploration, and quantile Q-values when scaling up RL algorithms. The findings from these extensive experiments lead to the novel BRO algorithm, which consists of a novel architecture with proper regularization and exploration. Empirical results demonstrate that BRO achieves state-of-the-art performance on 40 complex tasks from the DeepMind Control, MetaWorld, and MyoSuite benchmarks, outperforming leading model-based and model-free algorithms, especially in the challenging Dog and Humanoid tasks.", "questions": "See the weakness section. Is there any evidence that the proposed method can benefit from training on diverse tasks?", "limitations": "The authors discussed their limitations well in the paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper tackles an important problem of scaling up in reinforcement learning, especially in continuous action domains.\n\n2. The authors conduct extensive experiments on the effects of different methods on scaling up, which I found very informative.\n\n3. The proposed algorithm, BRO, achieves strong empirical performances on various domains, especially on the Dog & Humanoid domains.\n\n4. The paper is well-structured and well-written.", "weaknesses": "Usually, scaling up benefits more when a large amount of data is available, where large models can lead to positive transfer or generalization across different tasks. However, the current setup is the same as the standard setting, where the agent is trained for 1M steps on each task separately. The work would be more significant if the model is trained on and can be transfered across different tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback regarding our work. We are very pleased that the reviewer found our results on scaling interesting. Please find our rebuttal below. \n\n>Usually, scaling up benefits more when a large amount of data is available, where large models can lead to positive transfer or generalization across different tasks. However, the current setup is the same as the standard setting, where the agent is trained for 1M steps on each task separately. The work would be more significant if the model is trained on and can be transferred across different tasks. \n\nWe thank the reviewer for their excellent question related to generalization across different tasks. In this work, we intentionally focused on the single-task setup because it is the standard approach in previous work proposing base agents [1, 2]. Furthermore, it provides a clear and isolated setting to study the properties of RL algorithms and serves as a necessary foundational step before tackling multi-task learning. During the limited rebuttal period, we enhanced our single-task analysis by the experiments described in the single-task setup (e.g. longer training, offline, image-based, and BRO with TD3 backbone), further confirming that BRO is an attractive option for future studies. Nevertheless, conducting informative multi-task or continual learning experiments requires significantly more work, as highlighted by e.g. [3]. \n\nHowever, we commit to running simple preliminary multi-task experiments for the camera-ready version. We also added this discussion to the future work section.\n\n>Is there any evidence that the proposed method can benefit from training on diverse tasks?\n\nThere are generic arguments that suggest that BRO could perform well in a multi-task learning environment. For example, the increased network capacity combined with robust regularization techniques should theoretically decrease catastrophic forgetting. Moreover, these design choices should also help in reducing overfitting to individual tasks, thus increasing generalization across tasks. We do not have any insights into the quality of representations learned by BRO in a multi-task setup, but such research could be interesting when tackling multi-task RL. \n\nWe thank the reviewer again for their time and insights. We are happy to answer further questions if any arise. We also welcome further discussion on the inclusion of preliminary multi-task results in the camera-ready version of our paper. \n\n[1] Hessel, Matteo, et al. \"Rainbow: Combining improvements in deep reinforcement learning.\"\n\n[2] Schwarzer, Max, et al. \"Bigger, better, faster: Human-level atari with human-level efficiency.\"\n\n[3] Wolczyk, Maciej, et al. \"Disentangling transfer in continual reinforcement learning.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C7sh5UM8lR", "reviewer": "Reviewer_7YXC", "summary": "This paper investigates how reinforcement learning (RL) can benefit from parameter scaling. The authors introduce BroNet, a variant of ResNet with LayerNorm, as a well-regularized network structure for the critic that improves performance when scaled. They also find that when the critic is properly regularized, the common Clipped Double Q-learning trick can be replaced with optimistic exploration, further boosting sample efficiency. These findings are combined into a new algorithm called BRO.\n\nTo demonstrate the efficiency of their approach, the authors conduct extensive experiments on 40 challenging tasks across 3 benchmark suites. They also provide comprehensive ablation studies to justify their design choices.", "questions": "- The benefits of scaling the critic appear to saturate after 5M parameters. What are your projections for further scaling? Do you anticipate additional benefits beyond this point, or do you believe there are fundamental limitations?\n- Could you provide more details on the evaluation metrics used, particularly for the MetaWorld benchmark? Given the various reporting methods in the literature for MetaWorld success rates, it would be helpful to clarify which specific metric was employed in this study.", "limitations": "Yes, the authors have discussed the limitation in section 5.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper addresses the important and timely topic of scaling in model-free off-policy RL.\n- It offers extensive large-scale studies on various design choices, providing valuable insights to the RL community.\n- The proposed BRO algorithm shows promising results across a wide range of tasks.", "weaknesses": "As acknowledged by the authors, the study primarily focuses on state-based off-policy RL. The transferability of their conclusions to other domains such as image-based problems and offline RL remains unclear, which limits the paper's broader impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable suggestions. We are also happy that the reviewer found the insights provided by our manuscript valuable. Please share our rebuttal below:\n\n>As acknowledged by the authors, the study primarily focuses on state-based off-policy RL. The transferability of their conclusions to other domains such as image-based problems and offline RL remains unclear, which limits the paper's broader impact.\n\nWe thank the reviewer for pointing us towards these domains. As described in the joint response, we are happy to announce two major additions to our manuscript.\n\nFirstly, we add experiments on two offline RL benchmarks: AntMaze (6 tasks); and Adroit (9 tasks) [1]. We design these experiments to test the effectiveness of the naive application of BroNet to popular offline approaches. To this end, we run Behavioral Cloning (BC) in pure offline, Implicit Q-Learning (IQL) offline + fine-tuning, and SAC with RLPD buffer online with offline data setup. We run all these algorithms with the default network and BroNet backbones. We find that the naive application of BroNet leads to performance improvements across all tested algorithms.\n\nSecondly, we add experiments in discrete image-based RL benchmark Atari 100k [2]. Similarly to offline, we evaluate the application of BroNet to the popular discrete RL algorithm SR-SPR. In this experiment, we substitute the regular critic network with a BroNet (without changing the convolutional encoder on other parts of the model like learning rate). We run 5 seeds on 3 tasks and find that BroNet can improve the performance of SR-SPR, but its effectiveness is dependent on the reset configuration and thus demands future work. \n\nWe show these experimental results in the uploaded rebuttal PDF. Additionally, we added a passage to our limitations section where we state that the usefulness of BRO for offline/image-based setups should be studied further. We hope that these added experimental results are a valuable addition for readers interested in vision-based and offline control, as well as increase the reviewer’s confidence in the contribution of our manuscript. \n\n>The benefits of scaling the critic appear to saturate after 5M parameters. What are your projections for further scaling? Do you anticipate additional benefits beyond this point, or do you believe there are fundamental limitations?\n\nWe believe that performance saturation beyond 5M parameters is due to the low complexity of the environments studied. Therefore, we anticipate that larger models will be beneficial for more challenging tasks, such as real-world robotics, multi-task learning, and image-based control in varied environments. Scaling is expected to become an increasingly significant tool in reinforcement learning. However, it is not a universal solution, and complex challenges like exploration may need approaches beyond just scaled SAC or BRO. We added this remark to Section 3. \n\n>Could you provide more details on the evaluation metrics used, particularly for the MetaWorld benchmark? Given the various reporting methods in the literature for MetaWorld success rates, it would be helpful to clarify which specific metric was employed in this study.\n\nWe thank the reviewer for this suggestion. We added the following text to our manuscript detailing our evaluation method: “In the MetaWorld environment we follow the TD-MPC2 evaluation protocol. As such, the environment issues a truncate signal after 200 environment steps, after which we assess if the agent achieved goal success within the 200th step. We do not implement any changes to how goals are defined in the original MetaWorld and we use V2 environments”. We hope that this clarifies our evaluation procedure for MetaWorld tasks. Please inform us if otherwise. \n\nWe thank the reviewer again for their valuable insights and suggestions on how to improve our manuscript. We think that implementing these suggestions resulted in additional value for readers, especially those interested in applying BRO to other domains than continuous control (e.g. offline, image-based, or discrete RL). \n\n[1] Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\"\n\n[2] Kaiser, Lukasz, et al. \"Model-based reinforcement learning for atari.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3MBbqxGzA", "reviewer": "Reviewer_TNcA", "summary": "This paper investigates the sample efficiency problem in continuous control. The authors propose the BRO algorithm, i.e., Bigger, Regularized, Optimistic. The authors find that strong regularization allows for effective scaling of the critic networks, which, paired with optimistic exploration, leads to quite good performance. BRO achieves strong performance on numerous continuous control benchmarks, and is the first model-free reinforcement learning algorithm that can learn meaningful performance in DMC dog and humanoid tasks.", "questions": "- how do you expect BRO to be applied in the discrete control tasks? Will BRO beat other stronger MCTS-based approaches in Atari games? Any comments here?\n\n- can you elaborate more on explaining why scaling the actor network is not effective?", "limitations": "The authors have a good discussion of the potential limitations of this work in the main text.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- this paper is easy to follow and easy to understand\n\n- the studied topic is important to the RL community. It is vital to develop stronger and more powerful model-free RL algorithms for continuous control problems\n\n- despite that this work combines numerous previous well-developed tricks and strategies, the authors selectively incorporate them into one framework and demonstrate that such design choice incurs quite good performance.\n\n- the experiments are extensive and solid\n\n- the developed BRO algorithm is the first model-free RL algorithm that can achieve meaningful performance in DMC dog and humanoid tasks", "weaknesses": "this paper has the following drawbacks,\n\n- the quality of the figures could be significantly improved. Please try to export pdf with matplotlib instead of taking screenshots by convention.\n\n- BRONet seems to be a simplified version of ResNet. I do not seem to observe any significant network architecture difference between them. The authors should not over-claim on the network architecture. Any clarifications here?\n\n- It is often unclear how the figures are plotted and which environments they cover, e.g., Figure 4, Figure 6. This should not be vague and ought to be clearly stated in the main text\n\n- I am a bit concerned with the claim that *Algorithmic improvements matter less as the scale increases* (Line 184). Do you think that this is always correct? One should focus more on scaling instead of algorithmic improvements in the context of RL?\n\n- missing baselines and references. The authors should compare against some other recent strong model-free RL algorithm, e.g., TD7 [1]. Meanwhile, the authors should cite the REDQ [2] paper when referring to replay ratios. REDQ should be included as a baseline approach in the paper. Moreover, a recent paper introduces sample multiple reuse (SMR) [3] that updates a fixed batch multiple times to boost sample efficiency. I think this can be a very relevant work and should be included and discussed in the paper. Also, the authors write that they introduce another actor network, and incorporate regularization techniques into critics, it reminds me of another work, DARC [4], where they leverage double actors for exploration and introduce critic regularization for better performance. It should be included in the paper.\n\n[1] For sale: State-action representation learning for deep reinforcement learning\n\n[2] Randomized Ensembled Double Q-learning: Learning Fast Without a Model\n\n[3] Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse\n\n[4] Efficient Continuous Control with Double Actors and Regularized Critics\n\nDespite the aforementioned drawbacks, this is a solid paper that may have a potentially large impact on the RL community. I would be happy to reconsider the score if the aforementioned flaws are addressed during the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time reviewing our work and the suggestions on how to improve it. We are also very pleased that the reviewer found the experimental section solid. We leave our rebuttal below: \n\n>missing baselines and references...\n\nWe thank the reviewer for suggesting these algorithms. In response, we added an experiment comparing BRO and BRO (Fast) to TD7, SMR, and RedQ on the 15 DMC tasks from our original evaluations. Given the limited time and the marginal performance improvements reported over SAC/TD3, we excluded DARC but cited and discussed it in the related work section. Our findings show that BRO and BRO (Fast) significantly outperform the additional baselines, and we detail these results in the rebuttal PDF we uploaded.\n\n>The quality of the figures could be significantly improved...\n\nWe thank the reviewer for noticing that - indeed some of our figures were embedded as PNG files. We changed all figures to vector PDFs. \n\n>BRONet seems to be a simplified version of ResNet…\n\nBroNet uses layer normalization and adds an extra layer normalization before the first residual stream, as shown in Figure 12. This addition boosts performance by over 20% in DMC tasks, aligning with prior studies like [1] and [2] that reported declines when applying standard ResNets to Atari and DMC benchmarks. We named this configuration \"BroNet\" to highlight these crucial design choices. Additional offline RL experiments demonstrating BroNet's effectiveness are detailed in our joint response and Figure 3 of the rebuttal PDF. At the same time, we agree that it is crucial not to present BroNet as a new neural architecture but as an effective integration of existing modules for RL applications. We believe that naming this configuration aids in distinguishing specific architectures within the community. We added this discussion to the Method section. We hope this meets the reviewer’s expectations, though we are open to revising the presentation in the manuscript as needed.\n\n>It is often unclear how the figures are plotted and which environments they cover, e.g., Figure 4, Figure 6...\n\nWe thank the reviewer for noticing that. We checked all figures for links to task lists and added them when missing. We use two task sets in our experiments: all 40 tasks from DMC, MW, and MYO, and 10 tasks from DMC and MW. The majority of experiments (e.g. Figure 6) were performed on all 40 tasks. \n\n>I am a bit concerned with the claim that Algorithmic improvements matter less...\n\nWe thank the reviewer for this feedback. We acknowledge that scaling may not always be beneficial, such as in tasks with sparse rewards and complex exploration. However, our main takeaway is that scaling, alongside optimization techniques like layer normalization and replay ratio, can significantly enhance performance, often surpassing RL-specific algorithmic advancements (e.g., Figures 4 & 7). Many recent algorithms focus solely on these specific improvements while maintaining traditional architectures with two layers of 128-256 units (e.g., RedQ, SMR, TD7, SR-SAC [4]). We believe the synergy between algorithmic and scaling enhancements warrants more attention in future research. While we agree that algorithmic improvements are crucial for the field's progress, we aimed to suggest they matter \"less\" rather than \"not at all.\" We are open to modifying the wording based on the reviewer’s feedback.\n\n>how do you expect BRO to be applied in the discrete control tasks...? \n\nWe focused exclusively on continuous control because there are significant differences in best practices between continuous and discrete algorithms. For example, continuous algorithms usually require more mechanisms to curb overestimation [3] or different replay ratios [4]. However, we are happy to announce additional experiments in the Atari benchmark, where we use 3 tasks and augment the SR-SPR model with BroNet and find BroNet promising in these tasks. We summarize these results in the joint response. Due to the large computational demands associated with MCTS-style algorithms, we leave comparing BRO to those for future work. We summarize these results in the joint response and the rebuttal PDF. \n\n>can you elaborate more on explaining why scaling the actor network is not effective?\n\nWe interpret this result to be consistent with the previous works showing that off-policy actor-critic algorithms are more “critic centric” (e.g. [3] showing that actor can be updated two times less frequently, or [4] showing that resetting critic is more important than resetting actor). We hypothesize that the critic learning is thus more complex because it models the dimension of actions as well (and policy changes only wrt. state). Furthermore, since the policy optimizes for critic output, the policy complexity is inherently capped by the expressiveness of the critic. \n\nWe thank the reviewer again for their time, insights, and suggestions. Based on these we made several additions to our manuscript, which we describe in detail in the joint response. In particular, we hope that the additional baselines and added references address the issue of “missing baselines and references”. We also hope that the added results in the offline RL and discrete image-based RL benchmarks increase the reviewer’s confidence in the BroNet utility. If so, we kindly ask the reviewer to consider adjusting the initial score of our work. \n\n[1] Schwarzer, Max, et al. \"Bigger, better, faster: Human-level atari with human-level efficiency.\"\n\n[2] Bjorck, Nils, et al. \"Towards deeper deep reinforcement learning with spectral normalization.\"\n\n[3] Fujimoto, S., et al. Addressing function approximation error in actor-critic methods.\n\n[4] D'Oro, Pierluca, et al. \"Sample-efficient reinforcement learning by breaking the replay ratio barrier.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful feedback. We are pleased that our work was well received, and that all reviewers recognized the potential significance of our work for the RL community, the scope of our experiments, and the significant performance improvements our method provides over previous approaches. Following the reviewers' suggestions, we have made additions to our manuscript that we believe further increase its value. We have included graphs of the new results in the rebuttal PDF and summarize them here:

**New baselines** - Following the recommendation of reviewer TNcA, we have added three new baselines: TD7 (original codebase), RedQ (JaxRL), and SMR (JaxRL) and tested them on 15 DMC tasks. In these experiments, our proposed BRO achieves 250-350% better performance than the suggested algorithms. These results are displayed in Figure 1 of the rebuttal PDF.

**Extended training** - Following reviewer zd6E's suggestion, we expanded BRO training beyond 1M environment steps, although in a single-task setup. We trained BRO and BRO (Fast) for 3M and 5M steps respectively on 7 Dog and Humanoid tasks and compared them to TD-MPC2 and SR-SAC. BRO significantly outperforms these baselines and notably almost solves the Dog Run tasks at 5M steps (achieving over 80% of possible returns). We show the 3M results in Figure 2 of the rebuttal PDF. 

**Offline RL benchmark** - As suggested by reviewer 7YXC, we have added experiments on two offline RL benchmarks: AntMaze (6 tasks) and Adroit (9 tasks) [1]. We tested three scenarios: pure offline (comparing vanilla Behavioral Cloning (BC) to BroNet-based BC), offline with fine-tuning (comparing vanilla IQL [2] to BroNet-based IQL), and online with offline data (comparing vanilla SAC to BroNet-based SAC). Using BroNet led to noticeable improvements for all learners, as depicted in Figure 3 of the rebuttal PDF.

**Image-based benchmark** - Following Reviewer 7YXC, we added experiments on 3 tasks from the Atari 100k [3] benchmark. Here, we changed the regular Q-network of the SR-SPR (RR=2) model [4] to a BroNet, and considered changing the reset schedules to better fit the plasticity of the BroNet model. As depicted in Table 1 of the uploaded PDF, applying BroNet to discrete, image-based tasks is a promising avenue for future research.  

**BRO + TD3** - To evaluate BRO performance beyond maximum entropy objectives, we tested BRO and BRO (Fast) with a TD3 backbone across 15 DMC tasks. BRO with a SAC backbone slightly outperformed TD3, though TD3 remains a viable option. This result might be helpful for practitioners interested in applying BRO to models with TD3 backbone, such as the image-based SOTA algorithm DrM [5]. These findings are illustrated in Figure 1 of the rebuttal PDF.

We hope that these results show that our approach to scaling seems to be promising in other branches of RL as well, and will ultimately prove to be helpful for readers interested in problems beyond continuous control. We believe that these substantially increased the quality of our manuscript, and again, we are grateful to the reviewers for their suggestions. We invite the reviewers to inspect the new results in the uploaded rebuttal PDF and are happy to answer any further questions. 


[1] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning."

[2] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning."

[3] Kaiser, Lukasz, et al. "Model-based reinforcement learning for atari."

[4] Schwarzer, Max, et al. "Data-efficient reinforcement learning with self-predictive representations."

[5] Xu, Guowei, et al. "Drm: Mastering visual reinforcement learning through dormant ratio minimization."

Meta-review: The paper makes improves SOTA for deep Reinforcement Learning RL (both model-free and model-based).  This is achieved using well-known techniques (using big critic networks, regularisation (layer normalisation, weight decay) and optimism in the face of uncertainty. Overall, this is a well-executed empirical paper. The most important component of this type of paper (empirical evaluation methodology) seems solid. Crucially, the submission provides a link to code used to generate the results, which is crucial for this type of paper.
Decision: Accept (spotlight)