Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
Authors: Michal Nauman
Marek Cygan
Carmelo Sferrazza
Aviral Kumar
Pieter Abbeel
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
multi-task learning
data scaling
compute scaling
Primary area: reinforcement_learning
TL;DR: We find that online multi-task RL with high-capacity value models leads to SOTA sample efficiency and performance

Abstract: Recent advances in language modeling and vision stem from training large models on diverse, multi‑task data. This paradigm has had limited impact in value-based reinforcement learning (RL), where improvements are often driven by small models trained in a single-task context. This is because in multi-task RL sparse rewards and gradient conflicts make optimization of temporal difference brittle. Practical workflows for generalist policies therefore avoid online training, instead cloning expert trajectories or distilling collections of single‑task policies into one agent. In this work, we show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online RL, allowing for robust and scalable multi‑task training. We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL. We find that, despite its simplicity, the proposed approach leads to state-of-the-art single and multi-task performance, as well as sample-efficient transfer to new tasks.

Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
Michal Nauman
Marek Cygan
Carmelo Sferrazza
Aviral Kumar
Pieter Abbeel
Abstract
Recent advances in language modeling and vision stem from training large models on diverse, multi-task data. This paradigm has had limited impact in value-basee reinforcement learning (RL), where improvements are often driven by small models trained in a single-task context. This is because in multi-task RL sparse rewards and gradient conflicts make optimization of temporal difference brittle. Practical workflows for generalist policies therefore avoid online training, instead cloning expert trajectories or distilling collections of single-task policies into one agent. this work, we show that the use of high-capacity value models trained via cross- entropy and conditioned on learnable task embeddings addresses the problem task interference in online RL, allowing for robust and scalable multi-task training. We test our approach on 7 mullti-task benchmarks with over 280 unique tasks, spanning high degree-of-freet of-freedom humanoid control and discrete vision-based RL. We find that, despite its simplicity, the proposed approach leads to state-of- tf-th-artt single and multi-task performance, as well sample-efficient transfer new tasks.
Introduction
Aggregate
DeepMind Control 500k
HumanoidBench IM Low:DoF hazemvi talo)
HumaaoidBeench 2M High-DFF haauni tala)
0.8
Hagh-Duf lunezalie (? taio)
Single-tzak
6R0
BRO
BRJ
Mulli-aak
TD-MPCC
1
1D-MPU2
0.6
SAC BBC
BAC-BRC
SAC BRC
Nomalize soure
SAC BBC
BBC.
SAC BRC
Normdized rituna
Nomalioal roturne
Normdized
Meta World 500k
Shadow Hand IM High Dul romipibbbe (ef talo)
Ataci 100k
Law Dell wwwdgaleeen tolo))
Vidkes howe divene (3 oale)
PC
DDRGHEER
Dr2
0.2
CTPG
GAMI+
SPR
SAC-BRU
SAC-BRU
DrU-BRC
SAC-BRE+
SAC-BRU-
DiQ ISRC.
0.0
0.5
0.8
0.45
SAC
SAC+BRC
Suxcesa rite
Saccess site
Figure 1: Scaling multi-task training leads to state-of-the-ae performance. Naive scaling of SAC to multi-task decrcases the aggregate performance (left). Our proposed method (BRC) works both in single and multi-task learning and provides pronounced performance improvement over previous approaches, including optimized single-task learners (right). We denote multi-task with and share additional details Appendix D.
Large-scale neural networks trained on large, diverse datasets have led to substantial advances in natural language processing [28, 18, 105] and computer vision [30, 78, 56]. These models, typically trained in a multi-task setting [85, 24, 1], exhibit not only strong performance across tasks seen during training but also demonstrate impressive sample efficiency when transferred to new domains or new
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MetaWorld LowDDF zesipolttio tuaka)
HumanoidBench-Mooee Medium
HumanoidBench-HHae Hard High-DeF la taks)
Atari Yoita-booolliose todo)
Low-DVE humaadd tako)
1.0
62%
66%
0.6
68%
0.4
0.6
14%
0.5
0.3
0.2
0.3
0.0
0.0
IM 0.0 1M 2M 0.0
250k
500k
500k
50k 1ook
Figure 2: Scaling multi-task training allows for sample-efficient transfer new tasks. We compare the performance of single-task BRC agent trained from scratch (green), to an agent initialized with our pretrained multi-task BRC agent trained on different tasks (blue). We find that transferring a multi-task BRC model to new tasks leads to better sample efficiency than learning from scratch. Y-axis denotes the average final performance.
tasks, including those that these models were never trained on [1. 102, 105, 66]. This combination of generalization and transfer has become the cornerstone of modern machine learning, suggesting that scaling models and data in a unified training regime can yield powerful and versatile systems.
Despite growing interest in reinforcement learning (RL) [25, 27, 79, 53], little known about how RL scales [89]. This is particularly true for online RL algorithms,, where recent advances have mainly focused on scaling model capacity for learning with limited single-task data [91, 77], making difficult to assess how these methods perform when the variety of tasks and data increases [104, 46]. Addressing this gap especially important, as experience with large language models shows that genuine scaling gains arise only when model capacity is paired with sufficiently diverse data [52, 45, 47, 69]. Unfortunately, previous work shows that naively extending value-based methods to multi-task i far from straightforward due to issues like reward imbalance [46] and gradient interference [120]. As such, practical systems often fall back on distillation from single-task experts [88, 65, 38, 21, 114, 111], offline learning from curated demonstrations [20, 7, 87, 17, 123, 119] or specialized techniques to resolve gradient confficts [22. 92, 120, 23, 67]. As such, online training of high-capacity value- based agents on many tasks remains significant open challenge.
We demonstrate, for the first time, that value models trained with online comporal-differeme learning scale to the billion-panne parameter range, challenging the belief that such capacity requires offline datasets and behavioral cloning [98, 70]. We show that, similarly to supervised learning, combining model capacity with multi-task RL training leads to significant improvements over single-tast saage-ttsk oracles, as well as pronounced sample-efficient gains when transferring to new tasks. The key design choices that unlock this behavior include significantly increasing critic capacity using normalized residual architectures, stabilizing temporal difference gradients through cross-entropy loss, and modeling task diversity via task embeddings instead of separate network heads. We experiments using over 280 complex tasks from 5 benchmarks and find that the proposed combination of design choices produces simple agents that offer state-of- the-art performance in both single-task and multi-task scenarios, all while using a single hyperparameter configuration. Our contributions are as follows:
ofth-offthheart
Model scaling in online RL we show design choices that allow for scaling value-based RL up to
IB parameters (Figures 4 & 12) without the use of offline data [61] or behavioral cloning [98]. We show that using such high-capacit models addresses the issues of gradient magnitudes [46] and conflicts [120] that are associated with multi-task RL (Figures 3 & 5).
Data scaling in multi-task RL we show that multi-task RL can yield improved sample ef- ficiency and final performance as compared to single-task learning with state-of-the-art agents (Figure 1), despite using significantly less gradient updates (Figure 9). We find that, given common embodiment, agent performance scales with the number of tasks used in training (Figure 10).
Sample-efficient model transfer we find that initializing training from a pretrained, multi-task value model can lead significant improvements in sample efficiency when compared to learning from scratch (Figure 2). We find the sample efficiency and final performance of the pretrained model transferred new tasks is scaling with the capacity of the multi-task model, well as the number of tasks used in pretraining (Figure 10).
Our goal is to show that, much like vision and language modeling, larger models trained on more diverse streams of experience lead to policies that not only perform well in tasks used in multi-task learning but also transfer to new problems in a sample efficient manner. As such, we take step toward bridging the methodology gap between supervised learning and value-based control.

---

Variance between tasks
Performance improvement
10
1.0
Naire (MS6)
Relative vartance
Nomallzzd (MSE)
0.8
Nommalzzed
Normaiized
0.6
0.4
0.2
0.0
TD Loss
Grad Norm
Entropy
Single-task
Multi-ask
Figure 3: Cross-entropy loss stabilizes online multi-task learning, We investigate BRC with naive application of MSE loss (purple), MSE loss paired with return normalization (green) and cross-entropy paired with retumn normalization (blue) -B-MM..UM. Varying reward magnitudes in multi-task learning can destabilize learning of certain tasks, which translates to high variance of signals between tasks (left). Stabilizing this effect via cross-entropy loss allows for improved scaling when moving from single to multi-task learning (right).
2 Related Work
High-capacity value models. Previous work indicates that naive scaling of model capacity in single-task RL leads to divergent behavior when training via temporal difference loss [15, 91, 77]. Interestingly, it was shown that using layer normalization [77, 633, feature normalization [59, 61], weight normalization [80]. batch normalization [13, 19] or classification losses [40, 41, 33] can stabilize that divergent behavior and improve performance when using high-capacity models. Whereas these works investigated the above design choices in isolation, we explicitly combine normalized residual valuue architectures [77] with cross-entropy loss via categorical Q-learning [1 [] and show that using these is crucial for efficient scaling in multi-task RL.
Multi-task learning. Previous work showcased variety of issues with multi-task RL, such as
divergence due to heterogeneous reward scales [104, 46] or conflicting gradients [120, 67] and
proposed solutions such as reward [46] or gradient [22, 1 116] normalization, gradient projection [120]
or distillation of single-task expert policies [104, 1 114, 111]. In contrast to these works, we show that increasing model capacity paired with training via cross-entropy loss is very effective at stabilizing multi-task learning, leading to policies that significantly outperform state-of-the-art single-task oracles. Furthermore, previous work considered multi-task training with high-capacity models [61, 41, 98]. In these works, the authors focus on offline learning from fixed datasets containing high-return trajectories. In contrast to these works, we focus online multi-task learning without expert data.
Transferable generalist policies. Prior work investigated the possibility of learning a generalist agent that can solve many tasks and transfer to new ones [87, 17, 41, 103]. Due to multi-task RL complexities, many works train generalist agents via behavioral cloning on expert demonstrations [87, 31, 117]. These works often use pre-trained vision and language backbones [103, 68, 48] and achieve transfer via fine-tuning the pretrained model weights [54] or conditioning the pretrained model on text or visual task encoding [17, 123]. Although we are also interested in learning transferable generalist policies, our work assumes non non- stationary learning of value functions via temporal difference [100]. More RL-based approaches facilitate transfer by learning a task-agnostic encoder and retraining linear layer when the task changes [9, 115, 32, 2] or appending a learned task embedding to the model inputs, with heuristic selection of embeddings for new tasks [34, 43, 86, 97]. Whereas our work also considers task embeddings, we learn these online by backpropagating the temporal difference loss.
3 BRC Approach
In this section, we outline the Bigger. Regularized, Categorical (BRC) approach and its design principles that, when combined, allow effective scaling of online value-based agents (Figure 6). We discuss these design choices below and share additional details in Appendix C.
Cross-entropy loss via distributional RL. Recent studies have shown that reframing regression prob- lems as classification tasks can significantly improve performance in both supervised learning [106] and RL [33]. The Mean Squared Error (MSE) loss is known to depend on the scale of predicted values and is susceptible to outliers [14]. Note that this susceptibility is particularly problematic in multi-task learning. differences in reward magnitudes introduce an implicit task prioritization:
3

---

Single-task model scaling
Multi-task model scaling
1.0
1.0
Nornalized returns
Normalized lized returns
0.8
0.8
0.6
0.6
0.4
Vauillo SimBa
0.4
0.2
Itroaa (MSI)
0.2
BroNia (CB)
0.0
0.0
4
16
64
256
4
16
64
256
Total parameter count (M)
Total parameter count (M)
Figure 4: BroNet paired with cross-entropy loss scales in both single and multi-task RL. We compare scaling behavior of different architectures in single (left) and multi-task (right) when solving the 1B-MEDIUM benchmark. We pair SAC with the vanilla [39], SimBa [63], BroNet with mean squared error loss [77], and proposed BroNet with cross-entropy loss architectures. Both figures report final performance after IM steps.
Model size scaling
Gradient conflicts between tasks
1.0
1.0
Normalized returns
Gradient conflicts rate
0.9
.98
0.8
0.7
.96
Singlo-tas tank
0.6
Sepurste heads Task embedding.
0.5
.94
16
64
256
16
64
256
Total parameter count (M)
Total parameter count (M)
Figure 5: Using task embeddings is preferable to separate heads design. We compare the performance (Ieft) and gradient similarity [120] (right) different approaches for multi-task learning on HIB- MED-MM. We consider single-task, multi-tast via separate heads [46, 61] and via task embeddings variants of our proposed BRC. We find that the task embeddings design outperforms other variants at all considered model scales and, interestingly, the separate heads design performs better than single task oracle only past certain model scale.
tasks with higher rewards produce higher TD errors and stronger gradient signals [104, 22, 46, 1 116]. This in turn results in increase in the variance of TD errors, gradient norms, and policy entropy between tasks, destabilizing learning. We demonstrate this phenomenon in Figure 3, which demon- strates substantial variability in loss magnitudes, gradient norms, and policy entropy across tasks for a multi-task learner. Although reward normalization [46] partially mitigates this issue, we find that adopting a cross-entropy loss via categorical Q-learning [11] effectively balances gradients and ensures consistent learning signals across all tasks. We normalize the rewards in each task by dividing them by the maximal Monte Carlo return achieved in a given task that point in learning (see Appendix C for details). Finally, we note that previous work builds theoretical arguments on why cross-entropy loss might stabilize TD learning [112, 113].
Scaled Q-value model. Prior work proposed scaling the Q-value model in single-task RL by using regularized ResNets [15, 77]. In particular, the BroNet architecture [77] was shown to achieve state-of-the-ar performance on a variety of control tasks. To this end, we combine this architecture with the cross-entropy loss, and refer this variant as BroNet (CE). As shown in Figure we observe that the trends observed in single-task learning persist in multi-task scenarios. Similarly to single-task setup, vanilla architectures experience performance degradation when the Q-value model size is increased, despite being exposed to more diverse data than in single-task learning. Furthermore, regularized architectures achieve steady performance improvements when increasing the capacity, with our proposed BroNet (CE) yielding the best performance in both single and multi-task setups. Furthermore, previous work found that online multi-task learning leads to gradient conflicts that substantially decrease the learner performance [120, 67]. Interestingly, we find that the rate of gradient conflicts decreases as the number of parameters increases (Figure 5), showing that complex interventions such as gradient surgery might not be required after a certain model scale. We discuss the considered architectures used to represent Q-values Appendix C, as well as the definition of gradient conflicts in Appendix D.

---

Importance of BRC components
None
TE
CE TE
-Q
TE 8QQ
8Q
62.4%
CE TB SQ
0.5 0.6
0.7
0.8
0.9
0.0
0.1 Shapley values
0.2
0.5
Normalized returns
Figure 6: Scaled Q-value model is the most important design choice. (Left) We investigate the final performance of the base model paired with various combinations of scaled Q-value model (SQ), cross-entropy loss via categorical RL (CE), and learnable task-emhedding module (TE) HB-MEDIUM. (Right) SQ associated with the highest Shapley value, accounting for 62% improvements.
Task embeddings. To accommodate scalable learning across numerous tasks, we employ Iearn- able task embeddings [86, 41]. By providing task identifiers and using task embeddings effectively transforms the multi-task setting into single-task by combining multiple MDPs into one [34]. In this unified MDP, the state representation is augmented with the learned embeddings, thereby allowing the models to distinguish which task it is solving. In contrast to approaches such as separate value and policy heads [120, 121, 61], learnable embeddings facilitate the discovery of shared structure between tasks, positioning similar tasks closer within the embedding space, without assumption that the Q- values of different tasks are linear with respect the shared representation [81, 46, 2]. Crucially, not explicitly supervise the learning of shared structure between tasks. In con- trast, we optimize embeddings by backpropagating the cross-entropy temporal difference (TD) loss, thus encouraging embeddings that represent well the value structure. As shown in Figure 7, our approach learns embeddings that cluster similar tasks. Furthermore, as shown in Figure 5, using task embeddings performs better than separate heads various model scales. We detail our approach to using task embeddings in Appendix C.
Jump
Run
Tum-Easy
Run-Front
Turn-Hard
Run-Back
Walter
Campole
Chertah
Swingup
Ruacher (2)
Finger
Balance
Roscher
Itall
Balango-Har Hard
Swingup-Hard
Easy
Jump-Back
Hard
Stand Run-Back
Spin
Run
Walk
Hard
Walk-Back
Easy
Figure 7: Learnable task embeddings discover the underlying dynamics structure. graph first two principal components of the task embeddings learned online on MW +MC and find that the dis- covered embeddings cluster similar embodiments.
these approaches
Summary. The design choices that enable us to scale multi-task RL agents are: (1) using scaled and regularized residual network architecture to represent the Q-values, (2) using cross-entropy instead of MSE loss to train the critic, and (3) conditioning models on task embeddings which are learned by backpropagating the temporal difference loss. We refer to the resulting approach as Bigger, Regularized, Categorical (BRC). We apply BRC to two popular model-free algorithms for continuous and discrete control, and study their scaling and transfer behavior. For continuous control experiments, utilize Soft Actor-Critic (SAC) [39], whereas for discrete vision-based control, use DrQ-e [58] (we discuss hyperparameters in Appendix F). As shown in Figures 3 and 5, BRC tackles the problems of stability [104, 22, 46, 116] and gradient conflicts [120, 67] in multi-task RL. To assess the importance of BRC components, we evaluate the performance of every combination of these choices. As shown in Figure 6, find that combining the above techniques leads to synergy effects, where removing any block always leads to a decrease in performance. Finally, we calculate the exact Shapley values [94] associated with each component and find that using the scaled Q-value model is the most important factor, accounting for 60% of the presented improvements.
Experimental Evaluation
We now discuss experiments designed to evaluate the effectiveness of our BRC approach in scaling with data and compute. Our goal is to answer the following questions: (1) How does our proposed method enhance the effectiveness of model capacity scaling in multi-task RL? (2) Does the perfor-
5

---

C
Figure 8: We consider 283 tasks from simulation benchmarks. We test our approach with SAC [39] on MetaWorld, DeepMind Control, HumanoidBench and ShadowHand and with DrQ-e [58] on rision-based Atari. In both approaches we use single set of hyperparameters across all tasks and multi-task configurations.
mance of our multi-task learner improve when scaling the number of tasks? (3) Do the representations learned in online multi-task learning transfer to new tasks and result more efficient learning than fresh initialization? (4) How does model capacity scaling impact the task-scaling and transfer ca- pabilities of agents? We answer these questions through experiments described in the subsections Appendix D.
below and analyzed in Section 5. For more details our experiments
Benchmarks. We consider a wide range of tasks, with a total of 283 diverse, complex control problems spanning five domains: DeepMind Control (DMC) [101]. MetaWorld (MW) [121]. Hu- manoidBench (HB) [93]. Atari [10], and ShadowHand (SH) [49]. The tasks considered include locomotion and manipulation, different embodiments, numerous degrees of freeddom ([AA reaching 60 dimensions), fully sparse rewards, and ausion-based control. We use the following multi-task configurations: DMC- with dog (4 tasks) and humanoid (3 tasks) locomotion tasks trained for 500k environment steps per task [77]. MW with 50 manipulation tasks for a Franka robot [121] trained for 500k environment steps per task, HB-MEDIUM with 9 humanoid locomotion tasks trained with 20 whole-body locomotion and manipulation
for I M environment steps per task, tasks for a humanoid equipped with Shadow Hands [93] trained for 2M environment steps per task, and SH with high DoF Shadow Hand manipulating 85 diverse shapes with fully sparse rewards [49] trained for M environment steps per task. We also consider an 80-task MW+DMC with multiple robot embodiments considered in Hansen et al. [41]. For vision-based experiments, we consider 26 tasks from Arcade Learning Environment (ATARI) [51]. Finally, we consider an additional 64 tasks from all benchmarks for transfer experiments. We list all the task sets considered in Appendix E.
Training. We compare approach against a variety of modern single and multi-task approaches. We consider TD-MPC2 [41], Dreamer [40]. BRO [77], SAC [39], TD7 [37]. SimBa [63], Sim- BaV2 [64], MH-SAC [121], PCGrad [120], PaCo [99], CTPG [44], GAMT [49], DDPG+HER [5, 6], DrQ-e [58], SPR [90], SimPLe [51], and DER [108]. Whenever possible, we show previously reported results, otherwise we run official repositories associated with the given method. For our approach, we use a single set of hyperparameters across all benchmarks, in both continuous and discrete action experiments. Specifically, we use hyperparameter values proposed in previous works for our backbone algorithms ([77) for SAC and [3] for DrQ-e). We use the 250M variant of the single and multi-task BRC model except when directly stated otherwise. In both cases, following prior works [41]. we increase the batch size 4x to accommodate multi-task learning. In contrast to previous works [41, 64], we do not use learning rate scheduling or discount heuristics for our method. Furthermore, all multi-task task agents interact with all tasks in parallel and keep apdates-per-dee (UTD) fixed at UTD- [77]. As such, the multi-task learners perform fewer gradient steps than their single-task counterpart (e.g. with 20 tasks, the multi-task model performs 20 times fewer updates while performing the same number of environment steps). We detail hyperparameters in Appendix
Transfer learning. In our transfer experiments, we evaluate three adaptation protocols inspired by previous work. Firstly, we consider model transfer by initializing single-task learner with the weights of the pretrained multi-task agent and continue learning only on data from the new transfer, we keep the pretrained parameters frozen [34, 60] and adapt only through a liw-dimensional task embedding inferred from less than a thousand target-task transitions. Finally, as a form of a baseline in transfer approaches, we consider an approach leveraging prior data [8, 122] in which we initialize the experience buffer of a fresh single-task learner with transitions stemming from multi-task pretraining (we refer to this approach as data transfer). In all cases, the multi-task model is not trained on the transfer tasks, mimicking the train-test split used in supervised learning [14]. We report results for transfer experiments in Figures 2, 10 and 11. We list the tasks used in multi-task and transfer learning in Appendix E.
task [61, 41J. Secondly, in what we call embedding-tuninn

---

Sample Efficiency
Compute Efficiency
0.8
0.8
Nomalized returns
Normalized returns
SAC-BRC (MT)
SAC-BRC
0.6
2x improvement
0.6
SumBaV2
40x improvement
BRG
0.4
0.4
TD-MPC2
0.2
0.2
0.0
0.0
0.5
1.0
1.5
2.0
0.04
0.2
25
125
Environment Steps (M)
Gradient Steps (M)
Figure 9: Multi-task learning is sample and compute-efficient. We compare the performance of our proposed single and multi-task BRC agents on HB- HARD benchmark and find that multi task learning leads 2x improvement in sample efficiency (left) while performing 40x less gradient updates (right) when compared to SOTA single-task learners. This result shows that multi task training can very efficient.
Evaluation. Since tested environments different reward scales, we report performance normalized to [0, 1] according to practices drawn from previous works: for DMC, we divide the returns by 1000 [77]: for MW, we report success rates [121]: for 1B-MEDIUM and HB-HARD we divide the returns by their success score [93, 63]; and for ATARI we divide the returns by scores achieved by humans [72]. When reporting aggregate, we report the sample mean across tasks with 95% confidence interval calculated via bootstrapping over seeds [3]. We detail the implementations in Appendix F, normalization values in Appendix E and report the unnormalized scores in Appendix H.
Analysis
This section summarizes the results of our experiments with the BRC approach, particularly high- lighting its scaling and transfer properties. We adhered to the setup discussed in Section
Efficiency of multi-task learning. Perhaps most importantly, we demonstrate that given the design choices outlined in Section 3, a single value model trained jointly multi-task setup not only matches but surpassss strong single-task specialists, establishing clear generalist advantage in value-based RL (Figure 1). This result fills the gap left by recent work reporting parity with single-task oracles [87, 111]. Furthermore,
shown in Figure 9, the performance improvements are complemented by significantly lower compute load, with our multi-task model performing 40x fewer gradient updates to achieve the performance of the best single-task model. Since the multi-task model learns from all tasks in parallel, it uses more data than any individual single-task learner - we find that the outcome is consistent with observations in supervised learning, where data and compute reach a given performance level [71, 52, 47]. We note that BRC achieves this level of performance while using single set of hyperparameters across all tasks proposed by prior work and tuned for single-task DMC [77]. We present detailed results, including training curves, in Appendix H.
be traded off to
Scaling model capacity. Similarly to single-task learning, increasing the model capacity can lead to substantial performance improvements in the multi-task setting (Figure 4). In fact, we observe that the increased data diversity resulting from multi-task learning allows the performance to scale beyond that of single-task RL - whereas in single-task the performance saturates at around 64M parameters, the multi-task model still improves beyond 250M parameters (Figures 4 and 12). Furthermore, as shown in Figure 5, we observe that increasing the parameter count reduces the rate of gradient conflicts for both single and multi-task learners, suggesting that high-capacity models interventions such as gradient surgery. To this end, as shown in Figure 1, we observe that our method outperforms multi-task specific approaches like CTPG [44], PaCo [99] PCGrad [120]. Finally, we find that scaling the Q-value model the most important design decision, accounting for more than 60% of the improvements of our approach according the Shapley values (Figure 6).
at least partly substitute
Scaling number tasks. We find that the BRC performance scales with the number of tasks used in pretraining, with the multi-task variant performing equally or better than single-task counterpart (Figure I). Interestingly, we also observe that our proposed method trains robustly even when the number of robot embodiments grows (Figure 12). Furthermore, we find that given the common
7

---

Pretraining tasks scaling
Model transfer scaling
Normalizz saaiizzdd returns
1.0
Single
Normalizon izzd returns
0.5
Mulii-usk (3)
Mulii-ask (f)
0.4
0.8
Mutti-akk (9)
0.3
0.6
0.2
0.4
0.1
16
64
256
16
64 Total parometer count (M)
256
Total parameter count (M)
Figure 10: Increased data and compute improves pretraining and transfer performance. We train our multi-task agent on HB-MEDIU MEDIUM using 3. 6 or 9 tasks for 1M environment steps. We find that the final performance 3 shared tasks improves when increasing the number of pretraining tasks, with the trend most pronouneed for the biggest evaluated models (left). We also investigate the final performance of the same models when transferring to 3 new tasks. The transfer performance improving with both model capacity and the number of pretraining tasks (right). We detail the setup in Appendix D.
Model transfer
Embedding- tuning transfer
Normalizell performance
DDPG-HER
1.5
GAMT
1.0
BRC (Zero-Shot)
0.5
Model transfer
Data transfer
0.0
From scratcls
BRC (Few-Shot)
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.2
0.4
0.6 Success rate
0.8
1.0
Environment steps (M)
Figure 11: Value-based RL agents are transferable. (Left) We compare the performance of model transfer with the BRC agent to fresh agent initialized with the buffer used in pretraining the transferred BRC model (data transfer). We consider 15 transfer tasks from MW, HB-MEDIUM and HB-HARD listed in Appendix E.2. (Right) We also inspect the potential of fine-tuning solely via adjusting the task embedding, while keeping the policy and value models frozen (embedding tuning transfer). We report success rates on 29 transfer tasks from the SH manipulation benchmark listed in Appendix E.2. We consider using random embeddings from pretraining tasks (zero-shot) and choosing the best best- cest-performin embedding from the pretraining set (few-shot). Surprisingly, the few-shot embedding transfer achieves over 90% success when manipulating new objects.
embodiment, pretraining using more tasks leads to both improved performance on pretraining tasks and improved capabilities when transferring to out-of-distribution tasks (Figure 10). This indicates that akin to supervised learning, increased data diversity in online RL leads to more general features that allow for sample-efficient transfer. Whereas our work showed empirically that such scaling effect exists in high-DoF humanoid tasks, we believe that establishing more comprehensive theory of task similarity in RL can be an impactful avenue for future research.
Model transfer from multi-task RL We observe that multi-task agents can learn features that transfer to new tasks, leading improved sample efficiency and performance on various benchmarks (Figure 2). This transfer capability seems to be related to the size of the multi-task learner, with bigger models being better transfer learners (Figure 10). Examining the learning curves Figure 11, we observe that model transfer not only outperforms learning from scratch but also converges more rapidly than the data transfer baseline. Although both methods rely on the same set of pretraining transitions, initializing with multi-tast task weights yields higher returns throughout training. Interestingly. as shown in Figure 11, the model transfer procedure (i.e. transferring weights of the pretrained model) is often more practical than data transfer (i.e. initializing fresh agent with replay buffer from multi-task pretraining [8]. This result consistent with recent observations made in single-task offline-to-online fine-tuning [122]. We also note that the data transfer procedure differs from traditional offline-to-onn tonlne RI. [8, 122] or network resets [29] that the data transfer method uses data generated when solving auxiliary tasks. Although the exact reason for the effectiveness of model
8

---

Offline learning
Online sample efficiency scaling
TD3+BC (6M samples)
3.7 GPU days
Transitions required
256
0.75
1.00
128
TD-MPC2 (6M samples)
GPU days
64
BRC+BC (6M samples)
GPU
32
16
BRC (0.5M samples)
8.8 GPU days
8
0.0
0.2
0.6 N Normalized score
0.4
0.8
1.0
16
64
256
1024
Total parameter count (M)
Figure 12: Scaling online multi-task TD learning past 1B parameters. We investigate the performance of online (BRC) and offine (BRC+BC) agents the multi-task MW+DMC benchmark consisting of 12 different robot embodiments [41]. (Left) Online multi-task learning is significantly more compute and data-efficient than offline learning. (Right) We investigate the number of samples required for BRC agents to reach a percentage of offline TD-MPC2 final performance. The find that 1B BRC model is the most sample-efficiet,, reaching the performance of offline TD-MPC2 with 64k transitions, almost 100x less than TD-MPC2.
transfer requires further study, pretrained weights encode the dataset without inducing distribution mismatch when learning new tasks.
We also investigate the possibility of transferring to new tasks by solely adjusting the task embeddings, while keeping the entire network frozen (embedding- transfer). Here, we consider the Shadow Hand manipulation setup proposed in previous work [49], with pretraining focused on manipulating 85 distinct shapes, with evaluation on 29 out-of-distribution objects. As shown in Figure 11, using only 300 transitions for task embedding selection allows the BRC agent to achieve over 90% success rates when manipulating new shapes, outperforming specialized baselines using privileged information [49] or hindsight relabeling [5]. Furthermore, using a random embedding from the pretrained set yields satisfactory performance, showing that our multi-task policy conditioned on task embedding did not overfit to pretraining shapes. We believe that this behavior stems from the ability of the model to discover semantics of the underlying task space (Figure 7). This result shows that multi-task training with auskembeddingg can lead to robust transfer and skill discovery just by manipulating the network input space, without updating the parameters of the policy or value model. We expand our analysis of embedding- -unnnng transfer in Appendix C.
sabbedding-tuningg transfer
Multi-task value models learn more efficiently with online data Recent work performed offline model-based training on 6M expert transitions per task from 80 MW and DMC tasks, containing 12 different embodiments [41]. We investigate the performance of BRC on this benchmark in both offline and online setup, where in the offline learning we use a behavioral cloning auxiliary objective following prior work [35, 82, 83] which we detail in Appendix C. As shown in Figure 12, offline multi-task RL remains challenging, even when using high-capacity value models, with online BRC performing nearly 20% better than its offline counterpart. Furthermore, in contrast to results in online learning (Figure 1), we find that the offline BRC agent slightly underperforms TD-MPC2, indicating that Q Q-learning with offline data might be harder than offline model-based learning. To better contrast the efficiency of offline and online multi-task value learning, we investigate the minimal number of online transitions required by BRC agents of different capacities (16M, 64M, 256M and IB) to achieve a fraction (0.5, 0.75, and 1.0) of final TD-MPC2 performance on the MW+ DDC benchmark. As shown in the right part of Figure 12, BRC sample efficiency still improves past 1B parameters, with logarithmic curves modeling the efficiency improvements well at different performance fractions.
Conclusions
Our study showed that online training of value functions by temporal difference loss scales beyond billion parameters and to many tasks. As discussed throughout the manuscript, scaled Q-value model trained via cross-entropy loss and conditioned on learnable task embeddings stabilizes the previously described issues of varying gradient magnitudes [104, 22, 46] and conflicting gradient [120, 67]. resulting sfate-of-the-arr performance across a variety of tasks. Perhaps surprisingly, we found that scaled multi-task RL can be extremely computationally efficient, achieving the final performance of

---

single-task experts with 40x fewer gradient steps on the challenging whole-body humanoid control tasks. Finally, we showed that pretrained multi-task value models can be transferred to new tasks, improving both sample efficiency and final performance. We discuss the limitations of our approach in Appendix We hope that our work challenges the belief that "one agent per task" is the preferable workflow to training a scaled multi-task agent and provides concrete foundation for online training of generalist value models in RL.
Acknowledgments
We also gratefully acknowledge the Polish nigh-perrformanee computing infrastructure, PLGrid (HPC Center: ACK Cyfronet AGH), for providing computational resources and support under grant no. PLGG/224401 7817. We also thank NVIDIA for providing compute resources through the NVIDIA Academic DGX Grant. Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and as Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon. Marek Cygan was partially supported by National Science Centre, Poland, under the grant 2024/54///ST6/00538 We would like to thank the Python [109], NumPy [42], Maiplotlib [50], SciPy [1 10] and JAX [16] communities for developing tools that supported this work.
References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.085 087744 2023.
[2] Agarwal, A., Song, Y., Sun, W., Wang, K., Wang. M., and Zhang, X. Provable benefits of representational transfer in reinforcement learning. In The Thirty Sixth Annual Conference Learning Theory, pp. 2114-2187. PMLR, 2023.
[3] Agarwal, R., Schwarzer, M., Castro, S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
[4] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep rein- forcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 14:29304-22330, 2021.
[5] Andrychowicz. M. Wolski, F., Ray, A., Schneider, J., Fong, R.. Welinder, P., McGrew, B., Tobin, J.. Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[6] Andrychowicz, O. M., Baker, B., Chociej. M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[7] Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (ypt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-2466 2022.
[8] Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
[9] Barreto, A., Dabney, W., Munos, R., Hunt, J., Schaul, T., van Hasselt, H. P., and Silver, D. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[10] Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. J. Artif. Int. Res., 47(1):253-27, May 2013. ISSN 1076-9757.
[11] Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 458. JMLR. org, 2017.
10

---

[12] Bellman, R. Dynamic Programming. Princeton University Press, 1957.
[13] Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., and Peters, J. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity, In International conference on learning representations (ICLR), 2024.
[14] Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006.
[15] Bjorck, N., Gomes, C. P., and Weinberger, K. Q. Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems, 34:8242- 8255, 2021.
[16] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Honderman-Milher S., et al. Jax: composable transformations of python+ numpy programs. 2018.
[17] Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu. J., et al. Rt-1: Roboties transformer for real-world control at scale. arXiv preprint 0rXiv:22/2.068//1 2022.
[18] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:11877-1901, 2020.
[19] Celik, O., Li, Z., Blessing, D., Li, G., Palenicek, D., Peters, J., Chalvatzaki, G., and Neu- mann, G. Dime: diffusion-based based maximum entropy reinforcement learning, arXiv preprint arXiv:2502.02316. 2025.
[20] Chen, L.,, Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-1500 2021.
[21] Chen, T., Xu, J., and Agrawal,
system for general in-hand object re-orientation. In Conference on Robot Learning, pp. 297-307. PMLR, 2022.
[22] Chen, Z.., Badrinarayanan, V., Lee, C.-Y,, and Rabinovich, Gradnorm: Gradient normaliza- tion for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pp. 794-803. PMLR, 2018.
[23] Chen, Z., Ngiam, J., Huang, Y., Luong. T., Kretzschmar, H., Chai, Y., and Anguelov. D. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances Neural Information Processing Systems, 33:2039 2050, 2020.
[24] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P.. Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko. S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H.. Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstenn K., Eck. D., Dean, J.. Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023, URL http:// jmlr org/ wwtgg/apers/eeeee ataaprss/24/// v2//222//////// 1144.144.4 html.
[25] Christiano, P. F. Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[26] Ciosek, K., Vuong, Q., Loftin, R., and Hofmann, Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
11

---

[27] Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D.. et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414
2022.
[28] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/N199 14233.9-12....
[29] D D'Oro, P. Schwarzer, M., Nikishin, E., Bacon, P.-.... Bellemare, M. G., and Courville, A. Sample-efficient reinforcement learning by breaking the replay ratio barrier. The Fleventh International Conference on Learning Representationss 2022.
[30] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D.. Zhai, X., Unterthiner, T., De- hghani, M., Minderer. M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
[31] Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pp. 8469-8488 PMLR, 2023.
[32] Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. Contrastive learning goal- conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 11603-35620, 2022.
[33] Farebrother, J.. Orbay, J., Vuong, Q., Taiga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value functions via classification for scalable deep rl. In Forty-first International Conference on Machine Learning.
[34] Finn, C., Abbeel, P.,, and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1226-1135. PMLR, 2017.
[35] Fujimoto, S. and Gu, S. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20144 2021.
[36] Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
[37] Fujimoto. S., Chang, W.-D,,
Smith, E., Gu, S. S., Precup, D., and Meger, D. For sale: State- action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36:61573-61624 2023.
[38] Ghosh, D., Singh, A., Rajeswaran, A., Kumar, V., and Levine, S. sonde--nd-cooquue reinforcement learning. In International Conference on Learning Representations, 2018.
[39] Haarnoja, T.. Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-18700 PMLR, 2018.
[40] Hafner, D., Pasukonis, J., Ba, J., and Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0410 2023.
[41] Hansen, N.. Su, H. and Wang, X. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv: 2310. 16828, 2023.
[42] Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D.. Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Amay programming with numpy. Nature, 585(7825)3357-3622 2020.
[43] Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., and Riedmiller, M. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
12

---

[44] Je, Li, K., Zang, Y., Fu, Hu, Q., Xing, J. and Cheng, J. Efficient multi-task reinforcement learning with cross-task policy guidance. Advances in Neural Information Processing Systems, 37:1 17997-1191111 18024, 2024.
[45] Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray. S., al. Scaling laws for autoregressive generative modeling. arXiv preprint 2r1iv:2010.1442000 14701, 2020.
[46] Hessel, M., Soyer, H., Espeholt, L., Czarnecki, W., Schmitt, S., and van Hasselt, H. Multi-task deep reinforcement learning with popart. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
2019.
[47] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E.,, Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et Training compute-optimal large language models. arXiv preprint arXiv:2203, 75556, 2022.
[48] Huang, H., Liu, F,, Fu, L., Wu, T., Mukadam, M., Malik, J., Goldberg, K., and Abbeel, P. Otter: A vision-language-action model with text-aware visual feature extraction. arXiv preprini arXiv:2503.0373: 2025.
[49] Huang, W., Mordatch, I., Abbeel, P., and Pathak, D. Generalization dexterous manipulation via geometry-awant multi-task learning. arXiv preprint arXiv:2111.0306 2021.
[50] Hunter, J. D. Matplotlib: A 2d graphics environment. Computing in science & engineering, (03):90-95, 2007.
[51] Kaiser, L.., Babaeizadeh, M., Milos, P., Osiński, B., Campbell, R. H., Czechowski, K., Erhan, D.. Finn, C.,, Kozakowski, P., Levine, S., al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2019.
[52] Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint urXiv:2001.0832 2020.
[53] Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):822-887, 2023.
[54] Kim, M. J.. Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.00 09246, 2024.
[55] Kingma, D. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
[56] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson,, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-.., al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
[57] Konda, R. and Tsitsiklis, J. N. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-1014, 2000.
[58] Kostrikov, I., Yarats, D.., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint (.ti::2004.16444 2020.
[59] Kumar, A., Agarwal, R., Ma, T., Courville, A., Tucker, G., and Levine, S. DR3: Value- Based Deep Reinforcement Learning Requires Explicit Regularization. arXiv preprint arXiv:2112.04716, 2021.
[60] Kumar, A., Fu, Z.. Pathak, D., and Malik, Rma: Rapid motor adaptation for legged robots. 2021.
arXiv preprint
[61] Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https :/ :tttpp::///paaae //tp//eeeeeeeeeeee//e/eeee..e.ee net/foruniid-4 ----ouumiddd---KKKaAaA
13

---

[62] Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data, Advances neural information processing systems, 33:19884-19895 2020.
[63] Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J., Subramanian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.0975:
2024.
[64] Lee, H., Lee, Y., Seno, T., Kim, D., Stone, P., and Choo, J. Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502. /5280, 2025.
[65] Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):133-1373, 2016.
[66] Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng. C., Zhang, C., Ruan, C., et al. Deepseek-v3 v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[67] Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34: 18:18888-118 18890, 2021.
[68] Liu, F., Fang, K., Abbeel, P., and Levine, S. Moka: Open-vocabulle robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation ICRA 2024.
[69] Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., OdrzygóždZ, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference Machine Learning, 2024.
[70] Mark, M. S., Gao, T., Sampaio, G. G., Srirama, M. K., Sharma, A., Finn, C., and Kumar, A. Policy agmostic rI: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint 1rXiv:2412.066m
2024.
[71] McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training. arXiv preprint arXiv:/812.06162, 2018.
[72] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King. H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 2015. URL http: h /doxxxxoiiioo org/10..0388 nature14236
[73] Mnih, V., Badia, A. P., Mirza, M. Graves, A.. Lillicrap. T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
[74] Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M. Tactical optimism and pessimism for deep reinforcement learning. Advances Neural Information Processing Systems, 34:1 1849--12633, 2021.
[75] Nauman, M. and Cygan, M. On the theory of risk-aware agents: Bridging actor-critic and economics. In JCML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2023.
[76] Nauman, M., Bortkiewicz, M., Mitos, P., Trzcinski, T., Ostaszewski, M., and Cygan, M. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, 2024. URL htttt:tst///aiii org/ wtt/pdf////poooooo/oop//g/ /4003 00514. PMLR 235:37342-37364.
[77] Nauman, M., Ostaszewski, M., Jankowski, K., Miloš, P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and samplc-efficient continuous control. arXiv preprint arXiv:2405.16158 2024.
[78] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research Journal, 2024.
14

---

[79] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray. A., al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744,
2022. 2022.
[80] Palenicek, D., Vogt, F., and Peters, J. Scaling crossq with weight normalization. arXiv preprint arXiv:2506.037. 2025.
[81] Parisotto, E., Ba, L.,, and Salakhutdinov, R. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv: /5/ 1.13442,, 2015.
[82] Park, S., Frans, K., Levine, S., and Kumar, A. Is value learning really the main bottleneck in offline rI? arXiv preprint arXiv:2406. 09329, 2024.
[83] Park, S., Li, Q., and Levine, S. Flow q-learning. arXiv preprint arXiv:2502.02533 2025.
[84]
Puterman, M. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[85] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PmLR, 2021.
[86] Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. Efficient off off-policy meta- reinforcement learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
[87] Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y. Kay, J., Springenberg, T., et al. A generalist agent. arXiv preprint arXiv:2205.06/ 2022.
[88] Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukeuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv:/511.0629/ 2015.
[89] Rybkin, O., Nauman, M., Fu, P., Snell, C., Abbeel, P., Levine, S., and Kumar, A. Value-based deep rl scales predictably. arXiv preprint arXiv:2502.043327 2025.
[90] Schwarzer, M., Anand, A., Goel, R., Hjelm, D., Courville, A., and Bachman, Data- efficient reinforcement learning with self-predictive representations. In International Confer- ence on Learning Representations, 2020.
[91] Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp. 30365-30380. PMLR, 2023.
[92] Sener, and Koltun, V. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018
[93] Sferrazza, C., Huang, D.-.., Lin, X., Lee, Y., and Abbeel, P. Humanoidhench: Simu- lated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
[94] Shapley, L. S. et al. A value for n-person games. 1953.
[95] Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387-395 PMLR, 2014.
[96] Silver, D., Schritwiesser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, I.., Lai, M., Bolton, A., al. Mastering the game of go without human knowledge. nature, 550(7676):354-355
2017.
[97] Sodhani, S., Zhang, A., and Pineau, J. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pp. 9767-9779. PMLR, 2021.
15

---

[98] Springenberg, J. T., Abdolmaleki, A., Zhang, J., Groth, O., Bloesch, M., Lampe, T., Brakel, P., Bechtle, M. E.. Kapturowski. S., Hafner, R., al. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning, pp. 46323-46350. PMLR, 2024.
[99] Sun, L., Zhang, H., Xu, W., and Tomizuka, M. Paco: Parameter compoistioom multi-task reinforcement learning, Advances in Neural Information Processing Systems, 35:21495- 21507, 2022.
[100] Sutton, S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
[101] Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L.., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint 100:008000000 2018.
[102] Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J.. Soricut, R., Schalkwyk, J., Dai, A. M.,
Hauth, A., Millican, K., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[103] Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna. J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
[104] Teh, Y., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
[105] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[106] Van Den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In International conference machine learning, pp. 1747-1756. PMLR, 2016.
[107] Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad. arXiv preprint /8/2:: 2204444 2018.
[108] Van Hasselt, H. P., Hessel, M., and Aslanides, When use parametric models in reinforce- ment learning? Advances in Neural Information Processing Systems, 32, 2019.
[109] Van Rossum, G., Drake, F. L., et al. Python reference manual, volume 111. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
[110] Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-772, 2020.
[111] Wan, W. Geng, H., Liu, Y.. Shan, Z., Yang, Y., Yi, L., and Wang, H. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialiss learning. In Proceedings of the IEEE/CV International Conference on Computer Vision, pp. 3891-3902, 2023.
[112] Wang, K., Zhou, K., Wu, R., Kallus, N., and Sun, W. The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in neural information processing systems, 36:2275-2312, 2023.
[113] Wang, K., Kallus, N., and Sun, W. The central role of the loss function reinforcement learning. arXiv preprint arXiv:2409. 12799, 2024.
[114] Xu, Y., Wan, W., Zhang, J., Liu, H., Shan, Z., Shen, H., Wang, R., Geng, H., Weng, Y., Chen, J., et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4737-4746, 2023.
[115] Xu, Z., Wu, K., Che, Z., Tang, J., and Ye, J. Knowledge transfer multi-task deep reinforce- ment learning for continuous control. arXiv preprint arXiv:2010.07494, 2020.
16

---

[116] Yang, E., Pan. J., Wang, X., Yu, H., Shen, L.. Chen, X., Xiao, L.. Jiang, J., and Guo, G. Adatask: task-aware adaptive learning rate approach to multi-task learning. Proceedings of the AAAI conference artificial intelligence, volume 37, pp- 10745-10753, 2023.
[117] Yang, J. H., Sadigh, D., and Finn, C. Polybot: Training one policy across robots while embracing variability. In Conference on Robot Learning, pp. 1955-29741 PMLR, 2023.
[118] Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL https: : /wwp/reepeeeeeeeeeeeeeeeeeeeeeee net. forum? 1dGG6-6sTvvat.
[119] Ye, S., Jang. J., Jeon, B., Joo, S., Yang, J., Peng, B., Mandlekar, A., Tan, R., Chao, Y.-.,, Lin, al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024.
[120] Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-588 5836, 2020.
[121] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pp. 1094-1100. PMLR, 2020.
[122] Zhou, Z., Peng, A., Li, Q.. Levine, S., and Kumar, A. Efficient online reinforcement learning fine-tuning need not retain offine data. arXiv preprini arXiv:2412.07762, 2024.
[123] Zitkovich, B., Yu, T., Xu, S., Xu, Xiao, T., Xia, Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. Rt-2: Vision- caaion-laguage-attionnn action models transfer web knowledge to robotic control, In Conference on Robot Learning, pp. (68331 PMLR, 2023.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We discuss experiments in Sections 4 and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the proposed method in Section 5 and Appendix
A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not propose any new theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss our experimental setting in Section and Appendix D. We submit code in supplementary material and will open source code soon.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
under the following https://github. cotp///niiiikkiiggeeuuuuuaooooot
Justification:
We release code
link:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips. ec/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips... public/guider "CodeSubmisss sssisnin wwsissssiiiyyyyyyocc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discuss these choices in Section 4 and Appendices E and F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide 95% confidence intervals calculated using bootstrapping [3].
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss the computer resources used in Appendix 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:///nsssseeeeeeeeeee http://rrpp.cpppplliii/cppo/p/ct csCuidel ines?
Answer:
[Yes]
Justification:
We conformed to the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact of our work in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any foundation model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit the open source software providers in the Acknowledgments section.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We did not LLMs beyond editing text
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https 220 for what should or should not be described.
24

---

Societal Impact
Our work does not have any specific societal impact besides improving the general capacity of value-based RL agents.
Hardware Information & Reproducibility
All experiments were conducted an NVIDIA A100 and H100 GPUs with 80GB of RAM and 16 CPU cores of AMD EPYC 7742 processor. We open-source our code under the following link:
A Limitations
Our study is restricted to model-free setup and studying the effectiveness of online multi-task learning in model-based RL might be an interesting avenue for future research. Furthermore, our discrete- action experiments we use the simple DrQ as base algorithm and our results in Atari100 are not competetive with state-of-the-art algorithms tuned for performance in Atari. such, we believe that combining our insights with approaches like BBF or EfficientZero might be promising avenue for further improvement of these state-of-the-an algorithms for Atari benchmark. We scaled both model capacity and task diversity and observed consistent performance gains, indicating that multi-task value-based RL benefits from larger networks and broader data. These results are encouraging but incomplete: they do not yet clarify when tasks help each other and when they do not. Our experiments suggest that tasks sharing the same embodiment usually reinforce one another, while suites show synergy in some cases and none in others. When synergy is absent, our method still learns, although at a slightly slower rate, whereas prior approaches often failed to converge. We believe that studying the conditions that create synergy between tasks in multi-task learning might be an important direction for future work. Finally, replay buffer memory usage is a limitation of method and multi-task learning in general, with the memory scaling linearly with the number of tasks. In practice, we find the memory footprint to be manageable in most settings. Assuming Float32, on the HB-Medium benchmark with tasks, the full replay buffer (IM transitions per task) occupies only 4.40 GB of memory.
B Background
We study an infinite-horizon Markov Decision Process (MDP) [12, 84], expressed by the tuple AS, ?)), where the state space S is continuous, while the action space A may be continuous or r,
discrete. The function r(s, a, s') gives the immediate reward received when action taken in state s leads to the next state s', and p(s' 8,a) is the corresponding transition kernel. The scalar r E (0, 1] the discount factor. policy T(a s) specifies a distribution over actions conditional on each state: its uncertainty measured by the entropy H(=(( s)). Following Haarnoja al. [39], we define the soft value in state s as the expected discounted sum of rewards augmented by the policy entropies accumulated along the trajectory:
VT (T()) Ea (rss', + (H)())))++++))))))))))))))))+))) + +>V3(5'))
The temperature parameter a 2 balances reward maximisation against exploration through entropy. The soft Q-value under policy क is:
Q" a) =
(s.a)) (r(s a.s") 1e. + (=))).
policy is optimal when it maximizes the expected soft value of the initial state distribution, according to x* arg max Eea>ppV"((su) with II the chosen policy class (for example, Gaussian policies). Soft values and soft Q-values are linked by (**)) = (aaaat-))) (x, a) a log (a x)]- In practice this expectation is often approximated with a single sample a ~ T(. s). Off-policy
actor-critic methods learn both the critic Qo and the actor Vs from transitions T = (s, a, s') stored in a replay buffer D 195, 73, 96]. The critic parameters minimize the squared soft temporal difference error 0 arg ming Er~p Qo s, a) s')) with v(r, s') + Vg(s'), where the target value
25

---

is computed with a slowly updated set of parameters 0 [107]. The policy parameters ascend the critic's estimate of the soft value & t arg max E,,DD a.1 (Q((s a) a log T s)]. These updates iterate until the policy approaches the solution [55, 57].
C Bigger, Regularized, Categorical
In this section, we discuss the implementation details associated with the proposed BRC method.
C.1 Base model
Continuous action environments. We use SAC [39] as the base model. Following previous work studying SAC sample efficiency [26, 74-76], we do not use clipped double Q-learning [36] while still using and ensemble of two critic networks, i.e. we update the critic networks according to the average output of the target networks, as opposed to taking the minimum. The proposed SAC+BRC approach is also related to the BRO algorithm. Conceptually, BRC can be understood as SAC without CDQL, with task embeddings and using BroNet critic.
Discrete action environments. We use DrQ-c as our base model [118, 3]. Following Schwarzer et al. [91], instead of performing hard target updates, we update the target network every learning step using Polyak averaging.
C.2 Cross-entropy loss via distributional RL
Categorical RL. We follow the categorical RL formulation as described in Bellemare et al. [I]] As such, instead of predicting only the mean return, we can model the full distribution of returns. Following Bellemare et al. [1 11, we define the support as a discrete vector with Natoss values denoted by 24, each defined by:
(i - 1) Unnx
Umin
24 = Umin
1,
Natoms
Natorns
At time step the parametric approximation of the true return distribution is a function of z; and the probability mass assigned to each atom (denoted as ( (g())00)) oz)). The critical observation is that return distributions obey a distributional Bellman equation [11]. The key implementation problem stems from the fact that categorical RL learns distribution with fixed support. This is particularly problematic in a multi-task setting, where each environment can have vastly different reward scales, leading to different scales of Q-values. To remedy this issue, rescale the rewards of each task according to a Monte Carlo approximation of the highest return encountered during the training.
Return normalization. The goal of reward normalization is twofold: firstly, we want to bound the returns such that there no regret stemming from using categorical bounded Q-value representation; and secondly, we want to maintain a similar scale of returns between tasks. To this end, we maintain a per-task normalization factor that we denote G2 with = 1, num tasks. training step, the normalization factor is proportional to the maximal absolute returns observed until the given training step. To this end, whenever an episode concludes for any task, we calculate the Monte-Caria returns for every state in that episode, and if the episode is truncated, bootstrap the Monte- Carlo estimate with the critic output. Then, we find the state with the biggest absolute value (i.e., max xxxx[] (s))) where (st))) represents the Monte Carlo returns for state st). Finally, if the new return estimate bigger than the current G, we update it:
a particular
G < max(max[[G((s))), G.]
We use G; to normalize the returns during model updates: when we sample new batch of data from the replay buffer, we normalize the returns according to:
r=(s,a) a) = ri(s, a) * Vmaz
(3)
G + A
26

---

Where (s, a) denotes the sampled rewards for the ith task, Vmaz denotes the maximal returns bucket for categorical RL, and X, corresponds the maximum entropy correction for ith task, which we estimate via sum of geometric sequence X; = atH:/(1 7)), with y denoting the discount factor, denoting the entropy temperature, and Hi denoting the empirical entropy of the policy conditioned on ith task embedding. Normalizing rewards according to Equation 4 guarantees that the modelled
returns are bounded by Vmaa=
C.3 Scaled Q-value model
Designing BRC, we built on previous work that studied scaling of Q-value models [61, 91, 77J. In this subsection, we detail the architectures used in our experiments.
Propricception-based control. For proprioception-banl -based control, we use the BroNet architecture. BroNet is an MLP with residual connections and layer normalization which detail in Figure 13), As shown in Figure 4, we also tested SimBa [63], follow-up work to BroNet that changes the initial layer normalization to running statistics normalization, but found BroNet to perform noticeably better (Figure 4).
N residual blocks
Dense
Layer Norm
ReLU
Dense
Lager
ReLU
Dense
Layer
Residual Addltiion
Dense
Figure 13: BroNet used in the SAC+BRC method. We use the exact residual architecture as presented in Nauman et al. [77].
Image-based control. In image-based experiments, we use the scaled Impala architecture [61, 91]. Following previous work [62, 77], add layer normalization between the encoder and Q-network modules. We show the architecture in Figure (14).
Value
Encoder
Impala
ReLU
Layer Norm
Dense
Layer Norm
ReLU
Advaniage Dense
Dense
Figure 14: Impala architecture used in the DRQ+BRC method. The architecture is based on observations made in previous works [61, 91. 77].
C.4 Task embeddings
We use an embeddings module! which we adjust by backpropagating the temporal difference loss. Following prior works [41], we constraint the LI norm of the embeddings to be equal to and leave experimentation with this setting for future work. We concatenate the states with task embeddings whenever we forward the actor or critic models according to Figure 15.
Tei Itdis
Izshedding
Noodule
trahecddgg
Trak lades
Mociie
Inehoddiggs
Creciianio
Imege
freela
Bacoding
(a) SAC+BRC
(b) DRQ+BRC
Figure 15: Our approach for concatenating observations with the learned task embeddings. We use slightly different approach for proprioceptive (left) and vision-based (right) control. In particular, in vision-based tasks, learn task-agnostic encoder by concatenating task embeddings to the output of the encoder.
As shown in Figure 15, we use slightly different approaches for proprioceptive and vision-basce control. In proprioceptive control, we simply concatenate the task embedding with the raw proprio-
'The embedding module is called Embed in JAX and Embedding in Torch.
27

---

ceptive state, so both the actor and critic are directly conditioned on the task. In Vision-based control, we first pass the RGB observation through the IMPALA encoder. We then concatenate the resulting image feature with the task embedding and feed the joint vector to the Q-network. The image encoder therefore task- aan--agno, while the Q-network conditioned on both the visual features and the task.
D Details on Experiments
In this Section, we describe the procedure used to generate each figure, as well as discuss the baselines used in the manuscript. Unless explicitly stated otherwise, we use hyperparameter configuration described in Appendix F for both single and multi-task BRC variants.
Table I: Description of model sizes shown in Figures 4, 5, 10 and 12.
Model Size
Number of blocks
Block width
R 1M
2
256
1M
2
512
16M
2
1024
64M
2
2048
256M
2
4096
R 1B
2
8192
D.1 Figures
Figure 1. We report the average final performance on every considered benchmark. We take average with respect to random seeds (we use a minimum of random seeds per method) and different tasks. We denote the training lengths directly on the graph. We normalize scores according Appendix E. We use tasks listed in Appendix E. In the aggregate graph on the left, we average the performance of SAC [39], MH-SAC [121], BRC (single-task) and BRC (multi-task) between all continuous action benchmarks shown in Figure 1, summing to 171 tasks. For the BRO algorithm, we use the fast implementation with UTD according to Nauman et al. [77].
Figure 2. We show the performance of single-task BRC initialized from scratch, as well as single- task BRC initialized with the parameters of pretrained multi-task BRC model. We use the task configurations described in Appendix E.2. To showcase the sample efficiency, we report the average performance in two time-steps denoted on the graph: in the final step of the training, well as in the middle step of the training. We normalize the scores according to procedure described in Appendix D.
Figure 3. We report metrics from the final step of the 1M step training in the HumanoidBench- Medium benchmark with tasks listed in Appendix E. We single and multi-task BRC algorithms. Besides regular BRC, we study two other variants: one using non-distributional Q-learming without reward normalization, and one using non-distributional Q-learning with reward normalization. In the left figure, we report the relative variance (a (1) for temporal difference (TD) loss, gradient norm, and policy entropy conditioned on different tasks. In the right figure, we report the average normalized final performance of single and multi-task learners. We use normalization detailed in Appendix and average over 5 random seeds for each method. We calculate 95% confidence interval using bootstrapping on random seeds [4].
Figure 4. We report final performance on the HumanoidBench--Meee Mediuu benchmark detailed in Appendix E. after IM environment steps per task. We consider single and multi-task BRC models with different critic architectures and varying width according to Table 5. In particular, we consider the vanilla MLP [39]. SimBa [63], BroNet with MSE loss [77] and our proposed BroNet with cross-entropy loss. We calculate 95% confidence interval using bootstrapping
random seeds [4].
Figure 5. We consider single-task BRC, multi-task BRC and multi-task BRC with separate heads architecture [121, 61] on the HumanoidBench-Meee trained for 1M steps. On the left Figure, we report the final performance for different widths of the critic model, as described in Table 5. In the right figure, we report conflicting gradients for the same models considered in the left figure. In particular, we report the percentage of conflicting gradients, with gradient conflict calculated according to the methodology presented in Yu et al. [120].
benchmark as described in Appendix E. 1,
28

---

There, gradient conflicts are defined by the cosine similarity between gradients stemming from different samples. Specifically, a conflict occurs when the cosine similarity is negative, indicating opposing gradient directions. When calculating gradient conflicts in multi-task setup, we calculate conflicts between transitions sampled from different tasks, whereas, in the single-task setup, we divide the batch into two parts and calculate conflicts between these two subbatches. We calculate 95% confidence interval using bootstrapping on random seeds [4].
Figure 6. We report the final performance after IM steps of training on HumanoidBench-Medinm
benchmark as described in Appendix E.1. We consider three design choices: scaled Q-value model denoted as SQ (we consider using BroNet with cross-entrope loss using 4096 units of width, otherwise using 512 units of width); cross-entropy denoted as C'E (we consider the cross-entropy loss via categorical RL and reward normalization, otherwise we use non-distributional RL with the mean squared error loss); and learnable task embeddings denoted as TF (we consider learning task embeddings by backpropagating the critic loss, otherwise we use the separate heads design). We evaluate all combinations of the described techniques, with CF + T + SQ being the base BRC model. We calculate 95% confidence interval using bootstrapping random seeds [4]. In the right figure, we report exact Shapley values, which we calculate based on the runs presented in the left part of the figure.
Figure 7. We present the learned BRC task embeddings after 500k steps of online training on the MWC benchmark. We take the 32-dimensional embeddings and extract first two principal components using the PCA algorithm.
Figure 9. We report the training curves for the HumanoidBench-Hardd benchmark as described in Appendix E. We consider single and multi-task BRC, SimBaV2 [64], BRO-Fast [77] and TD- MPC2 [41]. All baseline algorithms use the configurations proposed in their respective manuscripts. The left figure shows environment steps per task, and the right figure shows the total gradient steps when learning all tasks. We calculate 95% confidence interval using bootstrapping on random seeds [4]. BRC uses 5 random seeds.
Figure 10. We report the final performance of BRC trained on different subsets of the Beeraandddeenc--eeieee
benchmark for 1M environment steps, given different critic widths as described in Table 5. We consider subsets of 3, 6, and 9 tasks from the benchmark: the 9 tasks variant uses all tasks listed in Appendix E. (i.e. walk, stand, run, stair, crawl, pole, slide, hurdle and maze); the 6 tasks variant trains on walk, stand, stair, crawl, pole and slide tasks and is consistent with the transfer setup described in Appendix E.2; and the 3 tasks variant trains on walk, stair and slide tasks. In the left figure, we report the average final performance on the tasks used in the training of all variants (i.e. walk, stair and slide tasks). In the right, we report the performance of the models trained on 6 and 3 tasks when transferred to 3 new tasks (run, hurdle and maze). On both graphs. we show the performance of single-task BRC for comparison and calculate 95% confidence interval using bootstrapping on 3-5 random seeds [4].
HlumanoidBench-Medin
Figure 11. On the left Figure, we show the training curves averaged over 15 transfer tasks from HumanoidBeench- Medium, HumanoidBench- Hard and World listed in Appendix E.2. We report the first 500k steps of training, and show three algorithms: single-task BRC trained from seratch; single-task BRC initialized with the multi-task BRC trained on different tasks (results consistent with Figure and description in Appendix E.2); and single-task BRC initialized with the replay buffer stemming from pretraining of the multi-task BRC model (basing on the setup proposed in Ball al. [8]. In the right figure, we report the performance on 29 transfer tasks from the ShadowHand benchmark listed in Appendix E.2. In both graphs, we transfer 3 model seeds and calculate 95% confidence interval using bootstrapping on random seeds [4].
We focus on the MW+DMC benchmark. In the left, we report the final performance resulting from offline training according to Hansen al. [41]. Additionally, we report the GPU days required to finish the training, assuming an 80 GB A100 graphics unit. On the right, we report the minimal amount of environment steps required to reach fraction (50%, 75% and 100%) of the final multi-task TD-MPC2 performance on the MW+DMC benchmark [41], depending on the BRC model width. We additionally show linear fit in the logarithmic space achieved using ordinary least squares regression. We run 5 seeds per model, except for the 1B for which we run seeds.
Figure 12.
29

---

D.2 Baselines
SAC. Soft Actor-Critic [39] is an off-policy actor-critic method that maximizes the sum of expected returns and an entropy term, using twin Q-functions and a stochastic Gaussian policy. We use hyperparameters provided in Nauman al. [77] and we rerun SAC for all environments except for HumanoidBench-HHan where we use the results provided in Sferrazza et al. [93].
MH-SAC. Multi- Head SAC [120] shares common encoder but attaches separate head for each task, for both actor and critic. Thus the network produces task-specific outputs while retaining shared feature extraction. Similarily to BRC, we the hyperparameters of single-task SAC [77].
BRO. The Bigger, Regularized, Optimistic [77] builds on SAC and scales the critic network by using the BroNet architecture. BroNet architecture (Figure 13) leverages layer normalization to restrict the complexity of the learned Q-value function. We use the low UTD variant of BRO (BRO-Fast), and use the results provided in the original manuseript. For missing environments, we BRO-Fast using the original hyperparameters [77].
SimBa. SimBa architecture [63] is a follow-up work to BroNet, where authors propose to change the initial layer normalization to running statisties normalization. We run SimBa using the code from the official repository.
V2. SimBaV2 architecture [64] is a follow-up work SimBa, where authors propose to change the layer normalization to hyperspherical normalization. We run S2 using the official repository using the proposed hyperparameters [64].
TD-MPC2. TD-MPC2 [41] learns a latent dynamics model and performs model-predictivvn control by back- proaaaappggg through imagined latent trajectories. We use the results provided in the original TD-MPC2 manuscript [41] and for HumanoidBench. Hard we use the results provided in Sferrazza al. [93].
Dreamer D3eamerV3 [40] trains recurrent latent world model from pixels, imagines rollouts to update an entropy-regularized actor-critic. We use the results provided in TD-MPC2 manuscript [41].
PCGrad. PCGrad [120] multi-task method that resolves gradient interference in multi-task learning by projecting each task's gradient onto the orthogonal complement of any other task's conflicting component before the parameters are updated. We use the results provided in He et al.
[44].
Parameter-Comppositonnn Compositional learning [99] is multi-task method that factors each network's 's weights into a low-rank shared basis and task-specific composition vectors, so each task's 's parameters expressed linear combination in the common subspace. We use the results provided in He et al. [44].
PaCo. Parameter-
CTPG. Cross- Task Policy Guidance [44] is multi-task method that trains auxiliary selector that decides, at every step, which task-conditioned controller should act; the resulting trajectories provide data to update all controllers simultaneously. We use the results provided in He et al. [44].
GAMT. Geometry- Aware Multi-Task learning [49] a multi-task method that feeds a frozen PointNet encoder that converts each object's 's point cloud into a shape embedding, then conditions a shared dexterous-hand policy on that embedding, allowing manipulation behaviors to generalize across object geometries. We use the results provided in Huang et al. [49].
DDPG+HER. Deep Deterministic Policy Gradient combined with Hindsight Experience Replay [5] relabels each stored transition with alternative goals that were actually achieved, permitting goal- conditioned value and policy learning from sparse binary rewards. We use the results provided in Huang et al. [49].
E Considered Benchmarks
In this section, we describe the multi-task benchmarks that we considered our studies, as well as the normalization scheme that we used in the evaluation. We use default episode lengths for all environments, except for MetaWorld where we truncate after 200 steps following prior work [41, 77]. We do not use action repeat wrappers.
30

---

E.1 Multi-task benchmarks
MetaWorld (MW) - we use the full benchmark of 50 tasks [121]. The benchmark uses a relatively low observation and action dimensionality of |S| = 39 and |A| =4
Training tasks: assembly, basketball, bin-picking, box-close, button-presstoopdowr, button- press-topdown-walll bution-press, button-press-wall. coffee-button. coffee-pull, coffee-push, dial- turn, disassemble, door-close, door-lock, door-open, door-unlock, hand-insert, drawer-close, drawer-open, faucet-open, oope faucet-close, suucee-ppe.faaucet-looo,, hammer, handle-p poeele-ppess--idd handle-press, handle-pull- side, handle-pull, lever-pull, pick-place-wall, pick-out-of-hole pick-place, plate-slide, plate- slide-side, plate-slide slide-back, plate-slide-bace back-side,, peg tag-inneer-siiee peg- segg--uplg--sie side, soccer, stick- stick-pull, pull, push, push-wall, push-back, reach, reach-wall, shelf-place, sweep-ints prep-into, sweep, window-open open, window-close
push, stick
(HB-MEDIUM)- Here, we consider the benchmark of locomo- tion tasks without Shadow Hands [63]. We use all tasks with common embodiment with |S] 51 and |A| = 19.
HumanoidBench-Mediin
Training tasks: walk, stand, run, stair, crawl, pole, slide, hurdle, maze
HumanoidBench-Haan (HB-HARD) - Here, we consider the benchmark of locomotion and manipulation tasks with Shadow Hands [93]. We use 20 tasks with varying state dimensionality, leading [S] = 307, |A| = 61.
Training tasks: walk, stand, run, stair, crawl, pole, slide, hurdle, maze, sit- sit-simple, sit sit-hard, balance-simple, balance-hard, reach, spoon, window, insert-smale small, insert-normal, bookshelf- simple, bookshelf-hard
DeepMind Control Dogs (DMC-HARD)- - We use all dog tasks from the DMC bench- mark [101] leading to 4 common embodiment tasks with |S| = 223 and |A| = 38.
Training tasks: dog-stand, dog-walk, dog-trot, dog-run
Deep Control Humanoids (DMC-HARD)) We use all humanoid tasks from the DMC benchmark [101] leading to 3 common embodiment tasks with |S| 67 and [A] 24.
Training tasks: humanoid-stenn stand, humanoid lumanoid-rum
ShadowHand (SH) - We use all training tasks as proposed in Huang et al. [49], leading to 85 manipulation shapes with [S| 68 and |A| = 20.
Training tasks: e-toy-airplane, knife, faat-screwdriver, elephont, apple, scissors, i-cups, cup, foam-brick, pudding-box, wristwatch, padlock, power-drill, binoculars, b-lego-duplo. ps-controller, mouse, hammer, fllego-duplo, piggy-hank, can, extra-lagge-clmpp peach, lego-duplo. racquetball, tuna-fish-can, a-cups, pan, strawherry, d-toy-airplane, wood-block, mine-hole-peg
sugar-box. box, ball, torus, aitoy-arrplane chain, j-cups, c-toy-airplane, airplane, sooe-peg-test, test, water-bottle, c-cups, medium-clanp. large-marker, h-cups, b-colored-wood wood- blocks, j-lego-duplo. f-toylo.ftoy-rrrppp a-toy-airplane, toothbrush, tennis-ball, mug, sponge, Alego-duplo, phillips- screwdriver, f-cups, c-lego-duplo. d-marbles, d-cups. camera, d-lego-duplo, golf-bull, k-toy- airplane, b-cups. softball, wine-glass, chips-can, cube, master-chef-cane alarm-clock. gelatin- box, delgoo baseball, light-bulb, banana, rubber-duck. headphones, i-lego-duplo, b-toy- airplane, pitcher-base, j-toy-airplane, g-lego-duplo, cracker-box, orange, e-cups
Diverse embodiment (MW+DMC) We use the diverse embodiment benchmark proposed in Hansen et al. [41]. This benchmark uses 80 tasks from the Meta World and DeepMind Control benchmarks, with total of 12 different robot embodiments. The benchmark uses
[S] = 39 and |A| 6.
Training tasks: assembly, basketball, bin- bin-picking. box-close, button-press- press-topdown, button- press- possstoodoow--wll, button-pree press, matton-press-wll. coffee-bution button, coffee-pull, coffee-push. dial- turn, disassemble, door-close, door-lock, door-open, door-unlock, hand-insert, drawer-close, drawer-open, faucet-ppn,,, faucet-lccc- hammer, hundle-press-siee handle-press, handle-pull- side, handle-pull, lever-pull, pick-place pick-out-of-houl pick-place, plate-slide, plate- slide-side, plate-slide-bace plate-slide-buck bides,,, sig-insert--ide, peg- reg-unplug-sside soccer, stick- push, stick-pull, push, push-wall, push-backk reach, reach-wall, shelf-place, sweep-into, sweep. window- open, window-close walker- stand, walker-walk; walker-run, cheetah-run, reacher-easy, reacher-hand, acrobot- swingu,, swinggu cartpole-balanee ceapoll--bbance-pparree cartpole-swingup cartpole seaeodd-wwwnngp--ppe ball-in-cup cal--in-cup-cathh, finger-spin, finger-tur lasy.
31

---

finger-turn-uute fish-swim, hopper-stand. hopper-hop, Cheetah-run-backwear cheetah-rur- front, thteetah-run-beee cheetah-jump, walker-walk buckwards, walker-rune necteer-rnn-uueee run-buckwards. hopper- hop-haskwands, reacher-three-ee
walker-walk-uuckward,,
reacher-three-han hared, ball ball-in-cup-spe
pendulum-spit
Atari (ATARI) Here, we consider all 26 tasks from the Atari 100k benchmark [51]. We use full ALE actions space [10], leading to [S| 84 x 84 x 4 and |A| 18.
Training tasks: alien, amidar, assault, asterix, bankheist, baitlezone, boxing, breakoul, chop- percommand, crazyclimber, demonattack, freeway, frostbite, gopher, hero. jamesbond, kangaroo, krull, kungfumuster, mspacman, pong, privateeye, qbert, roadrunner, seaquest, upndown
E.2 Transfer learning benchmarks
Here, we detail the multi-task and transfer configurations used in our experiments. The difference with respect to previous subsection is that in these benchmarks we had leave from multi-task training tasks for transfer evaluation.
Meta World (MW) for transfer, we left 5 hardest tasks according to Hansen et al. [41].
Training tasks (45 tasks): basketball, bin-picking, box-close, pitton-pres-topdowee button- press-toplown-wall, button-press, button-press-wall, coffee-button, coffee-pull, coffee-push, door-close, door-lock, door-open, door-umlock, hand-insert, drawer-close, drawer-open, faucet- open, faucet-close, hammer, handle-press-side, handle-press, handle-pull-side, handle-pull, pick-place-wal wall, pick-out aal-of--hole pick-place, plate-slide, plate-slide-side plate- ciite-slide-bat back, plate- slide-back side, reg-insert-side peg pog-unplug-aan side, soccer, stick tick-push, stick-pull, push, push wall, reach, reach-wall, shelf-place, sweep-into, sweep, window-open open, window- close Transfer tasks (5 tasks): assembly, disassemble, dial-turn, lever-puil, push-back
HumanoidBench-Medhin (HB-MEDIUM)- for transfer, we left 3 hardest tasks according
to Lee et al. [63].
Training tasks (6 tasks): walk, stand, stair, crawl, pole, slide
Transfer tasks tasks): run, hurdle, maze
for transfer, we left 7 tasks categorized as "hard manipulation" in Sferrazza et al. [93].
HumanoidBench-HHun (HB-HARD)
Training tasks (20 tasks): walk, stand, run, stair, crawl, pole, slide, hurdle, maze, sit-simple, sit sit-hard, balance- simple, balance-hard, reach, spoon, window, small, insert-normal, bookshelf simpl,, bookshelf-hun
Transfer tasks (7 tasks); truck, powerlift, room, door, baskethall, push, cabinet ShadowHand (SH) - we followed the training/transfe division proposed in Huang et al. [49].
Training tasks (85 tasks): e-toy-airplane, knife, flttscrrwwdriver elephant, apple, scissors, i-cups, cup. foam-brick, pudding-box, wristwatch, padlock, power-drill, binoculars, b-lego- duplo, ps-controller, mouse, hammer. f-lego-duplo, piggy-bank. can, extra-larg-ccamp peach, a-lego-duplo. racquetbull, tuna-fish-can, a-cups, pan. strawberry, d-toy-airplane, wood-block. small-marker, sugar-box, ball, torus, i-toy-airplane, chain, j-cups, c-toy-airplane, airplane, nine-hole-peg peee--et--,,,,s,, water-bottle, c-cups, medium-clanp, large-marker, h--cps, b-colored-wood- blocks, j-lego-duplo. f-toy-airplane, toothbrush, tennis-ball, mug, sponge, k-lego-duplo, phillips- screwdriver, f-cups, duplo, d-marbles, d-cups, camera, d-lego-duplo, golf-ball k-toy- airplane, b-cups, softball, wine-glass, chips-can, cube, master-chef-cant alarn-clock, gelatin- box, h-lego-duplo, baseball, light-bulb, banana, rubber-duck, headpkones, i-lego-duplo, b-toy- airplane, pitcher-base, taarr-aase--tt--ppppppe j-toy-airplane, e-lego-duplo, cracker-bor box, orange, e-cups
Transfer tasks (29 tasks): rubiks-cube, dice, bleach-cleanser pear, e-lego-duplo, pyramid, stapler. flashlight, large-clamp, a-toy-airplane tomato momato-soup-coe soup-can, fork, cell-phone, m-lego-duplo. toothpaste, flute, stanford bunny, a-marbles, potted-meat- meat-man can, timer, lemon, utah-teapot train, g g-cups, Alego-duple, bowl, door-knob, mustard bottle, plum
Atari (ATARI) we used the tasks from Atari 100k [51] for multi-task training.
Training tasks (26 tasks): alien, amidar, assault, asterix, bankheist, battlezone, boxing. breakout, chopperconmand, crazyclimber, demonattack, freeway, frostbite, gopher, hero, james- bond, kangaroo, krull, kungfumaster, mspacman, pong, privateeye, qhert, roadrunner, seaquest, upndown
32

---

Transfer tasks (20 tasks): asteroids. atlantis, beamrider, berzerk, bowling, centipede, double- dunk, fishingderby, gravitar, icehockey, namethisgame, pitfall. riverruid, stargunner, timepilot, futankham, venture, videopinball, wizardofwor, yarsrevenge
E.3 Score normalization
We normalize the scores according to practices from previous works, with the goal of bounding returns between and I. MetaWorld, we just report unnormalized success rates (which definition or bounded). In HumannidBBenc-MMediun and HumanoidBench-Harte we report returns, which we normalize according to Equation and scores sourced from Sferrazza et al. [93]. In DMC, report returns, which normalize by diving by 1000 (according to Nauman al. [77]). In ShadowHand, we report success rates. Finally, in Atari tasks, we report human normalized returns [72] according to Equation 4 and values presented in Agarwal et al. [4].
Returns Rondom Returns
Normalized Returns Optimal Returns Random Returns
(4)
For Atari, we use the scores provided in Agarwal et al. [4]. The table below details the normalization scores used in the HumanoidBench benchmark.
Table 2: Random and success scores for HumanoidBench tasks
Task
Random Score
Optimal Score
hl-crawl-vO
272.658
700.0
hl-hurdle-vO
2,214
700.0
hl-maze-v0
106.441
1200.0
hl-pole-v0
20.090
700.0
hl hl-run-vO
2.020
700.0
hl-slide-vO
3.191
700.0
h1-stair-vO
3.112
700.0
h1-stand-v0
10.545
800.0
h1-walk-v0
2.377
700.0
hlhand-balance hard-vO
10.032
800.0
(lhand-bllane 0
10.170
800.0
blland-bssketball-va
8,979
1200.0
hand-bookshelf_haarde hard-v0
14.848
2000.0
h1
hhand-booksheff_ssmpl.....
16.777
2000.0
hhand-crawl--0
278.868
800.0
hl
hhhaddddoor-OO
2,771
600.0
hhhand-hurdle-v0
2.371
700.0
no
1,673
350.0
thanddinsert_smalll
1,653
350.0
llhand-maze-e
106,233
1200.0
hlhand-pacce package-v0
10040.932
1500.0
hhand-pole-v0
19.721
700.0
h1
17.638
800.0
hhand-push-vo
520.800
700.0
blhand-reach-v0
-0.024
12000.0
3.018
400.0
hlhand-run-v0
1.927
700.0
hhand-sit_harrdvdd
2.477
750.0
hhandd-sit siipssistc-oo
10.768
750.0
hhand-slide--v
3.142
700.0
h1 hhand-spponn-v0
4.661
650.0
hthand-stair-v0
3.161
700.0
h
11,973
800.0
hlband-truck-vd truck-v0
562,419
3000.0
Shhand-wwlk---e
2,505
700.0
hlhand-windww-w
2.713
650.0
33

---

Hyperparameters
We detail the hyperparameters used in our experiments in Tables and below. As discussed in Section 4, we use a single hyperparameter configuration across all tested tasks, showcasing robustness of our approach. We release the code at the following link: https://github:
Table 3: Hyperparameters for SAC+BRC
Table 4: Hyperparameters for DrQ+BRC
Hyperparameter
Value
Hyperparameter
Value
UTD
2
UTD
Action repeat
1
Action repeat
Embedding size
32
Embedding size
32
Discount rate
0.99
Discount rate
0.99
Optimizer
AdamW
Optimizer
Adam W
Num atoms
101
Num atoms
101
Vmin
-10
Vinin
-10
Vmax
10
VIna
10
Polyak т
5e-3
Polyak
5e-3
Target update frequency
1
Target update frequency
Weight decay
le-4
Weight decay
le-4
Batch size
1024
Batch size
256
Buffer size per task
le6
Buffer size per task
lc5
Actor learning rate
3c-4
Learning rate
1c-4
Critic learning rate Temperature learning rate
3e-4
Encoder
Impala
3e44
Encoder channels
(128, 256, 256)
Initial temperature
0.1
Pre-critic
Layer norm Dense
Target entropy
|Al/2
Critic
Actor architecture
BroNet
Critic depth
Actor depth
1
Critic width
512
Actor width
256
Data augmentation
True
Critic architecture
BroNet
Start
1.0
Critic depth
e decay steps N-step
Critic width
4096
5000
Num critics
2
G Additional Efficiency Results
We include wall-clock measurements comparing efficiency of BRO [77] and multi-task BRC across four model sizes for HumanoidBench (HB-Medium benchmark). We generate these wall-clock estimates using an uniform setup using single H100 GPU with 16 CPU cores of AMD EPYC 7742 processor. The results, summarized in the table below, confirm that multi-task learning via BRC offers significant gains in efficiency, even at reduced model scales.
Table 5: Multi-task learning via BRC offers wall-clock improvements, when compared to BRO which uses 8x smaller batch size and 64x smaller critic.
BRO-4M
BRC-MM
BRC-16M
BRC-64M
BRC-256M
Final Performance
0.57
0.65
0.69
0.83
0.95
Minutes to perform IM steps (all tasks) Minutes reach BRO final performance
1013
242
256
336
658
1013
154
263
Wall-clock improvement
6.4
7.3
6.6
3.9
H Training Curves
34

---

DMC-Hard tasks)
HamanoidBenchs- Easy (9 tacks)
HomsnoidBench- Hard (20 tacks)
0.8
1.0
0.8
Normalned rerumie
Normaltzed manallepdd retumns
Narmalized petunss
0.6
0.8
10.6
0.4
4-44171 211
0.4
0.1
0.2
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.1
0.6
0.8
1.0
0.5
1.0
1.5
2.0
Environment stops
Enoiinomment sleps
slops
MetaWorld tasks)
Shadow Hand (85 tasks)
MW-DMC (8D tacks)
1.0
1.0
DET1
Sucees rete
0.8
Success ralc
Nanualzed scote
0.8
0.6
0.6
CENGOITS
0.1
0.4
0.2
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.2
0.1
0.6
0.8
1.0
0.2
0.1
0.5
Enviromment stops
slops
Enviromment stops
Figure 16: Aggregate training curves. Y-axis denotes the performance metric, and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping.
Ous (MT)
Ous (ST)
MEI SAC (MT))
SAC(ST)
SanBeV2 (ST)
TD MPC2 (ST)
BRO Fast (ST)
hl-stand
h1-mum
hl-craw]
1000
500
600
400
600
400
400
200
55
1.0
0.0
0.0
0.5
1.0
0.5
1.0
bil-polo
bi -il-siid
bl-hurdle
bl mmzz
400
750
200
300
000
400
200
103
Figure 17: Unnormalized training curves for HumanoidBenci-MMciiie benchmark. Y-axis denotes sum of episodic returns and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping
35

---

(MI))
(SC
DDPMPC2
biland-war walk
hthaand-stadd
hibaand-rnn
blband-stair
hihand-crawe
1000
790
300
100
400
200
600
250
250
200
2.0
0.0
2.0
2.0
bllaand-pole
b landd-sidee
bi hibamd--uurlle
11 Thand-maae
hltand-sit simple
300
1000
750
750
300
100
200
250
250
bihandd-st
hhhnd-balanes simple
hlhand-halance hard
bihand-rench
Haand-spoon
200
5000
000
250
100
50
100
0.0
1.2
2.0
0.8
1.0
2.0
0.0
1.3
2.0
0.3
1.0
2.0
0.0
I.laandwwiie window
hilannd insert small
hand inseiiinseeiiieeeeeeeeeee pormal
hibandd bookchelf simple
h.Baand bookshelf hard
800
200
800
400
000
600
200
100
400
200
0.0
1.0
2.0
0.0
0.0
1,0
2.0
0.0
1.0
2:0
0.0
20
Figure 18: Unnormalized training curves for the HumanoidBench--Hi benchmark. Y-axis denotes sum of episadic returns and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping.
Oum (MT)
Oues (ST)
MIESAC (MT)
SACISD)
BRO-Past (ST)
TD7 (ST)
SimBoV2 (8T)
humanoid-stan stand
homsnnod-waik
humsnnodd-run
dog-stand
dog-walk
1000
000
750
750
000
400
500
600
250
200
200
250
0.0
0.25
0.5
0.0
0.25 dog--nun
0.5
0.0
0.25
0.5
0.0
0.26
0.6
0.0
0.25
0.5
dog--to
600
500
400
200
200
Figure 19: Unnormalized training curves for the DMC-Hard benchmark. Y-axis denotes sum of episodic returns and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping.
36

---

Ours (MT)
Ours (ST)
MH-SAC
SAC (ST)
BRO-Fast (ST)
TD-MPC2 (ST)
acsabbly
baaksthall
Nospicking
bon-closc
bulun poa.---tssllplloeeedd..
100
153
355
8:0
030
350
850
8:5
025
1,26
0.6
50
0.25
65
03
1.29
05
0,t codoo-pull
0.20
03
0.25
buttes-presi
bustoo-rraawwlll
coffze-brtton
caflo-puhh
1.00
1.00
1.80
1.00
1.0
3.50
8:0
0.00
0.00
0.50
020
0.5
0.35
1:35
0:33
doon-3lse
door-opes
-100
1.80
-1.00
-1.00
130
0:0
8:30
030
850
8:00
95
4:25
4.5
9:35
0.25
Grawepclloe
drower-spen
faucct-rpen
Gascci-cloee
banall-peco-iide
1.00
1.0
1.80)
1.00
153
3.5
$30
430
410
8M
3:25
6:00
436
8:0
05
0.25
68
3.24
05
0.26
0.26
13.2
handle- press
puidi--pllliide
handt-ppll
lever-pll
pick-plac--ell
pick-oat
1.00
1.50
1.00
3.5
810
0.60
0.10
0:2
0.25
1:35
3.35
p4dk place
plato
site-slide-sid
plate
pag:
1.00
1.80)
1.00
1.5
8:0
8:0
0300
0:50
823
0.25
3:35
0.35
pog Aos-oppigg-ee
soccer
posk
stclepooll
prahhwwall
1.00
1.00)
1100
153
8.50
0:50
0.00
0.50
6:20
4.25
3.35
0.25
puib-back
sealh-wall
place
sweep
180
1.50
1.00
1.801
1.60
818
8.80
850
625
0.25
133
open
1.00
Figure 20: Unnormalized training curves for the MetaWorld benchmark. Y-axis denotes the success rate
and X-axis denotes cnvironment steps. We report 95% confidence interval calculated via bootstrapping.
37

---

Ours (MT)
Ours (ST)
MIT SAC (MT)
SAC (ST)
J. argane
degun
LOD
1.00
320
poNr
0.0
324
DELL Iege dlep
20th
8. deta
TaL, 160
L
pV
bury
Jaria:
redd
PAB Jux
Lay. Arblax
chick
vart
1.80
331
S.A
X-
sprage
S.
14
15
14
1.00
3.1
14
14
14
3.1
14
p4
0.4
E1
330
32
15
3.0
Figure 21: Unnormalized training curves for the Shadow benchmark. Y-axis denotes the success rate and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping.
38

---

Ours (MT)
Ours (ST)
MH-SAG (MT)
SAC (ST)
Colke sted
Add
dlwnsk
026
625
0,55 Eageraaxs mar
9.25
9.25
Sape-tan say
hepperbop
Chandrren Bucknad
Chestbbee
1000
chacubera
valkewokk bndonart
valeevra backwards
tepaoiian backnards
raadhan-err hard
218
2.36
1:24
236
baokcbbt
bespiciigg
outicondten
calloogu
671
135
E13
033
2.55 dvenegoon
335
223
695
336
Axnreopm
Eder
6m
276
605
65
6d5
1:5
925
3.5
80
135
625
0,55
65
035
330
035
615
plate Wie-diillak
Auta- 122-sidohuk-ddk
454
6.5
270
0.15
0/5
625
68
0J5
3.25
1.5
628
0,55
65
605
4.00
0.5
605
APUp
675
025
570
2.0
025
02
025
Figure 22: Unnormalized training curves for the MW+DMC benchmark. For MetaWorld tasks, Y-axis denotes the success rate whereas for DMC tasks it denotes sum of episodic returns, X axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping.
39

---

Table Atari-100k benchmark scores.
Game
Human
SimPLe
DER
SPR
DrQ
Ours (ST)
Ours (MT)
Alien
7127.7
616.9
739.9
841.9
863.2
1005.8
906.6
Amidar
1719.5
88.0
188.6
179.7
137.8
192.1
147.8
Assault
742.0
527.2
431.2
565.6
579.6
629.0
503.4
Asterix
8503.3
1128.3
470.8
962.5
763.6
1442.2
1285.4
BankHeist
753.1
34.2
51.0
345.4
232.9
959.7
104.5
BattleZonc
37187.5
5184.4
10124.6
14834.1
10165.3
14905.0
20316.0
Boxing
12.1
9.1
35.7
-0.1
45.1
Breakout
30.5
16.4
1.9
19.6
19.8
24.9
20.6
ChopperCommant
7387.8
1246.9
861.8
946.3
844.6
1867.5
1848.4
CrazyClimber
35829.4
62583.6
16185.3
36700.5
21539.0
79276.5
94606.8
DemonAttack
1971.0
208.1
508.0
517.6
1321.5
503.2
479.6
Freeway
29.6
20.3
27.9
19.3
20.3
31.6
28.4
Frostbite
4334.7
254.7
866.8
1170.7
1014.2
1413.9
715.7
Gopher
2412.5
771.0
349.5
660.6
621.6
512.0
1477.6
Hero
30826.4
2656.6
6857.0
5858.6
4167.9
10228.0
7472.3
Jameshond
302.8
125.3
301.6
366.5
349.1
249.0
288.6
Kangaroo
3035.0
323.1
779.3
3617.4
1088.4
3644.0
1184.0
Krull
2665.5
4539.9
2851.5
3681.6
4402.1
5277.8
6764.8
KuntFuMasstr
22736.3
17257.2
14346.1
14783.2
11467.4
6310.5
13723.2
MsPacman
6951.6
1480.0
1204.1
1318.4
1218.1
6310.5
1682.1
Pong
14.6
12.8
-19.3
-5.4
-9.1
1.9
0.2
PrivateEye
69571.3
58.3
97.8
86.0
3.5
42.7
99.8
Qbert
13455.0
1288.8
1152.9
866.3
1810.7
4781.2
1954.6
RoadRunner
7845.0
5640.6
9600.0
12213.1
11211.4
11887.0
18021.2
Seaquest
42054.7
683.3
354.1
558.1
352.3
434.6
635.2
UpNDown
11693.2
3350.3
2877.4
10859.2
4324.5
5290.3
7102.2
Mean
1.0
0.443
0.285
0.616
0.465
0.667
0.849
Model transfer
Data trans
From scratch
assembly
pusb-back
lever-pull
0.75
0.75
0.70
0.50
0.00
0.00
0.50
0.50
0.25
025
0.25
0.25
0.86
6.0
0.0
0.25
0.5
5:25
0.5
53
6.25
0.3
3.25
05
0.0
0.5
hi-mmaee
B1 A1-hunlie
h 11-
hilband-track
h: Thhnn-pwwwrii
300
1500
200
200
400
1250
1000
100
100
760
h1hannd-cm rolaadd-ooo
hi hilhnddde door
hihand basketball
hand band-push
bi hand cabinet
500
600
800
40
100
0.0
1.0
255
Figure 23: Unnormalized training for the transfer tasks. For MetaWorke tasks, Y-axis denotes the success rate whereas for other tasks denotes sum episodic returns. X-axis denotes environment steps. report 95% confidence interval calculated via bootstrapping.
40

---

Embedding-tuning transfer
88
Test tasks
Training embeddings
Figure 24: Effectiveness of pretrained embeddings. We show the performance of using pretrained embeddings manipulate new shapes. Interestingly, most of lesting shapes can be manipulated via majority pretrained embeddings, showcasing that the individual policies do not overfit to the particular task.
41


Reviews:
[{"review_id": "ghc9GbkpZu", "reviewer": "Reviewer_yjFR", "summary": "The paper presents the BRC method, which scales value-based reinforcement learning (RL) models to achieve multi-task learning and higher efficiency both in terms of sample-wise and computation-wise. The paper proposes to leverage pure RL path rather than imitation learning or distillation to acquire multi-task policies by 1. cross-entropy based value learning; 2. high-capacity value function; 3. task-embedding. The authors did comprehensive and concrete experiments to demonstrate the efficacy of the proposed method. In conclusion, BRC enables scalable, efficient multi-task RL with high-capacity models and improved transfer learning.", "questions": "1. How do you define gradient conflicts? Why is it still high even in the single-task learning setting, as demonstrated in Fig.5 right?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Scaling value network has been a popular topic that hopes to make RL agents generalizable and perform better in multi-task settings. This paper is the first one to make impressive improvements on empirical results by conducting evaluations on almost all mainstream benchmarks.\n2. Categorial learning + return normalization + task embedding combination is reasonable for multi-task design, especially for diverse reward margins.\n\nWeakness:\n1. This paper didn't include model-based RL algorithms into base models, which is expected because SAC+BRC has achieved stronger performance than the most advanced model-based RL algorithms on DMC. However, the author didn't take this into consideration.\n2. Authors are selective about Atari baselines. It seems they exclude some of the SoTA model-based RL baselines like EfficientZero, which could lead to overclaiming about the improvements on the Atari-100k benchmark.", "was_revised": true, "final_justification": "Authors' rebuttals mainly resolved my concerns. They didn't claim SoTA performance on Atari, although they were selective on baselines.  Overall, I think this paper has been rigorously and comprehensively evaluated on common benchmarks, where the strong performance proved its efficacy. They also make clarifications on the definitions of gradient conflicts. I would recommend this paper to be accepted.", "rebuttal": "We thank the reviewer for helping us to improve the quality and impact of our work. We are also very glad that the reviewer acknowledges the importance of our empirical results. To address the reviewer’s concerns and improve the quality of our work, we have made a few changes to our manuscript: \n\n1. We discuss our focus on the model-free setting in the limitations section, clarifying that the scope of this work does not extend to model-based approaches due to our goal of understanding how the proposed design choices impact scalability of model-free RL\n\n2. We clarify that our SOTA results are in the continuous control setting, which is the primary focus of our work, and do not extend to discrete control. The discrete control (Atari) experiments are included as secondary validation, meant to demonstrate that the same design principles generalize to multi-task learning in discrete domains, even when using a simple, compute-efficient baseline. \n\n3. We expand our discussion on gradient conflict computation, including how we define and measure gradient conflict. These clarifications appear in Section 3 and the Appendix.\n\n4. We add results measuring wall-clock performance of our proposed approach as compared to BRO. These results are supposed to better contextualize the efficiency improvements stemming from multi-task learning.\n \nPlease find our detailed rebuttal below:\n\n> Additional results\n\nWe would like to point the reviewer’s attention to newly added evaluations. To better ground our considerations on compute efficiency of multi-task learning, we now include wall-clock measurements comparing wall-clock times for BRO and multi-task BRC across four model sizes for HumanoidBench (HB-Medium benchmark). We generate these wall-clock estimates using an uniform setup using a single H100 GPU with 16 CPU cores of AMD EPYC 7742 processor. The results, summarized in the table below, confirm that multi-task BRC offers significant gains in efficiency even at reduced model scales. As discussed in Sections 4 and 5, these wall-clock improvements stem from improved sample efficiency, as well as performing less gradient steps over the course of the training and are proportional to the number of tasks trained at once.\n\n|                                           | BRO (4M) | BRC (4M) | BRC (16M) | BRC (64M) | BRC (256M) |\n|-------------------------------------------|----------|----------|-----------|-----------|------------|\n| Final performance                         | 0.57     | 0.65     | 0.69      | 0.83      | 0.95       |\n| Minutes to perform 1M steps in all tasks  | 1013     | 242      | 256       | 336       | 658        |\n| Minutes to reach final performance of BRO | 1013     | 157      | 138       | 154       | 263        |\n| Wall-clock improvement                    | 1        | 6.4      | 7.3       | 6.6       | 3.9        |\n\n> This paper didn't include model-based RL algorithms into base models...\n\nOur vision is to enable RL algorithms to successfully train large models on diverse multi-task datasets. We agree that the natural next step would be to extend our method to other settings, such as model-based or on-policy RL. For this study, however, we had to balance the scope with our compute budget, and we decided to focus on model-free settings. Model-free RL is a well-established setup, with previous works showing that multi-task learning requires techniques like gradient projection or distillation of single-task policies. This raises the question whether online value-based RL can successfully scale to many tasks at all without using multi-task-specific techniques. Our work shows that a simple algorithm can be used for both single and multi-task learning, with significant gains when used in a multi-task context even when compared to model-based TD-MPC2. Furthermore, we note that a lot of RL applications rely on a streamlined process in which a model is learned completely separately from the policy, and the policy is learned via a general model-free algorithm [1,2]. As such, we believe that our results are a good starting point for future works investigating model-based extensions to our approach. **We expanded our limitation section** to underscore that our results are restricted to the model-free setup. Furthermore, if the reviewer believes that adding model-based results would improve the reader’s experience, **we are happy to add basic experiments to the camera-ready version**.\n\n[1] Kaufmann, Elia, et al. \"Champion-level drone racing using deep reinforcement learning.\" Nature 2023\n\n[2] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" NeurIPS 2022\n\n> Authors are selective about Atari baselines...\n\nWe thank the reviewer for pointing this out. Our primary focus is on demonstrating that training with diverse, multi-task data combined with specific design choices (cross-entropy, scaled critic, and task embeddings) improves value-based RL performance, especially in continuous action problems, where we do claim state-of-the-art results. In contrast, our Atari100k (discrete action) experiments serve as a secondary validation, aimed at testing the generality of our findings rather than achieving SOTA. To keep these experiments compute-efficient, we use a simple baseline (Efficient DQN + DrQ) and focus on comparing its multi-task and single-task variants. We acknowledge that we do not include stronger model-based baselines such as EfficientZero, and we agree that this limits the scope of our discrete-action claims. To make our intentions clear for the reader, **we now clarify this in the paper** by adding the following to the results section: “The results on the Atari100k benchmark show that the proposed design choices allow the agents to benefit from multi-task learning. While in our experiments we use the simple DrQ as base algorithm, combining these insights with approaches like BBF or EfficientZero might be a promising avenue for further improvement of these state-of-the-art algorithms.” We also add a note in the limitations section acknowledging that our discrete-action results are not competitive with the strongest existing baselines and serve as a proof of concept. \n\n> How do you define gradient conflicts? Why is it still high even in the single-task...\n\nWe define gradient conflicts as in [3, page X], where they are defined by the cosine similarity between gradients stemming from different samples. Specifically, a conflict occurs when the cosine similarity is negative, indicating opposing gradient directions. When calculating gradient conflicts in multi-task setup, we calculate conflicts between transitions sampled from different tasks, whereas, in the single-task setup, we just divide the batch into two parts and calculate conflicts between these two subbatches. **We now clarify this definition in the main text** to avoid ambiguity. Regarding why gradient conflicts still appear in the single-task setting (as shown in Fig. 5, right): this stems from the observation that even within a single MDP, diverse data can give rise to competing optimization signals. As discussed in Section 3, using task embeddings can unify a multi-task setup into a single-task formulation by treating the multiple tasks as variations within a single MDP. For example, a point-mass agent with different goal locations can be considered a single-task MDP with high intra-task diversity, or a multi-task MDP where goal position can be encoded into a \"task embedding\". This intra-task diversity can lead to gradient conflicts if samples (e.g., reaching widely separated goals) produce gradients that point in conflicting directions. Therefore, gradient conflicts are not exclusive to multi-task learning and they can arise in single-task settings whenever the task exhibits sufficient structural diversity. Figure 5 empirically supports this by showing that the single-task setting still exhibits non-zero levels of gradient conflicts.\n \n[3] Yu, Tianhe, et al. \"Gradient surgery for multi-task learning.\" NeurIPS 2020\n\n\nWe thank the reviewer for their detailed feedback. Please let us know if the above rebuttal clarifies our results, especially with respect to the Atari benchmark. We also would like to point the reviewer’s attention to the added results on multi-task training efficiency, highlighting the practical impact of our work. Please let us know if there are any remaining issues. If not, we kindly ask the reviewer to consider adjusting their score."}, {"review_id": "s6XF29LtqG", "reviewer": "Reviewer_wYfY", "summary": "The paper introduces BRC, a recipe that lets online value-based RL scale to billion-parameter models and hundreds of tasks without offline data or expert distillation. BRC combines three design choices: (1) a scaled, regularised residual critic (BroNet) that remains stable at large widths, (2) replacing MSE TD loss with cross-entropy via categorical Q-learning to equalise gradients across tasks with disparate reward scales, and (3) a single network conditioned on learnable task embeddings rather than separate heads. Multi-task BRC achieves comparable performance to task-specific specialists while performing up to 40 times fewer gradient steps, and the resulting networks transfer to unseen tasks with markedly better sample efficiency.", "questions": "1. Can BRC handle the observation noise, latency, and dynamics shift typical of physical robots?\n\n2. Why does replacing the MSE TD loss with cross-entropy categorical Q-learning reduce gradient conflict across tasks?\n\n3. How do BRC's gains scale when the critic is restricted to parameters less than 10M or when only one or two GPUs are available?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "***Strengths***:\n\n1. BRC is evaluated on 283 tasks across five benchmarks. Robust ablations and Shapley analyses pinpoint the scaled BroNet critic as the main source of gains, lending credibility to the claim that careful architectural and loss design, rather than task-specific tuning, drives the performance.\n\n2. The paper combines (i) a scaled, regularised residual Q-network (BroNet), (ii) cross-entropy TD loss via categorical Q-learning to equalise gradients across tasks, and (iii) learnable task embeddings in place of multiple heads. This triad is argued to act synergistically—removing any component degrades results.\n\n***Weakness***:\n\n1. Training billion-parameter critics on 283 tasks almost certainly requires very large GPU budgets, but the paper gives no wall-clock times, FLOPs, or carbon-footprint estimates. Without such data, reviewers cannot judge whether the reported efficiency (fewer gradient steps) translates into practical savings in energy or hardware.\n\n2. It remains unclear whether resource-constrained settings (e.g., on-device robotics) reap any benefit, or whether BRC simply overpowers interference by brute force. Additional results at small/medium scales or a theoretical account of why cross-entropy stabilises TD learning would strengthen the contribution.\n\n3. All benchmarks are simulators with well-shaped dynamics; no physical robot or real-data test is attempted. Moreover, although the paper briefly contrasts online vs. offline variants, deeper analysis of offline RL is missing.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helping us to improve the quality and impact of our work. We are happy that the reviewer found our experiments to be robust. To address reviewer’s concerns and further strengthen the manuscript, we propose the following changes:\n\n1. We include new results comparing the wall-clock of BRC using different model sizes and BRO. These results demonstrate that multi-task BRC significantly reduces training time while improving the final performance.\n2. We add experiments evaluating BRC under noisy observations and stochastic dynamics. Both single and multi-task BRC outperform BRO, highlighting the applicability of our approach to noisy tasks.\n3. We have made a number of improvements to the manuscript based on reviewer feedback. These are detailed in our point-by-point responses.\n\nPlease find our responses below:\n\n> Multi-task BRC achieves comparable performance to task-specific specialists\n\nAs shown in Figures 1, 2, 4 and 9, multi-task BRC consistently outperforms single-task specialists across a wide range of tasks and specialists. To the best of our knowledge, this is the first work to demonstrate that a multi-task value-based RL training can lead to substantial performance improvements over its single-task counterpart in an online setting. \n\n> Training billion-parameter critics...\n\nWhile we present results for a 1B-parameter model in Figure 12, all core comparisons against baselines were conducted using a 250M parameter model to ensure reproducibility and accessibility. This detail is discussed in Appendices D and F, and **we clarify it more prominently in Section 4**, with explicit references to the relevant appendices. We hope that this change will improve the clarity of our work.\n\n> The paper gives no wall-clock times, FLOPs, or carbon-footprint estimates...\n\nWe thank the reviewer for this suggestion. To better illustrate the practical efficiency of multi-task learning, **We now include wall-clock measurements** comparing efficiency of BRO and multi-task BRC across four model sizes for HumanoidBench (HB-Medium benchmark). We generate these wall-clock estimates using an uniform setup using a single H100 GPU with 16 CPU cores of AMD EPYC 7742 processor. The results, summarized in the table below, confirm that multi-task BRC offers significant gains in both performance and efficiency, even at reduced model scales, reinforcing its scalability and real-world applicability. As discussed in Sections 4 and 5, the wall-clock improvements stem from improved sample efficiency paired with performing less gradient steps over the course of the training and are proportional to the number of tasks trained at once. \n\n|                                        |BRO-4M|BRC-4M|BRC-16M|BRC-64M|BRC-256M|\n|----------------------------------------|------|------|-------|-------|--------|\n| Final performance                      | 0.57 | 0.65 | 0.69  | 0.83  | 0.95   |\n| Minutes to perform 1M steps (all tasks)| 1013 | 242  | 256   | 336   | 658    |\n| Minutes to reach BRO final performance | 1013 | 157  | 138   | 154   | 263    |\n| Wall-clock improvement                 | 1    | 6.4  | 7.3   | 6.6   | 3.9    |\n\nAs follows, multi-task BRC offers wall-clock improvements, even when compared to BRO which uses 8x smaller batch size and 64x smaller critic. We also note that the FLOPs per gradient and simulation step are identical between the single and multi-task BRC (250M parameters, 1024 batch size, and simulation of all tasks). Therefore, the “40x improvement in gradient steps” can also be interpreted as a “40x improvement in FLOPs”. Our result highlights that the data-compute trade-off observed in supervised learning [1] persists value-based RL where multi-task training uses more diverse data to reduce compute cost per performance unit, especially in large-scale settings.\n\n[1] Kaplan, Jared, et al. “Scaling Laws for Neural Language Models”. arXiv\n\n> It remains unclear whether resource-constrained settings reap any benefit...\n\nWe appreciate the reviewer’s point. To address the concern about resource-constrained settings, we note that several of our experiments (Figures 4, 5, and 10) are conducted using small models (<20M parameters). In particular, as shown in Figure 5 (left), our approach outperforms single-task specialists even when the critic model has fewer than 5M parameters (using ~20MB of memory). In all experiments we keep the actor network fixed at fewer than 200k parameters, incurring negligible memory overhead. This demonstrates that the benefits of BRC are not solely due to large-scale compute and that our method offers gains even under modest resource budgets. Furthermore, given the small size of the actor network, inference-time costs remain equivalent to SAC or PPO, making BRC practical for deployment scenarios such as on-device robotics. To better communicate this result to the reader, **we add a discussion of limitations of large models** to the limitations section, as well as **expand the discussion of performance with smaller models** in Section 5. We also do not believe that BRC overcomes interference simply through “brute force.” As shown in Figure 6 and as noted by the reviewer, all components contribute to performance gains, with removal of each component resulting in a statistically significant performance drop. \n\n> Theoretical account of why CE stabilises TD learning would strengthen the contribution\n\nWe thank the reviewer for this suggestion. To make this clearer for readers, **we expand our discussion of previous works discussing the importance of loss functions in TD learning** [2,3] in Section 3. \n\n[2] Wang, Kaiwen, et al. \"The benefits of being distributional: Small-loss bounds for reinforcement learning.\" NeurIPS\n\n[3] Wang, Kaiwen, et al. \"The central role of the loss function in reinforcement learning.\" Statistical Science \n\n> No physical robot, deeper analysis of offline RL... \n\nWe thank the reviewer for raising this point. Our goal in this work is to investigate whether RL models can effectively learn from diverse multi-task data in the online learning setting. Given resource constraints, we focused on simulation-based benchmarks, which allow controlled and repeatable evaluation across a wide range of tasks. To this end, we evaluated BRC across over 200 simulated tasks. We believe this constitutes one of the broadest evaluations in value-based RL to date, and provides a strong foundation for future works. Regarding offline RL, we agree that a deeper analysis would strengthen the paper. While our offline experiments are smaller than online counterparts, they suggest that our design principles also transfer to offline learning. We are particularly excited about applying BRC to real-world robotic learning, where sample efficiency and model transfer (as shown in Figure 10) are especially valuable and we plan to explore this direction in follow-up work. To point the reader to the potential of such experiments, **we expand the discussion of future work in the conclusion section**. \n\n> Can BRC handle noise...\n\nWe thank the reviewer for this question. To evaluate BRC’s robustness to real-world conditions such as observation noise and stochastic dynamics, **we added a new experiment** using two environment wrappers: observation wrapper with noise ($\\sigma$ = 0.05) added to each observation, simulating noisy state estimation; and action wrapper with noise ($\\sigma$ = 0.05) applied to actions before they are executed, simulating stochastic or imprecise actuation. We summarize the results in the table below. Across both settings, BRC maintains strong performance and consistently outperforms BRO, indicating its robustness to moderate noise. We are happy to rerun this experiment with noise level that the reviewer finds appropriate. \n\n|                    |BRC(multi)|BRC(single)|BRO(single)|\n|--------------------|----------|-----------|-----------|\n| No noise           | 680      | 574       | 487       |\n| Noisy observations | 638      | 586       | 494       |\n| Noisy dynamics     | 417      | 373       | 338       |\n\n> Why does CE reduce gradient conflict across tasks?\n\nWe thank the reviewer for this question. While our results in Figure 5 (right) suggest that critic model size and task embeddings have a primary impact on reducing gradient conflicts, we also observe that replacing the MSE with a CE contributes to stabilizing training dynamics in multi-task settings. Although we do not claim that CE loss directly minimizes gradient conflict, we show in Figure 3 that the CE leads to more consistent gradient norms across tasks, stabilizing learning. Overall, while CE may not directly align gradients across tasks, it contributes to a more stable optimization landscape. We see this as an important area for future theoretical and empirical analysis, and we hope our results help motivate such work.\n\n> How do BRC gains scale when restricted to less than 10M or when only one or two GPUs are available?\n\nWe evaluate BRC performance across several critic sizes (1M, 4M, and 16M) in Figure 4. These results show that BRC delivers substantial performance gains even in low-resource settings. For example, for 1M parameters, multi-task BRC outperforms single-task SAC (denoted as \"vanilla\") by over 3x, and single-task BRO (denoted as \"BroNet (MSE)\") by over 2x. Regarding GPUs, all experiments in our paper were run on a single A100/H100 GPU, and no experiments required multi-GPU parallelism. We have also developed an optimized implementation of BRC that supports training on consumer-grade GPUs, and we will release this code to ensure accessibility.\n\nWe thank the reviewer again for their helpful suggestions. We hope that the rebuttal clarified all issues, and that the added results strengthen our claims on the efficiency and practicality of BRC. If so, we kindly ask the reviewer to consider changing their score accordingly. Please inform us if we can further clarify any issues."}, {"review_id": "0RWTAKoUSS", "reviewer": "Reviewer_9y5s", "summary": "The paper proposes a new state-of-the-art method on multi task RL, spanning across continuous/discrete action space and state/vision-based input space. Their main sources of improvement were (1) using a well-normalized model (BRO), (2) parameter scaling the model, (3) employing cross-entropy loss, and (4) learnable task embedding. The normalized model allowed its size to be scaled up to 256x (Figure 4), while the cross-entropy loss effectively unified the scale of the loss between tasks and thus their gradient magnitude.\n\nThe authors extensively verify their method's effective in 7 multi-task benchmarks (DMC, HB, Metaworld, Atari, ...), showing that (1) the proposed method performs on-par with previous single task SOTA models and (2) surpasses those methods in multi-task setting. This leads to significant training speedup, taking 2x less samples and 40x less gradient steps to achieve the same performance as prior works.\n\nExperiments were also made in pretrain-then-finetune regime (transfer learning), where they consider three types of transfer: full-finetuning (model transfer), tuning embedding only, and creating expert transitions (data transfer), and show that they outperform from-scratch models as well as prior works on transfer learning.", "questions": "1. I think the compute efficiency in Figure 9 (right) is not fairly compared to prior works. To my understanding each gradient step would take significantly different compute to perform, depending on batch size, buffer size, parameter size, etc. In this case I think the x-axis should be either FLOPs or wall-clock time.\n\n2. The data transfer method sound more like JSRL [1] (or WSRL [2] but that's already cited) than RLPD.\n\n3. In Figure 10 (right), juding by the closing gap between 3 task and 6 task as parameter scales, it seems like parameter scaling and task (data) scaling are not complementary, which is surprising. What do you think of this result? In that sense, I am curious of the results with 9 tasks as well.\n\n4. Also just curious: in Figure 11 (left), how would embedding-tuning transfer perform compared to other transfer methods?\n\n[1] Jump-Start Reinforcement Learning., Uchendu et al.\n[2] Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data., Zhou et al.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n- Extensive experiment clearly shows the effectiveness of the method\n- They provide sufficient analysis that justify each component well, as well as exhaustive ablation of their 3 components.\n- The writing is clear and straightforward, easy to understand.\n\nWeakness\n- The method likely suffers from heavy memory constraint, due to large replay buffer (1M per task * num tasks) and large batch size (1024).\n- The components are relatively common in the RL field, although I don't think this as a huge weakness.", "was_revised": true, "final_justification": "The paper makes great contribution to the RL community by show that multi-task RL has superior scaling properties than when learning from individual tasks, and provides a simple method (learnable task embedding) to avoid gradient interference between tasks. Scaling the model size in RL has been one of the major challenges recently, and I think this paper is a meaningful step in this direction.", "rebuttal": "We thank the reviewer for their help in improving our manuscript. We are glad that the reviewer finds our experiments convincing. Following up on reviewer’s suggestions, we made the following improvements to our work:\n\n1. We include new results comparing the wall-clock training time of our method using different model sizes and BRO on the HumanoidBench tasks. These results demonstrate that multi-task BRC significantly reduces training time while improving the final performance.\n2. We have made a number of textual and structural improvements to the manuscript based on reviewer feedback. These are detailed in our point-by-point responses.\n\nPlease find our responses below:\n\n> The method likely suffers from heavy memory constraint\n\nWe agree that memory usage from the replay buffer is a limitation of our method. However, in practice, we find the memory footprint to be manageable in most settings. Assuming Float32, on the HB-Medium benchmark with 9 tasks, the full replay buffer (1M transitions per task) occupies only 4.40 GB. On MW with 50 tasks, the total memory usage reaches 16.65 GB. Such requirements are often within the capacity of standard research or cloud hardware. In scenarios with a significantly larger number of tasks, the memory cost would scale linearly, but practical workarounds exist. For instance, a lazy loading or prioritized task sampling strategy could reduce the need to keep the full buffer for all tasks in memory at once. **We now mention this in the limitations section** to make this trade-off more clear.\n\n> The components are relatively common in the RL field, although I don't think this as a huge weakness.\n\nWe agree that the components we use are individually well-established in the RL literature. Following the bitter lesson we chose simple, scalable components to maximize performance while minimizing complexity. We think that the novelty of our work lies in how these elements are combined and in the new perspective we offer on their interaction in the context of stabilizing multi-task RL. Our empirical results (e.g. Fig 6) demonstrate that their combination leads to synergistic effects not observed when using each component in isolation.\n\n> I think the compute efficiency in Figure 9...\n\nWe thank the reviewer for this suggestion. We agree that gradient steps alone do not fully capture compute cost, since they can vary significantly depending on batch size, model architecture, and hardware utilization. In response to this, **we have added a supplement that includes wall-clock time measurements**, along with a discussion on the trade-offs between different compute metrics (gradient steps, FLOPs, and wall-clock time). Our initial proposition focuses on gradient steps because while FLOPs are a useful theoretical measure, they do not account for practical factors like GPU parallelism and memory access patterns. Similarly, wall-clock time can vary substantially based on implementation details and hardware setup. That said, to better illustrate the practical efficiency of multi-task learning, we now include wall-clock measurements comparing wall-clock times for BRO and multi-task BRC variants across four model sizes for HumanoidBench (HB-Medium benchmark). We generate these wall-clock estimates using a uniform setup using a single H100 GPU with 16 CPU cores of AMD EPYC 7742 processor. The results, summarized in the table below, confirm that multi-task BRC offers significant gains in efficiency even at reduced model scales. As discussed in Sections 4 and 5, these wall-clock improvements stem from improved sample efficiency paired with performing less gradient steps over the course of the training and are proportional to the number of tasks trained at once.\n\n|                                           | BRO (4M) | BRC (4M) | BRC (16M) | BRC (64M) | BRC (256M) |\n|-------------------------------------------|----------|----------|-----------|-----------|------------|\n| Final performance                         | 0.57     | 0.65     | 0.69      | 0.83      | 0.95       |\n| Minutes to perform 1M steps in all tasks  | 1013     | 242      | 256       | 336       | 658        |\n| Minutes to reach final performance of BRO | 1013     | 157      | 138       | 154       | 263        |\n| Wall-clock improvement                    | 1        | 6.4      | 7.3       | 6.6       | 3.9        |\n\n> The data transfer method sound more like JSRL...\n\nWe thank the reviewer for the observation. We agree that our “data transfer” setup shares similarities with JSRL and WSRL as well. We have updated the relevant section to reflect this similarity. The key distinction between these approaches and \"data transfer\" is that in our approach, the agent is initialized with offline data from different tasks (e.g. solving humanoid-run using data from humanoid-stand), whereas in JSRL/WSRL/RLPD basic configuration, the offline data comes from the same target task. This cross-task transfer framing differentiates our setup from prior methods. **We’ve added this discussion to the revised manuscript**. If helpful for clarity, we’re also open to renaming “data transfer” in the final version to better align with existing terminology.\n\n> It seems like parameter scaling and task (data) scaling are not complementary...\n\nWe appreciate the reviewer’s observation. While the narrowing gap between 3-task and 6-task performance in Figure 10 (right) is interesting, we do not believe it necessarily indicates that parameter scaling and data/task scaling are not complementary. One plausible interpretation is that, with enough training, the performance gap between generalist and specialist models on a given task naturally decreases. This phenomenon is also observed in LLMs: a generalist model pretrained on many tasks often performs well with minimal fine-tuning, but a smaller specialist model can eventually match or surpass its performance on a single task given sufficient data and training [1]. We also note that Figure 10 reflects only final performance at 1M steps, and does not capture differences in sample efficiency earlier in training (where we also see improvements). Finally, regarding results with the 9-task model: unfortunately, we cannot evaluate it on the same validation tasks used for the 1, 3 and 6 task settings, since the 9 task set includes those validation tasks and would introduce data leakage.\n\n[1] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\" NeurIPS 2022\n\n> How would embedding-tuning transfer perform compared to other transfer methods?\n\nWe thank the reviewer for this question. The “embedding tuning transfer” experiments were designed to explore the potential for sample-efficient adaptation and meta learning, rather than raw performance. Unlike “data transfer” and “model transfer” which involve full actor-critic training and require significant data (>500k transitions in our experiments), “embedding tuning transfer” freezes the actor-critic weights and only adjusts the task embedding vector using a very small amount of data (~300 transitions per task). Given this constraint, we expect embedding tuning to underperform in final performance compared to the more resource-intensive transfer methods. However, its key advantage lies in efficiency and rapid adaptation. We believe its potential would become more apparent in larger-scale settings, where rich task embeddings can be learned and reused across many tasks. We view this as a promising direction for few-shot RL, and plan to explore it further in future work.\n\nWe thank the reviewer again for their helpful questions and suggestions. We hope that our rebuttal and implemented changes address all issues raised. If so, we kindly ask the reviewer to consider adjusting their score. We are also happy to further clarify or address new questions."}, {"review_id": "C5sxV1L3vF", "reviewer": "Reviewer_zxeb", "summary": "This paper investigates three existing techniques in reinforcement learning (RL) to improve the capacity and learning of the value function in multi-task reinforcement learning. Specifically, the three techniques are distribution RL cross-entropy loss, larger ResNet-based critic network with LayerNorm, and learned task embeddings. The paper presents extensive experimental results on over 280 tasks across five benchmarks, providing insights for how improving and scaling critic learning can boost performance and generalization in the multi-task setting.", "questions": "Below questions might impact the evaluation:\n1. Figure 11 (left) suggests that model transfer is more efficient than data transfer. However, this appears to contradict findings in [1] and the BRO paper, where data transfer is shown to be useful in single-task settings, particularly concerning network resets. Could you elaborate on this apparent discrepancy?\n2. Is SAC+BRC the same as BRO-SR+CE (where SR represents Scaled-by-Resetting from [1] and CE is the cross entropy loss used in distribution RL)? \n3. Are task IDs provided to the agent? This detail, along with a clearer definition of the learning problems, appears to be missing or not explicitly stated in the main text. Clarifying these aspects is crucial for a complete understanding of the experimental setup.\n4. Error bars appear to be missing from several plots, including Figures 1, 2, and 3 (left). Could you please add these to represent the variability and statistical significance of the results?\n\nBelow are suggestions for improving the paper:\n1. It is crucial to explain statistical significance in DRL experiments. I’d suggest at least mention Appendix D.1 in the caption of Figure 1.\n2. In Figure 5 left, The comparison between \"separate heads\" and \"single-task\" isn't statistically significant. Their confidence intervals overlap, and their rank order flips at various points. To strengthen this claim, consider increasing the number of runs or adding more data points (e.g., testing additional network sizes).\n\n[1] D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P. L., Bellemare, M. G., & Courville, A. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. In The Eleventh International Conference on Learning Representations.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper's primary strength is its comprehensive and rigorous empirical investigation:\n- It conducts experiments on a diverse set of benchmarks, including continuous and discrete control, with coverage of both dense and sparse reward settings.\n- It provides further insights into performance improvement through a variety of analyses, including variance analysis of different variables, transferability to new tasks, the parameter scaling effect, and ablation studies.\n- The paper is clearly written and relevant works are properly cited, which is helpful considering the extensive results and literature.\n\nA potential weakness of the paper is its limited novelty. The proposed approach is a combination of well-established techniques, and while their application to the multi-task setting is new, the resulting performance improvements might be considered predictable by researchers familiar with these methods. The paper's main contribution is therefore an empirical one, demonstrating the effectiveness of this combination, rather than a conceptual one. Nevertheless, this is a valuable and well-executed study that will be of interest to the community.", "was_revised": true, "final_justification": "During the rebuttal, the authors' response addressed my questions regarding the distinction between model transfer and data transfer in this paper, as well as some algorithmic details. I have also read other reviews and the author's responses. While there are directions mentioned by other reviewers to further improve the paper, I think the current contributions of the paper are strong and of interest to the community. I maintain my positive assessment and support the acceptance of this paper.", "rebuttal": "We thank the reviewer for the helpful review and suggestions. We are glad that the reviewer appreciates the comprehensiveness of our empirical results. To the best of our knowledge, this constitutes one of the broadest evaluations in value-based RL to date. Following reviewer’s suggestions, we made a number of changes to our manuscript that we discuss in detail below:\n\n> The proposed approach is a combination of well-established techniques… \n\nWe appreciate the reviewer’s thoughtful summary. We agree that the individual components used in our approach are well-established in the RL and supervised learning literature. However, their systematic combination in the context of multi-task value-based RL has not been explored in prior work, and we believe this empirical contribution is both non-trivial and timely. In particular, many recent efforts in multi-task RL have relied on specialized architectural tricks or task-specific strategies. The fact that a relatively simple combination of scalable design principles can match or exceed specialist performance across various tasks and allow for efficient transfer, is in our view a surprising and valuable result. We hope this simplicity and robustness will make BRC a useful baseline for future research.\n\n> Figure 11 (left) suggests that model transfer is more efficient than data transfer…\n\nWe thank the reviewer for this insightful comment. The key difference lies in the source of the offline data: our “data transfer” method initializes the replay buffer using data from different tasks than the target, whereas approaches like RLPD or network resets (as in the BRO paper) rely on offline data collected from the same target task. This difference in data relevance likely explains the performance discrepancy. As expected, initializing with high-quality target task data (as in RLPD/resets) can be more effective for accelerating learning than using cross-task data. However, a strength of our “data transfer” approach is that it does not require any data from the target task, making it applicable in settings where such data is unavailable or expensive to collect. To better highlight this difference, **we add the above argument to Section 5**. We also agree that the observed performance gap between data and model transfer in Figure 11 (left) is intriguing and highlights a promising direction for future research into cross-task transfer dynamics.\n\n> Is SAC+BRC the same as BRO-SR+CE? \n\nNo, SAC+BRC is not equivalent to BRO-SR+CE, although there are overlapping components. Our approach (SAC+BRC) shares some design elements with BRO, such as using BroNet for architectural structure and distributional RL (we use categorical cross-entropy instead of quantiles). However, SAC+BRC does not include several BRO components, including: Scaled-by-Resetting (SR), Optimistic exploration and Quantile-based value heads. Instead, we use task embeddings and categorical values (CE loss), which we found to be both simpler and more scalable across tasks. For example, unlike BRO which reduces batch size for stability, our method scales well with large batch sizes needed in diverse multi-task settings. Our design goal was to retain components with high practical impact while simplifying the overall method. As a result, SAC+BRC can be seen as a simplified yet more performant variant of BRO, that also works well in multi-task settings. To make these differences clearer for the reader, **we add a paragraph to the Appendix in which we discuss the differences and similarities between BRC and BRO**.\n\n> Are task IDs provided to the agent?\n\nWe thank the reviewer for highlighting this. Yes, task identifiers are provided to the agent. Specifically, we assign an integer ID to each task (from 1 to NumTasks), and use this ID to retrieve a corresponding task embedding vector. This embedding is concatenated to the observation and fed into both the actor and critic networks. As described in Section 3, the task embeddings are learned by backpropagating the critic loss. To improve clarity, **we have now added a paragraph in Section 3 explicitly detailing this mechanism, and we include a diagram in the appendix to visually illustrate the process**. \n\n> Error bars appear to be missing from several plots / link to Appendix D... \n\nWe thank the reviewer for this suggestion. **We will add error bars to the listed figures in the camera-ready version, as well as link to Appendix D in Figure 1**. \n\n> The comparison between \"separate heads\" and \"single-task\" isn't statistically significant…\n\nWe thank the reviewer for this observation. You are right that the performance of \"separate heads\" and \"single-task\" in Figure 5 (left) is not statistically different for models smaller than 250M based on the current confidence intervals. That said, we find this result consistent with trends observed in prior work - when model capacity is limited, \"separate heads\" often underperforms compared to single-task specialists. Our intent in Figure 5 was to explore whether increased model capacity mitigates this gap and the results suggest that \"separate heads\" learners can surpass single-task performance at larger model scales. We agree that the analysis would be stronger with more random seeds - **we are happy to add these in the camera-ready version**. \n\nWe thank the reviewer again for their thoughtful review. We sincerely think that incorporating these changes will improve the reader’s experience and the impact of our work. Please inform us if the rebuttal answered reviewers' hesitations."}]

Author rebuttal:

Meta-review: The paper proposes BRC, a simple and scalable recipe for online multi-task value-based RL combining: (1) a scaled, regularized residual critic (BroNet-style), (2) cross-entropy TD via categorical Q-learning with return normalization, and (3) learnable task embeddings instead of separate heads. The authors conduct a broad empirical study over 283 tasks spanning DMC, MetaWorld, HumanoidBench, ShadowHand, and Atari, using unified hyperparameters. Results show that multi-task BRC surpasses strong single-task specialists in continuous control with substantially fewer gradient steps; rebuttal-added wall-clock measurements further support practical efficiency. The work also demonstrates transfer to unseen tasks, including efficient embedding-only adaptation.

Strengths and weaknesses (largely addressed in rebuttal): 
- Reviewers praise the breadth and rigor of experimentation, the clarity of writing, and the first convincing demonstration within online, value-based RL that multi-task training can surpass single-task specialists at scale. 
- Concerns raised include modest conceptual novelty (the components are individually known), resource demands (large batch sizes and multi-task replay buffers), limited Atari baselines (authors have clarified that Atari is a proof-of-concept rather than SOTA), absence of real-robot experiments, and only limited exploration of model-based and offline RL. The authors’ rebuttal addresses gradient-conflict definitions, clarifies the Atari scope, and adds wall-clock comparisons and noise/stochasticity robustness results.

Overall, this is a good empirical and integrative contribution that meaningfully advances multi-task, value-based RL, showing that a simple, scalable recipe can outperform specialists and transfer effectively. Despite integrative novelty, the breadth, rigor, and practical impact justify acceptance. All four reviews are positive, and the rebuttal addressed key concerns. Please make sure the camera-ready incorporates reviewers' suggestions including scope and claims, compute and resources, return normalization and categorical support, and baselines and statistics.
Decision: Accept (poster)