Title: Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

multi-task learning

data scaling

compute scaling
Abstract: Recent advances in language modeling and vision stem from training large models on diverse, multi‑task data. This paradigm has had limited impact in value-based reinforcement learning (RL), where improvements are often driven by small models trained in a single-task context. This is because in multi-task RL sparse rewards and gradient conflicts make optimization of temporal difference brittle. Practical workflows for generalist policies therefore avoid online training, instead cloning expert trajectories or distilling collections of single‑task policies into one agent. In this work, we show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online RL, allowing for robust and scalable multi‑task training. We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL. We find that, despite its simplicity, the proposed approach leads to state-of-the-art single and multi-task performance, as well as sample-efficient transfer to new tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates three existing techniques in reinforcement learning (RL) to improve the capacity and learning of the value function in multi-task reinforcement learning. Specifically, the three techniques are distribution RL cross-entropy loss, larger ResNet-based critic network with LayerNorm, and learned task embeddings. The paper presents extensive experimental results on over 280 tasks across five benchmarks, providing insights for how improving and scaling critic learning can boost performance and generalization in the multi-task setting.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper's primary strength is its comprehensive and rigorous empirical investigation:\n- It conducts experiments on a diverse set of benchmarks, including continuous and discrete control, with coverage of both dense and sparse reward settings.\n- It provides further insights into performance improvement through a variety of analyses, including variance analysis of different variables, transferability to new tasks, the parameter scaling effect, and ablation studies.\n- The paper is clearly written and relevant works are properly cited, which is helpful considering the extensive results and literature.\n\nA potential weakness of the paper is its limited novelty. The proposed approach is a combination of well-established techniques, and while their application to the multi-task setting is new, the resulting performance improvements might be considered predictable by researchers familiar with these methods. The paper's main contribution is therefore an empirical one, demonstrating the effectiveness of this combination, rather than a conceptual one. Nevertheless, this is a valuable and well-executed study that will be of interest to the community.\n\nQUESTIONS:\nBelow questions might impact the evaluation:\n1. Figure 11 (left) suggests that model transfer is more efficient than data transfer. However, this appears to contradict findings in [1] and the BRO paper, where data transfer is shown to be useful in single-task settings, particularly concerning network resets. Could you elaborate on this apparent discrepancy?\n2. Is SAC+BRC the same as BRO-SR+CE (where SR represents Scaled-by-Resetting from [1] and CE is the cross entropy loss used in distribution RL)? \n3. Are task IDs provided to the agent? This detail, along with a clearer definition of the learning problems, appears to be missing or not explicitly stated in the main text. Clarifying these aspects is crucial for a complete understanding of the experimental setup.\n4. Error bars appear to be missing from several plots, including Figures 1, 2, and 3 (left). Could you please add these to represent the variability and statistical significance of the results?\n\nBelow are suggestions for improving the paper:\n1. It is crucial to explain statistical significance in DRL experiments. I’d suggest at least mention Appendix D.1 in the caption of Figure 1.\n2. In Figure 5 left, The comparison between \"separate heads\" and \"single-task\" isn't statistically significant. Their confidence intervals overlap, and their rank order flips at various points. To strengthen this claim, consider increasing the number of runs or adding more data points (e.g., testing additional network sizes).\n\n[1] D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P. L., Bellemare, M. G., & Courville, A. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. In The Eleventh International Conference on Learning Representations.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new state-of-the-art method on multi task RL, spanning across continuous/discrete action space and state/vision-based input space. Their main sources of improvement were (1) using a well-normalized model (BRO), (2) parameter scaling the model, (3) employing cross-entropy loss, and (4) learnable task embedding. The normalized model allowed its size to be scaled up to 256x (Figure 4), while the cross-entropy loss effectively unified the scale of the loss between tasks and thus their gradient magnitude.\n\nThe authors extensively verify their method's effective in 7 multi-task benchmarks (DMC, HB, Metaworld, Atari, ...), showing that (1) the proposed method performs on-par with previous single task SOTA models and (2) surpasses those methods in multi-task setting. This leads to significant training speedup, taking 2x less samples and 40x less gradient steps to achieve the same performance as prior works.\n\nExperiments were also made in pretrain-then-finetune regime (transfer learning), where they consider three types of transfer: full-finetuning (model transfer), tuning embedding only, and creating expert transitions (data transfer), and show that they outperform from-scratch models as well as prior works on transfer learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- Extensive experiment clearly shows the effectiveness of the method\n- They provide sufficient analysis that justify each component well, as well as exhaustive ablation of their 3 components.\n- The writing is clear and straightforward, easy to understand.\n\nWeakness\n- The method likely suffers from heavy memory constraint, due to large replay buffer (1M per task * num tasks) and large batch size (1024).\n- The components are relatively common in the RL field, although I don't think this as a huge weakness.\n\nQUESTIONS:\n1. I think the compute efficiency in Figure 9 (right) is not fairly compared to prior works. To my understanding each gradient step would take significantly different compute to perform, depending on batch size, buffer size, parameter size, etc. In this case I think the x-axis should be either FLOPs or wall-clock time.\n\n2. The data transfer method sound more like JSRL [1] (or WSRL [2] but that's already cited) than RLPD.\n\n3. In Figure 10 (right), juding by the closing gap between 3 task and 6 task as parameter scales, it seems like parameter scaling and task (data) scaling are not complementary, which is surprising. What do you think of this result? In that sense, I am curious of the results with 9 tasks as well.\n\n4. Also just curious: in Figure 11 (left), how would embedding-tuning transfer perform compared to other transfer methods?\n\n[1] Jump-Start Reinforcement Learning., Uchendu et al.\n[2] Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data., Zhou et al.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces BRC, a recipe that lets online value-based RL scale to billion-parameter models and hundreds of tasks without offline data or expert distillation. BRC combines three design choices: (1) a scaled, regularised residual critic (BroNet) that remains stable at large widths, (2) replacing MSE TD loss with cross-entropy via categorical Q-learning to equalise gradients across tasks with disparate reward scales, and (3) a single network conditioned on learnable task embeddings rather than separate heads. Multi-task BRC achieves comparable performance to task-specific specialists while performing up to 40 times fewer gradient steps, and the resulting networks transfer to unseen tasks with markedly better sample efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n***Strengths***:\n\n1. BRC is evaluated on 283 tasks across five benchmarks. Robust ablations and Shapley analyses pinpoint the scaled BroNet critic as the main source of gains, lending credibility to the claim that careful architectural and loss design, rather than task-specific tuning, drives the performance.\n\n2. The paper combines (i) a scaled, regularised residual Q-network (BroNet), (ii) cross-entropy TD loss via categorical Q-learning to equalise gradients across tasks, and (iii) learnable task embeddings in place of multiple heads. This triad is argued to act synergistically—removing any component degrades results.\n\n***Weakness***:\n\n1. Training billion-parameter critics on 283 tasks almost certainly requires very large GPU budgets, but the paper gives no wall-clock times, FLOPs, or carbon-footprint estimates. Without such data, reviewers cannot judge whether the reported efficiency (fewer gradient steps) translates into practical savings in energy or hardware.\n\n2. It remains unclear whether resource-constrained settings (e.g., on-device robotics) reap any benefit, or whether BRC simply overpowers interference by brute force. Additional results at small/medium scales or a theoretical account of why cross-entropy stabilises TD learning would strengthen the contribution.\n\n3. All benchmarks are simulators with well-shaped dynamics; no physical robot or real-data test is attempted. Moreover, although the paper briefly contrasts online vs. offline variants, deeper analysis of offline RL is missing.\n\nQUESTIONS:\n1. Can BRC handle the observation noise, latency, and dynamics shift typical of physical robots?\n\n2. Why does replacing the MSE TD loss with cross-entropy categorical Q-learning reduce gradient conflict across tasks?\n\n3. How do BRC's gains scale when the critic is restricted to parameters less than 10M or when only one or two GPUs are available?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper presents the BRC method, which scales value-based reinforcement learning (RL) models to achieve multi-task learning and higher efficiency both in terms of sample-wise and computation-wise. The paper proposes to leverage pure RL path rather than imitation learning or distillation to acquire multi-task policies by 1. cross-entropy based value learning; 2. high-capacity value function; 3. task-embedding. The authors did comprehensive and concrete experiments to demonstrate the efficacy of the proposed method. In conclusion, BRC enables scalable, efficient multi-task RL with high-capacity models and improved transfer learning.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Scaling value network has been a popular topic that hopes to make RL agents generalizable and perform better in multi-task settings. This paper is the first one to make impressive improvements on empirical results by conducting evaluations on almost all mainstream benchmarks.
2. Categorial learning + return normalization + task embedding combination is reasonable for multi-task design, especially for diverse reward margins.

Weakness:
1. This paper didn't include model-based RL algorithms into base models, which is expected because SAC+BRC has achieved stronger performance than the most advanced model-based RL algorithms on DMC. However, the author didn't take this into consideration.
2. Authors are selective about Atari baselines. It seems they exclude some of the SoTA model-based RL baselines like EfficientZero, which could lead to overclaiming about the improvements on the Atari-100k benchmark.

QUESTIONS:
1. How do you define gradient conflicts? Why is it still high even in the single-task learning setting, as demonstrated in Fig.5 right?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03