Learning Massively Multitask World Models for Continuous Control
Authors: Nicklas Hansen
Hao Su
Xiaolong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
world models
continuous control
Primary area: reinforcement learning
TL;DR: We introduce a new benchmark with 200 diverse tasks spanning many domains and embodiments, and train a language-conditioned multi-task world model on all 200 tasks via online interaction.

Abstract: General-purpose control demands agents that act across many tasks and embodiments, yet research on reinforcement learning (RL) for continuous control remains dominated by single-task or offline regimes, reinforcing a view that online RL does not scale. Inspired by the foundation model recipe (large-scale pretraining followed by light RL) we ask whether a single agent can be trained on hundreds of tasks with online interaction. To accelerate research in this direction, we introduce a new benchmark with 200 diverse tasks spanning many domains and embodiments, each with language instructions, demonstrations, and optionally image observations. We then present Newt, a language-conditioned multitask world model that is first pretrained on demonstrations to acquire task-aware representations and action priors, and then jointly optimized with online interaction across all tasks. Experiments show that Newt yields better multitask performance and data-efficiency than a set of strong baselines, exhibits strong open-loop control, and enables rapid adaptation to unseen tasks. We release our environments, demonstrations, code for training and evaluation, as well as 200+ checkpoints.

Published as a conference paper at ICLR 2026
LEARNING MASSIVELY MULTITASK WORLD MODELS FOR CONTINUOUS CONTROL
Nicklas Hansen', Hao Su Xiaolong Wang"
" Uniiersity of California San Diego, 'Equal advising Inihasenn, haosu, xiw012 x @uucsdeed.
ABSTRACT
General- purpose control demands agents that act across many tasks and embod- iments, yet research on reinforcement learning (RL) for continuous control re- mains dominated by single-task or offline regimes, reinforeing view that online RL does not scale. Inspired by the foundation model recipe (large-scale pretrain- ing followed by light RL) we ask whether a single agent can be trained on hun- dreds tasks with online interaction. To accelerate research in this direction, we introduce new benchmark with 200 diverse tasks spanning many domains and embodiments, each with language instructions, demonstrations, and option- ally image observations. We then present Newt, iinguage-conditione multitask world model that is first pretrained on demonstrations to acquire task-aware rep- resentations and action priors, and then jointly optimized with online interaction across all tasks. Experiments show that Newt yields better multitask performance and data-efficiency than a set of strong baselines, exhibits strong open-loop con- trol, and enables rapid adaptation to unseen tasks. We release our environments, demonstrations, code for training and evaluation, as well as 200+ checkpoints.
Webpage: https://www.nickk.ee http///ww..cklasaass....n com/NewtWWW
INTRODUCTION
Learning generalist control policy that can perform a wide variety of tasks is an ambitious goal shared by many re- searchers, and significant progress has already been made towards that goal (Jang et al., 2021; Reed et al., 2022; Bro- han et al., 2023; Open X-Embodiment Collaboration et al., 2023; Black et al., 2024). However, the dominant approach among current efforts is to train a large policy with super- vised learning tories collected by egg. human teleoperation.
Average score
200 tasks
0.5
0.4
0.2
enormous dataset of near-expert trajec-
0.1
This approach has major drawbacks: (i) it greatly lim- its the amount of data available for training, and (ii) per- formance of the resulting policy is ultimately bounded by the quality of demonstrations. For these reasons, the com- munity is increasingly turning to reinforcement learning (RL) for continuous improvement of large models. While this new paradigm of large-scale pretraining followed by light RL has led to impressive capabilities in game-playing (Baker et al., 2022; Vasco al., 2025) and reasoning (Ope- nAI, 2024; Guo al., 2025; Su et al., 2025), the continuous control community remains dominated by narrow training tasks (Tassa et al., 2018; Cobbe et al., 2019; Kostrikov al., 2020; Hafner et al., 2023; Cheng et al., 2024; Joshi et al., 2025) or strictly offline regimes (Lee et al., 2022; Hansen et al., 2024; Park et al., 2025), reinforcing a view that RL. from online interaction does not scale in this domain.
0.0 0
20M
40M
60M
BOM 100M
Environment steps
BC
Single-task BC Ours (no demos)
PPO
FastTD3
Ours
Figure 1. Massively multitask RL.
Average score when training a single agent via online interaction 200 tasks spanning 10 task domains.
In this work, we challenge this assumption and ask the following question: can single policy be trained with online RI. hundreds of control tasks once? To answer this question, we in-

---

Published as a conference paper at ICLR 2026
DMControl (21 tasks)
DMControl Ext.
SeradWoorld (49 tasks)
ManiSkill3 (36 tasks)
MuJoCC (6 tasks)
(16 tasks)
MiniArcade
Box2D
RoboDesk (6 tasks)
OGBench (12 tasks)
Atari (27 tasks)
(19 tasks)
(8 tasks)
Figure 2. Tasks. Our proposed benchmark, MMBench, consists of 200 distinct tasks across 10 task domains, including 41 tasks. See Appendix C for detailed overview of our task set.
troduce MMBench: the first benchmark for massively multitask RL. MMBench comprises of 200 diverse tasks spanning multiple domains and embodiments, each with language instructions, demonstrations, and optionally image observations, enabling research on both multitask pretraining. offline-to-online RL, and RL from scratch.
We then present Newt, language-conditionee multitask world model based TD-MPC2 (Hansen et al., 2024), which we first pretrain on demonstrations to acquire task-aware representa- tions and action priors, and then jointly optimize with online interaction across all tasks. To extend TD-MPC2 to the massively multitask online setting, we propose a series of algorithmic improve- ments including refined architecture, model-based pretraining on the available demonstrations, additional action supervision in RI. policy updates, and a drastically accelerated training pipeline.
We validate our method on our proposed benehmark, and demonstrate that effective policy learning across hundreds of tasks is feasible with online RL and in fact can lead to strong multitask policies. Our experiments demonstrate that Newt (7) outperforms set strong baselines when training from state observations, (2) can be rapidly adapted to unseen tasks and embodiments by finetuning with online RL, (3) capable of open-loop control over surprisingly long time horizons, and (4) benefits from access to high-reson resoutrool image observations. In support of open-source source science, 200+ model checkpoints, 4000+ task demonstrations, code for training and evaluation of Newt agents, well all 220 MMBench tasks (including 20 test tasks) considered in this work.
release
In the following, we first introduce our benchmark MMBench, and then present our Newt world model. Readers can refer to Appendix A for a comprehensive discussion of related work on bench- marks, training infrastructure, and scaling of RL and control policies more broadly. We also provide recommendations for future work in Appendix B.
2 BENCHMARK FOR MASSIVELY MULTITASK REINFORCEMENT LEARNING
the first bench- mark of its kind. MMBench contains a total of 200 unique continuous control tasks for training of massively multitask RL policies. The task suite consists of 159 existing tasks proposed in pre- vious work, 22 new tasks and task variants for these existing domains, well as 19 entirely new arcade-style tasks that we dub MiniArcade. The overarching goal of MMBench is to provide a com- mon framework and infrastructure for research and prototyping of massively multitask RL. Figure provides an overview of the 10 domains included MMBench, Figure shows tasks included in MiniArcade, and Appendix C provides detailed description of each task domain. In the following, we discuss key features of our benchmark, as well as our efforts in making MMBench accessible to the broader research community.
To study the feasibility Massively Multitask RL, we introduce MMBench:
2

---

Published as a conference paper at ICLR 2026
Figure 3. MiniArcade. We release a new task suite, dubbed MiniA reade, that consists of 22 tasks spanning 14 unique arcade-style environments (depicted). All tasks support both low-dimensional state representations and RGB observations, and have well-defined reward functions for RL.
Ant Ball (OGBench)
Embodiment: Ant (quadruped) with 8 controllable joints (4 legs).
Instruction: Push the soccer ball to the goal location.
Rocket Collect (MiniArcade)
Embodiment: 2D space rocket with controllable actions: main engine and side thruster.
Instruction: Collect coins while avoiding asteroids.
Figure 4. Sample language instructions. All instructions in MMBench provide description of embodiment and action space followed by task description. Refer to Appendix D for more samples.
2.1 PROBLEM FORMULATION
Online reinforcement learning (RL) aims to learn a policy (agent) from interaction with an (in our case: multitask) environment. This interaction commonly modeled tially Observable Markov Decision Process (Bellman, 1957; Kaelbling et al., 1998) formalized a tuple AS,AT R2 7). Here, environment dynamics T: A are governed by gener- ally unobservable states E S approximated as s (S Simg Slang) where Satate Simg: Slang are low-dimensional state inputs, image observations, and language instructions, respectively, a A are actions, R: S A + R is reward function that produces task-specific rewards r, and task-specific discount factor. Our overarching goal is learn single policy s.t. return E, - 7're], R(s:, (S:)) is maximized in expectation across all time steps and tasks, i.c. a massively multitask policy trained to perform hundreds of tasks simultaneously via online RL.
infinite-horizon Par-
2.2 ENVIRONMENTS
Observations, actions, and rewards. Our task suite comprises of diverse continuous control tasks span- ning locomotion, tabletop manipulation, navigation, arcade games, classic control problems, and more. Tasks vary greatly in observation and action space di- mensionality, task horizon, and reward specification. All tasks support three observation modes: (7) low- dimensional states, (2) 224 x 224 RGB images, or (3) both. To unify state observations and actions across tasks and domains, we provide mask such that in- valid dimensions can taken into account during pol- icy learning and inference. Unification of visual ob- servations done on a per-domain basis, with some domains readily supporting rendering at 224 x 224, while others require resizing (Box2D) or zero zero-paddeee cero-ngdding (Atari). Refer to Appendix C for table detailing the observations, actions, rewards, and episode lengths in each task domain.
Pick Cube
Pick Banana
Air Hockey
Pong
Open Slide
Pointmass
Open Drewer
Door Open
Open
Point Maze
Landing
DMControl
Ant
Meta- World
Happer Hop
Maniskil
MuJoCoo
Quadruped Run
Box2D
Hopper
RoboDesk
Cheetah Half-cheetah
OGBench
MiniArcade
Hopper Hop
Figure 5. Language embeddings. First
2 principal components of CLIP-ViT/E embeddings shown for a subset of tasks.
3

---

Published as a conference paper at ICLR 2026
Environments 200 tasks
Task description
Newt Architecture
Embodiment:: Quadruped (ant) with 13 concrollable joints (4 lege). Instruction: Riin sa fast ss possible using all lege.
Sell-predccive model TD-MPC2
RGB obs
CLIP
Actions
DINOV2
Newt World Model
pieA
paysay
enc
State obs
Rewards
[-H
stop-grad grad
stop
Figure 6. Method. Our agent iteratively collects data via multitask environment interac- tion, and optimizes its world model on the collected data. The world model takes a state vector, language instruction, and optionally RGB observations as input, and outputs actions via planning.
Language instructions. While some tasks can be differentiated solely based on observations, this is universally true. For example, the manipulation tasks RoboDesk share a common observation space (pose and velocity of robot and objects), and thus it is not clear from observations alone which simple solution would be to provide agents with a one-hot encoding
object is to be manipulated. of task indices, but this greatly limits the potential for transfer to new tasks since it provides no mechanism for representing unseen tasks. Instead, we choose to provide language instructions for every task in MMBench. Two sample language instructions shown in Figure 4, with additional samples available in Appendix D. To verify that language instructions adequately differentiate tasks and embodiments, encode all language instructions from MMBench with CLIP-ViT TBB, and visualize their first two principal components in Figure 5.
2.3 ACCESSIBILITY
Training agents with online RL on large number of tasks is challenging from learning perspective and may be prohibitively expensive for researchers and practitioners with limited computational resources. To make MMBench accessible the broader RL community, we have devoted significant time and effort to reduce computational costs.
Demonstrations. Exploration has historically been key challenge in online RL, and a large body of literature dedicated to exploration strategies (Brafman & Tennenholtz, 2003; Bellemare et all, 2016; Pathak al., 2017; Sekar al., 2020; Ecoffet et al., 2021; Ladosz al., 2022). However, as problem scope and task complexity grows, learning an agent from scratch ("tabula rasa") via RI. is becoming infeasible. Demonstrations serve exploration burden and variance of online RL, creating a practical path for researchers with limited compute still contribute the field. To this end, provide 10-40 demonstrations for each task in MMBench, collected by single-task TD-MPC2 (Hansen et al., 2024) agents trained from low- dimensional state observations. While we do not directly leverage the trained single-task agents in this work (beyond collecting demonstrations). we make all 200 model checkpoints available for use by other researchers, and are excited see what the community will use them for.
strong behavior prior that dramatically reduces the
Asynchronous environments. Our benchmark consists of hundreds of environments in multiple robotics simulators, 2D game engines, and emulators, which complicates parallelization and overall software integration. To aid adoption, we provide ready-to-use docker image and fast, easy-to- use environment wrappers that enable asynchronous environment stepping and rendering, batched Trame-stacking and image encoding using large pretrained backbones, cached language embeddings, easy handling of diverse observation and action spaces, as well as auto-resetting whenever a task completes. These measures drastically reduce wall-time and allow users get started immediately.
4

---

Published as a conference paper at ICLR 2026
3 NEWT: A MULTITASK WORLD MODEL FOR CONTINUOUS CONTROL
To learn RL agents effectively in a massively multitask setting like ours, the algorithm of choice needs to satisfy the following criteria: needs (i) scale with increasing model and data size, (ii) be robust to various observation and action spaces, reward functions, and task horizons, (iii) be able to adequately differentiate tasks, and (iv) train in a reasonable time frame. We choose to base agent, Newt, on model-based RL algorithm TD-MPC2 (Hansen et al., 2022; 2024) as it satisfies the first two criteria. Concretely, TD-MPC2 performs trajectory optimization (planning) in the latent space of a learned self-predictive (decoder-free) world model, and it has demonstrated robust learning across a variety of single-task (online RL) and multi-task (offline RL. environments. We extend TD-MPC2 the massively multitask online RL setting, and describe our algorithmic improvements in the following. Figure 6 summarizes our approach.
3.1 LEARNING A MASSIVELY MULTITASK WORLD MODEL
TD-MPC2 learns its world model with a combination of joint-emeedding prediction (self- prefiprreee dynamics modeling), reward prediction, and TD-leanning (Sutton, 1998); this is in contrast gen- erative world models that are typically trained to decode raw future observations (e.g. RGB images) using auxiliary decoder network. A key benefit of self eef-prredetive world models is, in addition to being computationally cheap due to not having a decoder, that they are trained to be control-centric: accurately predicting outcome (return) conditioned model extends the TD-MPC2 architecture to support language instructions and optionally RGB ob- servations in addition to low-dimensional state observations. Specifically, our world model consists of the following components:
sequence of actions. Our proposed world
Language encoder
g CLIP text (Sangg
Encodes natural language instruction
Image encoder
x DINOv2 (Simg)
Encodes RGB image observation (optional)
State encoder
2 = 11Saattt: X, g)
Computes latent state representation
Latent dynamics
d(z, a. g)
Predicts latent forward dynamics
(1)
Reward
F = R(z, g)
Predicts reward r of a transition
Terminal value
Â Q a. a.
Predicts discounted sum of rewards (return)
Policy prior
á = p(z,g)
Predicts optimal action a*
where s = {Stang: Simg: } are language, image, and state observations, respectively, a are actions. (frozen) pretrained backbones for language and image inputs which allows to
In practice, cache embeddings, and we choose to implement all other components as MLPs. Components that take multiple arguments input fuse their inputs via concatenation before feeding them into the first dense layer, i.e., we let (SSsuuee, x, g) 1a( (Saate, x, g)) where [t] denotes concatenation.
Following TD-MPC2, we jointly optimize h, d, R, Q by gradient descent the objective
(0) = TBB
Σ X t=
|
sg( ( x g))1 + (CE(Ferr) + (celee- 2t)
Solf-prediction
Rewards
Valuees
where r = (s, a, r, s') s'). is subsequence sampled from a replay buffer B, X E (0, 1] constant coefficient which weighs temporally distant samples exponentially less, sg is stop grad operator that helps mitigate representation collapse (Grill et al., 2020), fCE is the cross-entropy loss, and predictions (z/, fr, ).) are defined in Equation 1. Regressing rewards and values using a MSE loss is challenging in a multitask setting as reward distributions may be drastically different between tasks and task domains. Therefore, we opt for discrete regression objective (the cross-entropy loss) and model values in log-transformed space such that we are able to model a wide range of values with single prediction head (Bellemare al., 2017: Kumar et al., 2023; Hafner al., 2023; Hansen et al., 2024; Farebrother et al., 2024). We use z. step TD-target (Sutton, 1998), and let Qig be an exponential moving average (EMA) of the online Q network (Lillicrap et al., 2016). In practice, we optimize small ensemble of Q-networks and define the target Q-value as the minimum of a random subset of Qigs estimates (Chen et al., 2021). per-task discount factors (7) since episode lengths vary drastically between tasks; we use the domain-defaur discount factor when one is available and otherwise estimate it using a heuristic described in Appendix H.
+ Q2g p(z)), ), g) as our one-
We choose
5

---

Published as a conference paper at ICLR 2026
The policy prior p is formulated stochastic maximum entropy policy (Ziebart al., 2008) that learns to maximize Q Q-values as estimated by the Q-network defined above. However, we find that naive application of this policy objective to problem setting leads to subpar performance in tasks where Q-values are difficult to estimate. Instead, we choose to leverage a small set of demonstrations (more on this in Section E) and add additional behavior cloning loss the policy prior of TD-MPC2. This serves two purposes: (i) directly leveraging expert demonstrations as action supervision, and (ii) explicitly distilling actions selected via planning into the less expressive policy prior. Concretely, we define the policy objective as
H
1x(0)= = E TBB
X t=0
1pp(z,, g) - ael3 - -Q(z, Q(z,, p(2z,)).g)
H(p(-Ize: g:)
(3)
Model-based BC
Q-valuc
Entropy
where Zt+1 d(zz,au at. g). Zo h(ssaueg Xu, g) is the latent rollout of т. During environment interaction, TD-MPC2 selects actions by planning with the learned world model, with the planning procedure warm-started by the policy prior p. Refer to Appendix G for details on our planning algorithm. In the following, we describe other ways in which we leverage demonstrations.
3.2 LEVERAGING DEMONSTRATIONS FOR WORLD MODEL LEARNING
To overcome the difficulty of exploration in massively multitask online RL, we choose to leverage a small number of demonstrations for each task. Although one could naively add demonstrations to the replay buffer at the start of training and indeed benefit from the model-based BC term introduced in Equation 3, we would like to utilize demonstrations to their full extent. Concretely, we propose to use demonstrations in four distinct ways:
(/) Model- pretraining. Prior to any online interaction, we first pretrain all learnable compo- nents from Equation 1 on the provided demonstrations. Specifically, we assume that demonstrations consist of (s, a, r) tuples and jointly optimize all components by minimizing C(0) + (O), but with the Q-value term in Equation 3 temporarily disabled such that we can fully leverage the strong action supervision from the demonstrations. This is contrast to prior work (Zhan et al., 2020; Hansen et al., 2023a) that pretrains only encoder and/or policy.
(2) Constrained planning. When transitioning from pretraining to online RL, we empirically ob- serve that (at the start of RL) planning with the world model yields weaker behavior policy than directly using the pretrained policy due to a (comparably) inaccurate value function. To maximally retain performance when switching to planning, we initially bias the planner towards the pretrained policy and linearly anneal this bias to zero during the first 12% of training. See Appendix G for details.
(3) Oversampling of demonstrations. We maintain separate replay buffers for demonstrations and online interactions, and sample subsequences at equal proportions (50% from each) during agent updates (Feng et al., 2023; Ball et al., 2023). This means that demonstrations are (artificially) overrepresented in the training data, and ensures that the demonstration data remains available the agent throughout training regardless of the capacity of the online interaction buffer.
(4) Action supervision RL policy updates. As discussed in Section 3.1, adding -aaeel- BC loss term in the policy objective of Equation 3 provides direct action supervision and helps regularize the RL-based policy objective when Q-value estimation is inaccurate (Lin et al., 2025).
In summary, our Newt agent is model-based RL method for language- and optionally image- conditioned massively multitask continuous control. aims to fully leverage available demonstra- tions as well as online interaction data, resulting in data-efficient yet computationally inexpensive method. addition to the algorithmic improvements discussed in this section, we also drasti- cally accelerate training speed by distributing both model updates, environment interactions, and replay buffers across multiple processes and GPUs, compiling training and inference code with torch. compi le (Bou et al., 2023), and other implementation details described in Appendix H.

---

Published as a conference paper at ICLR 2026
DMControl
DMControl Ext.
Meta-World
ManiSkill
MujoCo
0.6
0.6
0.8
0.6
0.6
0.4
0.4-
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
Box2D
RoboDesk
OGBench
MiniArcade
Atari
0.4
0.8
0.6
0.6
0.3
0.6
0.4
0.4
0.2
0.2
0.4
0.2-
0.2
0.2
0.1
0.0
0.0
0.0
0.0
0.0
50M
100M
50M
100M
50M
100M
50M
100M
50M
100M
BC
Single-task BC
PPO
FastTD3
TD-MPC2
Ours
Figure 7. Per-domain performance. Average score of a single state-based agent on MMBench (10 task domains; 200 tasks). See Appendix for more baselines, and Appendix for per-task curves.
4 EXPERIMENTS
We evaluate Newt on our proposed benchmark MMBench which consists of 200 tasks across 10 task domains: DMControl (Tassa et al., 2018), DMControl Extended, Meta-World (Yu et al., 2019), ManiSkill3 (Tao et al., 2025), MuJoCo (Todorov et al., 2012), MiniArcade (a contribution of this work), Box2D (Brockman et al., 2016), RoboDesk (Kannan et al., 2021), OGBench (Park et al., 2025), and Atari (Bellemare et al., 2013; Farebrother & Castro, 2024). See Figure for visual overview task domains, and refer to Appendix C for a detailed description of each task domain. Although method can readily be applied visual RL, experiments in this section use state obser- vations unless we explicitly state otherwise. In support of open-source science, we publicly release 200+ model checkpoints, 4000+ task demonstrations, code for training and evaluation of Newt agents, as well as all 220 MMBench tasks considered in this work. We seek to answer:
(QI) Performance. Can single agent be trained on hundreds of unique tasks with online RL? How does our model-based approach (Newt) compare to a set of strong baselines?
(Q2) Learning from demonstration. Do demonstrations alleviate the difficulty of exploration in a massively multitask setting? How do we leverage demonstrations effectively for model- based RL?
(Q3) Model capabilities. What are the downstream capabilities of a massively multitask world model? Can we leverage our trained model for zero-shot or few-shot transfer to unseen tasks/em- bodiments? Can we perform open-loop control?
(Q4) Analysis & ablations. What makes good multitask world model? What role does language and vision play? What are the current capabilities and limitations of Newt?
Baselines. Our baselines represent the state-of-ther the-thr-tt in data-efficient RL, learning from demon- strations, and RL with large amounts of parallel environments. Specifically, our baselines include:
Behavior cloning (BC) (Pomerleau, 1988; Atkeson & Schaal, 1997) on demonstrations. We com- pare against conditige- multitask BC policy, as well as 200 single-task BC policies.
Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an on-policy actor-critic method that maximizes clipped surrogate objective, and has gained popularity wall-time training when combined with massive parallel simulation. We base our experiments on the cleanRL (Huang et al., 2022b) implementation, and extend it to support language-conditionin and per-task discount factors. We also tune hyperparameters: see Appendix J for results before/after.
part due to its fast
Fas (Seo et al., 2025), a modern implementation of TD3 (Fujimoto et al., 2018) designed for RL with parallel environments. We extend FastTD3 to support mnnguugc--ooddiioniie and per-task discount factors, and find that n-step returns are critical in our challenging multitask setting.
TD-MPC2 (Hansen al., 2022; 2024) trained with multitask online RL. This baseline can be con- sidered a more naive implementation of Newt without language-conditionngg pretraining, demon- strations, nor the BC loss term introduced in Equation 3. We match the parameter count of Newt.
200 single-task TD-MPC2 (Hansen et al., 2022; 2024) agents trained on individual tasks for 5M environment steps (1B total steps). We collect demonstrations for BC and Newt using these agents.
7

---

Published as a conference paper at ICLR 2026
Model size 200 tasks
Batch
Language instructions
Demonstrations
200 tasks
200 tasks
200 tasks
0.5
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.2
0.2-
0.2
0.1
0.1
0.1
0.1
20M 40M 60M
100M
20M
100M
20M
100M
B0M
100M
Environment steps
Environment steps
Environment steps
Environment steps
2M
5M
128
256
None
Task ID
+demos
+BC
20M
8OM
512
1024
CLIP
*pretrain
Figure 8. Ablations of all key design choices. Our default formulation of Newt is shown in bold.
Model-based pretraining of our Newt world model the same demonstrations as for BC. During pretraining, we optimize the model as described in Section 3.2.
See Appendix F for more information on baselines, including implementation and hyperparameters.
Implementation details. Language is encoded using CLIP CLIP-ViT/E (Radford et al., 2021) and images using DINOv2/5 BB (Oquab et al., 2023), which results in embeddings of dimensions 512 and 768, respectively. State observations are 128-dim vectors, actions are 16-dim vectors, and our Newt agents have 20M learnable parameters unless stated otherwise. See Appendix H for more details.
4.1 RESULTS
Benchmarking algorithms. We evaluate the performance of our method, Newt, and baselines on proposed MMBench task suite. Methods are trained for 100M enviromment steps (in total across all tasks) using oowddimeniional state observations. Our main result is shown in Figure 1, and per- domain scores are shown in Figure 14. These results indicate that Newt is more data-efficient and achieves a higher overall performance than PPO, FastTD3, and TD-MPC2, which we attribute in part its significantly better performance in DMControl, DMControl Ext., ManiSkill, and MiniAr- cade. However, we also observe subpar performance across all RL methods in the MuJoCo, Box2D, and Atari domains, with the performance of Newt often being similar to that of the simpler BC base- line. We conjecture that this may be due to the relative uniqueness of tasks in these domains; for example, many Atari games have little common beyond their action space. While we observe that the addition of demonstrations in many cases leads to better asymptotic performance as it alleviates the difficulty of exploration, we recognize that there is still room for improvement; developing meth- ods that yield more consistent improvement across tasks is thus an exciting future research direction. Please refer to Appendix and Appendix K for additional experiments and bascline comparisons.
Analysis & ablations. We ablate all key design choices, including model and batch size, use of lan- guage instructions, well all the different ways in which we can leverage demonstrations. Ablations conducted with 20M parameter agents on the full 200-task set. Our main ablations are shown in Fig- ure additional experiments and per-domain results can be found in Appendix J. We make the following observations:
Table 1. Language sometimes inhibits zero- shot generalization. Success rate with seen and unseen instructions in 10 unseen manip- ulation tasks. 100 trials per task.
Instruction
Success (%)
(Defanlt)
Push
<unseen>
0.3
Push <cube>
21.0
(Defanlt)
Pick <unseen>
10.5
(I) Scaling model and batch size is beneficial when the number of training tasks increases. In contrast to previous work that shows only marginal improve- ments from model scaling in a single-task RL setting (Hansen et al., 2024), we see a clear benefit in scaling model and batch size in multitask setting, up point; see Appendix H for details on scaling. We conjecture that there exists a compute-optimal (model, batch) size for any given number of tasks in which learning is stable, and that further scaling of training tasks will require proportion- ally larger models and batches than presently.
Pick <cube>
(2) Language helps differentiate tasks. We find that conditioning the agent on language instructions provides clear performance benefits (0.371 + 0.438 normalized score), with the greatest improve- ment in domains where tasks cannot be differentiated by observations alone (e.g. RoboDesk);
8

---

Published as a conference paper at ICLR 2026
Walker Walk
(DMControl)
Lunarlander Takeoff (Box2D)
1=0
t= 16
48
t=
t= 16
48
A
Point Maze (OGBench)
Pick Screwdriver
ManiSkill 11)
8
16
1 24
f 0
t 8 t= 16
24
P Figure 10. Open-loop control. Exccuting open-loop plans without any environment feedback. These results indicate that our world model learns meaningful representations of the environment.
Appendix for a detailed performance comparison of Newt with and without access to language instructions. Additionally, shown in Appendix I., this difference in downstream performance also translates to a lower training loss for the language-condittonee agent. fact, we find language con- ditioning match the performance of task indices (as used in TD-MPC2) on training tasks while also providing mechanism for generalization to unseen tasks.
(3) Demonstrations improve performance in hard exploration tasks. We find that any individual way of using demonstrations (pretraining, oversampling, model-based BC loss) is helpful, but that using them all in conjunction yields the best data-efficiency and asymptotic performance.
Task transfer. To investigate whether our multitask agent transfers unseen tasks/embodiments, we develop held- out task set that spans multiple task domains and finetune our agent to each task individually using online RI. and no demonstrations. Transfer tasks and aggregate finetuning re- sults are shown in Figure 9. We observe that the pretrained Newt agent achieves zero-shot score of 0.192 compared to 0.013 when trained from scratch, and reaches an average score of O. 0.868 100k environment steps just 0.480 for the baseline. These results demonstrate non-trivial transfer- ability, and expect transfer results to get better and more training data becomes available. As shown in Ta- ble we find that unseen language instructions can greatly inhibit zero-shot generalization of our agent, depending on the task. Replacing the noun describing the object to be ma-
Few-shot finetuning (20 unseen tasks/embodiments)
1.0
Normalized score
0.8
0.6-
nipulated (unseen instruction) with cube (inaccurate but
0.2
seen instruction) improves zero-shot success rate by 20.7% across 6 pushing tasks, whereas we see the reverse trend for pick-and-plee place tasks. To best reflect the true capabili-
0.0
20k
40k
60k
80k
100k
Environment steps
ties of our agent, use unseen instructions in all transfer experiments.
From scratch
Finetuned
Figure 9. Few-shot finetuning. Aver- age score when transferring our agent to new tasks/embodiments. seeds.
Open-loop control. Model-based approaches uniquely positioned to perform open-loop control (planning and ex- ecuting actions without environment feedback). To better understand the current capabilities of Newt, we compare its open-loop planning performance relative to the default closed-loop planning. We evaluate 8 diverse tasks and with planning horizons of up to 48 time steps 16x longer than its training horizon of 3. Qualitative results are shown in Figure 10, and the full quantitative results are provided in Appendix I. We find that Newt can plan over long time horizons without being explicitly trained to do with performance closely matching closed-loop control in most tasks. Common failure modes include drifting dynamics (Walker Walk, DMControl), failing to de- celerate after reaching the target (Lunarlander Takeoff, Box2D), or the inability to predict stochastic elements (Assault, Atari). Refer to Appendix for more open-loop results.

---

Published as a conference paper at ICLR 2026
Visual observations. The majority of our exper- iments are conducted with low-dimensional states due to (i) added computational costs of visual RL, and (ii) lack of available baselines for massively multitask visual RI.. However, hypothesize that some task domains stand to benefit more from vi- sion than others. To test this hypothesis, we add 224 x 224 RGB inputs to our state-based agent de- scribed in Equation and finetune the entire model for 30M environment steps. We report average score across all domains as well as for a select few do- mains where the score change is noteworthy. We ob- serve only a marginal overall improvement (++0.004) with visual observations, but find that performance improves significantly for manipulation (RoboDesk and Meta- -eta-World) and decreases in domains such as DMControl where learning from vision is known to be more challenging than from state (Hafner al.,
Table Visual RT. Agent performance after
finetuning with visual inputs for 30M steps.
Domain
Score
Gain
State +RGB
All
0.438
All
0.442
+0.004
RoboDesk
0.500
+0.125
4
WerldWorld
0.572
+0.069
MiniArcade
0.437
-0.04
4
DMControl 0.471
-0.029
Table 3. Training cost. Expected wall-time
when training on 200 tasks for 100M envi- ronment steps in total (20M params). Re- ported for different hardware configurations.
Observation
Hardware
Days
State
x RTX 3090
11.2
2019; Srinivas et al., 2020; Kostrikov et al., 2020).
State
2x RTX 3090
7.3
State
2x RTX 5090
4.6
Training cost. Table shows the expected wall-time when training on 200 tasks from MMbench, for var- State+RGB
2x RTX PRO 6000
4.7
ious hardware configurations and observation modes (state and state+RGB with rendering at 224 x 224). Machines are equipped with an AMD EPYC 9354 CPU and 128 GB of RAM. The demonstration dataset requires 32 GB of disk space.
5 CONCLUSION
We present MMBench, a benchmark for massively multitask RL, as well as Newt, a language- conditioned multitask world model trained with online RI. We demonstrate that online RI. on hun- dreds of tasks simultaneously is feasible, and that it can lead to world models with surprisingly strong generalization and open-loop control capabilities. While we recognize that some challenges remain (rate of improvement may differ substantially between tasks and domains, generalization is currently inhibited by poor language understanding, as well as the significant hardware requirements for fast visual RL), we remain very optimistic about the future of massively multitask RL training.
To encourage further research in this area, we provide a comprehensive discussion of related work in Appendix A as well as recommendations for future work in Appendix B.
ACKNOWLEDGMENTS
The authors thank the following people for helpful discussions and feedback on paper drafts, in al- phabetical order: Adrian Remonda, Arth Shukla, Bo Ai, Hansen Lillemark, Jiajun Xi, Lars Paulsen, Stone Tao, and Yutao Xie. A special thanks is also extended to the open-source RL community, without whom this project would not have been possible; particularly the original developers of Mu- JoCo, DMControl, Meta-World ManiSkill3, RoboDesk, OGBench, Arcade Learning Environment, Gym and Gymnasium, TorchRL (Vincent Moens), as well as their numerous individual contributors.
This project was supported, in part, by NSF CAREER Award IIS-2240014 and NSF CCF-2112665 (TILÓS), as well as gifts from Amazon, Meta, and Qualcomm.
STATEMENTS
A statement on reproducibility. Reproducibility is important to us. We release 200+ model check- points, 4000+ task demonstrations, code for training and evaluation of Newt agents, as well as all 220 MMBench tasks (200 training tasks and 20 test tasks) considered in this work; all of our arti- Net//Newtw..... We also
facts are made publicly available at https: / /www. nicklashanseeee ashansen... provide extensive implementation details and empirical results in the appendices. Most notably, Ap-
10

---

Published as a conference paper at ICLR 2026
pendix C provides an overview of task domains, Appendix E provides details demonstrations, Appendix F and Appendix H provide implementation details for baselines and Newt, respectively, and Appendix K provides learning curves for all 200 tasks.
statement on ethics. We propose benchmark, MMBench, that is comprised of both entirely new tasks, tasks that derivative work based on existing task domains, and existing tasks that are used as is. All task domains and baselines for which we rely on third-party code are open-source and have permissible licenses. For example, DMControl is licensed under Apache 2.0 License, OGBench is licensed under an MIT License, and Arcade Learning Environment (ALE; also known as Atari) is licensed under a GNU General Public License v2.0. Any derivative work of existing task domains is licensed under their respective licenses; code contributed as part of our work (e.g., MiniArcade and Newt) is licensed under an MIT License.
REFERENCES
Christopher G. Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, 1997.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. Advances in Neural Information Processing Systems, 2016.
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, 2016.
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International Conference Machine Learning, pp. 459-455. PMLR, 2017.
Richard Bellman. A markovian decision process. Indiana Univ. Math. J., 6:679-684, 1957. ISSN 0022-2518.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 19/2,0660,, 2019.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, al. piO: A laiion-lnguago--cction flow model for general robot control. arXiv preprint 4rX1v.2410.24164,, 2024.
Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. Torchrl: A data-driven decision-making library for pytorch, 2023.
Ronen I. Brafman and Moshe Tennenholtz. R-max general polynomial time algorithm for near- optimal reinforcement learning. J. Mach. Learn. Res., 3:213-231, March 2003, ISSN 1532-4435. doi: 10. 10.1162/1
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
11

---

Published as a conference paper at ICLR 2026
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-lane lawtonn-angugg-aaae models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15816 2023.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. International Conference on Learning Representations, 2021.
Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 11443- 11450. IEEE, 2024.
Jack Clark and Dario Amodei. Faulty reward functions in the wild. OpenAl Blog, 2016.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. arXiv preprint arXiv: 2019.
Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, P. Abbeel, and Wojciech Zaremba. One-shot imitation learning. ArXiv, abs/1703.07326, 2017.
DYNA Robotics. Dynamism v1 (dyna-l) model: A breakthrough performance and production- ready embodied ai. https: hwww://wwwwww. dyna. co/eseaacc, June 2025.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then
explore. Nature, 590(7844):580-556, 2021.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scal- able and efficient deep-rl with accelerated central inference. arXiv preprint arXiv:1910.06591, 2019.
Jesse Farebrother and Pablo Samuel Castro. Cale: Continuous arcade learning environment. Ad- vances in Neural Information Processing Systems, 37:134927-134946 2024.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Ir- pan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In Forty-first International Conference on Machine Learning, 2024.
Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Fine- tuning offline world models in the real world. Conference Robot Learning, 2023.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Bean-Bastien Grill, Florian Strub, Florent Altch'e e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self-super learning. Advances in Neural Information Processing Systems, 2020.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. 2019.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 040044, 2023.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmen- tation. In International Conference on Robotics and Automation, 2021.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In ICMI., 2022.
12

---

Published as a conference paper at ICLR 2026
Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang. Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations. 2023a.
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting learning- from-- scratch baseline. In International Conference on Machine Learning (ICML), 2023b.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control, 2024.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances neural information processing systems, 29, 2016.
Shengyi Huang. Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Truck, 2022a. URL http:::s///rrlrllos/se 25/ 25/ppo- imp ementat inlleatiiondettan on- implementation--dtaalli
-gi 1o/2022/03/
Shengyi Huang. Rousslan Fernand Julien Dossa, Chang Ye. Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep rein- forcement learning algorithms. Journal of Machine Learning Research, 23(27)):1-88, 2022b.
Marcel Hussing, Claas A. Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, and Eric Eaton. Dissecting deep rl with high update ratios: Combatting value overestimation and divergence. Reinforcement Learning Conference, August 2024.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In CoRL, 2021.
Viraj Joshi, Zifan Xu, Bo Liu, Peter Stone, and Amy Zhang. Benchmarking massively parallelized multi-task reinforcement learning for robotics tasks. In Reinforcement Learning Conference, 2025. URL https:///// /ttp//ppppeeeeieeeeeeeeeeeeeeeet met//forum weet//oormmmdd22200000000000
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 1998.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic re- inforcement learning at scale. ArXiv, abs/2104.08212, 2021.
Harini Kannan, Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task rein- forcement learning benchmark. https: // wtt.b..ccooooooo httgg/ooollleeeeeeeeeeeee robooeekk 2021.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mereat, Abdul Rehman, Pan- nag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad. Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang.
13

---

Published as a conference paper at ICLR 2026
Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg. Ken Goldberg. Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph Lim, Jitendra Malik, Roberto Martin-Martin, Subramanian Ramamoorthy. Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey zero-shot gener- alisation in deep reinforcement learning, Journal of Artificial Intelligence Research, 76:201-264,
2023.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference on Learning Representations, 2020.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. International Conference on Learning Representations, 2023.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Samuel Lavoic, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, and Aaron Courville. Simplicial embeddings in self-supervised learning and down- stream classification. In International Conference Learning Representations, 2023. URL https: http://ppneeeeveeeeeee .we.iiewweeeeeeeee...e wetest/fooumiddddeee wow.mmiidwttttrrrrprr.
Kuang-Huse Hueni Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi Multi-game formers. Advances in Neural Information Processing Systems, 11-27921-279935 2022.
-ame decision trans-
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin. Michael Lin- gelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, You-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. Behavior- Ik: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.0922 09227,,92222 2024.
T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1 02971, 2016.
Haotian Lin, Pengcheng Wang. Jeff Schneider, and Guanya Shi. Td-m(pe)2: Improving temporal difference mpc through policy constraint. arXiv preprint arXiv:2502.03:: 03550,, 2025.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning, arXiv preprint 1xxxiv2555.18711 2025.
AJ Miller, Fangzhou Yu, Michael Brauckmann, and Farbod Farshidian. High-performance re- inforcement learning spot: Optimizing simulation parameters with distributional measures. arXiv preprint, 2025.
Diganta Misra. Mish: A self regularized non monotonic neural activation function. arXiv preprint arXiv: arXiv:/908.00 ww0. 08681, 2019.
14

---

Published as a conference paper at ICLR 2026
Jun Nakanishi, Jun Morimoto, G. Endo, Gordon Cheng, Stefan Schaal, and Mitsuo Kawato. Learn- ing from demonstration and adaptation of biped locomotion with dynamical movement primitives. Robotics and Autonomous Systems RaS, 2004, 01 2003. doi: 10. 100.1999/smmmmm22004.... 2.
Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, and Pieter Abbeel. Bigger, regu- larized, categorical: High-capacin value functions are efficient mulli-task learners. arXiv preprint arXiv:2505.23156 2025.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo. Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong. Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. arXiv preprint arXiv:1711.009377 2017.
Open X-Fmbodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Mad- dukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai, Anchit Gupta, Andrew Wang. Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian lchter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang. Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton. Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim. Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador, Joseph J. Lim, Jun- hyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krish- nan Srinivasan, Kuan Fang. Kunal Pratap Singh, Kuo-Hao Zeng. Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Out, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart'in-Marr in, Rohan Baijal, Rosario Scalise, Rose Hendrix. Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque. Ryan Julian, Samuel Bustamante, Sean Kir- mani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subrama- nian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park. Suraj Nair, Suvir Mirchan- dani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Dar- rell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Vitor Guizilini, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu,
15

---

Published as a conference paper at ICLR 2026
Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Ja- son Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X mod- org/ abs/a31s1 08864, 2023.
els. attps::/arra
OpenAL.
Introducing openai ol o1-preview introddccing-operaalll--p-pperep
September 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:230..07193, 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787. PMLR, 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantago- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177,,22 2019.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2817-2826. PMLR, 06-11 Aug 2017.
Dean A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackic Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06173 2022.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with leamed model. Nature, 588(783)):604-6666 2020.
16

---

Published as a conference paper at ICLR 2026
Ingmar Schubert, Jingwei Zhang, Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin Riedmiller, Jost Tobias Springenberg. Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A generalist dynamics model for control. arXiv preprint arXhv:2305. 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06:: 06347, 2017.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 8583-8592. PMLR, 13-18 Jul 2020.
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505 2244,, 2025.
Lucy Xiaoyang Shi, Joseph J. Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. Conference on Robot Learning, 2022.
Arth Shukla.
Speeding up htpp://urtssuukk...ssssukkkccom Mar 2025.
with massively parallel simulation. URL https://arthshutt substack. honm:p/apeeding-uu---eee sac-eeeedinn-------------------- -e --w-mmiely-ppraalllee.
Arth Shukla, Stone Tao, and Hao Su. Maniskill-hab: A benchmark for low-level manipulation in home rearrangement tasks. arXiv preprint arXiv:2412. 122/12..321// 13211, 2024.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.04136, 2020.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503. 23829, 2025. URL https: / //tpxv/xvvv////////////ea////wav ievivi.orrrr. org/ has///b///// abs/2503. 23829.
R. Sutton. Learning predict by the methods of temporal differences. Machine Learning, 3:9 3:9 1998.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arna Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Ca- landra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. Robotics: Science and Systems, 2025.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, et al. Deepmind control suite. Technical report, DeepMind, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEFE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10. 109/ROOS.2012.6333330
Miguel Vasco, Takuma Seno, Kenta Kawamoto, Kaushik Subramanian, Peter R. Wurman, and Peter Stone. A super-human vision-based reinforcement learning agent for autonomous racing in Gran Turismo. Reinforcement Learning Journal, 4:1674-1710. 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 170 2017.
17

---

Published as a conference paper at ICLR 2026
Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, et al. Envpool: A highly parallel reinforcement learning environment execution engine. Advances in Neural Information Processing Systems, 35:22409- 22421, 2022.
Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control using covariance variable importance sampling. ArXiv, abs/1509.01145 2015.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. International Conference on Learning Representations, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2019.
Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin. A framework for effi- cient robotic manipulation. arXiv preprint arXiv:2012.07975, 2020.
Jesse Zhang, Minho Heo, Zuxin Liu, Erdem Biyik, Joseph Lim, Yao Liu, and Rasool Fakoor. EXTRACT: Efficient policy learning by extracting transferrable robot skills from offline data. In 8th Annual Conference on Robot Learning. 2024.
Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence, volume 3, 2008.
18

---

Published as a conference paper at ICLR 2026
APPENDICES
Webpage: http::/www.niccleelaeen.../cc//
Related work: Appendix A
Recommendations for future work: Appendix B
MMBench: Appendices C-F
Newt: Appendices G-H
Additional results: Appendices I-L
A Related Work
20
Recommendations for Future Work
21
Task domains
23
D Sample language instructions
26
E Demonstrations
28
Baselines
28
G Description of Planning Algorithm
30
H Implementation Details
31
I Open-Loop Control
34
J Additional Experiments
35
K Per-Task Results
37
Loss Curves
44
19

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Our work spans multiple research topics including (i) the development of new training environments, (ii) benchmarks for evaluation of multitask policies, (iii) algorithms and infrastructure for large-scale RL, and (iv) learning from demonstration. In the following, we aim to provide a comprehensive yet concise overview of related work along each of these axes.
Benchmarks for multitask RL. Historically, most RL benchmarks have been designed for single- task training and evaluation (Bellemare et al., 2013; Brockman et al., 2016; Tassa et al., 2018; Cobbe al., 2019; Tao et al., 2025). While some benchmarks evaluate policy learning and generalization within narrow task (Cobbe et al., 2019; Hansen & Wang, 2021; Kirk et al., 2023) or task family (Kolve et al., 2017; Yu et al., 2019; Savva et al., 2019; Kannan et al., 2021; Liu et al., 2023; Shukla et al., 2024; Park et al., 2025; Joshi et al., 2025). multi-domain learning and generalization in the context of RL remains mostly unexplored. Similarly, several works propose multi-embodiment datasets for imitation learning in robotics (Open X- X-Embbdiment Collaboration et al., 2023; Li et al., 2024; Khazatsky et al., 2024) but we have yet to see a comparably large initiative for multitask and multi-embodiment RI, let alone multi-domain RL. Our proposed benchmark MMBench consists of 10 distinct task domains, each of which contains a variety of different tasks and embodiments that all have reward functions, demonstrations, language instructions, and optionally visual observations.
Scaling RL. Existing literature has predominantly explored scaling of control policies in the context of imitation learning, where access to large demonstration datasets for supervised policy learning (e.g. behavior cloning) is assumed (Jang et al., 2021; Reed et al., 2022; Schubert et al., 2023; Brohan et al., 2023; Open X- X-Embodiment Collaboration et al., 2023; Octo Model Team al., 2024; Black et al., 2024). Perhaps most similar to ours, Reed et al. (2022) learns a multi-domant control policy via supervised learning on more than 63M demonstrations collected in various simulation environments. Although this is an impressive feat, relying on the availability of large amounts of expert demonstrations is highly impractical if not infeasible for many downstream applications, and the capabilities of the resulting agent is inherently limited by the behavior policy (e.g. human or learned specialist policy) that generated the demonstrations. For this reason, the community is increasingly turning to RL for training and finetuning of agents with superhuman capabilities in game-ple game-playing playing (Berner et al., 2019; Schrittwieser et al., 2020; Lee et al., 2022; Baker et al., 2022; Vasco al., 2025), reasoning and agentic AI (OpenAl, 2024; Guo et al., 2025; Su et al., 2025), and most recently robotics (Hafner et al., 2023; Hansen et al., 2024; Cheng et al., 2024; Miller et al., 2025; Nauman et al., 2025; DYNA Robotics, 2025). In particular, TD-MPC2 (Hansen al., 2024) presents a model-based RL algorithm that can be trained on up to 80 tasks across 2 domains (DMControl and Meta- (eta-World) using offline RL on large dataset of 545M transitions collected by specialist policies. However, RL still remains brittle to changes in learning dynamics (Fujimoto et al., 2018; Chen et al., 2021; Kostrikov et al., 2020; Hansen et al., 2023b; Farebrother et al., 2024), hyperparameters (Huang et al., 2022a; Hussing et al., 2024), and task specification (Clark & Amodei, 2016; Kirk et al., 2023) as evidenced by our ablations in Figure and our zero-shot results in Table 1. Additionally, scaling of RL is often bottlenecked by training infrastructure and interaction throughput (Espcholt et al., 2019; Weng et al., 2022; Tao et al., 2025; Shukla, 2025; Seo al., 2025). We provide a comprehensive benchmark for multi-domain RL accompanied by robust training infrastructure and a practical RI. algorithm Newt) that consumes various data sources and modalities.
Learning from demonstration. Demonstrations and offline data have been used to bootstrap poli- cies (Nakanishi et al., 2003; Ross al., 2011; Ho & Ermon, 2016; Pinto et al., 2017: Duan al., 2017; Peng et al., 2019; Kalashnikov et al., 2021; Baker et al., 2022; Ball et al., 2023), overcome the difficulty of exploration in sparse reward tasks (Vecerik et al., 2017: Hester et al., 2018; Rajeswaran et al., 2018; Zhan et al., 2020; Hansen et al., 2023a; Feng et al., 2023), and initialize multitask skills before online improvement with RL (Shi al., 2022: Shukla et al., 2024; Zhang et al., 2024; Lu et al., 2025). By providing demonstrations for 200 tasks, MMBench provides a new, challenging testbed for massively multitask imitation learning, online RL, and offline-toonnline RL in a common task suite that allows for fast, reliable, and reproducible measures of algorithmic improvement, and our agent Newt serves as a strong baseline for the community to build upon in this new paradigm. A research direction that we believe MMBench is especially suited for is finetuning of pretrained models (VLAs) using online RL as explored in Lu
generalist models such as assion-languge--ae et al. (2025) for narrower set of tasks than consider in this work.
20

---

Published as a conference paper at ICLR 2026
B RECOMMENDATIONS FOR FUTURE WORK
This work introduces MMBench, a benchmark for massively multitask RL, as well as Newt, a Inguuag-ccondtiones multitask world model trained with online RL. We demonstrate that online RL on hundreds of tasks simultaneously is indeed feasible, and that it can lead to world models with surprisingly strong generalization and open-loop control capabilities. We are excited by these encouraging results, and look forward to seeing in which ways the research community will build upon our work. In the following. we detail interesting open research questions and opportunities for future work in this area in hopes of inspiring new innovations. Specifically, we believe that the following topics will have great potential for impact over the coming years:
Visual RL. While MMBench and Newt fully support high-resolution (224 x 224) RGB obser- vations, the majority of our experiments in this work are limited to low-dimensional (128-d) state observations. Training visual RL policies has substantially higher hardware requirements, particularly so in terms of GPU memory if one is to store the replay buffer in GPU memory for fast sampling - and this is especially true in the massively multitask setting since it is necessary retain some amount of data for each task. We believe that improving the practicality of vi- sual RL remains an important research goal, and we see two future directions that could help alleviate the computational cost: (1) further pretraining of the visual encoder and world model on the provided demonstrations and/or large out-of-domait datasets, and (2) innovations in data pipelines that go beyond the typical first-in-first-ou replay buffer.
Language understanding. We find that language understanding is currently a bottleneck for task generalization despite leveraging pretrained language embeddings from CLIP (Radford et al., 2021). This may, in part, be due to the limited number of language instructions: one fixed instruction per task for total of 200 tasks. We expect language understanding to improve as the number of training tasks increases, but we also believe that additional techniques such as external datasets that contain language instructions, learning
data augmentation, pretraining from language at the token-level rather than embeddings, or any other techniques that seek to improve the agent's ability interpolate instructions will be immensely helpful.
Pretraining and improved base models. Pretraining of the Newt agent is currently limited to supervised learning on a demonstration dataset. Our ablations show that improving the pretrain- ing stage leads consistently better performance both pre- and post-RL, so we expect further improvements and scaling of the model pretraining to be a valuable research direction. We expect training of agents for embodied docision-making and control to eventually resemble the training recipe of contemporary reasoning models (OpenAI, 2024; Guo et al., 2025) in which significant resources are invested into large-scale pretraining of an agent before any RL is performed.
Neural architectures. Our Newt agent based on the TD-MPC2 (Hansen et al., 2024) archi- tecture which has proven to perform well across a variety of tasks and model sizes. However, the architecture remains rather simple: each component of the world model architccture is a de- terministic MLP (except for the Gaussian policy prior). We believe that leveraging more recent architectural innovations such the Transformer (Vaswani et al., 2017) and Diffusion Policy (?) could potentially lead to further performance improvements, but their integration into model- based RL. algorithms remains relatively unexplored.
Model capabilities. Our initial explorations into the capabilities of our Newt agent indicate that it is capable of open-loop control across hundreds of tasks, and that it can be rapidly adapted to new tasks and embodiments using online RL. However, further research into the emerging capabilities (and current limitations) of massively multitask agents is warranted. For example, the structure of the emerging (learned) latent state space and by extension dynamics modeling is not well understood, but we believe that improved understanding of emerging behavior in agents will help inform the design of future agents and neural architectures for mooell-based RL.
Learning strategies. We find convergence rate to differ substantially between tasks and task domains. This is perhaps not surprising as tasks vary greatly in complexity, degree of random- ization, and reward specification. While our experiments show that the provided demonstrations greatly improve data-efficiency as well as asymptotic performance of our Newt agent, we believe that more sophisticated training strategies may also lead to better data-efficiency and asymp- totic performance. Two directions that appear particularly interesting are (I) learning curricula that dynamically balance environment interaction (data collection) for each task based task
21

---

Published as a conference paper at ICLR 2026
progress, and (2) non-ume non-uniform sampling methods or training objectives that prioritize sampling from tasks or subtrajectories that are particularly useful.
Training environments and datasets. We believe that further scaling the environments and datasets used for training of world models will continue to drive performance and lead to bet- ter generalization abilities. Development of additional RI. environments using egg. procedural generation, agentic AI, other generative methods is thus promising research direction, in addition to training techniques that incorporate existing large-scale datasets from other domains.
Overall, we remain very optimistic about the future of massively multitask RL training, and hope that our emphasis on transparency and sharing of code, data, and checkpoints will inspire other researchers to do the same.
Appendices continue on the next page
22

---

Published as a conference paper at ICLR 2026
MMBench Appendices B-E
C
TASK DOMAINS
We consider 200 tasks across 10 domains. Our task set is comprised of diverse continuous control tasks spanning robot manipulation, locomotion, navigation, arcade games, and classic control prob- lems, each varying in task complexity, time horizon, observation and action space dimensionality. and reward formulation. Although camnot provide detailed information about every task due to the sheer number tasks considered, this section aims to give the reader a brief overview of the rypes of tasks used in our experiments. Table 4 provides overview of the task domains considered.
Table 4. Overview of task domains. Our selection of tasks cover wide range of task types, state and action dimensionalities, time horizons, and reward formulations. We summarize them below.
Task domain
State dim min
Action dim min
Ep. length min
Reward
RGB
max
max
max
DMControl
3
78
1
12
500
500
dense/sparse
224 224
X
DMControl Ext.
3
19
1
500
500
dense/sparse
224 224
Meta-World
39
39
4
100
100
dense
224 224
ManiSkil13
10
42
1
12
25
500
dense/sparse
224 224
MuJoCo
4
105
1
8
50
1000
dense/sparse
224 224
MiniArcade
4
44
1
2
200
500
dense/sparse
224 224
Box2D
8
24
2
4
500
500
dense
224 224
RoboDesk
73
73
5
5
100
100
dense
224 224
OGBench
9
42
2
8
100
1000
dense
224 224
Atari
128
128
3
3
1000
1000
sparse
224 224
C.1 DMCONTROL
DMControl (Tassa al., 2018) is benchmark for continuous control and features a variety of lo- comotion, manipulation, and control tasks with simple embodiments. It is available at https:// github. com/ eepogle-dess wedee-eeiieeeem nd/ com__oottlll... All tasks are designed as infinite-horioon MDPs (no termination conditions), and all tasks have a fixed episode length of 500 when an ac- tion repeat of 2 is applied, as is standard practice for this benchmark (Yarats et al., 2021; Hafner et al., 2023; Hansen et al., 2024). Reward formulation varies between tasks; some tasks have dense (shaped) rewards whereas others are sparse and only provide a non-zero reward on success. DM- Control tasks do not have notion of success, but episode returns are designed to be in [0, 1000] for every task. so we produce normalized score for this benchmark by scaling the returns to the [0. 1] interval. Note that this does guarantee that score of 1.0 can be obtained in all DMControl tasks,
23

---

Published as a conference paper at ICLR 2026
it is simply an upper bound. We consider 21 tasks from this domain. Examples of tasks included in our DMControl training set are Finger Turn Hard, Fish Swim, and Quadruped Run.
C.2 DMCONTROL EXTENDED
DMControl Extended is, as the name implies, an extended task set based on the original DMControl environment. We include custom tasks previously proposed by Hansen et al. (2024) (which are all available at https:/ www..t.mmpp..
com), and design an additional 5 locomotion tasks across new embodiments: Jumper, Spinner, and Giraffe. Tasks included in this task set follow the same recipe the original DMControl benchmark: a fixed episode length of 500 steps, and episode returns in [0, 1000] which we normalize by scaling down to [0, 1]. We consider 16 tasks from this domain. Examples of tasks included in our DMControl Extended training set are Walker Run Backward, Cheetah Jump, and Spinner Spin.
C.3 META- WORLD
Meta- World (Yu et al., 2019) is a benchmark for robotic table-top manipulation tasks with a Sawyer robot and end-effector position control, and is readily available https: / tig......... com/ Farama-Founde
on Son/Meawwolldd It consists of 50 diverse manipulation tasks that all share a common observation- and action space to facilitate multitask learning. We use a fixed episode length of 100 steps (after applying an action repeat of 2), and no terminal conditions. Meta-World provides dense reward functions and success criteria for each task in the task suite; we report binary task success at end of episode the normalized score for Meta- Meta-World tasks. We exclude one task, Shelf Place, from our training set due to an issue with the simulation, and consider the remaining 49 tasks from this domain. Examples of tasks included in our World training set Assembly, Bin Picking, and Lever Pull.
C.4 MANISKILLL3
ManiSkill3 (Tao et al., 2025) is an open-source simulator and task suite for embodied AL. It fea- tures a wide range of tasks and embodiments including tabletop manipulation, quadruped loco- motion, whole-body humanoid control, mobile manipulation, lar control tasks from MuJoCo and DMControl. Official documentation is available at https: / (raniiii1...ddddeeecss io. Observation and action spaces are defined a per-task ba- sis but is relatively consistent within a specific group of tasks. For example, most ManiSkill3 table- top robotic manipulation tasks share a common observation space, but the action space depends on the specific robot embodiment and control mode. Robot embodiments in our ManiSkill3 task set include Franka Emika Panda, S0100, and xArm6 for manipulation, and Anymal for locomotion, in addition to simpler MuJoCo-like embodiments such as Ant, Hopper, and Cartpole. Control modes include end-effector position control, end-effector 6D pose control, and joint position control. We use the default episode lengths for each task but apply an action repeat of 2 and use no terminal conditions. ManiSkill3 provides dense reward functions and success critera for most tasks in the task suite, but not all. We use a dense reward function when one is available. We report binary task success at end of episode as the normalized score for ManiSkill3 tasks whenever one is available, and episode return normalized to the j0, 1) interval otherwise. Note that this does not guarantee that a score of 1.0 can be obtained in all ManiSkill3 tasks, is simply an upper bound. We consider a to- tal of 36 tasks from this domain, of which 5 tasks new and created specifically for our paper; we intend to submit a pull request to the official ManiSkill3 code repository with our proposed task ad- ditions. Examples of tasks included in our ManiSkill3 training set are Stack Cube, Pick Screwdriver. and Anymal Reach.
well as reproductions of popu-
C.5 MUJoCo
MuJoCo (Todorov et al., 2012) is an open-source physics simulator and classic benchmark in the reinforcement learning and continuous control research communities. Official documentation for the tasks is available at https:/ gymmas um. farama, in this suite are diverse in terms of embodiments and all have different observation and action spaces. We use the default episode lengths for each taks but apply an action repeat of 2 for some of the tasks.
wegg/enviroe wonvvroommentss. s / /m joco. Tasks
24

---

Published as a conference paper at ICLR 2026
We use the v4 version of the tasks and disable all termination conditions to be in line with our other task domains. All tasks have pre-defined reward functions, some of which are sparse (only non-zero reward when success is reached), and report episode return normalized to the [0. 1] interval as normalized score for MuloCo tasks. We consider a total of 6 tasks from this domain. Examples of tasks included in our MuJoCo training are Ant, Half-cheetah, and Reacher.
C.6 MINIARCADE
MiniArcade is new task suite created specifically for this paper, implemented using pygame (https: gi thub. com/ ppmm///gam/////ee pygame). It consists of 22 tasks spanning 14 unique arcade-style environments. Tasks vary greatly in task objective, episode length, observation and action space dimensionality, well as reward formulation. All tasks have fixed episode length and no termination conditions. We design each task to fully support low-dimensional state repre- sentations, RGB image observations, and combination of the two. We define normalized return by pre-defined success criteria for tasks where it is appropriate to do so, and otherwise use episode return normalized to the [0, interval. As result, there is guarantee that tasks, it is simply an upper bound. We consider total of 19 tasks
of 1.0 can
be obtained in all Mini from this domain for training, and have fully open-soureed the entire task suite for the community to build upon. Examples of tasks included in our MiniArcade training set are Spaceship, Cocomut Dodge, and Phase-Evade
C.7 Box2D
Box2D is a small collection of 2D environments created for the original release of OpenAl Gym (Brockman al., 2016). Official documentation for the environments is available at https: / /gymmas i um, fa farama orgg /envi ronmentss // box2d. The tasks vary in task objective, episode length, and observation and action space dimensionalities. We use a fixed episode length for every task by disabling termination conditions. The Box2D tasks that we consider were originally designed for low-dimensional state observations, so we have modernized the implementations and added support for visual observations, including (visual) domain randomization to promote more generalizable visual control policies. Tasks in this task suite have pre-defined dense reward func- tions that we use for policy learning, and report normalized score as episode return normalized to the [o, 1] interval. Note that this does not guarantee that a score of 1.0 can be obtained in all Box2D tasks, it is simply upper bound. We consider total of 8 tasks and task variations based on the original OpenAl Gym environments; 5 for the Bipedal Walker environnment and 3 for the Lunarlander environment. Examples of tasks included in Box2l) training set are Bipedal Walker (Obstacles), Lunarlander Land, and Lunarlander Takeoff.
C.8 ROBODESK
RoboDesk (Kannan et al., 2021) is small suite of robotic manipulation tasks designed for mul- titask RL research. It includes object manipulation tasks in a single desk-themed environment, and all tasks share common observation and action space, An implementation of RoboDesk is available at https: : //gitt/////t// h/g/tttuut com/ /oom//golleeeeooooooooo researee robodesk. The benchmark ex- plicitly designed for visual RL but also supports low-dimensional state observations. All tasks have a fixed episode length and no terminal conditions, and both dense reward functions for policy learn- ing and success criteria for evaluation are provided; RoboDesk. We consider tasks from this domain. Examples of tasks included in our RoboDesk training set are Push Green, Open Drawer, and Place Flat Block in Bin.
the success rate normalized score for
C.9 OGBENCH
OGBench (Park et al., 2025) is a benchmark designed to facilitate research in offline goal- conditioned RL. It consists of multiple distinct environments and embodiments, and supports both low-dimensional state representations as well as RGB image observations. OGBench was developed with multi-goal but not eulti-emmbdimment learning in mind, and the action space dimensionality thus depends on the particular task. Tasks vary greatly in episode length, but we use a fixed episode length for each task without any terminal conditions. Because OGBench was not designed for online
25

---

Published as a conference paper at ICLR 2026
RL, we define dense reward functions for tasks where the observation space to include all relevant task information (e.g. goal position). Unlike the orig- inal OGBench environments available at https: //gttttubbb/// com/ seohongpark, /ogaanch, we treat different goal positions within the same scene configuration as the same task. To increase environment difficulty and diversity, create a series of novel scene configurations for the 21) point mass and Ant (quadruped) goal-conditioned navigation environments. We use success rate as normalized score for all but one task (Ant locomotion) which does not have clear metric of success; we thus use episode return normalized to the [0, 1] interval for this task. We consider 12 tasks based on the original OGBench environments; 5 tasks for the 2D point mass embodiment and 7 for the Ant embodiment. Examples of tasks included in our OGBench training set are And Soccer, Point Mass (Maze), and Ant (Bouleneck).
not previously defined, and modify
1.10 ATARI
The Arcade Learning Environment (Bellemare et al., 2013), often referred to as simply Atari or ALE, is an interface for a collection of diverse and often challenging Atari 2600 games origi- nally designed for human play. The ALE is long-standing benchmark for RL algorithms,, and has traditionally been used primarily for single-task visual RL benchmarking with discrete ac- tion space. More recently, Farebrother & Castro (2024) has proposed a non-linear continuous- to-discrete action transformation that extends support to agents with continuous action spaces, which makes it suitable task suite for our purposes. An implementation of ALE with support for continuous action spaces is available at /ttps:///gituii com/ Fa aama-- Foundat i Arcade Lea Learning- Envi ronment. However, several properties of ALE still make it chal- lenging to apply directly to our problem setting: (1) the ALE is built for visual RL and unlike our other task domains it does not support low-dimensional state representations for each game; and (2) games vary by several orders of magnitude in terms of episode length, with a single episode in games like Assault sometimes exceeding 100, 000 steps with human-level agent. To address the first challenge, we opt for the (admittedly not ideal) solution of using the raw game RAM state, which can be represented as a 128-dim vector, as our low-dimensional state representation. While we have verified that single-task agents are able to achieve non-trivial performance in all games con- sidered, we recognize that using the raw RAM state as observation limits the potential for cross-task knowledge transfer. We therefore expect visual policies to have better potential for generalization in the Atari domain. To address the second challenge, we simplify our problem setting by limiting agents to a fixed episode length of 1, 000. Each Atari game varies greatly in their reward formula- tion, but games generally assign sparse rewards only at key events such as scoring a goal or losing life. The games have no clear success criteria and we thus choose to use episode return normal- ized to the [0, 1] interval as normalized score for this domain. The linear transformation between episode return and normalized score is defined for each task and takes the shorter episode length into account. However, this does not guarantee that a score of 1.0 can be obtained in all Atari games, it is simply upper bound. We consider 27 tasks (games) from this domain. Examples of tasks included in our Atari training set are Ms. Pacman, Bowling, and Space Invaders.
D SAMPLE LANGUAGE INSTRUCTIONS
We manually define language instructions for all 200 training tasks considered in this work. Our language instructions have two components: (1) description of the embodiment, including how many controllable joints or action dimensions the given task has; and (2) a short task instruction in plain language. We make an effort to be consistent in our description of embodiments, action spaces, and tasks, while still providing enough language variation for generalization to be feasible. We provide a small subset of our language instructions in the following.
Walker Run (DMControl)
Embodiment: 2D walker with 6 controllable joints across 2 legs. Instruction: Run forward as fast as possible.
26

---

Published as a conference paper at ICLR 2026
Walker Run Backward (DMControl Ext.)
Embodiment: walker with controllable joints across 2 legs. Instruction: Run backward quickly.
Jumper Jump (DMControl Ext.)
Embodiment: jumper with controllable joints across legs. Instruction: Jump as high as possible.
Plate Slide (Meta-Worid)
Sawyer robot with 2-finger gripper and delta end-effector position control. Instruction: Slide the hockey puck into the goal.
Embodiment:
Anymal Reach (ManiSkill3)
Quadruped (Anymal robot) with controllable joints across 4 legs. Instruction: Reach target.
Embodiment:
Stack Cube (ManiSkill3)
Franka robot with 2-finger gripper and delta end-effector 6D GD-pose control. Instruction: Stack the red cube on top of the green cube.
Embodiment:
Half-cheetah (MuJoCo)
Embodiment: 2D MuJoCo half-cheetah with controllable joints.
Instruction: Run forward as fast as possible.
Bipedal Walker Obstacles (Box2D)
Embodiment: 2D cartoonish bipedal walker with controllable joints across legs. Instruction: Traverse the various obstacles quickly.
Lunarlander Land (Box2D)
Embodiment: 2D lunar lander with 2 controllable thrusters.
Instruction: Land safely on the landing pad.
Open Slide (RoboDesk)
Embodiment: Franka robot with 2-finger gripper and delta joint position control. Instruction: Open the sliding door.
Ant Maze (OGBench)
Embodiment: Ant (quadruped) with 8 controllable joints (4 legs). Instruction: Navigate through maze to reach the red goal.
Bird Attack (MiniArcade)
Embodiment: 2D shooter game with controllable actions: move and fire.
Instruction: Eliminate all enemies in a space-invaders style game.
Chase & Evade (MiniArcade)
Embodiment: 2D chase and evade game with X and movement. Instruction: Take turns being the chaser or evader.
Ms. Pacman (Atari)
Embodiment: Atari game with controllable actions: radius, theta, and fire. Instruction: Eat pellets and avoid ghosts in game of Ms. Pacman.
27

---

Published as a conference paper at ICLR 2026
E DEMONSTRATIONS
To aid in research pretraining + finetuning of massively multitask agents provide pert demonstration dataset for MMBench. This dataset contains 10-40 demonstrations per task (depending task difficulty) for a total of 4020 demonstrations, which are all collected by single- task TD-MPC2 (Hansen et al., 2024) agents trained from low-dimensional state observations. Each single-task agent is trained for 5M environment steps (Atari: 10M) ensure that performance has converged, although we observe that training converges carlier than that on most tasks. Training single-task agent for 5M environment steps takes approx. hours on a single NVIDIA RTX 5090 GPU, totaling an estimated 1,300 GPU hours to train expert policies for every task in MMBench.
To ensure that only successful demonstrations are collected, we reject episodes that do not result in success (when available) or have episode returns that are substantially (> 25%) lower than the median for that task. Table provides an overview of our demonstration dataset. For completeness. also release all 200 checkpoints that generated the provided demonstrations; these checkpoints can be used to generate additional checkpoints, or be used for e.g. teacher-student policy distillation.
Table 5. Demonstrations. An overview of the demonstrations provided for each task domain.
Task domain
Num. tasks
Demos per task
Total demos
DMControl
20
420
DMControl Ext.
16
20
320
World
10
490
ManiSkil13
20
720
MuJoCo
20
120
MiniArcade
10
190
Box2D
40
320
RoboDesk
20
120
OGBench
20
240
Atari
40
1080
Total
200
4020
F BASELINES
BC. Our multitask behavior cloning (BC) implementation closely follows that of our pretraining stage for Newt, except that only consists of two components: the encoder h and policy p. Both encoder and policy are conditioned on language instructions, and are trained with a BC objective that mask invalid action dimensions such that all action vectors contribute equally the loss regardless of number of valid dimensions for that particular task. Our single-task BC policies are identical to the multitask BC policies except they are not conditioned on language instructions.
PPO. We base our PPO implementation off of the cleanRL (Huang et al., 2022b) implementation available here: https: thub. com vwxyz wttp///aa////eeeeeeeeeeeeeeeeeee ppo_cont intoossatttososnooon_oooo saacgym, and extend it to support language-conditionig per-task discount factors such that comparison to our method is fair. We find that the default imple- mentation and hyperparameters performs poorly in our massively multitask RL setting, a number of additional changes. Specifically, we find that (i) using a tanh-Normal policy improves training stability, that (ii) increasing the number of trainable parameters from 0.5M to 3.5M leads to marginally better performance throughout training, and that (iii) using longer policy rollouts (128), more mini-batches and epochs (8), smaller value loss coefficient (1), scaled down environment
master c e
and
make
28

---

Published as a conference paper at ICLR 2026
rewards (0.1 x), and clipping of both the surrogate objective and value function. An empirical com- parison of our PPO implementation before and after these changes can be found in Appendix J; we find that these changes approximately double its average score at 100M environment steps, but that both implementations perform relatively poorly compared to FastTD3 and Newt.
33stTD3. We base TD3 implementation off of FasTTD3 (Seo et al., 2025) which is avail- able here: httpss:/ge // trithhu.. com/ haan//yoogggseeooooo Sacassse//aatttee Fastt an improved implementation of TD3 designed for the era of RL with large amounts of parallel environments. FastTD3 demon- strates robust learning across a number of different benchmarks described in Seo et al. (2025), but has not previously been evaluated in a multi- setting. We extend FastTD3 to support language-con condiiiiionn for compatibility with MMBench, as well as per-task discount factors to make value estimation easier across tasks. We find that n-step returns of are critical our chal- lenging multitask setting and we thus use that value by default. Appendix J shows the performance of FastTD33 for different values of n.
Model-based pretraining. This baseline is equivalent to the pretraining stage of Newt. Specifically, we pretrain all components from Equation as described in Section 3.2 for total of 200, 000 iterations, and report performance results without planning (simply using the policy prior as behavior policy) as we find it to perform the best immediately following the pretraining stage; planning eventually outperforms the policy prior but initially suffers from an inaccurate value function.
Appendices continue on the next page
29

---

Published as a conference paper at ICLR 2026
Newt
Appendices F-G G
G DESCRIPTION OF PLANNING ALGORITHM
Our planning algorithm closely follows that of TD-MPC2, which variant of the Cross- Entropy Method (CEM) or Model Predictive Path Integral (MPPI; (Williams et al., 2015)) for tra- jectory optimization with the learned world model. Specifically, TD-MPC2 plans trajectories by fitting a time-dependent multivariate Gaussian with diagonal covariance to a value landscape esti- mated with rollouts from the world model. Starting from an initial distribution parameterized by (H",
μra o E R" A E R", where t is the current time step and H is the planning hori- zon, we iteratively refit parameters (H. of "elite" trajectories (those whose estimates are in the top-K of trajectories sampled that given iter- ation), and sample another set of trajectories based on the updated parameters. After a fixed number of planning iterations, the algorithm samples trajectory from the final optimized distribution and the agent exccutes the first action in the returned sequence.
weighted average of the value estimates of small set
When evaluating our agent in a closed-loop manner, we replan trajectories every time step and initialize nº, o with the one-step shifted parameters the previous time step (receding horizon MPC). When evaluating in an open-loop mamner, the planning procedure returns the entire action sequence and we execute them one by one without replanning nor any environment feedback. To warm-start the planning procedure, TD-MPC2 generates fixed number of action sequences by rolling out the model (in the latent space) using actions coming from the policy prior and adds them the set of action sequences sampled at every iteration of the planning procedure. Our proposed constraint on the planning procedure early in training (when the model is less accurate than the policy prior pretrained on demonstrations, as we discuss in Section 3.2) initializes (μ² o²) as a linear interpolation between the action distribution coming from the policy prior p and the Guassian prior used TD-MPC2. Our annealing schedule is visualized in Figure II. We refer the reader to Hansen et al. (2022) and Hansen et al. (2024) for more information about the planning procedure.
Constrained planning
Annealing schedule
1.0
0.5
0.0
20M
40M
Environment steps Ours
Figure 11. Constrained planning. We find that biasing planning towards the action distribution of the pretrained policy helps with exploration early in training and mitigates a drop in performance when switching from the pretrained model-free policy planning with world model. We use simple linear annealing schedule (from 2M to 12M steps) as shown here.
30

---

Published as a conference paper at ICLR 2026
H IMPLEMENTATION DETAILS
Language embeddings. We use text embeddings from (CLIP; (Radford et al., 2021)) which outputs continuous embeddings of 512 dimensions. Embed- dings are visualized in a low-dimensional space in Figure 5.
Image embeddings. We use RGB image embeddings from facebook/ NOv2; Oquab et al. (2023)) which outputs continuous embeddings of 768 dimensions. We do use frame stacking our experiments, but expect it to improve downstream performance as the added history information allows models to infer e.g. velocity information from visual observations.
(DI-
Network architecture. All learnable components from Equation 1 are implemented MLPs. The encoder h contains variable number of layers depending on the architecture size, and we also vary the number of Q-functions in our ensemble when training larger or smaller models than our default model size of 20M learnable parameters. MLPs consist of linear layers followed by LayerNorm (Ba et al., 2016) and Mish (Misra, 2019) activation functions, and the latent state representation normalized using simplicial embedding as originally proposed by Lavoie et al. (2023) and later applicd in an RL context by Hansen et al. (2024). We summarize our Newt architecture for the default 20M parameter state-based agent with language-conndttonnng using a Py -ircch-like notation:
Encoder (2, 235, 904 parameters): ModuleDiet
(state): Sequential
Normedtinear Seatureeure, out_ 1224 bias-True, act-Mish)
(1): Norme NormedLizeaz fen_ffeature=1021, out cat_features=122) bias-True, act-Mish)
hormedlinear fon_fessures=1224, out_features=512, bieseIrue, actmaimplicial)
Dynamice (2, 601 pazameters): Sequential
(0): NormedLinear (in_feaatree=10000 out out_features=1020, bias=True, ast-Mish)
KormedLinea: (in, fea, out_Leaturee-1021, bias-True, act-Mish) Sut_featuras=512 bias=True, act=Simplicisi)
(2): KormedLinear (in_festures=100
Reward (2,223, 205 parameters) Sequential
(01: KormedLinear (in, 1e40 out_Leatuvee-1123, bias-True, act-Mish)
(1): KormedLinear (in_fest:ues--001, out_featurre=1021, bias-True, act-Mish)
(2): Linear Linear (ia_aa/urssfttre222222 ont_featuues=100, biasstrue)
Policy prior 12, 12,136,08 096 parameters): Sequentiel(
(0) Kormadtinear fin_footurcs=1024 aut_foatures=1124, bias=True, ast=Mish)
(11: Kormedt fatii__eaaureeeeaa out_ uut_features=1024 bias-True, act-Mish)
(2): Linear [in__eeteeeeeeeee=1122222 out_festures=32, bias-True!
O-functions (11,116,02 025 parametera): (Ensemble(
(Qs): Kodulelist
(0-4): 5 Sequential(
(0) NormedMdned it 1040 oar_featurea-1024 blaseTrue, act-Misht
(1): NormedLineat tin teauuereeee, out_featurea-1024 blas-True, act-Miah)
: Linear (in out 1el bias-True!
When the agent is conditioned on image embeddings in addition to state and language, the encoder expanded as follows:
Encoder (3, 022, 336) NoduleDict
(state): : Sequentiall
HormedLineez 10__ Let__eeeure==1224, bias-True, act-Mish) (1): Normedt:/ near (1n_ to__turusss10024 out out_ reatures-1024,, blas-True, act
(2): HormedLineaz (in fen_teaaure--121,, out fet_ueeau2---512 biss-Tzue, act-Simplicial)
and the total parameter count increases to 21M. We experiment with four model sizes: 2M pa- rameters, 5M parameters, default model with 20M parameters, and a larger model with 80M parameters; Table 6 details the specific architecture for each model size.
Latent state normalization. The simplicial embedding (Lavoie et al., 2023) is simple method for normalization of the latent representation by projecting into L fixed-dimensional simplices using softmax operation, which first applied in an RL context by Hansen et al. (2024). A key benefit to e.g. discrete representation or squashing) that
of embedding z as simplices (as opposed naturally biases the representation towards sparsity without enforcing hard constraints. Intuitively,
31

---

Published as a conference paper at ICLR 2026
the simplicial embedding be thought of "soft" variant of the rector-of-categorice approach to representation learning proposed by Oord et al. (2017) AVQ-VEE) Whereas VQ-VAE represents latent codes using set of discrete codes (L vector partitions each consisting of one-hot encoding), a simplicial emhedding partitions the latent state into I. vector partitions of continuous values that each sum to 1 due to the softmax operator. This relaxation of the latent representation is akin to softmax being relaxation of the arg max operator. We follow Lavoic et al. (2023) and TD-MPC2 (Hansen et al., 2024) without modification and implement the simplicial normalization layer using Py Tikeh-like notation as follows:
def simplictal (salf, 5, V=8)
shape 2. 2.appe 2.vicM (ogtem(sahhpppp (*shape [:-1], -1, V)
sottmax (8, dim--1)
return 2. view vive (Ashape)
Here, is the latent representation z, and is the dimensionality of each simplex. The number of simplices L can be inferred from V and the dimensionality of z. We apply softmax to each of L partitions of z to form simplices, and then reshape to the original shape of Refer to Lavoie et al. (2023) for more details.
Table 6. Model sizes. Architecture changes that we make for our scaling experiments. Encoder dim is the dimensionality each hidden layer in the state encoder, MLP dim is the dimensionality of hidden layers in all other parts of the agent, Latent dim is the dimensionality of the latent state representation, enc layers is the number encoder layers, and Q-ensemble is the number Q- functions in the ensemble. We highlight our default model parameters in bold.
Parameters
2M
5M
20M
80M
Encoder dim
128
256
1024
2048
MLP dim
256
512
1024
2048
Latent dim
384
512
512
704
# enc layers
2
2
3
4
Q-ensemble
5
5
7
Heuristic for discount factors. Following TD-MPC2, we use the following heuristic to determine suitable discount factor for tasks for which no default discount factor is available:
T
1
7 = clip(
[0.95, 0.995])
(4)
where T' is the episode length after applying action repeat, and clip bounds the discount faccor to the interval [0.95, 0.995]. This heuristic ensures that tasks with short episodes are assigned a low discount factor, and vire-versa for tasks with long episodes.
Hyperparameters. Our hyperparameters are listed in Table 7 for the default 20M parameter agent.
32

---

Published as a conference paper at ICLR 2026
Table Hyperparameters. We detail relevant hyperparameters for method below, corre- sponding to our default 20M parameter agent. Our choice of hyperparameters closely follow that of TD-MPC2 (Hansen al., 2024) with a few exceptions which we highlight bold.
Hyperparameter
Value
Planning
Horizon (H)
3
Iterations
Population size
512
Policy prior samples
24
Number of elites
64
Minimum std.
0.05
Maximum std. Temperature Momentum Constrained planning
2
0.5
No
Anneal from 2M - 12M steps
Policy prior
Log std. min. Log std. max.
-10
2
Replay buffer Capacity Sampling
10, 000, 000
Uniform
Architecture (5M)
Encoder layers
3
Encoder dim
1024
MLP dim
1024
Latent state dim Activation
512
LayerNorm + Mish
Number of Q-functions
5
Number of reward/value bins
101
Simplicial dim (V)
8
Simplicial temperature (r)
1
Optimization
Update-to-data ratio Batch size
0.075
1024
Joint-embedding coef.
20
Reward prediction coef.
0.1
Value prediction coef.
0.1
BC coef.
10
Temporal coef. (A)
0.5
Q-fn. momentum coef.
0.99
Policy prior entropy coef.
1 x 10-4
Policy prior loss norm
Moving (5%, 95%) percentiles
Optimizer
Adam
Learning rate
3 x 10
Encoder learning rate
1 x 10
Gradient clip norm
20
Discount factor
Heuristic
33

---

Published as a conference paper at ICLR 2026
Additional results
Appendices H-K
I OPEN-LOOP CONTROL
Finger Turn Hard (DMControl)
Assault 16
(Atari)
1=0
=
16
t=
32
t 48
t 0
t 32
48
Poke Cube
(Maniskill)
Push Green (RoboDesk)
1= 0
==8 8
=
t= 24
t= 0
1=8
t= 16
24
Figure 12. Additional qualitative results for open-loop control. Executing open-loop plans with- out any environment feedback. Our results demonstrate that Newt learns meaningful representations.
Table 8. Open-loop control. Quantitative results when planning for 48 or 24 consecutive time steps, depending on the task. We report the fraction of original closed-loop control performance achieved in open-loop manner after 33%, 66%, and 100% of the planned time steps. Higher is better
Task
1 16
32
48
Walker Walk
0.68
0.40
0.25
Finger Turn Hard
0.41
0.39
0.29
Assault
0.00
0.10
0.08
Lunarlander Takcoff
0.67
0.40
0.34
Average
0.44
0.32
0.24
Task
[ 8
16
24
Point Maze
0.88
0.67
0.37
Pick Screwdriver
0.95
0.89
0.83
Poke Cube
0.50
0.38
0.35
Push Green
0.95
0.89
0.83
Average
0.82
0.71
0.60
34

---

Published as a conference paper at ICLR 2026
J ADDITIONAL EXPERIMENTS
DMControl
DMControl Ext.
Meta-World
ManiSkill
MujoCo
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
Box2D
RoboDesk
OGBench
MiniArcade
Atari
0.4
0.8
0.6
0.6
0.3
0.6
0.4
0.4
0.2
0.2
0.4
0.2
0.2
0.1
0.0
0.0
0.0
0.0
0.0
50M
100M
50M
100M
50M
100M
50M
100M
50M
100M
BC
Model-based pretraining
No demos
No language
Ours
Figure 13. Per-domain performance (additional baselines). Average score of single agent in each of the 10 task domains MMBench (200 tasks). shown in Figure 14.
These baselines complement the results
DMControl
DMControl Ext.
Meta-World
ManiSkill
MujoCo
0.6
0.6
0.3
0.4
0.4
0.2
0.2
0.2
0.1
Box2D
RoboDesk
OGBench
MiniArcade
Atari
0.4
0.8
0.6
0.6
0.3
0.6
0.4
0.4
0.2
0.2
M
0.2
M
0.2
0.1
0.2
0.0
0.0
0.0
0.0
0.0
50M
100M
50M
100M
50M
100M
50M
100M
50M
100M
No planning
Ours
Figure 14. Per-domain performance (ablation on planning). Average score of single agent in each the 10 task domains MMBench (200 tasks). We compare the performance our method with and without planning with MPC. The default formulation of Newt uses planning; the baseline without planning instcad uses the learned policy prior for decision-making Our results indicate that planning plays key role in massively multitask online RL, but that the policy prior in rare cases (MuJoCo) may outperform planning.
35

---

Published as a conference paper at ICLR 2026
FastTD3 n-step
PPO hyperparams
200 tasks
200 tasks
0.5
0.4
0.2
0.0
20M 40M
60M 80M 100M
20M
40M
60M BDM 100M
Environment steps
Environment steps
n
n 16
Our PPO
Ours
n
Ours
Default PPO
Figure 15. Ablations on baselines. We compare baseline performance across different hyperparam- eter configurations, and find that has moderate impact on data-efficiency and overall performance.
Table Comparison to single-task TD-MPC2 agents. Performance (normalized score; higher is better 1) comparison between our multitask Newt agent trained for total of 100M environment steps across all tasks (500k per task), and single-task TD-MPC2 agents trained until convergence (5M environment steps per task; IB total steps across all tasks). We also include more fair compar- isons in which the population of single-task agents have been exposed to the same total number of environment steps
Newt agent. We observe that the multitask agent generally lags behind the population of single-task agents in terms of performance but that the extent to which that is the case varies drastically between task domains.
Multitask
Single-task
Task domain
Num. tasks
Newt
TD-MPC2
20M
@ 100M
@20M
@ 100M
@ @IB
DMControl
0.49
0.50
0.32
0.67
0.81
DMControl Ext.
0.26
0.37
0.24
0.60
0.78
Meta- World
0.32
0.50
0.53
0.70
0.79
ManiSkil13
0.52
0.69
0.36
0.77
0.92
MuloCo
0.18
0.20
0.35
0.64
0.82
MiniArcade
0.40
0.44
0.50
0.75
0.94
Box2D
0.19
0.33
0.60
0.87
0.86
RoboDesk
0.42
0.38
0.48
0.67
0.82
OGBench
0.29
0.30
0.43
0.65
0.93
Atari
27
0.14
0.13
0.13
0.24
0.51
Total
200
0.31
0.44
0.38
0.65
0.80
36

---

Published as a conference paper at ICLR 2026
K PER-TASK RESULTS
Walker Stand
We lker
Walker Run
Cheetah Run
Reacher Easy
MMMM
0.5-
m
0.00
Swinqup
Pendulum Swinqup
Cartpale
Sparse
Am
Cartpole Swinngup
Cartpole Swingup Sparse
Catch
Finger Spin
Finger Tumn Easy
M
MMM
MM
Finger Turn Hard
Fish Swim
Hopper Stand
Hopper Hop
Quadruped Walk
MyMa
0.5
M
Amg
0.0
Quadruped
1.0
S0M
100M
Environment steps
Ours
Figure 16. DMControl. Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
37

---

Published as a conference paper at ICLR 2026
We
Cheetah
Cheetah
Cheetah
1.0
M
0.5
Mmma
Cheetah Jump
Hopper Hop Backward
Reacher Three Easy
Reacher Three Hard
Spiin
M
0.0
Pendulum Spin
Jumper Jump
Spinner Spin
Spinner Spin Backward
Spinner Jump
1.0
wwwmm
Giraffe Run
0.5
0.0
50M
100M
Environment steps
Ours
Figure 17. DMControl Extended. Per-task normalized scores for our 20M parameter Newt agent jointly trained on 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
38

---

Published as a conference paper at ICLR 2026
Assembly
Basketball
Button Press Topdown
Button Press Tapdown Wall
Button Press
1.0
r
w " W
0.5
Button Press Wall
Coffee Button
Coffee Pull
Coffee Push
Dial Turn
MY / W
mmm
0.5
wwm
0.0
Disassemble
Open
Drawer
Drawer Open
M
0.0
Foucet Open
Faucet Close
Hommer
Handie Press Side
Handle Press
1.0
MamM
M V
M
you Lever Pull
0.5
Handle Pull Side
Handle Pull
Peg Insert Side
Pog Unplug Side
mapoo
0.5
0.0
Wall
Plate Slide Side
M Monnmim
0.0
Plate Slide Back
Plate Slide Back Side
Push Back
Push
Push Wall
1.0
Reach
Reach Wall
Soccer
Stick Push
Stick Pull
Myl
0.5
0.0
Sweep
Sweap
Window Open
Window Close
Bin Picking
M M'M
M
M
0.0
1.0
0
50M
100M 0
S0M
100M 0
S0M
100M
S0M
100M
Environment steps Environment steps Environment steps Environment steps
Ours
Figure 18. Mela-World. Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
39

---

Published as a conference paper at ICLR 2026
Ant Walk
Ant Run
Cartpole Balance
Cartpole Swingup
Hopper Stand
MM
Hopper Hop
Pick Cube
Pick Cube EE: EE-pose
Pick Cube S0
Poke Cube
Amw
0.5
0.0
Push Cube
Pull Cuse
Pull Cube Tool
Stack Cube
Place Sphere
MmM
0.5
0.0
Lift Peg
Pick Apple
Pick Banana
Pick Can
Pick Hammer
1.0
mammmm
0.5
0.0
1.0
Pick
Knife
Mug
Orange
Pick Screwdriver
0.5
0.0
Pick
Tennis
Baseball
Pick Ammm
Pick Sponge
1.0
m
Anymal Reach
Reach
Reach EE
Reach xArmó
Cartpole Balance Sparse
Cartpole Swingup Sparse
0.0
0
S0M
100M
Environment steps
Figure 19, ManiSkill, Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench, Environment steps refers to total steps across all tasks.
40

---

Published as a conference paper at ICLR 2026
Ant
Half-cheetah
Hopper
Inverted Pendulum
Reacher
mw
0.0
0.5
100M
Environment steps
Ours
Figure 20. MuJoCo. Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
Bipedal Walker Flat
Bipedal Walker Uneven
Bipedal Walker Rugged
Bipedal Walker Hills
Bipedal Walker Obstacles
1.0
Aund
0.0
Lunarlander Land
Lunarlander Hover
Lunarlander Takeott
mm
mym
0.0
50M
100M
50M
100M
50M
100M
Environment steps
Environment steps Environment steps
Ours
Figure 21. Box2D. Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
Push Red W
Push Green
Push Blue Mmmap
Open Slide
Open Drawer
0.5
0.0
0.5
100M
Environment steps
Ours
Figure 22. RoboDesk. Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
41

---

Published as a conference paper at ICLR 2026
Push
Open Open
1.0
m
0.0
Flat Block In Bin
0.0
50M
100M
Environment steps
Ours
Figure 23. OGbench. Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
Cowbey
Coinrun
Spaceship
Pong
mww m
Highway
Hockey
Collect
mmm
My
mm mmmmam Mmm
Dodge
Cartpole Swingup
Sparse
Cartpole Swingup Sparse
1.0
S0M
100M
50M
100M
S0M
100M
50M
100M
Environment steps
Environment steps
Environment steps Environment steps
Ours
Figure 24. MiniArcade. Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Emvironment steps refers to total steps across all tasks.
42

---

Published as a conference paper at ICLR 2026
1.0
wwmmmm
Battle Zone
Beamrider
Boxing
Chopper Command
Crazy Climber
0.5
wwm.aa
www.mm
Double Dunk
Gopher
Ice Hockey
Jamesbond
Kangaroo
Wmwmmmm
mmm.mm
0.0
L Para
1.0
Pacman
This Game
Pong
www.
Nm
Road Runner
Robotank
Seaquest
Space Invaders
Tutankham
0.5
Up N'Down
Yars Revenge
0.0
50M
100M 0
SÓM
100M
Environment steps Environment steps
Ours
Figure 25. Atari Per-task normalized scores for our 20M parameter Newt agent jointly trained on all 200 tasks from MMBench. Environment steps refers to total steps across all tasks.
43

---

Published as a conference paper at ICLR 2026
L Loss CURVES
Total loss
Self celf-prediction loss
Reward loss
200
200 tasks
200 tasks
0.52
0.0020
0.16
M
0.50
0.0015
0.48
0.12
0.46
0.0005
20M
40M 60M aDM 100M
20M 40м
60M adM 100M
20M
40M
60M 80M 100M
Environment steps
Environment steps
Environment steps
No language
Ours
No language
Ours
No language
Ours
Value loss 200 tasks
Policy loss 200 tasks
Grad norm 200 tasks
0.80
0.00
0.75
-0.05
-0.10
Mamam MM
0.65
00.55
0.4
0.60
-0.20
0.2
20M 40M
60M 80M
100M
20M 40M
6OM 100M
20M 40M 60M 80M 100M
Environment steps
Environment steps
Environment steps
No language
Ours
No language
Ours
No language
Ours
Figure 26. Loss curves for our method with and without language instructions. These curves indi- cate that learning is stable across both agents, but that the agent with access to language instructions generally has a lower loss during training.
44


Reviews:
[{"review_id": "W8pvGCVoVm", "reviewer": "Reviewer_nJ1K", "summary": "This paper proposes MMBench, a multitask RL benchmark containing 200 different control tasks across 10 domains. TDMPC2 agents on each single task are trained to collect expert demonstrations, while both the model checkpoints and demonstrations are open-source. In addition, this paper substitute language descriptions for task indices for better distinguish among tasks, which makes Newt,  a multi-task world model based on TD-MPC2. Experiments show comparable results with a strong baseline FastTD3.", "questions": "1. To make it clear, is there only one agent being trained to interact with all 200 environments and collect online trajectories?\n\n2. What does the \"Language Instructions: None\" refer to in Line 399-405? Is there a task index provided for each trajectory?\n\n3. Is there any results that support the claim in Line 147 that one-hot encoding limits the potential for transferring to unseen tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes a benchmark which integrates domains that are popularly studied in the RL community, releasing single-task checkpoints and dataset which is significant not only for multi-task RL but also offline, O2O and continuous RL research.\n2. The empirical results show advantages over baselines on ManiSkill and DMControl. \n3. Model info such as training time, model architecture is detailed presented.\n4. The figures are well drawn and easy to understand.", "weaknesses": "1. There is no preliminary description so the problem setting confuses me at the beginning. In the multi-task RL setting a task label $n$ should be added to the original $(s, a, s', r)$ but in Line.275 there is only $(s, a, r)$. \n\n2. Newt only shows performance boost over baselines in DMC and Maniskill out of all 10 domains. In Meta-World, MuJoCo, Box2D, Robodesk and Atari it's just on par with FastTD3, while in OGBench and MiniArcade it's on par with behavior cloning. \n\n3. Selected baselines are not strong enough. FastTD3 is a strong baseline and in its original paper is compared with strong model-based baselines such as TDMPC2 and Dreamerv3, but it mainly reports results on humanoidbench, mujoco playground and Issaclab, neither of them are included in MMBench. Moreover, as long as Newt is built upon TDMPC2, it surprises me that TDMPC2 is not listed as a baseline in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55C1gPr24b", "reviewer": "Reviewer_73Ch", "summary": "This paper tackles the scalability challenge of online RL for continuous control by introducing MMBench, a large-scale benchmark spanning 200 diverse tasks across 10 domains (e.g., DMControl, Meta-World, Atari, and a newly proposed MiniArcade). Each task includes language instructions, demonstrations, and multi-modal observations. The authors also propose Newt, a language-conditioned multitask world model built upon TD-MPC2. Newt first pretrains on demonstrations to obtain task-aware representations and action priors, then performs online optimization across tasks with architectural refinements and action supervision. Experiments show that Newt outperforms several baselines (BC, PPO, FastTD3) in multitask performance and exhibits generalization to unseen tasks.", "questions": "1. The paper employs masking to handle inconsistent action/state dimensions. Could masking lead to sparse gradient updates or optimization inefficiencies for high-dimensional action spaces (e.g., humanoid control with 50+ joints)? Have you explored task-specific action embeddings or shared latent action representations as scalable alternatives?\n\n2. Table 3 reports Newt’s training time, but it is unclear whether this includes the cost of training 200 single-task TD-MPC2 agents used for demonstration generation. Please clarify whether the total computation time includes both stages (demo collection + multitask training) to ensure fair comparison and full cost transparency.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper has several notable merits:\n- MMBench provides a unified framework across 10 heterogeneous domains with consistent data handling and language-conditioned tasks.\n- The paper introduces reasonable design choices such as discrete regression for reward/value prediction and per-task discount factors, with comprehensive ablations supporting their impact.\n- The paper is well-organized, figures effectively illustrate key results, and open-sourced resources (200+ checkpoints, 4000+ demos) significantly enhance reproducibility.\n\nOverall, the paper represents a solid step toward scalable, general-purpose control systems.", "weaknesses": "### 1. Novelty concerns in core contributions\n\nWhile MMBench contains 200 tasks, most of them are directly inherited from existing benchmarks (e.g., DMControl, Meta-World). This limits its novelty compared to benchmarks such as ManiSkill3, which introduces fundamentally new task paradigms. Similarly, Newt—although incorporating CLIP/DINOv2 encoders and demonstration conditioning—builds incrementally upon TD-MPC2, without a clear paradigm shift.\n\n### 2. Missing analysis of task scalability\n\nThe paper emphasizes scaling to hundreds of tasks but lacks quantitative analysis on scaling behavior. For example, performance is not evaluated as the task count increases (e.g., 50 → 100 → 200 tasks), leaving unclear whether multitask training indeed benefits from more tasks. Moreover, since MMBench spans disjoint domains (e.g., Atari vs. Meta-World), the paper should analyze cross-domain transfer—whether training on visually distinct domains interferes with or enhances learning in others—and include ablations comparing full multitask vs. domain-specific training.\n\n### 3. Incomplete baselines in MBRL\n\nExperimental comparisons primarily focus on model-free RL and behavioral cloning baselines, while omitting competitive model-based RL counterparts. In particular, DreamerV3, which is explicitly designed for multitask continuous control, is absent, as is a multitask-adapted TD-MPC2 baseline (used for demo collection). Without these, it is difficult to attribute performance gains to the proposed architectural innovations rather than inherited advantages from TD-MPC2.\n\n### 4. Insufficient evaluation under state-limited conditions\n\nMost evaluations assume access to full low-dimensional states, which is unrealistic in real-world control settings. The paper lacks experiments under state-limited or purely visual conditions (e.g., partial observations or agent-only states), which would better demonstrate robustness and practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZyO9oP7m6d", "reviewer": "Reviewer_uUxq", "summary": "The paper presents an architecture to perform online RL on a bigger scale, training one policy for many different tasks. The learning algorithm requires as a first step a behavioural cloning approach to warm-start the network and make the exploration problem simpler. It then performs online RL, outputting actions after an MPC planning step trough a world model. It hence combines model-based and model-free RL approaches.", "questions": "how useful is the mpc planning? \nhow much does learning of the world model helps performance?\nhow much does the initial bc training help?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents good results on continous  control benchmarks which is still an interesting problem. Especially training one policy over this big variety of tasks is interesting to see. The algorithm is presented clearly and is easy to follow, and the model-based MPC aspect of it is interesting. I really appreciate the effort of the authors of making the code and the checkpoints accessible, this makes it possible to reproduce the results and build on-top of them.", "weaknesses": "My major concern is the applicability of this to real-world continuous control problems. While in simulation the results look good, it requires over 100M steps to train this policy which would be unfeasable on a real-world application. I also think the paper would benefit from ablating the usefulness of the different components - specifically interesting would be to understand how useful is the mpc planning is, how much learning of the world model helps performance as well as  how much does the initial bc training helps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)