One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning
Authors: Yuan Pu
Yazhe Niu
Jia Tang
Junyu Xiong
Shuai Hu
Hongsheng Li
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
multi-task reinforcement learning
world model
MCTS
latent space planning
Primary area: reinforcement learning
TL;DR: We improve multi-task learning by exploring architectures, finding MoE best for gradient conflicts, and using a Dynamic Parameter Scaling (DPS) strategy for adaptive model capacity.

Abstract: In heterogeneous multi-task decision-making, tasks not only exhibit diverse observation and action spaces but also vary substantially in their underlying complexities. While conventional multi-task world models like UniZero excel in single-task settings, we find that when handling a broad and diverse suite of tasks, gradient conflicts and the loss of model plasticity often constrain their sample efficiency.
In this work, we address these challenges from two complementary perspectives: the single learning iteration and the overall learning process. First, to mitigate the gradient conflicts, we systematically investigate key architectural designs for extending UniZero. Our investigation identifies a Mixture-of-Experts (MoE) architecture as the most effective approach. We demonstrate, both theoretically and empirically, that this architecture alleviates gradient conflicts by routing task-specific representations to specialized sub-networks. This finding leads to our proposed model, \textit{ScaleZero}. Second, to dynamically allocate model capacity throughout the learning process, we introduce an online Dynamic Parameter Scaling (DPS) strategy. This strategy progressively integrates LoRA adapters in response to task-specific progress, enabling adaptive knowledge retention and parameter expansion. Evaluations on a diverse set of standard benchmarks (Atari, DMC, Jericho) demonstrate that ScaleZero, utilizing solely online reinforcement learning with one model, performs on par with specialized single-task agents. With the DPS strategy, it remains competitive while using just 71.5\% of the environment interactions. These findings underscore the potential of ScaleZero for effective multi-task planning. Our code is available at \textcolor{magenta}{https://github.com/opendilab/LightZero}.

Published as a conference paper at ICLR 2026
ONE MODEL FOR ALL TASKS: LEVERAGING EFFICIENT WORLD MODELS IN MULTI-TASK PLANNING
Yuan Pu
Yazhe Niu 1.2
Jia Tang :3
Junyu Xiong
Shuai Hu
Hongsheng L
Shanghai Artificial Intelligence Laboratory
The Chinese University of Hong Kong MMLab
8 Naajjnng University of Aeronautics and Astronautic
Universitt Science and Technology of China
Novosibirsk State University
Centre for Perceptual and Interactive Intelligence
ABSTRACT
In heterogeneous multitask decision-making, tasks not only exhibit diverse observa- tion and action spaces but also vary substantially in their underlying complexities. While conventional multitask world models like UniZero excel in single-task set- tings, we find that when handling a broad and diverse suite of tasks, gradient confficts and the loss of model plasticity often constrain their sample efficiency. In this work, we address these challenges from two complementary perspectives: the single learning iteration and the overall learning process. First, mitigate the gradi- ent conflicts, we systematically investigate key architectural designs for extending UniZero. Our investigation identifies Mixture-of-Expert (MoE) architecture as the most effective approach. We demonstrate, both theoretically and empiri- cally, that this architecture alleviates gradient conflicts by routing task-specific representations to specialized sub-networks. This finding leads to our proposed model, ScaleZero. Second, dynamically allocate model capacity throughout the learning process, we introduce an online Dynamic Parameter Scaling (DPS) strategy. This strategy progressively integrates LoRA adapters in response to task- specific progress, enabling adaptive knowledge retention and parameter expansion. Evaluations on a diverse set of standard benchmarks (Atari, DMC, Jericho) demon- strate that ScaleZero, utilizing solely online reinforeement learning with model, performs on par with specialized single-task agents. With the DPS strategy, it remains competitive while using just 71.5% of the environment interactions. These findings underscore the potential of ScaleZero for effective multitask planning, Our code is available
INTRODUCTION
Unified world models (Reed et al. 2022 Lee et al. 2022 Gallouédec al. 2024) represent significant step towards generalist agents, offering a cohesive framework for multi-modal perception (Zhang et al. 2022). long-horize prediction (Ni et al.) 2024}, and decision-making by learning a shared latens space representation. By encoding the environment into a compact latent space, these learned representations enable effective planning. Algorithms like Monte Carlo Tree Search (MCTS) (Swiechowski et al. 2023) are well-suited for this task, as they can perform lookahead efficiently within this abstract representation (Schrittwieser et al., [019 202T). MCTS excels at dynamically balancing exploration and exploitation, combination that has achieved well-established success in homogeneous task domains like board games (Schrittwieser et al. 2019 Ye et al.) (221)) Extending this paradigm heterogeneous multitask reinforcement learning (MTRL.) YYu al. 2024
a key objective for creating generalist agents. However, this ambition is impeded by a formidable obstacle: training single shared model diverse tasks with potentially conflicting dynamics and
objectives is notoriously difficult (Cho et al., 2024). While prior work (Sun et al., 2022. Fernando
al. 2023 Shi et al. 2023) has attempted mitigate inter-tak interference, these methods have
"These authors contributed equally to this work.
CCorresonniigg author.

---

Published as a conference paper at ICLR 2026
been predominantly studied in supervised learning and model-free RL settings. The challenges and dynamics of multitask planning within a shared world model remain largely unexplored.
Within this specific domain of multitask planning, our work identifies and addresses two critical, intertwined obstacles not fully resolved by existing methods: (1) Representational bottlenecks and plasticity collapse: In diverse multitask settings, a shared model is susceptible gradient dominance from simpler, faster-converging tasks (Cho et al., 2024). This imbalance leads to representation interference (Yu et al. 2020; Bejnordi et al. (024), where the learning signals for more complex tasks are suppressed. This culminates in a progressive loss of network plasticity (Dohare et al. 2024
Todorov et al., 2025 the fundamental ability of a model to adjust its parameters to learn from new data inta-imppsing a hard ceiling on the model's 's overall learning capacity and leading to performance collapse on challenging tasks. (2) Static resource allocation: Conventional architectures employ a uniform, one-size-fits-al forward pass, applying the same learning resources (e.g., data collection and model updates) to every task irrespective of its intrinsic difficulty. This static strategy results in profound computational inefficiency, resources are squandered on converged or show diminishing returns, instead of being directed toward those that require further learning. To dissect these issues, we begin with a quantitative diagnosis using UniZero (Pu et al. 2024), contemporary unified world model, as a representative testbed. We train a single model via online reinforcement learning across eight canonical Atari games and observe a stark failure pattern, as shown in Figure [ While the model rapidly masters simple tasks like Pong, it exhibits initial progress followed by a catastrophic performance collapse on complex games with disparate dynamics and visual styles, such as Seaquest. whose complex exploration demands starkly contrast with the reactive gameplay of tasks like Pong. We provide quantitative evidence linking this failure to specific internal model dynamies (Lyle et al.) 2022): an uncontrolled inflation of the latent state norm and corresponding spike in the dormant neuron ratio (Sokar et al., 2023) within the transformer backbone, signaling a collapse in model plasticity, rendering the network unable to adapt to new information from challenging tasks.
Based on this diagnosis, we tackle these challenges from two complementary perspectives. First, as an internal, architectural solution, we undertake principled exploration of the design space across the core dimensions input representation, model architecture, and optimization strategy. This systematic investigation, which evaluates five key axes task conditioning, encoder architectures (ResNet (He) et al.) 2016) vs. VIT (Dosovitskiy et al. 2020),, latent normalization schemes, backbone design, and multitask optimization strategies (?) culminates in new, powerful model we term ScaleZero. Among the evaluated design choices, the integration of a sparse Mixture-of Expextssss (Shazeer et al.) 2017) backbone yields the most significant gains by fundamentally addressing plasticity collapse. To provide a deeper understanding of this key architectural choice, we present a dedicated empirical and theoretical analysis in Section [5.3] explaining MoE's s effectiveness in multitask planning. Validated on comprehensive suite of online RL benchmarks spanning diverse modalities coodalttes-iincuudnng 26 disually-coples, Atari games (Bellemare et al. 2013). 18 state-based DeepMind Control (DMC) Suite tasks (T (Tunyasuvunakool et al.) 2020). and 4 taxt- Jericho environments (Hausknecht et al., 2020) ScaleZero robustly matches and often surpasses the performance of single-task expert agents.
Second, as an external, procedural strategy to optimize the overall learning process, we introduce Dynamic Purameter Scaling (DPS), novel online mechanism that couples model capacity to learning progress. DPS adaptively curates the set of active tasks based on real-time return feedback and orchestrates a phased expansion of model capacity by injecting lightweight LoRA adapters (Hu) ct al. 2022). By strategically freezing previously trained parameters, DPS creates a curriculum of model capacity that directs computational resources where they are most needed. Our experiments demonstrate that when augmented with this strategy, our method achieves performance nearly on par with single-task agents while reducing total environment interactions by around 28.5% on the DMC benchmark, offering a superior trade-off between final performance and computational budget.
Our main contributions are summarized as follows: (1) We provide the quantitative diagnosis of plasticity collapse in unified world models within heterogeneous MTRL, establishing concrete link between performance degradation and internal learning dynamics. (2) We conduct a systematic architectural exploration that yields ScaleZero, a unified world model that demonstrates exceptional performance and generalization across distinct tasks from three distinct benchmarks. (3) We propose Dynamic Parameter Scaling, an adaptive training strategy that dynamically allocates model capacity and computational resources, reducing total environment interactions by around 28.5%.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
MCTS with Learned World Models. Planning in a learned latent space, popularized by MuZero (Schritwieseer al. 2019), is dominant RL paradigm (Hubert al. 2021 Antonoglou
et al. Danihelka et al. 2022 Xuan et al. 2024 Niu et al. (024). Recent works have integrated
Transformers enhance representational capacity (Micheli et [022] Pu al. 2024 Zhou et al.)
2024), achieving state-of-the-art performance in single-task domains. However, their monolithic de-
sign a critical liability in heterogeneous multitask settings, where they suffer from representational interference and plasticity collapse. Our work directly confronts this architectural bottleneck.
Multitask Reinforcement Learning. MTRL aims improve data efficiency by sharing knowledge across tasks (Vithayathil Varghese & Mahmoud 2020, Kumar et al. [202 Hansen et al., (223) Lec
al. 2022 Gallouédec al. 22024). Existing methods mitigate interference through architectural solutions like task-specific modules (Schmied et al., 2023 Sun et al. 2022) or optimization based approaches that manage gradient conflicts (Fernando et al. 2023 Ma et al. 2023). These strategies, however, have been predominantly studied outside of latent-space planning, where the unique challenge of disentangling dynamics prediction remains largely unaddressed.
Sparse and Parameter-Effliiene Architectures. We draw inspiration from two complementary paradigms. Sparse MoE models offer a natural architectural prior for multitask specialization by enabling conditional computation (Dai al. 2024 Obando-Ceron et al. 2024). Concurrently, parameter-efficient methods like LoRA (Hu et al. 2022) provide a lightweight mechanism for adaptation. However, conventional LoRA approaches
typically restricted to static, offline fine- tuning (Agiza et al. 2024] Huang et al. 2023). To adapt these principles to non-stationary online data streams, our approach diverges from standard methods that depend on predefined task boundaries. Instead, we introduce a mechanism that autonomously allocates parameter space in response to distinct distribution shifts. By unifying MoE with this adaptive DPS strategy within a transformer- based world model, we create a system that effectively balances architectural specialization with dynamic plasticity for large-scale MTRL. See Appendix F for more discussion.
BACKGROUND
3.1 MCTS WITH LEARNED WORLD MODELS
Monte Carlo Tree Search is powerful planning algorithm that has demonstrated remarkable success in domains with known rules (Silver et al., 2016 2017). Methods employing MCTS for planning within learned latent space represent the of in complex sequential decision-makin (Schrittwieser et al. 2019 Ye et al. 202I). These approaches learn a world model comprising three components: (i) representation model hg that encodes observation-action history into a latent state z;; (ii) a dynamics model go that predicts the next latent state and reward ($++1, fz) = go (2(2,, a:): and (iii) prediction model fo that outputs a policy and value (pt, U4) guide the MCTS planner. Recent architectures, exemplified by UniZero (Pu et al. 2024). unify these components into a single, monolithic Transformer. This design enables end-to-end optimization via composite loss that aligns the model's 's predictions with targets derived from the MCTS planner: policy target (x₂) based on the root node's 's visit counts, and a bootstrapped TD value target (otars) (Schrittwieser et al. 2019). These are combined with objectives for predicting the reward (rt) and the next latent state (2++1):
H Uniieed Σ
(Cvalue blarg) + Cpolicy (Pe, TT) + Lreward r2) + Ldynamics (f+41, sg(2++ 1)))
(1)
where H is the context length, sg(-) means sto--gradient. While sample-efficient and hyperparameter-
insensitive for single-task training, this unified scheme exposes critical vulnerability: in multitask settings, the shared transformer backbone is updated by an aggregated gradient. This shared update mechanism becomes the nexus of interference, impeding a single model's ability to learn multiple complex task and leading directly to the plasticity collapse we identified in our diagnosis.
3.2 GRADIENT CONFLICT IN UNIFIED ARCHITECTURES
The challenge of MTRL is to train a single, lisk-conditioned policy Ta(als, s,, k) on a distribution of K tasks, (Ti,..,,TR) TR). Even within a single task, the composite losses of a world model (e.g..
3

---

Published as a conference paper at ICLR 2026
for dynamies, reward, and policy) can conflict, creating a challenging multi-objective optimization problem (Ma et al.] 2023). In MTRL, this challenge is compounded by inter-task gradient conflicts. In unified architecture with shared parameters 9, the total loss is the sum of individual task losses, CMTRL (0)
k L+, where C is the loss from Eq. | for task k. Consequently, the learning dynamics are driven by the sum of per-task gradients: Greaa = 2==1 9k. where 9h = VeC The core problem, known as gradient conflicr (Liu et al., 2021). arises when these per-task gradients are misaligned, a condition quantified by their cosine similarity: cos .9)) < 0. A negative similarity indicates that an update improving performance on task actively degrades performance on task j. In a monolithic architecture where all core parameters are shared, frequent gradient conflicts lead to destructive interference (Sodhani et al., 2021). This dynamic forces the model into suboptimal compromise, creating the representational bottlenecks and catastrophic performance drops characteristic of plasticity collapse (Dohare et al. 2024).
3.3 ARCHITECTURAL PARADIGMS FOR MITIGATING INTERFERENCE
To overcome the limitations of monolithic designs, architectural solutions have been proposed to enable parameter specialization and mitigate interference. Two prominent paradigms are particularly relevant. Sparse
Exxpers replaces the dense feed-forward network in Transformer block with a set of N parallel "expert" networks (Cai et al.] 2025). For each input, a trainable gating function (xx) sparsely selects small subset experts (c.g.. top-k) to process the token: MoE(z)
EN (z) Expert,(x)) This conditional computation creates specialized pathways within the model, allowing different inputs (e.g., from different tasks) be processed by distinct subsets of parameters, thereby reducing interference and increasing model capacity without a proportional rise in computational cost. Recent designs also explore hybrid approaches, such including a shared expert alongside specialized ones, to balance generalization and specialization (Dai et al., 2024). Low-Rank Adaptation offers a parameter-efficient method for adapting large pre-trained models (Hu et al.) 2022).
Instead of fine- tue-ttning the entire weight matrix Wo. LoRA freezes G3 and injects a trainable, low-rank "update" matrix, A BA. The modified forward pass becomes (0g + ABA... By training only the small low-rank factors (A, B), LoRA can efficiently learn task-specific modifications while preserving the general knowledge in the pre-trained weights. This approach has proven effective for creating extensive multitask capabilities on top of single base model, where different LoRA modules can be composed to handle diverse tasks (Huang et al. 2023).
4 METHOD
We begin in Sec. 4.1 by quantitatively diagnosing plasticity collapse in multitask training. To address this, Sec. 4.2 presents a systematic design space exploration across five axes-which culminates in our proposed model, ScaleZero. Finally, to resolve the static allocation issue, we introduce Dynamic Parameter Scaling in Sect. 4.3 a LoRA-basee strategy that adaptively allocates model capacity.
4.1 DIAGNOSING PLASTICITY COLLAPSE IN MULTITASK PLANNING
To empirically substantiate the challenges of representational bottlenecks and plasticity collapse, we conduct a quantitative analysis of a unified world model's internal dynamics during multitask training. We track two key metrics indicative of network plasticity degradation: (1) DormantRatio(()
No 1 EM 1! (Ih/) S <) This metric quantifies the proportion of neurons whose activation magnitude N
[K]] falls below a threshold A high ratio signifies loss of active pathways and reduced network plasticity. (2) Latent State Norm: We compute the average L2 norm of the latent state (|2g|2). An uncontrolled inflation of this norm is a keol-known indicator of training instability (Team et al. 2025).
Diagnostic experiments on UniZero, conducted across multitask Atari8 suite (Appendix B reveal a critical failure mode in complex learning scenarios. While simpler tasks like Pong achieve stable convergence, more complex ones such as Seaquest exhibit initial learning followed by a catastrophic performance collapse (Figure D. This collapse coincides with sharp increase in the dormant neuron ratio and an inflated latent state norm. The representational nature of this failure corroborated by marked decline in the feature effective rank (Dohare et al., 2024), indicating a degradation of the learned feature space (Figure 55.. We attribute this failure mode to two intertwined factors: (1) Gradient Competition, where gradients from simpler tasks overwhelm those from complex ones,
4

---

Published as a conference paper at ICLR 2026
Perig
Hern
Seaqueet
Choppeecemmen
1070
000
300
100K
1966
mac
100
PIBC 130
19C
1204
11.0 Picc
Dommant Ratio
Labent Norts
Crepotaement
digs
Figure 1: Plasticity collapse in the baseline (UniZero) on a multitask Atari benchmark. While simple tasks like Pong and Hero show stable learning, complex tasks such as Seaquest and ChopperComman suffer a catastrophic performance collapse in later training (Top). This failure is precisely correlated with sharp spike in the dormant neuron ratio of the transformer (Bottom Left) and uncontrolled inflation of the latent state (Bottom Right), empirically validating the link between external performance and internal learning dynamics.
and (2) Representation Interference, where the shared network fails to maintain diverse features, leading representational bottleneck. Based on this diagnosis, our method tackles plasticity collapse from two complementary perspectives: (1) an internal, architectural solution, ScaleZero (Section 4.2), designed to mitigate interference within single learning iteration. (2) an external, procedural strategy, Dynamic Parameter Sculing (Section 4.3 which optimizes resource allocation across the entire learning process.
4.2
ARCHITECTURE DESIGN OF SCALEZERO VIA PRINCIPLED EXPLORATION
Habe Parem
Had
Bacdte
Mall
Labed Nere
Tak
Stuge1
Stag
Sage
Stige Leuring Stigo
Corpeltatnee Octpeldhiippy EmppeStaad
Clecsotta Comppedding
(a) Design Space of UniZero for Multitask learning
(b) Dynamic Parameter Scaling
Figure 2: (a) systematic exploration of the UniZero design space across five axes: task conditioning, encoder architecture, latent normalization, backbone design, and optimization. This investigation informs the design of our proposed ScaleZero model. (b) A conceptual diagram of Dynamic Parameter Scaling (DPS), DPS progressively expands model capacity by injecting LoRA adapters stages, triggered by learning progress. This creates a curriculum of model, directing resources toward unsolved tasks while preserving existing knowledge.
Motivated by the above diagnosis of plasticity collapse, we conduct principled exploration of the UniZero design space to forge robust solution on Atari8 multitask benchmark (see Appendix for experimental details). We systematically evaluate five key design axes (Figure 2a) drawn from three core dimensions (input representation, model architecture, and optimization strategy): task condition- ing, encoder architecture, latent normalization, backbone design, and optimization strategy. Specific implementation details are provided in Appendix AA.3 The comprehensive results, summarized in Figure 3 reveal that the integration of MoE backbone provides uniquely significant and robust.
5

---

Published as a conference paper at ICLR 2026
Normaalized
Normalized Median
UMDDe
0.4
Tar Evtedding
0.2
Scofe
0.2
0.0
-0.3
-0.2
-0.3
0.0
0K 50%
100K
150K
200K
250K
30DK
04 50K
100K
150K
200K
250K
300K
Ern Steps
Ery Stops
Figure 3: Performance impact of architectural modifications on the Atari8 multitask benchmark. This ablation across the UniZero design space reveals that replacing the Transformer backbone with a Mixture-of- Experts architecture yields the most significant and consistent performance gains. In contrast, other interventions, with the partial exception SimNorm, provide marginal or inconsistent benefits. These results underscore centrality of the MoE's conditional computation in overcoming the limitations of a shared, dense backbone.
performance uplift, directly counteracting the failure modes identified in our initial diagnosis. This is further corroborated by the complete plasticity metric analysis in Figure 7 which demonstrates that the MoE-based model consistently maintains healthy latent dynamics (i.c., lower dormant ratios and stable latent norms) across all tasks. Below, we detail the findings of each axis.
observations may face ambiguity in discerning the underlying task, especially when initial states are similar across environments (Rakelly et al., 2019; Sodhani al., 2021). To test if providing
Explicit Task Conditioning. A shared model processing
explicit task identity & could resolve ambiguity, we concatenate learnable task embedding éx to the latent state z. While this approach accelerates initial convergence on simpler tasks, it offers marginal long-term benefits for complex ones and fails to mitigate plasticity collapse, as evidenced by minimal improvement in metrics like the dormant neuron ratio (Figure 7. This indicates that such input-level task conditioning are insufficient to resolve deep-seated representational conflicts in online MTRL.
Encoder Architecture (ResNet vs. ViT). We contrast standard ResNet-lik encoder in Pu et al. (2024) with a Vision Transformer to assess the encoder's 's impact visual representation quality. On the Atari8 benchmark, the choice of encoder does not yield a significant performance difference. We attribute this to the limited visual heterogeneity in this specific task suite, but posit that the scalability and powerful feature extraction of ViT are critical for broader, more diverse multitask domains.
Latent Normalization (LayerNorm vs. SimNorm). To directly combat the observed latent norm inflation, we evaluate SimNorm (Hansen et al. 2023) against the standard LayerNorm (Ba et al. 2016). SimNorm, which projects latents onto simplex to enforce norm constraint, effectively stabilized training and prevented performance collapse, consistent with its reported success less heterogeneous settings (Hansen et al. (202)). However, in our highly diverse multitask setup. this hard constraint appears to curtail representational expressiveness, leading to suboptimal final performance. This highlights critical trade-off between stability and capacity for online heterogeneous RI. training.
Mixture-of- ofxturr-of-Expprtss Backbone. The most impactful modification is replacing the dense forward networks with sparse MoE layers in UniZero's transformer. This conditional computation paradigm directly addresses the core challenges of multitask planning by mitigating the intertwined issues of representational interference and gradient conflict, as it routes task-specific computations and their corresponding gradient updates through distinct, specialized expert sub-networks, As validated in Figures 3 and 7 the MoE backbone yields substantial performance gains. We attribute this success to its inherent ability to mitigate gradient conflict, mechanism analyze in depth in Section 5.3 There, we provide both empirical validation and a theoretical analysis demonstrating that the upper bound on gradient conflict for an MoE layer is strictly lower than that of its dense counterpart.
Multitask Gradient Correction. We also explored a dynamic gradient re-weighting scheme inspired by MoCo (Fernando et al. 2023) directly mitigate gradient interference between tasks. While this approach showed promise on some tasks, it introduced significant computational overhead by increasing per-step training time by approximately 40%, and demonstrated inconsistent efficacy across our heterogeneous task suite. Given this unfavorable performance-con trade-off, we defer the investigation of more efficient gradient correction methods to future work.

---

Published as a conference paper at ICLR 2026
The ScaleZero Architecture. The result of our systematic investigation is ScaleZero, novel architecture designed as a efficient and scalable world model. is defined by the integration of three core components: (1) a ViT-like Encoder for powerful and scalable feature extraction: (2) an MoE Backhone that uses sparse, conditional computation to increase model capacity while mitigating task interference; and (3) an explicit Layer Normalization applied to the encoded latent state ensure robust balance between stability and representational expressiveness. This core is complemented by a modular design that decouples task-specific encoders and heads, enabling flexible handling of diverse input and output modalities. As validated in Section [5] ScaleZero provides an effective blueprint for building generalist world models. Differences from UniZero are detailed in Appendix A.22
4.3 DYNAMIC PARAMETER SCALING FOR EFFICIENT MULTITASK LEARNING
To tackle the second challenge multitask planning- static resource allocation we propose Dy- namic Parameter Scaling (DPS). As illustrated in Figure 2b this strategy departs from the conven- tional sne-size-fits-eee paradigm by dynamically aligning model capacity with learning progress. DPS operates two synergistic principles: (1) adaptive task curation, which focuses computational effort exclusively on unsolved tasks, and (2) progressive capacity expansion. which injects new parameters only when necessitated by task demands. This combination establishes "curriculum of model complexity," directing resources precisely where they are most required. DPS employs dynamic, multi-stage training to manage tasks and parameters.
Adaptive Task Curation: Let T = {Ti) denote the full set of tasks. A task т; is deemed "solved" once its performance metric, Metric(T:), surpasses predefined threshold e.. At any training step t, we maintain an active set of unsolved tasks, Ue C T. To maximize efficiency, both data collection and gradient updates are performed only for tasks within Ue. Once solved, a task is removed from this active set, thereby ceasing all computational overhead associated with it.
The training proceeds in S + stages over total of Taax iterations. Stage 0 (Warm-up) trains the base model 03 for an initial To iterations. This stage establishes a robust foundation by training the shared base model on all tasks to learn general- reneral-purpose representations, which are crucial for the subsequent progressive introduction of specialized modules. Triggered by learning progress, each subsequent Stage s 2 (Expansion) incorporates a new, independent LoRA module, Ad, = B,A,. Concurrently, a set of learnable scaling factors is defined modulate the contribution of both the base model and all adapters. Specifically, a scaling factor Org is associated with the base model weights 0z. while each LoRA module A0, is assigned a corresponding scaling factor a, for j E S}. All scaling factors are initialized to I.
Staged Capacity Expansion:
Adaptive Stage Transition Triggers. The transition from stage - 1 to is triggered upon satisfying either of two criteria, balancing progress against a fixed computational budget: (1) Progress-baat bassd Trigger: A transition is triggered if the number of newly solved tasks during the current stage reaches quota Q.. (2) Budget-based Trigger: To prevent stagnation on persistently arduous tasks, a transition forced if the iteration count in the current stage exceeds pre-allocated limit, [TTmax TT)
Optimization and Parameter Isolation. The process starts with the base weight matrices from 8R. which are pre-trained in Stage 0 and subsequently frozen. For any base matrix Wo E On, its effective weight is progressively augmented. At the beginning of stage s 2 1, the effective matrix ((+)) is defined as: W's) = ao Wo + Σj=1 a,Ld, = an Wo + Σ5=1 a, A,y. A key principle of DPS is parameter isolation. Upon advancing to stage 8, all previously learned parameters- the backbone On and adapters {A0, }-; | -re frozen. To resolve scale ambiguity, the scaling factor a, for the currently active adapter A0, is temporarily fixed at 1. Optimization is thereby focused on the newly added LoRA module A0., the base model scaling factor ag. and the set of scaling factors corresponding to previously frozen adapters, {a,)5=1
This strategy yields two principal advantages. First, it implements a resource-efficient training curriculum, where new parameters are only introduced when required by the remaining unsolved tasks. Second, by isolating new learning within dedicated adapters, DPS provides targeted plasticity for difficult tasks while preventing catastrophic forgetting negative transfer to previously mastered skills, whose knowledge is preserved in the frozen backbone. DPS thus achieves a favorable trade-off between computational efficiency and final performance. (See Appendix A.4 for pseudocode).
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS AND ANALYSIS
Our experiments are designed to validate the two central claims of this work. First, in Section 5.1 we demonstrate that the proposed ScaleZero effectively mitigates the representational bottlenecks and plasticity collapse issue. We assess its ability to achieve powerful performance as single, generalist agent across a suite of heterogeneous benchmarks spanning visual, proprioceptive-bot -baiieee-addddeee and text- based domains (Atari (Bellemare et al.) 2013). DMControl (Tunyasuvunakool et al.] 20200), and Jericho (Hausknecht et al. 20200). Second, in Section 5.2 we quantify the efficiency gains afforded by our DPS strategy, showing its effectiveness in overcoming static resource allocation. Finally, in dectioo[5.3) provide dedicated empirical and theoretical analysis of the MoE, substantiating its role as a foundational element of ScaleZero. All experiments are conducted in a purely online RL setting, without reliance on expert data. Implementation details are provided in Appendix A
5.1 MULTITASK LEARNING BENCHMARKS
5.1.1 ATARI 100K BENCHMARK
Setup. We evaluate ScaleZero on the 26 games of the Atari 100k benchmark, a standard testbed for heterogeneous MTRL due to its diverse game mechanics. We compare one multi-task (MT) ScaleZero agent against the strong single-task (ST) UniZero baseline, where 26 separate mod- els are trained for each game. Performance is measured using the standard Human- Normalized Score (HNS) (Agarwal et al.] 2021), reporting both mean and median capture overall capability. Further experi- mental details are available in Appendix B
Table 1: Performance comparison of our multitask model, ScaleZero, against the single-task UniZero baseline across discrete (Atari) and continuous (DMControl) benchmarks.
Algocithm
Nomn. Menn
Norm. Median
UniiZero (ST)
0.38
Uni7er (MT)
0.31
0.16
(MT))
0.39
0.16
(a) Human-normalized scores on 26 games in Atari 100k benehmark. ScaleZero achieves higher mean score. Full training curves are in Appendix [ We omit other MT base- lines as, to the best of our knowledge, no prior work has tackled all 26 games with a single online RL agent.
Task
UniZero (ST)
Scale7ero (MT)
Results. As clearly summarized in Table single generalist agent, achieves a higher mean normalized score than the of 26 independently trained single-task specialists. This result demon- strates that ScaleZero effectively mitigates the negative interference- common chal- lenge in heterogeneous MTRL,, enabling positive knowledge transfer. Notably, per- formance gains are particularly marked on notoriously difficult exploration tasks like Seaquest a game we identified as a pri- mary failure case for the baseline in our ini- tial diagnosis (Figure T While the median score is marginally lower, influenced by a few hard-exploration games, the higher mean score confirms the overall advantage and generalization ability of our approach.
acrobot swingup
400.3
501.0
cartpole-aalaee
990.5
ScaleZero,
caripole balance_spaas
1000.0
1000.0
carpoll-swiingup
801.3
705.0
cartpole-swingee waspsssswiuupspppeeeee
752.5
708.1
cheetah run bail in_cup--tchh
517.6
510.9
961.6
954.2
finger-spin
810.7
574.2
finger fiager-tum, hard
1000.0
1000.0
982,0
hopper hop
120.5
138.0
hopper stand
602.6
553.1
pesdulum-swiagguu
865
$66.0
reacher easy
943.1
reaches-hord
988.8
943.5
walker-uun
587.9
562.7
walker-stand
976.4
919.9
walkos-walk
908.7
787.2
Modian
875
587.3
(b) Raw scores on DMControl tasks. ScaleZero achieves a superior median score, indicating robust generalist perfor- mance. Full training curves are in Appendix C For reference, 1.2M (MT) [Schmied et al. (2023) reported a mean score of 840 an easier 10-task henchmark that largely overlaps with ours, trained via offline supervised learning from expert data.
5.1.2 DEEPMIND CONTROL SUITE
Setup. To evaluate ScaleZero's effectiveness to continuous control, we conduct experiments on 18 tasks from the DeepMind Control Suite. These tasks, based on the MuJoCo engine (Todorov et al.) 2012), are characterized by low-dimensional state inputs and continuous action spaces, demanding precise dynamics modeling. Given the vector-based nature of observations, we use separate MLP encoders for each task. The experimental protocol remains consistent, comparing one MT ScaleZero agent against 18 individually trained single-task UniZero models. Details are provided in Appendix C
8

---

Published as a conference paper at ICLR 2026
demonstrate that ScaleZero achieves competitive performance, surpassing the baseline on several tasks and achieving a superior median score. This metric is particularly informative for evaluating generalist agent, as it shows robust performance across the majority of tasks rather than excellence limited to a simpler subset. This success validates the versatility of the ScaleZero, proving its effectiveness not only in visual domains but also in complex continuous control.
Results. Table lb
5.1.3 JERICHO: TEXT-BASED ADVENTURE BENCHMARK
Setup. To assess performance on tasks requiring language understanding and long-horize planning, we employ the Jericho benchmark. These text-based games present challenges through their combi- natorial action spaces and the sparse reward nature. We evaluate four representative games and compare against UniZero and CALM+OC (Sudhakar et al. 2023), a powerful language-modee model-based method. Both ScaleZero and UniZero use the same BGE text encoder (Xiao al. 2023
Results, As reported in Table 2 ScaleZero achieves performance on par with specialized single-task agents and is competitive with the strong CALM+OC baseline. This shows that the ScaleZero agent can effectively learn capable policies in environments where both states and actions are linguistic. These findings substantiate that the design ScaleZero are aagnsssy-agnostic, successfully extending its efficacy to the domain of text-based planning. Experimental details are provided in Appendix
Table Average returns comparison on four Jericho tasks. Detailed learning curves are in Appendix D
Algorihm
Acomcoent
Omniqqest
Detective
Zoek1
CALM+OC (ST) (Subakarr et
2013
288.5
28.0
UniZeru (ST)
10
10
295
44
ScaleZero (M1)
10
44
5.2 EFFICIENCY EVALUATION OF DYNAMIC PARAMETER SCALING
Having established the superior multitask performance of ScaleZero, gains afforded by our Dynamic Parameter Scaling strategy. We compare the standard ScaleZero against its ScaleZero-DPS variant the DMC18 benchmark. The primary metric is the number of environment interactions required to match the strong performance of the single-task baselines. As shown in Figure 4 ScaleZero-DPS consistently achieves this performance target using only 71.5% of the interactions required by ScaleZero. This corresponds to substantial 28.5% reduction in data sampling and subsequent training cost, directly validating the effectiveness in resolving the static resource allocation problem. The steeper learning curves shown in Figure IO further confirm that these efficiency gains persist throughout the dynamic training process. mechanistic analysis in Appendix C.4 reveals the agent's 's efficiency stems from an emergent hierarchical strategy: the model learns to preserve foundational knowledge in early layers while applying targeted plasticity later layers to adapt to unsolved tasks, providing clear mechanistic explanation for its performance.
evaluate the efficiency
Environment Interaction Cost Comparison (Budget: 0.4M/Task)
Por Interaction Cost
Total Cost
Ewwformmnnt Stogs
28/5%
BU%
Amohot Setrggon
Ball de Dap Catoss
Caripsie Exdarre
Cartonie Bstanie Starco
Cartbole Swviggu
Cartooet Setiggg,, Spreno
Finger Spin
Furger Easy
Heuper Part Herd
Happerr Hopp
Hoppor Stand
Perstaim Seriggo
Asserter Eary
Reserter Herr
Wainer Rant
Wallier Stand
Waller Rial
Todal
Cheeseat
ScaleZor
ScaloZero DPS
Figure Interaction cost comparison for ScaleZero vs. ScaleZero on DMControl. Tha latter reaches target performance with 28.5% reduction in the environment cost. Detailed curves are in Appendix C

---

Published as a conference paper at ICLR 2026
5.3 UNDERSTANDING MOE EFFICACY: AN EMPIRICAL AND THEORETICAL ANALYSIS
The superior performance of ScaleZero motivates a deeper investigation into its core architecture: the MoE transformer backbone. Here, analyze the mechanisms of MoE from both empirical and theoretical perspectives. Our empirical findings (detailed in Appendix E..) reveal that replacing dense MLP layers with MoE counterparts not only reduces gradient conflict within the MoE layers themselves but also in adjacent networks. To explain these observations, we provide a theoretical justification grounded the two-phase training characterization of MoE (Chen et al., 2022). We introduce the following informal theorem, which asserts that an MoE layer maintains a strictly lower upper bound on gradient conflict than dense counterpart- property governed primarily by the router's specialization. Detailed explanations, proofs and corollaries are provided in Appendix E.2
Theorem 5.1 (Upper Bound on Gradient Conflict in MoE Layers (informal)). In a Mixture-of Experts (MoE) layer, consider two tasks t and ty with routing weights Xm: X over M experts,
(mm) Let G be the maximum gradient and per-task gradients on expert m, denoted as 9t, (mn) and 9iz
conflict on any single expert, and cos(u, u, v) denote the cosine similarity between vectors and Then the following results hold: (1). (Generally) Define the m-th element of vectors u, as
[m]] [|, m I...., M. The full-layer gradient conflict admits Um >x 11 Um A 19
general upper bound: conflic((Gr, Gt2) S. G cos(u, v). (2). (Sparse Router) In the exploration
stage, where tasks uniformly select experts, the expected gradient conflict of the MoE layer is approximately bounded by G (1- (1MMM/((M- - K)! MK)) )), where K is the number of tasks. (3). (Sparse Router) In the router learning stage, where expert sets stabilize, the gradient conflict is roughly upper bounded by Elconslict, se G. U
ab where |S:|. b= |Sy] are the sizes of the expert sets for tasks and j. and U = [S, O Sj] is their intersection size.
6 CONCLUSION AND FUTURE WORK
In this work, we diagnose and addressed the critical issue of plasticity collapse in multitask world models. We introduced ScaleZero, unified architecture incorporating a sparse MoE backbone to cre- ate specialized computational pathways. Our experiments across 48 distinct tasks (Atari, DMControl, Jericho) demonstrate that ScaleZero robustly matches or surpasses the performance of specialized single-task agents. Furthermore, propose DPS, LoRA-based strategy that achieves competitive performance on DMControl while using just 71 71.5% of the environment interactions, highlighting a promising path toward greater sample efficiency. Our work provides a robust architectural foundation for developing more capable and efficient generalist agents. See Appendix G for full discussion.
ACKNOWLEDGEMENTS
We extend gratitude to several memmmeeberr of the Shanghai AI Laboratory for their invaluable assistance, support, and feedback on this paper and the associated codebase.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have read and adhered to the ICLR Code of Ethics. Our research focuses on the development of novel algorithms for multitask reinforcement learning, specifically proposing ScaleZero and the Dynamic Parameter Scaling (DPS) strategy. All experiments were conducted on standard, publicly available benchmarks (Atari, DMC, and Jericho), which are simulated environments. Our work does not involve human subjects, does use any personally identifiable sensitive data. The proposed methods are foundational and do not have immediate, direct real-world applications that could pose ethical risks societal harm. To the best of our knowledge, our research does not create or exacerbate discrimination or bias, as the training environments do not contain demographic or social data. We have strived to uphold the principles of scientific integrity and transparency by providing thorough description of our methods and experimental setup. We declare no conflicts of interest.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, we provide comprehensive details in the appendix of this paper. The appendix includes: (1) A detailed description of the implementation of our proposed model, ScaleZero, and the online Dynamic Parameter Scaling (DPS) strategy. (2) A thorough account of the experimental setup, including all hyperparameters, evaluation protocols, and the computational infrastructure used. (3) The complete theoretical proofs for the claims made in the paper. Furthermore, we commit to releasing our source code publicly upon the completion of the peer-review process to allow for full verification of our results and to facilitate future research in this area.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-2990 29320,, 2021.
Ahmed Agiza, Marina Neseem, and Sherief Reda. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 16196-16205, 2024.
loannis Antonoglou, Julian Schritwieser, Sherjil Ozair, Thomas Hubert, and David Silver. Planning in stochastic environments with learned model. In International Conference Learning Representations, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Babak Ehteshami Bejnordi, Gaurav Kumar, Amelie Royer, Christos Louizos, Tijmen Blankevoort, and Mohsen Ghafoorian. Interrogate: learning to share, specialize, and prune representations for muulii-ask learning. arXiv preprint 24Xi 16848, 2024.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning, 2022. URL htttt:t/aaxxiiiii www.wwoorgrgeee. abs /tt.s////2..... 02813
Myungsik Cho, Jongeui Park, Suyoung Lee, and Youngchul Sung. Hard tasks first: Multi-task reinforcement learning through task scheduling. In Forty-first International Conference on Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2026
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint 3XXi:24401006666 2024.
Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference on Learning Representations, 2022.
Carlo D D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement learning. arXiv preprint arXiv:2401.09561, 2024.
Shibhansh Dohare, Fernando Hernandez-Caariia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2020.
Heshan Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objective learning: A provably convergent approach. In International Conference on Learning Representations. International Conference on Learning Representations, 2023.
Quentin Gallouédec, Edward Beeching, Clément Romac, and Emmanuel Dellandréa. Jack of all trades, master of some, multi-purposs transformer agent. arXiv preprint arXiv:2402.09844, 2024.
Siddhant Haldar, Zhuoran Peng, and Lerrel Pinto. Baku: An efficient transformer for multi-task policy learning. Advances in Neural Information Processing Systems, 37:141208-141239, 2024.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint (xXxx22310.66222 2023.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Jinmin He, Kai Li, Yifan Zang, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Not all tasks are equally difficult: Multi-task reinforcement learning with dynamic depth routing. CoRR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp- 3796-3803, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -hu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269 2023.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4476-440 4486. PMLR, 2021. URL http: hubert 21 html
mlr. prwss/vis///
12

---

Published as a conference paper at ICLR 2026
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-tast data both scales and generalizes. arXiv preprint 4rX14:2211.1514, 2022.
Kuang- Mungg-HeiiLLa Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in neural information processing systems, 35:27921-279556 2022.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006. 15704, 2020.
Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang, Fei Liu, Zhenkun Wang, and Qingfu Zhang. Smooth tchebycheff scalarization for multi-objective optimization. arXiv preprint arXiv:2402.19078 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34: 15:1188888888 18890, 2021.
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski, Maciej Stefaniak, Michat Krutul, Jan Mataśnicki, Marek Cygan, Piotr Sankowski, Kamil Adamczewski, Piotr Miloš, et al. Joint moe scaling laws: Mixture of experts can be memory efficient. arXiv preprint arXiv:2502.051777 2025.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in reinforcement learning, arXiv preprint urXiv:2206.02120 2022.
Guozheng Ma, Lu Li, Zilin Wang. Li Shen, Pierre-Luc Bacon, and Dacheng Tao. Network sparsity unlocks the scaling potential of deep reinforcement learning. arXiv preprint arXiv:2506.17244 2025.
Haoyu Ma, Jialong Wu, Ningya Feng, Chenjun Xiao, Dong Li, Jianye Hao, Jianmin Wang, and Mingsheng Long. Harmonydream: Task harmonization inside world models. arXiv preprint arXiv:2310.00344 2023.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00888. 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1121v:13 2013.
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Lue Bacon. When do transformers shine in rl? decoupling memory from credit assignment. Advances Neural Information Processing Systems, 36, 2024.
Yazhe Niu, Jingxin Xu, Yuan Pu, Yunpeng Nie, Jinouwen Zhang, Shuai Hu, Liangxuan Zhao, Ming Zhang, and Yu Liu. Di-engine: A universal ai system/engine for decision intelligence. 2021.
https: /tttps://githube http://ttt.....ccoo////oe///
Yazhe Niu, Yuan Pu, Zhenjie Yang, Xueyan Li, Tong Zhou, Jiyuan Ren, Shuai Hu, Hongsheng Li, and Yu Liu. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024.
Johan Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprint arXiv:2402.088609 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vassil Tissier, Daniel Klein, Patrick Fernandez, Peter Jorgensen, Adriaan Rijks, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0711 193, 2023.
13

---

Published as a conference paper at ICLR 2026
Yuan Pu, Yazhe Niu, Zhenjie Yang, Jiyuan Ren, Hongsheng Li, and Yu Liu. Unizero: Generalized and efficient planning with scalable latent world models. arXiv preprint arXiv:2406. 10667 2024.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy meta-reinforcemmen learning via probabilistic context variables. In International conference on machine learning. pp. 5331-5340 PMLR, 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022.
Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions, 2023.
Ludger Rüschendorf. The wasserstein distance and approximation theorems. Probability Theory and Related Fields, 70(1): 1985.
Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, and Sepp Hochreiter. Learning to modulate pre-trained models in rl. Advances in Neural Information Processing Systems, 36:38231-38263 2023.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. CoRR, abs/1911.08265, 2019. URL
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadanin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. Advances in Neural Information Processing Systems, 34:27580-27591, 2021.
Shazeer, A Mirhoseini, Maziarz, A Davis, Q Le, G Hinton, and Dean. The sparsely gated mixture-of-experts layer. Outrageously large neural networks, 2, 2017.
Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing conflicting gradients from the root for multi-task learning. arXiv preprint a2X1 2023.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwesser, Joannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484488, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv: 2017.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In International conference on machine learning, pp. 9767-9779. PMLR, 2021.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference Machine Learning, pp. 32145-32168. PMLR, 2023.
Arjun Vaithilingam Sudhakar, Prasanna Parthasarathi, Janarthanan Rajendran, and Sarath Chandar. Language model-in-the-loop Data optimal approach to caam-to-recomment actions in text games. arXiv preprint 1.XXx:2311..066)) 2023.
Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: crramster-compositiioall multi-tan -uuti-taskkaa reinforcement learning. Advances in Neural Information Processing Systems, 35: 215077 2022.
14

---

Published as a conference paper at ICLR 2026
Maciej Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mañdziuk. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3): 2497-2562, 2023.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen. Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
Aleksandar Todorov, Juan Cardenas Cardenas-Cartagena, Rafael Cunha, Marco Zullich, and Matthia Sabatelli. pparsity-driven plasticity in multi-task reinforcement learning. arXiv preprint vrXiv:2508.0687/ 2025.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012. doi: 10. 109/1R0SS2001222. 201.6.880...02..888110 6386109.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https://doi.
org/10.1011// 2020. 100022. URL peatttss//www.... wttp://....eneeeeeooooooooeooeet rect. com/ www.scccceeeee.eeecceeoooee...oe / article/pii 52 http://p111/222666382228
Nelson Vithayathil Varghese and Qusay H Mahmoud. A survey of multi-task deep reinforcement learning. Electronics, 939):3363, 2020.
Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better multi-task learning. arXiv preprint arXiv:2311.1101, 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Yifan Xu, Nicklas Hansen, Zirui Wang. Yung-Chieh Chan, Hao Su, and Zhuowen Tu. On the feasibility of cross-task transfer with model-based reinforcement learning, arXiv preprint arXiv:2210. 10763, 2022.
Chunyu Xuan, Yazhe Niu, Yuan Pu, Shuai Hu, and Jing Yang. Rezero: Boosting mets-based algorithms by just-in-time and speedy reanalyze. arXiv preprint arXiv:2404. 163424404.6364,, 2024.
Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learnn with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777, 2020.
Yaming Yang, Dilxat Muhtar, Yelong Shen, Yuefeng Zhan, Jianfeng Liu, Yujing Wang. Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang, et al. Mtl-lora: Low-rank adaptation for multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 22010-22018, 2025.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances Neural Information Processing Systems, 34:25476-25488, 2021.
Weirui Ye, Yunsheng Zhang. Pieter Abbeel, and Yang Gao. Become proficient player with limited data through watching pure videos. The Eleventh International Conference on Learning Representations, 2022.
Jun Yu, Yutong Dai, Xiaokang Liu, Jin Huang, Yishan Shen, Ke Zhang, Rong Zhou, Eashan Adhikarla, Wenxuan Ye, Yixin Liu, et al. Unleashing the power of multi-task learning: A comprehensive survey spanning traditional, deep, and pretrained foundation model eras. arXiv preprint arXiv:2404.1888 18966, 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
15

---

Published as a conference paper at ICLR 2026
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint (tXXix:2033.36600 2022.
Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation. arXiv preprint arXiv:2504.07448, 2025.
Guoliang Zhao, Yuhan Fu, Shuaipeng Li, Xingwu Sun, Ruobing Xie, An Wang, Weidong Han, Zhen Yang. Weixuan Sun, Yudong Zhang, al. Towards comprehensive scaling law of mixture-of- experts. arXiv preprint arXiv:2509.245. 23678, 2025.
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint 20Xiv:2411.04983 2024.
16

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Implementation Details
Atari Experiment Details
28
DMC Experiment Details
Jericho Experiment Details
37
B
Gradient Analysis in Mol
40
Detailed Related Work
50
Limitations and Future Work
The Use of Large Language Models
55
17

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
This section provides a comprehensive overview of the implementation details for ScaleZero and the associated experimental methodology. For clarity, we first establish a baseline by providing a concise summary of the UniZero baseline (Pu et al. 2024), including training and MCTS procedures. Building upon this foundation, we then sequentially detail the core modifications that define ScaleZero, our explorations of key design spaces, the Dynamic Parameter Scaling (DPS) strategy, and finally, the full hyperparameter and computational setup ensure reproducibility. Our code is available at
A.1 UNIZERO
As outlined above, we begin by detailing the foundational framework our baseline, UniZero, to provide the necessary context for our contributions. The learning process revolves around core procedures operating in loop:
collect experience: The agent interacts with the environment to gather trajectories. At each timestep, it uses Monte Carlo Tree Search within its learned latent world model to generate improved policy, from which an action is sampled. The resulting experience tuples (observation, action, reward, done) and the MCTS-improved policy stored in replay buffer.
collect_experiente
2. seddat_wwolde model: The world model, policy network, and value network are jointly optimized using batches data sampled from the replay buffer. This step updates the model's understanding of the environment's dynamics and improves its decision decision-making capabilities.
A.1.1 MCTS IN THE LEARNED LATENT SPACE
The MCTS procedure is central to UniZero's planning capability and is used during the exxeriencee phase. For each action selection, search tree is built in the latent space over fixed number simulations. Each simulation consists of three phases:
Selection: Starting from the root node (the latent state of the current observation), the search traverses the by selecting actions that maximize upper confidence bound score. This score, calculated using the PUCT formula, balances exploiting high-value actions (Q-value) and exploring less-visited actions (policy prior P). The action a" is chosen according to:
((, 1+ N(2,a)
Σp N( +
1
a" arg max Q(²,a) P(2,a)
+ log log
(2)
C2
where N is the visit count, Q is the mean action- value, P is the policy prior, and c1. are exploration constants.
Expansion: When leaf node is reached, the dynamics network is called to predict the next latent state g++1 and reward /!. The decision network predicts policy p' and value of for the new state. This new node is added the tree, and its outgoing edges are initialized with the predicted policy priors.
Backup: The value rf predicted the new leaf node is used to update the statistics of the nodes along the simulation path. The Q-values and visit counts N for all parent state-action pairs updated recursively back to the root.
After all simulations are complete, the visit counts at the root node are normalized to form temperature-controlll controlled distribution, which serves as the improved policy for action selection.
A.1.2 MODEL AND POLICY OPTIMIZATION
The update wo model step optimizes the network parameters by minimizing a joint loss function over batches of trajectories sampled from the replay buffer. The optimization objective in
18

---

Published as a conference paper at ICLR 2026
UniZero is composed of four main prediction losses:
H
Lunizero (0)
E (... JouB
Σ
Ba |l++ s1( +B, TB(
r=
next-latent prediction
reward prodiction
+Bp CE(pe, m
+ CE farg
policy prediction
valuee prediction
where is the training context length and the loss components are defined as follows:
The model predicts the next latent state & The target is generated by slowly-updating target encoder h and is detached from the computation graph using stop-gradient operator (sg).
Next-latent Prediction:
Reward and Value Prediction: The model predicts the immediate reward f, and the state value e4. To handle varying scales across tasks, both predictions are framed as discrete classification problems and optimized using cross-entrope (CE) loss. The value target sery is bootstrapped n-step TD target computed using future rewards and a target value network.
The model's policy head Pt is trained to mimic the MCTS-improve policy me- This serves as a policy distillation step, which enhances training stability compared to direct policy gradient methods.
Policy Prediction:
The terms Be, Br, Sp. By fixed coefficients that balance the contribution of each component.
A.2
CORE MODIFICATIONS OF SCALEZERO OVER UNIZERO
Our core model, ScaleZero, is built upon the UniZero baseline, with key components upgraded to enhance learning efficiency and final performance in complex multitask environments. The primary architectural differences are summarized in Table 3 with detailed explanations provided in the subsequent sections.
Table Summary of core architectural differences between UniZero ScaleZero.
Component
UniZero (Baseline)
ScaleZero (Our Model)
Backbone
Standard Transformer blocks.
Mixture-of-Experts (MoE) former blocks. Standard LayerNorm.
Trans-
Latent State Normal-
SimNorm (Ll regularization).
ization
Visual Encoder
ResNet-like architecture trained from scratch.
Vision Transformer (VTT) trained from scratch.
A.2.1 BACKBONE: FROM DENSE TRANSFORMERS TO SPARSE MoE
The most significant modification lies in the model's backbone. While both models employ a Transformer architecture process state, action, and task tokens, their internal structure differs fundamentally.
UniZero (Baseline): The backbone consists of N standard Transformer blocks (c.g., N = 8 for Atari26).
To maintain comparable parameter count while increasing model capacity, the backbone is composed of N/2 (for Atari) MoE Transformer blocks. In each block. the standard bred-forward network (FFN) is replaced by MoE layer. This layer comprises 8 specialized expert networks and one shared expert, governed by a sparse top-1 1 routing gate. This design creates conditional computation pathways, mitigating task interference.
ScaleZero (Our Model):
19

---

Published as a conference paper at ICLR 2026
A.2.2 ENCODERS AND NORMALIZATION
Encoders For encoding observations, ScaleZero adopts similar strategy to the baseline but with clear decision justified by empirical results.
Visual Encoder (e.g., Atari):
Our primary encoder is Vision Transformer (ViT) trained from scratch. The imput image (64x64x3) is divided into 8x8 patches, projected into token embeddings, and processed by standard ViT-Base encoder to produce 768-dimensional latent state.
Text Encoder (e-g,, Jericho): For text-based environments, we use a pre-trained BGE (BAAI General Embedding) model (Xiao al. 2023) to encode textual observations into exed-dimensional vectors.
Justification for From-Scratch Visual Encoder preliminary experiments, we explored leverag- ing a powerful pre-trained visual encoder, specifically frozen DINOv2 model (dinov2_va (Oquab et af., 2023), to potentially improve feature quality. However, this approach did not yield significant or consistent performance gains over the from-scratch VIT. We hypothesize this is due to the domain gap between the natural images DINOv2 was trained on and the distinct visual char- acteristics of Atari frames. This finding justifies our final choice and highlights the importance of domain- -peain- visual representations. We leave a more in-depth investigation of pre-trained models, including fine-tuning strategies, future work.
Latent State Normalization ScaleZero replaces the Ll-based SimNorm used in UniZero with standard LayerNNrm for normalizing the latent state. This change was found to be effective within our modified architecture.
A.2.3 TASK-SPECIFIC ARCHITECTURAL CONFIGURATIONS
It is important to note that certain architectural choices are adapted based the benchmark, rather than being modification of ScaleZero over UniZero. These configurations apply to our entire framework.
Encoders: For the DeepMind Control (DMC) suite, which features heterogeneous vector observa- tion spaces, we employ independent MLP encoders for each task. For benchmarks with unified input modality (images for Atari, text for Jericho), a single shared encoder is used across all tasks.
Prediction Heads: To accommodate the distinet action spaces of each task, independent prediction heads (for policy, value, etc.) are universally employed across all benchmarks.
A.3 EXPLORATIONS ON KEY DESIGN SPACES
We conducted several exploratory extensions to key designs of UniZero to validate their effectiveness in multitask settings.
Task Information Encoding:
To mitigate representational conflicts multitask learning, we ex- plored explicit encoding of task information. By introducing a learnable task embedding matrix, nn. Embedding (task_id) and concatenating it with the state representa- concat ilatent_state, task_ task_embed), model to better distinguish between different tasks. In our experiments, the latent_stat dimension was 672, and the task_embed dimension was 96.
task embed task_embed
tion, new latlettet sttatestteeeeee
aimed for the
Vision Transformer (ViT); Our VIT implementation is adapted from the
lucidrains wrwiinn/vitpyttoree ViT-Base level, are summarized in Table
library.
The core hyperparameters, corresponding
Latent Normalization Strategy (SimNorm): As part of our ablation studies, explored the SimNorm strategy, inspired by the TD-MPC2 paper. SimNorm partitions the latent state into groups, where each group V-dimensional simplex g. The transformation is given by:
exp(24i4V\7)
norm [gt..... G6.
gr.,
9
(4)
j= exp (2
20

---

Published as a conference paper at ICLR 2026
Table 4: Hyperparameters for the Vision Transformer (ViT) encoder.
Hyperparameter
Value
image_size
64x64
patch_ size
8x8
dim (Embedding Dimension)
768
depth (Transformer Layers)
heads (Attention Heads)
mip_dim (MLP Hidden Dimension)
2048
dropout emb _op__rooout (Embedding Dropout)
0.1
0.1
In our experiments, we set the simplex dimension V = 8.
Mixture-of- (MoE): Our MoE layer implementation leverages the efficient design patterns from the mist mistral-infereneee
library. Specifically, we adopt a "I Shared + 8 Routed" con- figuration where every token is processed by a dedicated shared expert while gating network simultaneously routes it to the Top-1 expert among eight specialized experts. We do not claim this configuration is the theoretical global optimum, but rather a pragmatic trade-off between compu- tational efficiency and performance. This design aligns with ad LLM architectures (c.g., DeepSeek-Mee MoE Dai et al. (2024) that explicitly separate shared knowledge from specialized knowl- moderate sparsity (Routing) significantly mitigates the interference observed in dense baselines. In the future, drawing inspiration from Ludziejewski al.] (2025) and [Zhao al. (2025). plan to investigate the optimal MoE configurations specifically for online MTRL.
edge. As Proof of Concept, it demonstrates that introducing
Gradient Correction for MTRL.
Conflicting gradients are a primary challenge in MTRL,, often impeding stable optimization. To mitigate this, adopt the MoCo algorithm from the LibMTL library (Fernando al. 2023 MoCo is a gradient correction method that finds a consensual update direction by dynamically adjusting task weights. This process operates on comentum-smoothhee gradients, which provide a more stable estimate of each task's 's long-term descent direction compared to noisy single-step gradients. The procedure involves two main steps:
commentumm-based Gradient Smoothing. To obtain a stable estimate of each task's descent direction, we maintain a momentum- -based moving average, (t),), of its past gradients. For each task at step i, the update rule is:
(t()) (1 3) Bg
(5)
where gt( is the raw gradient (t) normalized and scaled by its task loss, and B is the momentum coefficient. This smooths out noisy single-step gradients.
Dynamic Weight Adjustment. The task weights X(t) E TRN are updated to minimize the squared norm of the aggregated gradient, formulated as minx / En (() We perform an approximate gradient descent step on this objective. The correct weight update is:
(6) where (t) the matrix whose columns are the momentum gradients (t)) and y the weight learning rate. The final corrected gradient applied to the shared parameters 0, is the weighted sum g, (f) Σ. Ac(t)):(t).
X(t) softmax (A(t - 1) - (X))
1))
Our implementation is tailored for a Distributed Data Parallel (DDP) environment. All gradient correction computations occur on the master process (rank 0), which then broadcasts the final corrected gradient to all other processes for synchronized updates. The procedure is detailed in Algorithm Key hyperparameters are listed in Table
in inference/mce.rr
21

---

Published as a conference paper at ICLR 2026
Algorithm MoCo in a Distributed Data Parallel (DDP) Setting
Require: Shared parameters 0s. task losses {
Require: Momentum gradients {3/ J task weights À E RR
Require: Hyperparameters: Momentum B, weight learning rate
Ensure: Updated shared parameters 0..
1: for each training step do
# On each process parallel
2: Compute local raw gradients (g[ and losses {L)cca).
3:
Aggregate all [g. and [L to the master process (rank 0).
# On the master process (rank 0) only
4: Let J.V and [Li]i= be the aggregated global gradients and losses.
5: for i to N do
6: Normalize and scale: gf f- 1..1i -- Li-
7: Update momentum gradient: y/ (1 B)y + Bgb.
8:
end for
9:
Construct gradient matrix: Y
10:
Update task weights: À f softmax(1 -artaxxxxxayy)") Y)A).
11:
Compute final corrected gradient: 9s f- Enn A,Bli-
Broadcast the corrected gradient Is all other processes.
# On each process parallel
13:
Apply corrected gradient: Ve, + 9s. Perform optimizer step to update 0..
14:
15:
end for
Table 5: Hyperparameters for the MoCo algorithm.
Hyperparameter
Symbol
Value
Momentum Coefficient
0.99
Weight Learning Rate
10.0
Weight Regularization
0.0
A.4 DYNAMIC PARAMETER SCALING (DPS)
This section provides a detailed breakdown of the Dynamic Parameter Scaling (DPS) strategy introduced in Section 4.3 DPS is designed to dynamically couple model capacity with learning progress, creating "curriculum of model complexity" that directs resources precisely where they are most required. Detailed pseudocode is provided in Algorithm 2
Task Curation and Active Set Management. As described the main text, we define the full set of tasks as T {T.).- A task is deemed "solved" once its performance metric, Metric(T,). surpasses a predefined threshold E,. At any time t during training, we maintain an active set of all unsolved tasks, U4 C T. To maximize computational efficiency, both data collection and gradient updates are performed exclusively for tasks within this active set Ut- Once a task is solved, is removed from the active set, thereby ceasing all computational overhead associated with it.
Staged Capacity Expansion. The training process partitioned into S + 1 stages, following the protocol outlined in the main text:
Stage 0 (Warm-up): During an initial period of To iterations, only the shared backbone network parameters, 0g. are trained. This stage establishes a robust foundation by training the shared base model on all tasks to learn general general-purpose purpose representations.
Stage 2 1 (Expansion): Each subsequent stage introduces a new, independent LoRA module, 20, B, Concurrently, set of learnable scaling factors is defined to modulate the contribution both the base model and all adapters. A scaling factor ag is associated with the base model weights 0z and each LoRA module A0, is assigned corresponding scaling factor aj. All scaling
22

---

Published as a conference paper at ICLR 2026
Algorithm 2 Dynamic Parameter Scaling (DPS)
Require: Set tasks T (Tx
} J.
Performance thresholds {e,]-=; Backbone params Op:
Require:
Hyperparameters: Total stages S, Warm-up iterations To. Max total iterations T maxx. Stage
quotas {Q.]--1-
Ensure: Trained parameters: DB, co. {20,, at a.)5 g== 1: Initialize active set of unsolved tasks: U 4- T.
2: Initialize base model scaling factor: on <- 1.
3: for stage 8 0 to 5 do
4: if 0 then
Stage O: Warm-up Phase
5: Set trainable parameters Ornin f- OB-
6: Set stage iteration limit L To-
7: for Estage 1 to L. do Sample batch exclusively from tasks in active set U. Perform forward pass with backbone: W(z) = Wor
8:
9:
Compute loss L and update Orain-
11:
Update active set: U {Ti E Metric(T:) < Ei). end for
12:
13:
else
Stage 2 1: Expansion Phase
14:
Initialize new LoRA module 20, = BgA, and its scalar a, f-
15:
Freeze OB and all previous modules [A0,)]
16:
Set trainable parameters Buain +- { U {ao) U fay
17:
Record solved tasks at stage start: Suant + T
Set budget for current stage: Tbudget [[fmax To)/57.
19:
Estage f- O.
while Estage budgee do
Budget-based Trigger
21:
Esage t- Esstag +
Sample batch exclusively from tasks in active set U4
Perform forward pass: (x)) (ao + Σド=1 (B()))
24:
Compute loss L and update Ounin-
25:
Update active set: U + { E T Metric(T:) < Ei
if (T \ / 2 Q, then break
Progress-basee based Trigger
27:
b Quota of newly solved tasks met, transition to next stage.
end
29:
end while
30: end if
31:
end for
factors are initialized These factors allow the model dynamically re-weigh and reuse knowledge from the base model and earlier adaptations.
Adaptive Stage Transition Triggers. The transition from stage to stage is governed by dual-trigger mechanism, balancing learning progress against a fixed budget:
1. Progress-based Trigger: A new stage is initiated if the number of newly solved tasks within the current stage reaches a predefined quota Q..
2. Budget-basee Trigger: To prevent stagnation particularly arduous tasks, transition is forced if the iteration count within the current stage exceeds a pre-allocated limit, calculated as [[Tmax T5)
Optimization and Parameter Isolation. key principle of DPS is parameter isolation. any given stage 2 1, the effective weight matrix W(+) is dynamically composed from the base weight Wo E 0g and all accumulated adapters:
((s) an Wo +
Σ a, = = aoWo
Σ a ByA, j=1
Upon entering a new stage s, all prior parameters the backbone O (pre-trained in Stage 0) and the matrices of all previously introduced adapters [A0)) frozen. To resolve scale ambiguity
23

---

Published as a conference paper at ICLR 2026
between the adapter and its scalar, the scaling factor as for the currently active adapter A9, is temporarily fixed at 1. Optimization is thereby focused exclusively on the newly added LoRA module A6.. the base model's scaling factor an, and the set of scaling factors corresponding to previously frozen adapters, {az]z=i-
This strategy yields two principal advantages:
It implements a resource-efficien training curriculum, as new parameters are only introduced when required by the remaining unsolved tasks.
By isolating new learning within dedicated adapters, DPS provides targeted plasticity for difficult tasks while preventing catastrophic forgetting negative transfer previously mastered skills, whose knowledge is preserved in the frozen backbone and prior adapters.
Stabilized Learnable Scaling Factors. To prevent training instability from unbounded learnable scalars, we constrain the scaling factors ag within narrow, stable range (e.g., centered 1.0). This achieved via a re- ee-aarametrrization trick, where each oj is computed from an unbounded underlying parameter a as follows:
03 offset + range tānh(ā;)
This design ensures that the contribution each adapter is modulated smoothly without causing drastic shifts in the output distribution. The specifie DPS hyperparameters used in our ScaleZero-DPP experiments are detailed in T
Table Hyperparameter configuration for Dynamic Parameter Scaling in our ScaleZero-DPS experiments.
Hyperparameter
Symbol
Value
Description
curriculum Saruuuuuuusstteeeeeeeeeee num
S ++1
5
Total number of stages (1 warm-up expansion).
lora_r lora_alpha
64
The rank of the LoRA matrices.
1
Conventional LoRA hyperparameter (scaling = alpha/r).
Note: This is distinct from our learnable a; factors. Initial value of the DPS learnable scaling faccoors,
Dora_scale_init _s____all____geeee__ee______ laange min
1.0
0.2
Range for DPS scalars, yielding ay E [0.8, 1.2].
vin_stage0_iten iters
10,000
Number of iterations for the warm-up stage.
max_stage_iters
5,000
Per-stage iteration budget, i.c., [Tmax To)/ST.
A.5 HYPERPARAMETER SETTINGS
Our hyperparameter configuration for ScaleZero largely follows the original UniZero paper to ensure a fair comparison. We employ the AdamN optimizer for all experiments. Table 7 lists the hyperparameters that are kept consistent across all benchmarks. Architectural configurations and loss weights are detailed subsequently, followed by benchmark-speciffe settings in Tables g 10 and I1
Our ScaleZero model incorporates Pecture-of-Experrr layers within its Transformer backbone for the three benchmarks to enhance model capacity and specialization. When MoE is used, each MoE layer consists specialist experts and shared expert, with the router selecting the top-1 expert per token. The final loss is a weighted sum of several components, with weights varying between discrete and continuous action space environments, as detailed in Table
A.5.1 ATARI-SPECIFIC SETTINGS
For Atari, observations are 64 x 64 x 3 images processed by a VIT encoder. Training is conducted for 400,000 total environment steps.
A.5.2
DMC-SPECIHC SETTINGS
For the DMC suite, state vectors are processed by an MLP encoder. Training runs for 400,000 total environment steps.
24

---

Published as a conference paper at ICLR 2026
Table 7: Common Hyperparameters. This table lists the hyperparameters that remain constant across all benchmarks (Atari, DMC, and Jericho tasks) for ScaleZero and UniZero.
Hyperparameter
Value
Planning
Number of MCTS Simulations (sim)
50
Temperature
0.25
Dirichlet Noise (o)
0.3
Dirichlet Noise Weight
0,25
Coefficient c₁
1.25
Coefficient C2
19652
Architecture
Embedding Dimension
768
Number of Heads
Dropout Rate (p)
0.1
Activation Function
GELU
Reward/Value Bins
101 for DMC and Jericho, 601 for Atari
SimNorm Dimension (V)
SimNorm Temperature (T)
Optimization
Optimizer
Adam W
Learning Rate
1 x 10 -4
Policy Entropy Coefficient
x 10-4 10-4
Weight Decay
Max Gradient Norm
Discount Factor
0.997
Soft Target Update Momentum
0.05
Temporal Difference (TD) Steps
Table 8: Loss function weights for different environment types.
Loss Term
Discrete Action Spaces (Atari, Jericho)
Continuous Action Spaces (DMC)
Value Loss
0.5
0.1
Reward Loss
1.0
0.1
Policy Loss
0.1
Observation Loss
10.0
10.0
Table 9: Key hyperparameters Atari experiments.
Hyperparameter
Value
General Training
Effective Global Batch Size
512
Discount Factor (7)
0.997
World Model
Num. Transformer Layers Training Context Length (H)
2 (Atari-8), 4 (Atari-26)
10
Encoder Type
VIT
MCTS
Inference Context Length (Hitter
4
Replay Buffer
Replay Buffer Size
500,000
Replay Ratio
0.25
25

---

Published as a conference paper at ICLR 2026
Table 10: Key hyperparameters for DMC experiments.
Hyperparameter
Value
General Training
Effective Global Batch Size
512
Discount Factor (7)
0.99
Frame Skip
(8 for Pendulum)
World Model
Num. Transformer Layers
Training Context Length (H)
5
Encoder Type
MLP
Num. Sampled Actions
20
MCTS
Inference Context Length (Hinter)
Replay Buffer
Replay Buffer Size
1,000,000
Replay Ratio
0.25
A.5.3 JERICHO-SPECIFIC SETTINGS
For the text-based Jericho suite, a pre-trained language model (BAA1/bge-base--ar-11.5)) used as the text encoder. Training runs for 500,000 total enviromment steps.
Table II: Key hyperparameters for Jericho experiments.
Hyperparameter
Value General Training
Effective Global Batch Size
256
Discount Factor (7)
0.997
World Model
Num. Transformer Layers
2
MoE Layers
None
Training Context Length (H)
10
Encoder Type
Text (BAtt/bge-bas--en-11.
MCTS 4
Inference Context Length (Hirtr
Replay Buffer
Replay Buffer Size Replay Ratio
500,000
0.1
A.6 TRAINING FRAMEWORK AND COMPUTATIONAL ANALYSIS
This section details the distributed training framework employed for our multi-task experiments and provides a comprchensive analysis the model parameter specifications and computational costs for the Atari, DMC, and Jericho benchmarks.
A.6.1 DISTRIBUTED MULTI- TASK TRAINING IMPLEMENTATION
Our multi-GPU, multitask training framework is implemented based on PyToh'''s Distributed Data Parallel (DDP) (Li et al. 2020). The design is optimized for scalability and efficiency when training a single unified model across highly diverse task sets. The core implementation logic is as follows:
Statie Task Partitioning and Resource Allocation: To handle the diverse set of environ- ments (e.g., 26 distinct Atari games), we employ static partitioning strategy. The total
26

---

Published as a conference paper at ICLR 2026
set of tasks is divided among the available GPUs (ranks). Each GPU is assigned exclu- sive subset of tasks and instantiates dedicated resources for each assigned task, including independent data collectors, evaluators, and replay buffers. This modular design isolates task-specific operations, ensuring robust memory management and preventing resource contention between tasks.
Heterogeneous Batch Construction: During each training iteration, every GPU constructs a local training batch by sampling data segments from the replay buffers of its assigned tasks. This results in a heterogeneous batch containing experiences from multiple different environments. To ensure load balancing, the micro-batch size sampled from each task's s buffer is uniform and pre-calculated based on global memory constraints.
Training transformer- world models requires significant memory. To simulate a large effective batch size necessary for stable convergence without exceeding GPU memory limits, we utilize gradient accumulation. The number of accumulation steps is dynamically adjusted in conjunction with the micro- batch size. This strategy ensures that the model parameters are updated using gradients derived from a sufficiently large and diverse dataset, promoting effective generalization.
Gradient Accumulation for Large-Scale Training:
Once the gradients are computed locally on the heterogeneous batches, the DDP framework synchronizes them across all GPUs via an A1 Reduce operation. This results in globally averaged gradient that reflects the collec- tive experience of all tasks in the benchmark. A single optimization step is then performed on the shared model parameters, ensuring that the unified model learns jointly from the entire multi-task distribution.
Synchronized Global Optimization:
This integrated approach enables the joint optimization of a single, powerful model using data streams from multiple environments in a distributed and highly efficient manner, which is key ScaleZero's strong performance in multitask scenarios.
A.7 PARAMETER ANALYSIS AND COMPUTATIONAL CoST
To provide granular analysis of resource utilization, Table I compares the network parameter scales and specific training costs between our multi-task approach, ScaleZero (MT), the multi-task baseline UniZero (MT), and the single-task baseline UniZero (ST). Experiments for multi-task models were conducted on a computing node equipped with 8x (4x for Jericho-4) NVIDIA A100 (80GB) GPUs, whereas single-task models were trained on individual 1x NVIDIA A.00 (80GB) GPUs per task. Table 12 summarizes the parameter counts and approximate wall-clock training times for the main results presented in sssectioon 5
Note on Parameter Counts: embeddings and auxiliary heads; thus, exceeds the simple sum of the Encoder, Backbone, and Head parameters. For UniZero (ST), the values listed represent the model size for single task. The total training time for ST is the aggregate time required train all tasks within the benchmark sequentially or in parallel resource equivalents.
The Total Params column encompasses all model components, including
A.8 JUSTIFICATION FOR THE MODEL-FREE MULTI-TASK BASELINES
In the experimental comparisons presented in Table Ta we focus primarily on Model-Based ap- proaches, excluding online model-free MTRL baselines. This exclusion is driven by fundamental disparities in data regimes and urchitectural paradigms:
The Atari-26 26 benchmark presents significant challenges due to its heterogeneity in visual dynamics and mechanics. A survey of recent literature (Xu al. [022 Ye et al., 2022; ?) indicates that existing multi-task methods predominantly rely on offline pre-training or fine-tuning rather than learning ab initio. While methods such as IMPALA with PopArt (Hessel et al. 2019 have shown promise, they operate in high-throughhut regime, typically requiring over 200 million frames to converge. In contrast, ScaleZero targets the cample-efficien regime (consistent with the MuZero/UniZero research trajectory). operating within strict budget of 100k-300k environment steps steps- approximately 0.2% of the data required by Hessel et al. (2019). Under such severe constraints, gigh-throughput model- free RI. methods fail to learn meaningful policies, rendering a direct comparison methodologically
27

---

Published as a conference paper at ICLR 2026
Table 12: Detailed comparison of network parameters, wall-clock training time, and hardware resources. "ulayer" denotes the number of transformer layers in the world model backbone. UniZero (ST), training time indicates the cost per and the total cumulative time for the benchmark. Colors indicate method type: Blue for ScaleZero (MT), Gray for UniZero (MT), and Orange for UniZero (ST).
Benchmark
Method
Network Parameters
Training Time
Hardware
Encoder
Backbone
Head
Total
ScaleZero (MT) (nbyeras)
26.8
254.9 M on
81,5
366.4 M
~6 Days
8x A100 (80G)
UniZero (nlayee=))
18.8
56.7
81.5
160.2
5 Days
A100
Atari-26
7 Hours/Tsk
UniZero (ST) (elayer=2)
7,8 M
14.2 M
3,9 M
43.8 M
1x A100 (80G)
Total: ~ Days
ScaleZero (MT) (nlipe-4)
10.8 M
254.9 M (013500)
66.9 M
339,9 M
2 Days
8x A100 (80G)
DMC-18
3 Hours/Takk
UniZero (ST) (#byor=2)
0.60
11.2
(800)
Total: 2.3 Days
ScaleZero (MT) (nlayee))
110.1
14.2 M
250.2
< Days
(800
Jericho-4
Hours/Task
UniZero (ST) (obyyrul)
110.1 M
14.2 M
3.2M
127.5 M
1x A100
Total: 2.1 Days
unsound. To date, no prior work has reported single online MTRL agent capable of mastering the full Atari-26 suite from scratch under this data budget.
Similarly, the current landscape for the DMC-18 benchmark is dominated by offline approaches or methods utilizing expert demonstrations (Schmied et al. 2023 Haldar et al. 2024). Comparing our online learning framework, which learns purely from interaction, against offline paradigms that benefit from extensive pre-collected datasets creates an inequitable experimental setting due to the vast disparity data availability.
Existing Model-Free MTRL methods, such as PaCo (Sun et al. 2022 and Soft Modularization (Yang al. 2020), introduce architectural innovations specifically tailored for Policy or Value networks. Conversely, ScaleZero operates within Model- -ased framework focused on learning environmental dynamics. The inductive biases required for accurate dynamics modeling differ fundamentally from those optimized for value estimation. Consequently, directly transplanting architectural mechanisms designed for policy optimization into a World Model structure lacks theoretical compatibility and straightforward implementation.
B ATARI EXPERIMENT DETAILS
B. BENCHMARK SETUP
This section elaborates on the experimental setup for the Atari benchmarks discussed in Section 5.1 We use the Arcade Learning Environment (ALE) (Bellemare et al. 2013) as our primary simulation platform.
Our evaluation protocol compares our multitask model, ScaleZero (MT), against a strong single- task baseline consisting of individually trained UniZero (ST) models. We adopt this comparative framework for two reasons: first, standard online MTRL baselines for the full Atari suite are currently lacking; second, the multitask variant of UniZero demonstrates severe performance degradation. Consequently, the UniZero ST baseline serves as rigorous "specialist" upper bound, providing challenging standard for evaluating the net positive knowledge transfer of our multitask approach.
The experimental evaluation is structured into two phases: first, we conduct model design and ablation studies on the Atari8 (multitask) benchmark (a subset of 8 games: Alien, Boxing, ChopperCommand, Hero, MsPacman, Pong. RoadRunner, and Seaquest). Second, we evaluate the final ScaleZero architecture on the full Atari26 (multitask) benchmark.
For all games, observations are preprocessed into 64x64 RGB images. The state input to the model is a single frame. yielding an input tensor of shape 3x64x64 without any frame stacking. We employ standard environment wrappers, including sticky actions (p=0.25) and frame skip of 4 (Mnih et al.) Score (HNS), with both mean and
2013). Performance quantified by the median scores reported assess aggregate performance and robustness. To ensure reproducibility and
28

---

Published as a conference paper at ICLR 2026
fair comparison, all environmental configurations strictly adhere to those in the UniZero paper (Pu et al. 2024) and DI-engine (Niu et al. 2021).
B.2
SUPPLEMENTARY ANALYSIS OF PLASTICITY AND REPRESENTATIONAL COLLAPSE
This section provides deeper empirical analysis of the plasticity loss and representational collapse phenomena diagnosed in Section
B.2.1 TRAINING DYNAMICS AND REPRESENTATIONAL COLLAPSE IN UNIZERO
While Figure in the main text illustrates the performance degradation and plasticity loss for UniZero complex tasks, |Figure presents the same metrics for the four other tasks in the Atari8 set.
To quantitatively diagnose the cause of this degradation, we measure the effective rank (Dohare et al. 2024) of the model's representation space. low effective rank indicates representational collapse, where the model fails to maintain the feature diversity required for multitask learning. For matrix E gaxm with normalized singular values pks the effective rank is the exponential of the Shannon entropy of its singular value distribution:
erank () = exp
=xx
Σ pk: log(pk)
Specifically, we compute the effective rank of a hidden layer's activation matrix from a minibatch of 256 sequence samples. As shown in FFigure 5] the performance collapse in the baseline model directly accompanied by decline in the representation' 's effective rank, providing quantitative evidence for the representational collapse hypothesis.
B.2.2 ANALYSIS OF REPRESENTATIONAL COLLAPSE
The correlations observed in our experiments reveal a critical failure mode in standard multitask architectures. These models face inherent difficulties in reconciling conflicting task demands, leading to severe gradient conflict. We hypothesize that as gradients from distinct tasks interfere destructively, the magnitude of the aggregated update signal diminishes phenomenon we term Gradient Norm Collapse. This hypothesis is supported by the experimental analysis in Ma et al.) (2025). which demon- strates that dense connectivity inherently induces such destructive interference, thereby impeding the model's ss ability to track non-stationary data distributions. Consequently, the feature space degenerates into a diwddiennsioal, task agno-tgi subspace, which is suboptimal for specialized tasks. This is the mechanism of representational collapse, quantitatively diagnosed by the low effective rank shown in Figure 5)
This contraction of the feature space renders many neurons redundant, leading to rising dormant neu- ron ratio. Specifically, when weights fail to adapt to shifting input patterns, neuronal pre-activations may systematically drift into negative regions. Once a neuron becomes inactive, its gradient vanishes, effectively freezing the associated weights and resulting in irreversible Plasticity Loss (Sokar et al.) 2023). To compensate for this impoverished representational capacity, normalization layers (e.g.. LayerNorm) amplify the magnitude of the few remaining active features, causing an inflation of the latent norm. Ultimately, this collapsed representation lacks the capacity to model task diversity, resulting in suboptimal performance.
In contrast, ScaleZero's 's MoE architecture is designed to mitigate this issue. By routing tasks to specialized expert sub-networks, it enforces dynamic sparsity. This design is consistent with the findings of Ma et al. (2025), which suggest that network sparsity is crucial for unlocking the scaling potential of deep reinforcement learning. By reducing the interference highlighted in their work, our MoE design theoretically lowers the upper bound of gradient conflict. This preserves distinct representational subspaces. As shown in FFiure 7] the MoE architecture effectively maintains neuron activity (low dormant ratio) and stabilizes representation magnitudes (controlled latent norm), thereby preventing representational collapse and enabling superior multitask performance.
29

---

Published as a conference paper at ICLR 2026
Return
Effective Rank
400
-10
304
200
304
0000
Figure 5: Analysis representation effective rank. figure supplements diagnosis Figure showing the relationship between game return (Left) and representation effective rank (Right) for the baseline model. The sharp drop in performance correlates strongly with decline in effective rank, substantiating the claim that performance failure is linked to collapse in the dimensionality of the model's 's representation space.
30

---

Published as a conference paper at ICLR 2026
Boxing
Mspacman
070
2000
-10
-15
08
20K
1006, 1OOK 2006 2006. 2008
0K
305
1008
1358
2006.
2005
300K
06
506
100K 100K:
2006
2306, 3006
06.
50K:
1900
150K. 2008
250K
2016
Steps
Stags
Dommnt Ratio
Latent Norm
Altn
Jile
Boeing
Booigg
32-
20
15
35
Era Steps
Figure 6: Training dynamics of UniZero on the other four Atari8 tasks. Complementing Figure 1 from the main this figure shows performance curves, dormant neuron ratios, and latent for UniZero Boxing, MsPacman, RoadRunner, and Alien.
B.3 FULL PERFORMANCE ON THE ATARI26 BENCHMARK
This section provides the complete learning curves corresponding to the summary results in Table la As reported in the main text, the single multitask ScaleZero (MT) agent achieves a higher mean HNS than the average of 26 specialized UniZero (ST) agents. This result demonstrates that ScaleZero achieves net positive knowledge transfer, which we attribute to the MoE architecture's ability to maintain plasticity and transfer general priors (e.g., object physies) from simple to complex tasks.
However, the median HNS of ScaleZero is lower than the ST bascline. This is primarily influenced by few cardeexploration mechanically unique games (e.g., Fye,,
suboptimal performance indicating that negative interference not fully eliminated. The complete learning curves for all 26 games are shown Figure 8
C DMC EXPERIMENT DETAILS
C.1 BENCHMARK SETUP
This section details the experimental setup for the DeepMind Control (DMC) Suite (Tunyasuvunakool al. 2020), which serves as our benchmark for continuous control as presented in Section [ Experiments conducted
suite of 18 tasks (e.g., walker_wadd cheetah. (eretahh_un),, following the experimental setup Pu et al. (2024). Following the methodology our Atari experiments, we compare single ScaleZero (MT) model against baseline of 18 individually trained UniZero (ST') models. Model inputs are low-dimensional state vectors from the environment, and the action space continuous vector normalized to [-1, 1]. Performance measured as the average return over 8 evaluation episodes, with results averaged across 2 random seeds.
C.2 PERFORMANCE COMPARISON: SCALEZZRO vs. UNIZERO
As summarized in Table 1b and detailed in Figure 9 the multitask ScaleZero model achieves performance competitive with, and often superior to, the single-task UniZero baseline. Notably, ScaleZero achieves a higher median score, indicating robust generalist performance across the majority of tasks rather than excelling on only a few. This validates that the architectural principles of ScaleZero are effective for complex, state-based continuous control problems. We hypothesize this success stems from the MoE layers learning to specialize in shared physical priors and control primitives (e.g., balancing locomotion), which then composed by the router based on the task.
31

---

Published as a conference paper at ICLR 2026
Return
Dormant Ratio
Latent Norm
Pong
-00
20
4000
92
2000
2000
1000
1290
8000
00
20
Alam
Boslng
40
2030
Roptomun
63
1000
20
12500
Roadrunner
3000
50K 100K
150K
200K
200K.
3C0K
50K 106K 150K 200K 2536
50C
150K 2006 2006 200K
Emt Steps
Emr Steps
Emr Steps
UNOWE
VIT
MCCA
Figure 7: Complete plasticity metric analysis for the ScaleZero design space exploration. This figure provides full data for the ablation study, detailing the evolution of performance (Retum), Dormant Ratio, and Latent Norm for different model variants across all tasks in the Atari8 multitask benchmark. The MoE-based model consistently outperforms others by maintaining superior plasticity metries.
32

---

Published as a conference paper at ICLR 2026
200
Retur
6004
0.00
6284
6.0M
0.0M
111M
6.00
2.004
0.04M
5.0M
6CMMM
6.10M
Ba siHees
SorrieZons
Boahng
Breakout
Peturn
13300
20
1039
-10-
0U6M
0464
0.64
020M
ChooperCommend
CranyClimer
Demonddtack
Freeway
3.0
13:0
Return
2200
1300
00.04
LUCK
1.1MM
0.004
11.0
0.1M
1.20
Gopher
Hero
Jameshond
2200
Returh
1300
C.0.
0.0.M
U.OM
Kancernz
Kung HuMeeer
Motecman
177m1
1500
Return
3300
250
5300
0.04
6.004
Peng
Privabelye
Obert
Rosdil.mme
400
1300
1900
-20
0.00M
0.00M
3.00
0.0H
0.04
3.30M
25M
6.20M
0.00M
0.00M
0.10M
0.10M
8.55
6.22M
Seacuent
UphiDosn
Normalzed Hean
Normmal Median
43.0
Raturn
63
623M
Steps
UniZero (ST)
ScaleZero (MT)
Figure 8: Learning curves for ScaleZero (MT) vs. UniZero (ST) on the Atari26 benchmark. This figure
shows the full per-game learning (mean and 95% confidence interval) for the multitask ScaleZero agent compared against 26 single-task UniZero baselines.
33

---

Published as a conference paper at ICLR 2026
Acretst Seingup
Sdl by Cup Catch
Cottook Bolance
Compole Saleno Soans
2009
003-
925m
6.M
63M
6.24
6.20
6/M
1.24
9.50
0.61
6AM
2:24
0.59
0J6
622
644
0.08
0.19
6.2
6.3%
3.9%
Carpooe rguu
Cartzdl Salnpup Starre
Cheeban Run
Enger Spt
300
600
825um
:00-
290
3000
600
5XC
16.
Perdulun Swingup
Raacher Easy
Neacher Hard
Walkrr
2009
500
400
203
200
0.0M
0.34
6.24
0/10
1.24
0.50
0.1
6/M
0.15
0
6.44
0.04
0.59
6.2
1.34
1.9%
Walker Stand
Whller
Nomolized Hean
Normalaed Median
3.00
2003
Emy Steps
UniZero
ScaleZero (MT)
Figure 9: Learning curves of ScaleZero (MT) vs. UniZero (ST) the DMC18 benchmark. This figure provides the full per-task learning curves, comparing the performance of the single multitask ScaleZero agent against 18 specialized UniZero baselines. Solid lines represent mean performance over random seeds, and the shaded area indicates the 95% confidence interval.
34

---

Published as a conference paper at ICLR 2026
C.3 EFFICIENCY EVALUATION OF DYNAMIC PARAMETER SCALING (DPS)
This section provides the detailed learning curves that substantiate the efficiency gains of Dynamic Parameter Scaling (DPS), as reported Section 5.2
Figure 10 compares the per-task sample
efficiency of ScaleZero-DPS against the standard ScaleZero model.
The curves demonstrate that ScaleZero-DPS (orange) learns significantly faster on most tasks, achieving target performance levels with fewer environment interactions. This validates the claim that DPS leads to a substantial reduction in sample complexity. The asterisks (*) denote tasks where training terminated early by the DPS policy upon reaching "solved" threshold, visually confirming the dynamic allocation of computational resources away from mastered tasks.
Acrobot Swingup
Ball In Cup Catch
Cartoole Balance
Cartpile Balence, Sparge
103-
201
Retum
-000
403
200
2.34
1.4
0.3
5.24
0.34
0.39
2:84
6M4M
64M
0.17
6.24
0.1d
0.54
8.24
6.3M
0
0.99
Cattpolc Sningup
Cartpolc Saingus Sparss
Chociahh Bun
Reger Spin
Retur
400
200
200
53M
0:M
2.54
6.12
0.28
3:94
6M
0.05
0.19
0.1M
0.3M
0:34
6:2M
0.50
Finger Tum Easy
Ploger Turn Hard
Plepper Hop
Heppper Stand
609
Patum
4o3
1.34
02M
2.24
0.30
0.26
6.4
0.39
0.0M
9:24
0.2M
0.30
0.49
Pendurum Swinoup
Reather Easy
Reacher Hard
We ler Run
800
retum
402
703
2.2M
0.M
0.32
0.5P
8.24
6.2M
6.60
Wibler Stand
No ker Walk
Normalas laod Mean
Nermalized Wedian
0.5
0.0
ScaleZero
Figure 10: Sample efficiency comparison of DPP standard Scale7ero DMC18. This figure shows the learning (return environment steps) for both models across 18 tasks. The faster rise of the ScaleZero-DDP curves (orange) illustrates its superior sample efficiency. asterisk (*) indicates that DPS policy halted training for that task, demonstrating the dynamic resource allocation mechanism in action.
C.4 MECHANISTIC ANALYSIS OF DYNAMIC PARAMETER SCALING
To understand the mechanisms by which Dynamic Parameter Scaling (DPS) achieves its sample efficiency, we conduct a granular analysis of the agent's ' intermal parameter dynamics. This section presents two complementary visualizations: a time-series plot showing the continuous evolution of key metrics (Figure IT), and a suite of "importance matrices" providing discrete snapshots at critical training junctures (Figure 12.
35

---

Published as a conference paper at ICLR 2026
C.4.1 SETUP
This figure provides a macroscopic view of the training process. The top panel correlates high-level putriss-normalizee performance scores and the number of solved tasks-with the progression of training stages. The bottom panel offers a view into the internal mechanics by tracking the average learnable importance ('alpha' scale) of LoRA adapters. It specifically compares the behavior of adapters in the first Transformer layer (Layer O, solid lines) versus the last (Layer 3, dashed lines), highlighting the emergence of different dynamic behaviors at different model depths.
Time-Series Dynamics (Figure I1):
Stage- Wise Importance Matrices (Figure 12): This multi figure offers more granular, event-based perspective. Each matrix is an "importance matrix" where a cell at '(row r, column c)* shows the average "alpha" scale of the adapter introduced in Stage 'c', as measured at the end of Stage 'r's training. This reveals how the model re-evaluates the contribution of all historical adapters after each phase of targeted learning. To provide comprehensive view, this analysis presented for:
The overall average across all layers and adapter types (Panel a).
A per-layer breakdown for Layer O and Layer to show hierarchical differences (Panels b-c). A per-type breakdown for adapters applied the Query, Key, Value, and output Projection linear layers to reveal functional specialization (Panels d-g).
C.4.2 RESULTS AND ANALYSIS
The analysis of these figures reveals several key dynamics of the DPS-managel learning process. The time-series plot (Figure 11) illustrates the staged training protocol. In this particular experiment, the stage transitions (vertical dashed lines) are triggered by a pre-allocated iteration budget, consistent with the method's budget-based trigger mechanism. These capacity expansions occur as overall performance and the number of solved tasks increase the course of training. The divergence in "alpha" scales between Layer O and Layer 3 suggests that the model applies different update strategies to shallow versus deep layers as training progresses.
The importance matrices in Figure 12 provide further detail on this behavior:
Overall Strategy (Panel a): The data is consistent with a dual mechanism of knowledge retention and plasticity. The importance scale of the foundational adapter from Stage 0 (column 0) is not static; it increases from 1.0 to 1.097 by the end of training. This suggests that the model increases the relative contribution its initial knowledge as encounters more complex tasks. Concurrently, the diagonal values show how the model focuses on the newest adapter during each stage, which is then re-weighted in subsequent stages as its knowledge is integrated.
Hierarchical Differentiation (Panels b vs. e): A notable difference is observed between Layer 0 and Layer In Layer (Panel b), the importance of the initial adapter (cell (4,0)) exhibits high stability with moderate growth (to 1. 1.058), indicating its role as stable foundation for low-level features. In contrast, Layer 3 (Panel e) shows more pronounced dynamic re-weighting. The foundational adapter's importance grows more significantly (to 1.099), suggesting that higher-level, abstract representations rely heavily on and amplify this core knowledge. The greater variance in Layer 3's alpha values indicates that later layers are more involved in adaptive, task-specific strategy modifications.
Functional Differentiation by Layer Type (Panels d-g): The analysis reveals distinct behaviors based on which linear layer within the attention mechanism an adapter modifies.
Query (Q) and Key (K) Adapters (Panels d, e): Adapters applied to the Q and layers, which produce the representations for calculating attention scores, generally exhibit importance scales that increase over time. For instance, the Stage Key adapter's scale reaches 1.104. This suggests continuous strengthening of the mechanisms that determine token-to-token relevance. Value (V) and Projection (Proj) Adapters (Panels f, g): contrast, adapters for the V layer (which provides the content to be aggregated) and the output Projection layer (which combines attention head outputs) show more complex dynamics, including suppression (alpha 1.0). For example, the Value adapter from Stage 3 is dlwn-weighte to 0.889 after Stage 4. This behavior may indicate a more cautious integration of new value-based information or a re-balancing of attention head outputs to integrate new skills without disrupting existing ones.
36

---

Published as a conference paper at ICLR 2026
In summary, the DPS method facilitates structured learning process. The model appears to preserve and amplify foundational knowledge, primarily in early layers, while enabling dynamic, adaptive strategy strategy-wwithhng in later layers. Furthermore, it modulates the importance of adapters based on their specific function within the self-attention mechanism (Query/Key vs. Value/Projection) This combination of knowledge retention and targeted, hierarchical plasticity offers mechanistic explanation for the method's observed sample efficiency.
DPS Performance and Task Curation Dynamics
Stage
Slege
Number of Solved Tesks
Normalized Score
0.6
0.5
Normalized Nean
Solved Tasks
0.4
Average Alpha Scale Dynamics per Layer
1.08
Query
Key
Layer
Value
Layer
3.04
Average Alpha
1.02
1.00
0.90
3k
10k
20k
30k
40k
Training Steps (Iterations)
Figure 11: Internal dynamics ScaleZero-DPS during training. (Top) Performance and task-solving progress across stages. (Bottom) Average 'alpha" scales for adapters in the first (Layer 0, solid) and last (Layer dashed) layers, showing the emergence of hierarchical specialization.
D JERICHO EXPERIMENT DETAILS
D.1 BENCHMARK SETUP
Jericho (Hausknecht al. 2020) a reinforeement-leariing benchmark built on classic text-adventure games, played entirely through natural-language interaction. Un- like Atari or DMC, demands robust language understanding of free-form scene, item, and state descriptions while contending with a combinatorial, effectively unbounded action space at each step the environment surfaces only small candidate set, yet many legal commands outside still execute-and and coping with sparse, delayed rewards that require long-horizon exploration and planning, making Jericho a stringent testbed for text-based RI..
Environment Overview:
We conduct experiments on four representative tasks from the Jericho benchmark, namely Detective, Acorncourt, Omniquest, and Zorkl. Examples expert interaction trajectories for Zorkl and Detective are illustrated in Figure 13] Following the protocol of the previous two benchmarks, compare our multitask ScaleZero (MT) with the single-task UniZero (ST) baseline. ScaleZero uses the same text encoder (bge-base base-en-vl.55 the same inference context length of 4, and the same per-step max sequence length 512 tokens as UniZero. The max action num and max steps for the four tasks given in Table 13
Results reported as the mean return
random seeds. If,
37

---

Published as a conference paper at ICLR 2026
LOGO
1.34
2
5.00
6865
L:33
Strgee
LO00
LO0
2986
LO00
Eunclatio
LiNZ
LB/3
LOWP
LOK
CSM
Adaparr Modiue orige Stage
Ada Poddd Stage
Aateste Podde Or Stage
(a) Overall Average
(b) Layer O Only
(c) Layer Only
Exclatier Denry Migpaar
Ingrraacce Der Sry edlggiis
Dolatee
Imporannt Peppl
(d) Query Adapters
(e) Key Adapters
(f) Value Adapters
(g) Projection Adapters
Figure 12: Evolution of Adapter Importance Across Training Stages and Model Components. Each matrix displays the learned importance (average 'alpha' scale) of adapters. A cell at '(row col c)' represents the importance of the adapter from Stage "c" after Stage 'r's training complete. (a) The overall average shows clear pattern knowledge preservation (column 0) and targeted plasticity (diagonal). (b-c) The layer-specific brcakdown reveals that foundational knowledge is more rigidly preserved in carly layers (Layer 0) while later layers (Layer 3) are more dynamic and adaptive. (d-g) The type-specific breakdown shows functional differentiation in how adapters are utilized over time, reflecting their distinct roles in the attention mechanism.
38

---

Published as a conference paper at ICLR 2026
dbuervriiee: Noppright 1981, 1987, 1981 Infvern All right waPORK
stceed trakenak Indacee, lac sRooaae 85 Sonal camber 540 Aeleleaa ciaaaciiiii ue sanning un coai fidd wos of white lonc wili beanked bax doee hec in sanill maat hame anrrooodd actions ('rpe rralibort booth wouth! wae' y
eleer awteeeinssrreaaaaaaaaaa byne ""Lapi" muse informtioon rear1s: inneen lobrayy d'1a Sibeler Chael office vor' canding Cliefvollice He kelling you Thee Moss nuslodd catslay rigltst 12.03 us. I wast sre to móne il feeere we aci aas laal publicty i le TBI la to cone in. "Yosd !f You spply Fe hank you shect of pape. Oase you lave reed it, go goneth or wesl yaale can sica pfoccof pepei woffow sore just gane
vasiial aecenneaaaaaaaeeeeeaaiea desivcallyy
side al vitabooo londded up. To secaati sancw patk wina Bhrough the iain
There
Valdd aeuaee Tond, took wodd"
poitssitaaad aclons Food, lake
actlen
STER1
ebservatien: "nttkerr hhur acre has pur gone by tm perms, je actire (eaett warl lox pare laa
obborcriier: Tooner Phe park sanding through drery fireed The heads
Ul ahaa boc, paralaaly
albuend low buaako staad lhe odge of puilia aialiial adians Dayt jos aoound loccal; looth, wolld waat keud""
oxtine:
STER2
abservation: "SOIINEREEN cntlenctio was created Man lammgrr worked
unding lllnggg But REALYY appreciae gareatzs Oheve wnd a ie
on you bete cajes
aburrvation" the abear faet aboce groend sented sere
iflf
adfa The ucased brands alove ahone cendiatoon oid en the leunch is seall
sik
buft rood cin the bich wol is Legesss upa axcusie will powiin aach. agp acelly waseaged to woggte. The agg iooeeera fine alay sevemnnted lapis Raglle Valid
to over a 888 will the Gleabester Centaal RBS (511g254 5457 Thore on Bully Gerris file ave Have fax Wilk geve hast cat over the IBB to any of cry gamo laai actien: [ouf wed jes paper downl, toon
take oggi take take
action: "q-
alive Drogged Nodd actonns Draat lake papel, 'woof "ronk??
"DUOP
"Nooh of oo woe
side of There
here and all the windons arr hea didddd To Tolhoooth samow park winch trroggh trees.
iroelaiiiaationn Fooe honth/ wee' T
selheer corynng nortieg Paaer paper, "ookyy"
actkes
(a) Zorkl
(b) Detective
STEP
ebvervatiens "ONNNIQU as PCP Reiesac Perse Vesson OP RUE ES! Traahs out- hing to dis and yours wes of tired, un you lay you hed and tank phikosptiic Eangttoon by memmnng of sts and of the world, dic) As you plalssphars, yeu bupa deac. tn up with
sais you plance srordl you you
You Commaderr Hoveio Vie
Quut 1966 Clrk Flandan and Cheis Enherdge
latt- Clais Badlanekkeees 2/ Senal mamber 0ab)2? i Lafera v622 Librury 6/10 SD-iinlarge Clonngg ane wanding an brgg clarting camorrded by denas frree Them io pars th exelii tweere car aee the (allid actione ('oanens taat laae""
Thec ies hazping dae You boiee iollddd adiona: /add,
ancki Jonal, you coosecke!"
actiun: rms"
rowerd: 5.0
deservatia:" 'cSmull Paatral path tranches have Thre patl die wos and wuch chat continac inao the firos. The paih cna leal into caller mndy inmiarylaa.. ald aaneee (weet tae wastr ppat exonce downt (pst on mreekl??
actioe:
rewarth:00
andiamat Vadd actisn. is soutleadd wolloral jos waskel down, jor cnsneskel, muell""
"aFak Rodi feloe
aetion:
row Nowardinh
Mand
ebeervatiess "slad Patioella aspaso to be the ondefiie puts. There cuil to the Shoue mreton can march here intlld scnoee yake meery lige macch 'ou sonsove satsenckl doen, jut on aas T
actions ye""
(c) Omniquest
(d) Acorncourt
First five steps of expert interaction trajectories across environments. At each step, environment supplics a set of valid candidate actions (used in our experiments); however, additional legal actions outside this set may also he exccuted, so the action space is, in principle, unbounded.
Figure 13:
any step, the valid action set is smaller than max action num, we pad to max action num and mark the padded entries with an action mask so they are excluded from sampling.
D.2 PERFORMANCE COMPARISON, SCALEZERO VS. UNIZERO (ST)
As shown in Figure 14, the multitask ScaleZero attains returns comparable to the single-task UniZero baseline on most Jericho games, with learning curves exhibiting "easy-first, hard-later" progression. Specifically. ScaleZero converges faster on Acorcourt, Omniquest, and Detective; on the most challenging Zorkl, it initially lags but accelerates after 150k environment interactions to reach parity with UniZero. We hypothesize that this stems from multitask training inducing transferable language priors and interaction routines, which help filter feasible commands and reduce fruitless exploration.
39

---

Published as a conference paper at ICLR 2026
Octective
CEFRQUASE
20kk1
233
332
32
Stagso
Steps
Stops
570 51895
Figure Learning curves ScalcZero (MT) vs. UniZero (ST) on the Jericho benchmark. This figure compares the performance of the multitask ScaleZero and single-task UniZero models on Jericho tasks. Solid lines represent the mcan performance over 2 random seeds, and the shaded area indicates the 95% confidence interval.
maximum valid actions (in 200 episodes)
maximum steps per episode (in 200 episodes)
Game
max action num
max steps
Acorncourt
34
17
45
100
Omniquest
24
78
25
100
Zorkl
53
396
55
500
Detective
11
51
12
100
Jericho tasks. Maximum valid actions denotes largest number of valid action candidates exposed by the environment at any step, observed over 200 evaluation episodes; this statistic informs the max action num parameter used in our experiments. Maximum steps per episode is the largest episode length observed the same 200 episodes, while steps parameter is the spisode-length cap adopted during experiments.
Table 13: Key game statisties
E GRADIENT ANALYSIS IN MoE
E.1 EXPERIMENTAL ANALYSIS
To understand why Mixture-of Experts (MoE) architectures excel in multitask settings, this study investigates the core mechanisms driving their performance. Focusing multitask reinforcement learning, we conduct dual analysis combining theoretical inquiry with empirical validation. The following section details the experimental protocol designed for this purpose.
E.1.1 EXPERIMENT 1: ANALYZING GRADIENT CONFLICTS IN MOE-BASED TRANSFORMERS
We conduct our experiments on Atari-8. Concretely, our network architecture consists of an encoder (ViT), backbone (Transformer), and corresponding heads. We compare two baseline methods with different backbones:
(1) Naive Transformer: The backbone consists four standard Transformer blocks.
The backbone also consists of four Transformer blocks, but the MLP layer in each block replaced with an MoE layer, which comprises one shared expert and eight non- shared experts(Liv et al. 2024). Shared experts provide cross-task general representations, enhancing generalization ability and ensuring training stability, while non-shared experts learn task-specific representations to strengthen the model's discriminability and task adaptability. During the forward pass, all non-shared experts are selectively activated by sparse gating network, which determines which specific expert use for each input.
(2) MoE-based Transformer:
For both baseline models, we investigate gradient conflicts at different components: (1) Input before the MoE layer. (2) Output of the encoder. (3) Parameters the MoE, including the shared expert, non-shared experts, and the entire MoE layer. We measure gradient conflict between tasks using the maximum negative cosine similarity, defined follows:
(-
10,-vo V v0,1l
VO, VO,
Max Gradient Conflict max i.j
(10)
where V0, and V0, denote the gradients of the i-th and j-th tasks, respectively. higher value of the Max Gradient Conftict represents greater degree of gradient conflict. We choose the maximum
40

---

Published as a conference paper at ICLR 2026
pairwise cosine similarity because it directly identifies the most severe gradient conflict between any two tasks. Unlike averaging operation, which can hide critical issues, the maximum value pinpoints the 'bottleneck pair' that most significantly impedes stable multitask training and overall convergence, even if other tasks cooperate well.
In MoE, when computing the gradient of an entire layer, if task A selects expert i while task B selects expert j with * j. then the gradient of task B on expert i is filled with zero. The reason that expert i is not involved in the forward propagation of task B, and thus makes no contribution its loss; consequently, its gradient during backpropagation should naturally be zero
10
2x
Drocker conte
M
6x
10
W Wilwast (aolly//MMia..... wolMaaaniio /i
10
10
6 20
2008
1000 1300
1
Thandin
(a) Conflict comparison of MoE layer gradients.
(b) Conflict comparison of en- coder output gradients.
(c) Conflict comparison of MoE input gradients.
Figure 15: Comparison of gradient conflicts between MoE and MLP baselines across different components. MoB-based Transformer exhibits fewer gradient conflicts MoB input and MoE layer.
Observation 1:
As shown in Figure 15 (a), the MoE-based Transformer exhibits fewer gradient conflicts in the MoE layer than its MLP counterpart. Furthermore, Figure Figure [5 (c) shows that introducing MoE also reduces gradient conflicts at the MoE layer's 's input, implying that MoE alleviates conflicts in other components to some extent. However, at the output of the shared encoder, as depicted in Figure Figure 15 (b), the gradient conflict levels for the MoE and MLP models are largely comparable and do not show a significant difference. This indicates that the advantage of using MoE in the backbone to mitigate gradient conflicts is primarily localized to the MoE layers themselves and their immediate downstream connections; this effect does not substantially propagate back to the upstream shared encoder.
We posit that since the encoder functions general feature extractor for all tasks, its gradient dynamics are likely dominated by the need to learn common representations, making it less sensitive to architectural changes in downstream modules.
1.2 EXPERIMENT 2: INVESTIGATING MoE GATING MECHANISMS
We are particularly interested in the internal gating mechanism of MoE. Previous StudiesChen et al. (2022) have shown that, under supervised learning, MoE can implicitly uncover latent cluster structures within input space. However, when faced with non-stationary data distributions generated through agent-environmen interactions, it remains unclear whether MoE experts still differentiate effectively. To answer the question, we analyze the entropy of expert selection distributions, which quantifies the uncertainty of the choice of which expert to use for a task, with low entropy indicating high specialization and decisive selection, while high entropy implies uncertainty or an average utilization across experts. We also record the Wasserstein dislance (Rüschendori 1985) between the expert selection distributions of different tasks, where smaller values indicate a greater proximity to the expert selection of two tasks. We aim to quantify this relationship find the underlying connections in expert selection among various tasks. The specific experimental steps are as follows:
(STEPI) During forward pass given training step, record the expert choices in the final MoE-based Transformer block.
(STEP2) For specific training step st, collect expert selection data past window of size S and compute the frequency of each expert's ' activation to form a probability distribution. For a given window size S, task and specific task j, we denote the task selection distribution P, and
41

---

Published as a conference paper at ICLR 2026
specific probability for task P.y. Different window sizes reflect attention data over different temporal scales in non-stationary learning.
(STEP3) Based on this probability distribution, calculate the expert selection entropy for each task Etask,
N
Etask;
Σ
log2 (PS) (
5=1
and Wasserstein Distance between task i and task 1' (We,s)
We consider two sizes S: immediate 100, short 1,000 We present the entropy results in
Figure [6] Due to space limitations, we show Figure 19 the Wasserstein distances between expert selections of different tasks multiple training stages.
2.0
19-
Entrapy
3.7
1.5
3.4
Training Interation
taui3 50864 taues Saaed
32007 glooul
Line plot showing the evolution of expert selection entropy in a multitask learning setting with eight tasks. The dashed lines correspond to the entropy values of individual tasks, and the solid red line represents the aggregated entropy across all tasks. Higher entropy reflects uniform and uncertain expert utilization, while lower entropy reflects more concentrated specialized expert selection.
Figure 16:
MoE differentiation and expert specialization. As shown in Figure 16 tasks progress, the entropy of the expert distribution gradually decreases, indicating a reduction in distri- butional uncertainty. This suggests that expert selection becomes concentrated on few outcomes, reflecting lower uncertainty. Furthermore, in Figure 19 we visualize the expert selection distribu- tions of different tasks at multiple training stages, offering additional evidence of the progressive specialization in expert utilization.
Observation 2:
1.1 EXPERIMENT 3: ANALYZING GRADIENT CONFLICTS BETWEEN SHARED EXPERT AND NON-SHARED EXPERT
To further investigate the relationship between expert selection and gradient dynamics within MoE, we employ a MoE-based Transformer to analyze gradient conflicts across different MoE components. Our experimental results are shown below.
Observation 3: As shown in Figure 17 the shared expert exhibits significantly higher gradient conflicts compared to the task-specific experts. Among the non-shared experts, the level of conflicts does not show substantial differences. Overall, the shared expert accumulates even more conflicts than the entire MoE layer, indicating that most gradient conflicts within the MoE layer concentrated on the shared expert. The introduction of several task-specific experts effectively reduces the overall gradient conflicts of the MoE layer, which is consistent with our theoretical analysis in Theorem E.4
We further investigated the conflicts among different experts within MoF. Results is shown in Figure We found that shared experts bear most of the gradient conflicts, whereas individual (non- shared) experts experience almost no conflict. A plausible explanation is that in standard gating MoE, different samples are routed different experts, each expert primarily receives gradient updates from specific type of sample. This effectively performs an implicit task partitioning, leading
Note that at the very beginning of training, when the window cannot be fully populated, we the available data within the window, which may lead to larger fluctuations early steps.
42

---

Published as a conference paper at ICLR 2026
Gradient Conflict
10-2
25K
50K
75K
100K
125K
150K
175K
200K
Training Iteration
MoE-layer -hare_expee
expertl
expent2
expert3
expertS
expertó
expert?
Figure 17: Gradient conflicts across experts in MoE training. The plot shows the evolution of gradient conflicts for the shared expert and 8 non-shared experts during training interations. The logarithmic y-axis reveals that different experts experience varying levels of gradient conflicts, with the shared expert and individual experts (xpert0-cxpeer)) demonstrating distinct conflict patterns, indicating effective expert specialization in the multitask learning framework.
to generally consistent gradient directions within the same expert and minimal conflicts. In contrast, in shared-expert MoF, samples pass through the shared experts, whose parameters must adapt simultaneously to multiple tasks, diverse semanties, and even different domain distributions. This can result in highly inconsistent gradient directions, significantly increasing conflicts and making difficult for shared experts to balance diverse task requirements during optimization. This observation naturally raises a simple question: is the alleviation of gradient conflicts mainly due to the sparse selection by the gating network? We provide theoretical justification from the perspective of gating.
E.2 THEORETICAL ANALYSIS
We observe that the magnitude of the gradient conflict in an MoE model is primarily influenced by the routing coefficient, P:. sparse MoE, this coefficient acts as a binary variable determined by the router network. For simlicity. let us consider top-1 sparse MoE, where only the expert with the highest routing score is selected for a given input. An effective router would learn to assign tasks that are prone to high gradient conflict to different experts. In this case, the routing coefficient for any conflicting task pair would be zero, thereby nullifying their contribution to the overall conflict. Conversely, a poorly performing router might allocate conflicting tasks to the same expert, in which case the conflict remains unmitigated.
Theorem F. (Upper Bound of Full-layer MoE Gradient Conflict with Sparse/Soft Routing). Con- sider a MoE layer with M experts. Let each expert's 's gradient norm satisfy [19: |ggT""|| S R, m 1....,M, M,, and let routing weights for a task be AI....,AM (M)) ()) Assume that for any task pair (21,t), the gradient conflict single expert is bounded by G. Then the full-layer MoE gradient conflict satisfies the explicit upper bound
O. The full-layer gradient is
G. = concat(X{ge) (1) "....,'.90 X1g9i
M
l 22 l1ge,
(m)) |19t,
(m²)
conflict (Gt, G1,) ( G.
7VL
G
M
M
(Ain
(m)) 1|
(2a))
(m))
(m)
m=1
m=1
Proof. From the block-level conflict bound we have
M 1a Aa
1,9 (m)
1191 (m)
conflict(Gr, Gta) S.. G.
Gr,,
Grgy ||
43

---

Published as a conference paper at ICLR 2026
with
M (A (119,1 (m)) 113,
M (Amn
|G =
G
(ma) 2119,, 112
m==
Define vectors u, e E AR as
(mn) ||. mm 21 llot,
U/ = 1aall (ma) ||-
Then the numerator equals (u, v), and the denominator equals ||u|| [|]]. By the Cauchy-Schwarz inequality,
(u, v) ||u| ||ll] which implies that the fraction most 1. Substituting this bound back, we obtain
conflict(G G12) G.
Equality holds if and only if and are collinear (i.e., there exists O such that cu), and each block-level conflict achieves its upper bound simultaneously.
Remark (Effect of Routing Strategies Full-layer Gradient Conflict). The upper bound full- layer MoE gradient conflict depends strongly on the routing strategy. We summarize two representative cases:
a. Dense Routing (Soft Gating, Emen Am 1)
All experts contribute to the full-layer gradient.
The fraction in Theorem can be interpreted as the cosine similarity between the weighted vectors u X @ |19t,]] and v = x ④ 119tal:
ΣM X Ma 1l (m) I119) (ma)
cos(u, S 1.
|Gr
||1aa]]
the expert norms roughly equal (1 (r")) determined by (
[]) the conflict upper bound is largely
Uniform weights (Am 1/M do not automatically reduce conflict; alignment of weighted gradients matters more.
Overlap alignment of routing vectors increases conftict, while orthogonal or disjoint routing vectors reduce it.
b. Sparse Routing (Top-1 gating, one Am 1, others 0)
Non-overlapping case (different tasks select different experts):Each task's 's full-layer gradient contains only its chosen expert. Inner products between task gradients are near zero because tasks lie in different expert subspaces. Full-layer conflict is strictly less than G, possibly near 0.
Collapsed case (all tasks select the same expert): Full-layer gradient reduces to the chosen expert's ' gradient. Conflict equals the single-expert bound G, since all gradients reside in the same subspace.
Takeaways
The gradient conflict of MoE with multiple experts is lower than that within a single MI.P-hased expert.
The mitigation of gradient conflict in MoE is closely related to the routers across tasks-more more orthogonal routing coefficients tend yield fewer gradient confticts.
44

---

Published as a conference paper at ICLR 2026
Therefore, the key mitigating gradient conflict lies in the router's s ability to foster specialization among experts, ensuring that tasks with potentially conflicting objectives are handled by distinct, differentiated experts. This naturally raises critical question: Can Mixture-of-Expene architecture indeed achieve this, and if so, what are the underlying mechanisms that enable this capability? In the following, answer this question affirmatively. By building upon existing foundational theories of contture-ff-Epertteeee
et al. (2022), we present formal derivation that explains precisely when and how this is achieved.
E.2. ANALYSIS
Previews work et al. 2022) theoretically demonstrated that, in single-task supervised learning setting. when the input distribution exhibits distinguishable cluster structures, the sparse gating Mixture-of-Expeee (MoE) rouler can automatically learn the cluster center features and route samples to the most suitable expert, thereby significantly improving performance. We further hypothesize that, in multitask learning, the input spaces of different tasks naturally correspond distinct clusters and the signal for the cluster center can be task ID or the latent pattern on the state space. Under this assumption, the conclusions from the original work can be directly extended to the multitask setting.
featurfeatures
Next, we adapt Chen et al. (2022) theoretical analysis to the reinforcement learning context using tabular example. Specifically, we treat each task as an independent Markov Decision Process (MDP) and construct a multitask MDP set with a shared state space but task specific transitions. In this setting, we derive the gradient conflicts that arise the MoE layer.
We also discuss the case where no clustering signal exists between tasks and point out that the introduction of task embeddings can significantly enhance the clustering structure between tasks, thereby promoting the isolation of expert selection across different tasks in MoE.
E.2.2 PROBLEM SETTING: MULTITASK MDP WITH TASK-SPECIAS LATENT CLUSTERS
We consider a Markov decision process (MDP) defined by the tuple A,, (,,, 7), where S is the state space, A the action space, P(s' s, a) the transition kernel, R(s, a) the reward function, and E [0, 1) the discount factor. There are K tasks
Each task Ti corresponds to disjoint
state subspace S(t), so that
K
S
s
U S(,
st0 s())) - = for 1 forl f'.
1=1
Patch-based state representation. Each state 8 E (( (i.e., belonging to task and cluster k) represented as an unordered collection of P patches Rdd The patches are randomly permuted before being presented to the model. Every state contains the following patch types:
1. Action-signal patch: exactly one patch equals a 1/ where 1y E Rd encodes the optimal- action feature.
2. Cluster-center patch: exactly one patch equals 3 cg indicating the (task-specific) cluster identity; router (e.g., a Mixture-of-Experr model) expected to detect this signal to decide routing.
3. Feature-noise (confounder) patch: exactly one patch equals eγ og where k is typically different cluster index in the same task f, modeling intra-task confounding feature.
4. Random-noise patches: the remaining P - 3 patches are i.i.d. draws from an isotropic Gaussian, A((0, o²
Thus the encoder receives unordered set {a1, (.1,..., Φr o containing exactly action signal, one cluster-center signal (task-specific), one intra-task confounder, and Gaussian noise patches.
MoE Model with Expert Specialization
We now analyze how a Mixture-of offExxerts (MoE) model can leverage expert specialization to address the aforementioned issues.
45

---

Published as a conference paper at ICLR 2026
Experts. We consider M linear experts. The value function of expert m is defined Vn wm) = (Wms ((s)).
(11)
Router. The gating network assigns
each expert hm (8; mm) = (0m, ((8)).
(12)
MoE Output. The overall value function is given by
VM (s; W) =
Σ Tmm (m) (s; wm).
(13)
where the softmax gating weights are
exp( (lom (8))
mm (8)
(14)
exp(hg(8))
Loss Function. We minimize the mean squared Bellman error (MSBE) with stop- soop-gradient:
L(W, = Σ
(VMoE(s) stop_grad ((T (MMO))))))))
Σ
8(s),
(15)
#
SES
where ((s) denotes the temporal-different (TD) emror
E.2.3 LEARNING DYNAMICS ANALYSIS
We follow analysis path similar to prior work et al. (2022), dividing the learning process into two stages: an early expert exploration stage and a later router learning stage.
Initialization.
The expert weights wm are randomly initialized from a small zero-mean Gaussian distribution:
(0) (O, o.
(16)
while the router weights 0 are initialized as zero vectors:
((0) 0.
(17)
Expert Exploration Stage.
According to [Chen al. (022). Lemma E.3], the zero initialization the router implies that the beginning of training (t 0),
P(mat = m) M ( for all i [m], m E [M].
max mE A
where P(m., = m) is the probability that input sample is routed to expert m at iteration t and (5) is negligible value. The equation means all experts have approximately uniform gating weights at the expert exploration stage. That is, the routing selection can be approximated as:
nm (s)
(18)
This ensures that each expert has an equal opportunity to learn from the data.
During the early stage of training, due random initialization, each expert m has a weight vector wm that exhibits a slightly larger inner product with some value basis vector vk- We define the initial preference cluster of expert
k = arg max kE(K]
(w) (0) vn.)
(19)
Under gradient descent updates, the weight vector wm of expert m will predominantly grow along the direction of its preferred basis vector Uk+ [Chen et al. (2022), Lemma E.5] In other words, during training, an expert progressively specializes in modeligg specific cluster ç₁. Building upon this, we derive upper bound the gradient conflicts during the Expert Exploration Stage.
46

---

Published as a conference paper at ICLR 2026
Theorem E.2 (Upper Bound on Single-Expert and Full-Layer MoE Gradient Conflict with Uniform Sparse Routing). Consider a Mixture-of of Experts (MoE) layer with M experts and K independent tasks. Each task independently selects expert uniformly at random: P M
Then:
1. For any single expert m, the expected gradient conflict is upper bounded by
Elconftict
(m) S G quigle Osingle
1/1 -
(
1 1 1-
K-1
M
M
For the full-layer MoE with sparse router, the expected gradient conflict is upper bounded by
M!
KSM
MoE Elconstict
G Qayeers
Bayer
(M KYMMK
1,
K>
Proof. Consider an arbitrary expert m and the tasks. Each task independently selects expert m with probability 1/M. Let X denote the number of tasks that select expert m. Then
X Bin (K, 1/M).
single-expert gradient conflict occurs if and only if X 2, and by assumption, the maximum conflict for single expert is G. Therefore, the probability that expert experiences a conflict is
Pr(X 2 2) =1 11
1 M
1 M
1
1)=
1
K-1
M
By linearity of expectation, the expected gradient conflict for single expert is upper bounded by
[[conflict (m)) < G Øsingle.
The full-layer gradient G, is constructed by concatenating all expert gradients. A full-layer conflict occurs at least one expert has a conflict (i.e., at least two tasks select that expert).
For K s .,, the probability that all K tasks select distinct experts (no collision) is given combina- torially by
M!
(M K)! MK*
so the probability of least one collision (full-layer conflict) is
M!
Qlayer
(M - KY!
For K > M, by the pigeonhole principle, at least one expert must be selected by two or more tasks, so
Qlayer 1.
Each conflict in an expert can contribute at most to the full-layer gradient conflict. By worst-case analysis and linearity of expectation, the expected full-layer MoE gradient conflict is therefore upper bounded by
E E/conffict MoE
< G Qayer
The single-expert bound G Gangle gives local (per-expert) expected conflict. The full-layer bound G Quyer accounts for collisions across all experts in the concatenated MoE layer. Together, they describe the expected gradient conflict behavior MoE layer with tasks and M experts under uniform sparse routing.
47

---

Published as a conference paper at ICLR 2026
Takeaways: Gradient Conflict in Expert Exploration
In the expert exploration stage, the full MoE layer shows a lower upper bound on gradient conflict, which further decreases as the number of experts M grows relative to lasks K.
Router Learning Stage
During the router-learning stage, the sparse MoE layer exhibits two decisive properties that justify its use in multitask setting.
First, the gating network rapidly identifies the latent task structure: after only
T 17 M -
iterations, its weight vectors On become strongly aligned with the cluster-center signal C₁ of every task k while simultaneously suppressing spurious correlations with label signals and noise. [Chen noise.[Chen et al. -lmma E.14] Consequently, any input drawn from task & routed to the corresponding subset of experts Mg with probability 1-of o( 2), even though the model is never given explicit task labels. [Chen et al. (2022)LLemma E.18]
Second, each expert in MA remains tightly specialised to its assigned task, but only chance-level E. E.12,Lemma 5.2].
performance on all other tasks. [Chen et al.
Taken together, these two properties guarantee automatic task separation and expert specialisation without external supervision, making the MoE layer an effective backbone for multitask problems in which tasks form separable clusters in the input space.
Theorem E.3 (Expected Gradient Conflict on Task-specific Expert Sets). Consider a Mixture-of- Experts (MoE) model with K tasks and M experts. Let tasks and j have expert sets S₁ and S with sizes [S = a, |Syl = b, and intersection U = |S: n Szl. Assume that each task selects an expert from its own expert set with probability - 0(1/d) uniformly, and from the complement set with probability O(1/d) uniformly. Let G denote the muximum gradient conflict if two tasks select the same expert. Then the expected gradient conflict between tasks and j satisfies
Elconslict, 5 (p()) + p(2) + +p(s)).
(20)
where
+
p()) U. ((1/1)
+ 0(1/d) ) ( (1-0(1/0)
b + 00//0)
M 0 p(2) (a U. b bb)- (1-00/10 + M - M-b'
a
M -6-1
(21)
0(1/d)
O(1/d)
(22)
a
p(3) (M (a bb) 0(1/d) 0(1/d)
(23)
M - M b In particular, the dominant contribution comes from the shared experts (first class), leading to
U Elconflict, 5, G ab + OG/dd)
(24)
Moreover, for thll full-layer MoE gradient by concatenating all expert parameters, the expected layer-level gradient conffict satisfies the same upper bound:
Jaeonflict)laerr S G. (p() + p(2) + p(3) G. U ab O(G/d)
(25)
Proof. Consider two tasks and with expert sets S, and Sj. Partition the set of all experts into three classes: (1) shared experts S₁ n nSJ, (2) task-specific experts S, \ Sj and Sj \ S,, and (3) unrelated experts outside S; U Sy. For each class, compute the probability that both tasks select the same expert.
For shared expert S,SS,, task i selects it either from its own set with probability (1-O(1/d)// or from outside with probability O(1 ((1/d)/( (M - a). Similarly for task j. Therefore, the probability that both tasks select is
1 O(1/d)
0(1/d)
L ((1//) (
O(1d))
10 =
a
M
b
M - b
48

---

Published as a conference paper at ICLR 2026
and there are U such experts, giving total contribution p()) = U p())
For task-specific experts E (S, \ S2) (S, \ St), one task can select it from its own set while the other selects from outside. This yields
O(1/d) 0(1/d) O(1/d)
M
M
with total of (a - U + b U) experts, giving P(2) (a U + U) p(
For unrelated experts & S; U.. both tasks must select from outside, giving
( O(1/d) O(1/d)
M M b
and there are M (a + b- U) such experts, yielding P(3) = (M - ba p(3)
Summing over all classes and multiplying by G, we obtain the expected single-expert gradient conflict:
E|conflictr,]l - G (P() + p(2) + p(3)).
Since the layer-level gradient is formed by concatenating all expert gradients, the layer-level conflict the sum over all expert contributions, hence
E| E[conflict)uuer S G (P(() + p(2) p(3)).
Finally, the dominant term is from the shared experts, giving the simplified approximation
E|conflict)nyer Re G U + O(G/d),
ab
where O(G/d) absorbs all higher-order small contributions from selecting non- con-preferred experts.
L
Takeaways: Gradient Conflict in Router Learning Stage
Gradient conflicts mainly arise from shared experts between tasks, while task-specific experts contribute little.
The full-layer MoE reduces overall gradient conflict compared to single experts, and conflicts decrease as tasks are routed to more disjoint expert subsets.
Corollary E.4 (Expected Full-layer MoE Gradient Conflict for K Tasks), Consider a MoE layer with M experts and K tasks. Each task has its own expert set S, with size [S,] 04- Each expert's s gradient norm satisfies l[g (m)) S R, and if two tasks select the same expert, the maximum gradient conflict is G. Each task chooses expert uccording to the following rule: with probability 1 1-0(1/o it selects uniformly from its own set, and with probability O(1/d) it selects uniformly from the remaining experts.
For any subset of tasks T C (1,2, ..,2...,,,,,. K}. let the number of shared experts
UT n S.
IET
Then the expected full-layer gradient conflict is upper bounded by
1- 1-1(1/
O(1/d)
Elconffict layer S
Σ
UT
II
(26)
a2
M a.
TEAKTTTI22
SET
and its leading term can be approximated
UT +OG//)
Elconftict)uwee RS G
Σ
(27)
ner 4
TCK,
49

---

Published as a conference paper at ICLR 2026
Proof. For each task subset T S 21, } with [7] 2, consider any expert e NIeT St. The probability that all tasks 7 simultaneously select this expert is given by the product of individual selection probabilities. For each task t T, the probability of choosing
O(1dd) 0(1/d)
M a
where the first term accounts for selecting from the task's own expert set and the second term accounts for selecting from outside the Since the tasks are independent, the joint probability that all tasks in T select the same shared expert is
O(1/d) II
+ 0(1/))
a₁
M M-a./
BET
Multiplying by the number of shared experts UT gives the total expected conflict contribution from this subset:
1 [1-0(1/))
0(1/d)
II
M -
FET
Summing over all task subsets with size at least and multiplying by G, yields the full-layer expected gradient conflict:
E/conflict
G
Σ PT-
TEIKT/IT122
In the leading order, neglect the O(1/d) small-probability contributions from selecting experts outside each task's 's own set, which gives
E(conflict R G
E
UT Ier + OGG/d),
TCKKLTT122
where the dominant contribution comes from shared experts that simultaneously selected by multiple tasks, and the higher-order terms are absorbed (G//d).
E.2.4 TASK EMBEDDING FOR IMPROVED TASK SEPARATION MULTITASK LEARNING
In multitask learning, tasks often share overlapping input spaces and underlying features (e.g.. sentiment classification tasks using the same text encoder), making it difficult for the routing network to distinguish tasks. This overlap leads to routing confusion, where data from one task may be misrouted to experts specialized in other tasks, reducing performance. To address this, task embedding introduces learnable vector e, for each task, which can be combined with input features. Since orthogonal across tasks (or constrained via a regularization term), it helps the routing network more clearly distinguish between tasks, even when inputs are similar. Task embedding also acts as regularization term, reducing expert load imbalance and improving the stability and performance of multitask MoE training.
Our theoretical analysis suggests that introducing distinet task embeddings provides crucial clustering signals, enabling the router to more effectively differentiate between tasks. This, in turn, mitigates gradient conflicts and enhances overall performance. However, our empirical results, as presented in Figure show that the inclusion of naive task embeddings did not yield significant improvement. We hypothesize that this due to the absence of explicit constraints to maintain separation between the embeddings during optimization, which could lead to their collapse in the later stages of training. For future work, we plan to investigate more effective methods for explicitly injecting task-specific information. The goal is to empower the router autonomously leverage the unique priors of each task, thereby further advancing multitask learning performance.
F DETAILED RELATED WORK
MCTS with Learned World Models. Planning within a learned latent space, popularized by MuZero (Schrittwieser al. 2019), has become dominant paradigm in reinforcement learning,
50

---

Published as a conference paper at ICLR 2026
non arce get aledtt
dict Cwe.Caaprrpeeeeeee
Mi. teT Deguge seNOl
308
C27
240
622
CM
141
(1) Interaiion
(b) inceation
(3 imisstire-6ik
Iou aedd acjdiit
Mic laaccppc bebdio
A22
inmertionn-033
(1)
352
421
14
662
010
100
Intenttttr100 18k
198c
Figure Nine heatmaps illustrating the evolution expert selection distributions former across different tasks and training interations. Each row corresponds to a specific task, where the color intensity reflects the frequency of expert selections within a sliding window of S = 1000. This visualization reveals how task- ass-spccifii expert utilization patterns emerge and evolve during training.
MoE-based Trans-
51

---

Published as a conference paper at ICLR 2026
Aassecrai dinanaa Mar staii bes secral
Tee Miseerai Ditt
604
412
Iechheermenn
(U) Routaar 1838.
Figure 19: Nine heatmaps illustrating the Wasserstein distances hetween expert selection distributions of
different tasks successive training interations with Semmestiote 100. Each heatmap corresponds to specific training interation, revealing how inter-task expert selection similarity changes over Diagonal elements are removed to exclude self-comparisons.
52

---

Published as a conference paper at ICLR 2026
t-SNE acroes Environments
60
20
1.5M Dimersion
Resking
Alln
Folddls trer
-20
-40
-0 -00
20
1-SNE Dimension
1
Figure 20: t-SNE visualization of the learned latent representations. Each point corresponds to 2D projection of a 7668-dimensional latent emhedding generated by the world model. Fmheddings are color-caded by environment ID to distinguish between distinct Atari games. The projection illustrates structured latent space where intra-game states exhibit high cohesion, while distinct environments maintain clear separation. This confirms that the model effectively captures fask-specific semantics while preserving discriminative global features. The analysis aggregates 8,000 data points derived from 400 observation sequenees, each spanning 20 time steps.
building upon the success of AlphaZero (Silver et al. 2016; 2017). This approach enables a model to predict its own dynamics, rewards, and policies, allowing for powerful lookahead search within a compact, learned representation (Hubert et al., 2021: Antonoglou et al. 2021 Danihelka et al.
2022] Xuan et al., 2024). Recent advancements have incorporated Transformers the backbone for these world models (Micheli et al. 2022, Robine et al., [2023] Pu et al. 2024 Zhou et al. 2024). dependencies. However, the monolithic nature of these architectures, while powerful for single tasks, becomes critical liability in heterogeneous multitask settings. They are prone to representational interference and plasticity collapse, where a single shared model struggles to accommodate the diverse and often conflicting properties of multiple tasks. Our work confronts this fundamental architectural bottleneck by systematically investigating and proposing more robust design.
significantly enhancing representational capacity and the ability to model
Multi-Task Reinforcement Learning (MTRL). MTRL seeks to improve data efficiency and gen- cralization by sharing knowledge across a distribution of tasks (Vithayathil Varghese & Mahmoud
2020 D' Eramo et al. 2024). A common strategy is to use a shared backbone with task -asc-fpec heads to handle diverse action or observation spaces (Kumar al. 2022 [Hansen al., 2023 Lee
ct al. 2022 Gallouédec et al. 2024). To mitigate inter-task interference within the shared backbone, various approaches have been proposed. Architectural methods aim to disentangle knowledge through context-based conditioning (Rakelly et al. 2019 Sodhani et al. 202 learnable modulation modules that adapt network activations (Schmied et al. 2023). or explicit modularization with task- awsre routing and parameter composition (Sun et al., 2002, He et al. 2023). In parallel, optimization- based methods focus on managing gradient conflicts at the training level by projecting gradients to avoid negative interference or re-weighting task losses dynamically (Fernando et al. 2023) Lin et al. 2024] Ma et al. 2023). However, these strategies have been predominantly studied in model-free RL or supervised settings, largely outside the domain of latent space planning, where the additional challenge of disentangling dynamics prediction is paramount. Few works have investigated the unique failure modes of multitask learning in this context. We address this gap by analyzing plasticity-relate metrics to identify the root cause of performance collapse- an architectural bottleneck in the world model-and propose a novel design to resolve it.
53

---

Published as a conference paper at ICLR 2026
Sparse and Parameter-Efficien Architectures. To overcome the limitations of dense, monolithic models, we turn to sparse and parameter-effiiient architectures. Sparse models like Mixture-of- Experts (MoE) (Dai et al. 2024) increase model capacity at near-constant computational cost through conditional computation. This provides a natural architectural prior for multitask special- ization, different tasks can be routed to different "expert" sub-networks (Obando-Ceron al.] 2024). Concurrently, parameter-efficient fine-tuning (PEFT) methods, notably Low-Rank Adaptation (LoRA) (Hu et al., 2022). enable lightweight, task-specific adjustments. By freezing the bulk of the model and training only small subset of injected parameters, LoRA can efficiently adapt model to new tasks or data distributions (Wang et al. 2023] Yang et al., 2025 Zhang al. 2025). However, we identify a paradigm shift required to apply these techniques to online multitask learning. Classic LoRA techniques are predominantly designed for the static fine-tuning
offline or nearly static datasets (Agiza et al.) 2024 Huang et al., (023). In contrast, our approach operates within non-stationary online data stream. Unlike standard methods that rely on predefined task boundaries, our DPS introduces in-training decision mechanism to make real-time judgments on when and how to expand model capacity. By allocating independent parameter spaces for distinct distribution shifts, DPS effectively balances plasticity and stability, surpassing the rigidity of standard fine-tuning. Our work bridges this gap by unifying MoE and this DPS mechanism within transformer-baxed world model, creating a system that both architecturally specialized and dynamically adaptable for large-scale MTRL.
G LIMITATIONS AND FUTURE WORK
While our main results demonstrate ScaleZero's robustness across 48 tasks and the efficiency of DPS on DMControl, we identify specific limitations in our current methodology. These limitations directly chart the course for our future research roadmap toward more capable generalist agents.
Generalizing DPS via Unified Training across Heterogeneous Domains. A primary limitation of this study is that the DPS strategy has been empirically validated only on the continuous DMC-18 suite. We acknowledge that claims of domain-agnosticism remain speculative without benchmarks on discrete video games (Atari) or language domains (Jericho). However, we posit that DPS is theoretically positioned to generalize beyond DMC due to two structural factors:
Structural Decoupling: DPS operates exclusively the world model backbone (the latent transition function), mathematically decoupling it from specific I/O modalities. Since differences in observation spaces (e.g., pixels vs. proprioception) and action spaces are encapsulated within the Encoder and Task Heads, DPS is designed to function effectively regardless of the environmental interface.
Hypothesis of Higher Utility via Heterogeneity: We hypothesize that the on-demand allocation mechanism of DPS will yield higher marginal utility in highly heterogeneous domains like Atari (e.g., the reactive simplicity of Pong vs. the planning depth of Seaquest) compared to DMC. Static architectures often struggle with such internal variance, whereas DPS can adaptively match capacity to the specific complexity of each task.
To validate these hypotheses, our immediate future work will transition from single- domain testing to a Unified Training regime involving Atari-26, DMC-18, and Jericho-4 simultaneously. This will allow to rigorously analyze whether the adaptive mechanism of DPS mitigate negative transfer across vastly different state-action spaces, thereby enhancing generalization in a truly multi-modal context.
Deepening MoF-LLoRA Synergy for Architectural Adaptation. Our current framework employs modular design where MoE handles architectural specialization and LoRA manages dynamic capacity scaling. However, a limitation of the current implementation that these components operate somewhat orthogonally. We believe that a deeper synergy is required to fully exploit the architecture's 's potential. Future work will investigate advanced coordination mechanisms, such as using LoRA to adapt the MoE gating network itself to stabilize routing during rapid distribution shifts, or utilizing MoE routing decisions to dynamically activate specific LoRA adapters. This aims to create a tightly integrated system capable of fine-grained, context-aware architectural adaptation.
54

---

Published as a conference paper at ICLR 2026
Addressing Sample Efficiency via Hybrid Offline-Online Learning. Currently, framework operates in a purely online learning setting. While effective, this approach inherently limited by the need to collect fresh data for every task, creating bottleneck for sample efficiency and "cold start" performance. A powerful extension to address this limitation is to integrate large-scale offline pre- training. By initializing the ScaleZero model with weights from foundation model pre-trained on vast offline datasets, we aim to leverage broad prior knowledge. This hybrid approach is expected to provide superior initialization, faster adaptation to new tasks, and higher peak performance, bridging the gap between specialized reinforcement learning and generalist foundation models.
H THE USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, the large language models Gemini 2.5 Pro and Gemini Pro were used to assist with language refinement. The primary goal was to enhance the clarity, precision, and readability of the text. The authors have reviewed and approved the final content.
55


Reviews:
[{"review_id": "1IUKwxZ86e", "reviewer": "Reviewer_QPFt", "summary": "The authors propose ScaleZero, a unified world model for heterogeneous multi-task RL that addresses plasticity collapse and gradient interference observed in latent space planners (e.g. UniZero).\nThe proposed approach particularly combines a Mixture-of-Experts backbone into previous work UniZero, and proposes a novel Dynamic Parameter Scaling technique using LoRA's paradigm to adapt the number of parameters with respect to the learning progress. \nExperiments on several tasks included in the benchmarks of Atari, DMControl, and Jericho demonstrate improved sample efficiency and at times overall performance versus training single-task experts via world models.", "questions": "- How does ScaleZero compare to a naive multi-task RL baseline that does not learn world models nor does planning?\n- From my perspective, I would expect ScaleZero to yield improvements in terms of median performance if anything, over mean performance. This is the case in Table I.b for DMC tasks, but could you provide more insights into why the opposite happens for Table I.a? This opposite trend for Atari games could mean that ScaleZero actually fails in solving plasticity collapse, and if anything introduces even more skewness in the performance distribution over different tasks.\n- Could you please provide an explicit comparison between ScaleZero and UniZero trained on single tasks by means of compute budget (GPU hours), wall-clock training time, inference time, and size of network parameters? It's currently unclear to me whether performance is also being traded off for efficiency or additional resource usage.\n- Despite the authors claiming MoE as the most impactful modifications, why does ScaleZero NOT include MoE layers for the Jericho benchmark experiments (Line 1179)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper investigates and directly tackles two relevant open problems in MTRL (task interference and plasticity collapse). This is done via thorough experimenn over multiple metrics: gradient conflict, mean and median performance, and more.\n- The authors carefully study the limitations of previous SOTA world-model approach UniZero in the realm of multi-task settings, providing useful insights for further research in the direction of bridging methods towards MTRL usage.\n- The paper is clearly written and relatively easy to follow despite the large amount of content included in the Appendix. Important claims and motivations are clearly highlighted.\n- The manuscript also contains relevant theoretical analysis on the underlying aspects of how sparse MoE affect gradient conflict.", "weaknesses": "- Absence of multi-task baselines: The authors only compare their approach as a modification to single-task experts trained individually on multiple tasks using UniZero. While the overall investigation is sound, it is currently unclear how the performance compares to state-of-the-art methods in MTRL. For example, [1] is a recent ICLR contribution that particular targets task interference via explicit task conditioning that ensures orthogonal experts. The authors' investigation in line 215 lacks any comparison with such baselines, limiting the impact and relevance of the overall analysis. As such, the current work mostly presents itself as an adaptation to an existing world-model approach (UniZero) to the MTRL paradigm; while this is a contribution in itself, insufficient evidence is provided on how these modifications compare to existing MTRL methods.\n\n\n[1] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. \"Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts.\" The Twelfth International Conference on Learning Representations, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iiiBQS8eMf", "reviewer": "Reviewer_o84j", "summary": "The paper addresses the challenge of heterogeneous multi-task online decision-making by proposing ScaleZero, a unified world model that extends UniZero with mechanisms for mitigating gradient conflicts and preserving model plasticity. ScaleZero integrates a sparse Mixture-of-Experts (MoE) backbone to handle conflicting gradients and a Dynamic Parameter Scaling (DPS) mechanism that progressively expands model capacity through LoRA adapters based on learning progress. The approach allows a single model to achieve competitive or near-expert performance (single task agent) across multiple diverse tasks (Atari, DMC, Jericho) while using only 71.5% of the environment interactions compared to specialized models.", "questions": "- Line 122: What exact failure pattern of “plasticity collapse” is being referred to, loss of gradient signal, representational collapse, or over-regularization?\n- Line 140: How does the sparse MoE approach differ from multi-head multitask methods like MHMTSAC (Yu et al., 2019)?\n- Line 232 / Figure 2: The figure caption refers to a \"systematic exploration\" but provides little quantitative evidence; can you provide clearer metrics or summaries for each axis of exploration?\n- Line 257: If the ViT encoder’s scalability claim is central, why is this figure placed in the appendix?\n- Line 259: Given the minimal impact of encoder choice, why retain it in the main text instead of moving it to supplementary materials?\n- Line 264: What empirical evidence supports the final preference for LayerNorm over SimNorm?\n- Line 279: Why include the “Multitask Gradient Correction” subsection if it was not part of the final method?\n- Line 323 (Section 3.3): How many LoRA modules can be added in total, is there a hard limit or a soft budget threshold?\n- Figure 4: Is the 28.5% reduction in environment steps consistent across different benchmarks, or specific to DMC18?\n- Reproducibility: What are the approximate computational requirements (GPU hours, memory footprint) of training ScaleZero compared to UniZero?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addresses a relevant and challenging problem: Tackles multi-task world model learning; an important step toward generalist RL agents capable of handling heterogeneous tasks with diverse observation and action spaces.\n- Principled design and justification: The paper builds on solid theoretical foundations around gradient conflict and plasticity collapse, grounding architectural and procedural design decisions with both empirical and analytical justification.\n- Well-motivated extensions to UniZero: The combination of sparse Mixture-of-Experts (MoE) layers and LoRA-based Dynamic Parameter Scaling (DPS) is technically sound and thoughtfully motivated.\n- Novel and interpretable component: Introduction of Dynamic Parameter Scaling (DPS) as an adaptive LoRA integration strategy provides a clear and interpretable mechanism for resource-efficient multitask learning.\n- Comprehensive experimentation: Evaluation across three benchmarks (26 Atari games, 18 DMC control tasks, and 4 Jericho text-based environments) convincingly demonstrates that one unified model can rival single-task specialists.\n- Strong empirical efficiency: Demonstrates competitive results while requiring only 71.5% of the environment interactions of comparable single-task agents, highlighting improved sample efficiency.\n- Thorough analysis: Empirical and theoretical analysis of MoE gradient conflict (Theorem 4.1) provides valuable insight into why conditional computation is effective for multitask planning.", "weaknesses": "- Limited performance gain in some benchmarks: In Table 1a, the reported human-normalized mean improvement on Atari is minimal (0.39 vs. 0.38), raising questions about statistical significance of the claimed parity or superiority.\n- Lack of code availability: Reproducibility is currently unverifiable as no code release accompanies the submission, despite promising to release post-review.\n- Exploratory overreach: Section 3.2 includes several explorative design discussions (e.g., LayerNorm vs. SimNorm, ResNet vs. ViT) that provide little scientific contribution to the main narrative and could be shortened.\n- Figure legibility issues: Figure 2’s small text and visual clutter reduce readability and comprehension.\n- Marginal novelty of core techniques: The combination of MoE and LoRA, while effective, builds primarily on well-established concepts. The primary novelty lies in their integration into a unified online multitask planner rather than in the individual components.\n- Theoretical depth could be improved: The provided theorem on gradient conflict upper bounds is only informally stated and lacks empirical connection to practical convergence or sample efficiency.\n- Benchmark contextualization: The paper could better explain the specific setup for comparisons—e.g., whether “single-task UniZero” agents were tuned separately and if “competitive” results include confidence intervals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lGqPMFt65R", "reviewer": "Reviewer_7ZaB", "summary": "This work investigates the challenges of training a unified world model for multi-task planning. The obstacles centered around representational bottlenecks and plasticity collapse of the shared model. The authors attribute these issues to gradient conflicts as well as the diversity of the shared features. To address these issues, this work investigates and studies several modifications to the UniZero [1] model, focusing on three core axes: input representation, model architecture, and optimization strategy. Consequently, the method addresses plasticity collapse using two approaches: an internal, architectural fix called *ScaleZero* to reduce interference during a single learning step, and an external, procedural strategy called *Dynamic Parameter Scaling (DPS)* to better manage resource allocation throughout the overall learning process. ScaleZero matches the performance of specialized agents on various benchmarks (Atari, DMC, Jericho) using only a single model and standard online reinforcement learning. Furthermore, when the DPS strategy is added, it maintains competitive performance while significantly reducing environmental interactions by nearly 30% (using only 71.5% of the original). This suggests that ScaleZero is a highly promising method for effective multi-task planning.\n\n[1] Yuan Pu, Yazhe Niu, Zhenjie Yang, Jiyuan Ren, Hongsheng Li, and Yu Liu. Unizero: Generalized and efficient planning with scalable latent world models. arXiv preprint arXiv:2406.10667, 2024.", "questions": "I have some questions:\n1. What is the specific advantage of using a world model over model-free reinforcement learning (RL) in the multi-task context?\n2. Could you elaborate on the relationship between gradient conflicts and the loss of plasticity? This could be explained either qualitatively or by comparing against relevant methods from the literature.\n3. In the Dynamic Parameter Scaling (DPS) strategy, what metric is used to compute performance for task curation? Specifically, is it the training return or a dedicated online evaluation metric?\n4. In DPS, are tasks re-included if their performance drops? If not, how does the method mitigate the issue of forgetting excluded tasks? Is this addressed by the expansion and freezing mechanism of the LoRAs?\n5. Could you provide a plot tracking which condition (Progress-based Trigger or Budget-based Trigger) was activated to initiate the switch to the next expansion stage?\n6. How is the number of expansion stages chosen? Is it directly correlated with the number of tasks? Additionally, how does changing the number of expansion stages affect overall performance?\n7. What is the effect of varying the number of experts on overall performance? An ablation study investigating this factor would be appreciated.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and structured.\n- Although the paper is dense, the authors present the content effectively.\n- This work combines existing techniques to provide a proper solution to challenges in multi-task planning. That constitutes novelty in my opinion.\n- I appreciate the example in Figure 1, which highlights the relationship between task performance and plasticity collapse using the Dormant Ratio and Latent Norm metrics.\n- A strength of this work is the diverse set of benchmarks addressed.", "weaknesses": "- Although the paper is well-written and the solution is well-motivated, the justification for using a world model for multi-task planning, as opposed to model-free multi-task RL, is not entirely clear. I advise the authors to highlight the importance of world models in comparison to model-free RL, either in the text or through benchmark evaluations.\n- The claim that model plasticity collapse is related to gradient conflicts is not well presented or supported. I recommend adding a gradient conflicts metric (such as the cosine between task gradients), similar to the ones in the appendix, for the example in Figure 1. This would allow us to observe the relationship between the loss of plasticity and gradient conflicts.\n- As previously stated, there is no comparison to model-free multi-task RL methods, and the paper lacks baselines from that category. I recommend benchmarking some advanced multi-task methods [1,2,3] with similar model capacity on benchmarks like DMC.\n- Some key ablations are missing. One important ablation would be varying the number of experts, and another would be varying the number of task curation stages. I believe these ablations were not studied in this work.\n- The limitations of the proposed method are not clearly stated in the paper. In my opinion, one limitation could be that there are many moving components to achieve that performance.\n\n[1] Sun, Lingfeng, et al. \"Paco: Parameter-compositional multi-task reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 21495-21507.\n\n[2] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. \"Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts.\" The Twelfth International Conference on Learning Representations.\n\n[3] Cho, Myungsik, et al. \"Hard tasks first: Multi-task reinforcement learning through task scheduling.\" Forty-first International Conference on Machine Learning. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmUhHBBZ9G", "reviewer": "Reviewer_K2YD", "summary": "This submission proposes an architectural (Soft-MoE) and a fine-tuning method (LoRA) applied to the task of heterogeneous multi-task RL & planning. The authors suggest that their are limitations of the current Resnet architecture through several axes such as (a) task conditioning, (b) encoder architecture, (c) normalization method, (d) backbone design, and (e) optimization strategy. The authors then introduce their Soft-MoE for online multi-task RL (MTRL), with adaption to new tasks being performed through low rank adaption.", "questions": "1. How many parameters are in the UniZero model vs the ScaleZero model?\n2. What other measures of task progress could be used instead of reward/success thresholds?\n3. Are there other parameter efficient fine-tuning methods that could be experimented with for fine-tuning a pre-trained model?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "Approaches an important problem & limitation within MTRL where typical MTRL problems have homogeneous tasks, this paper attempts to expand the task distribution further by introducing heterogeneous tasks into the task distribution. While this problem has been attempted before, it is still an unsolved problem.", "weaknesses": "1. The authors of this work should review accepted methods of reporting results in RL such as [1]. Only reporting the mean/median does not give the full performance profile and clouds whether the reported results are significant or not. I cannot judge this work effectively without this information. \n\n2. The presentation of this paper could use some work. Right now it jumps around a bit\n\n(a) Method section: I found this section extremely confusing and hard to read. There were multiple missing references to figures that show experimental results (i.e. Latent Normalization paragraph -- what figure is being referenced?), and there were also multiple references that I thought were out of place here (i.e. Rakelly et al 2019 & Sodhani 2021 -- they are both solving fundamentally different problems and don't explore the difficulties of state identification). The method section also starts introducing components and experiments immediately which clouds the clarity of what the authors are intending to add to the UniZero method. \n\n(b) Background Section: The authors begin the background section with a discussion of Monte Carlo Tree Search and then a discussion of gradient conflicts, but then later on line 258 mention \"online multi-task RL.\" Thus it's a bit confusing exactly what algorithm & method are being used here. I assume that this is just a terminology issue but this needs to be explicitly clear. The background section should precisely outline the algorithm (online RL? MTRL?) that is being used to optimize the agent. Terminology from this section should then be consistent through the rest of the work. \n\n(c) Related Work: This section is missing a ton of relevant MTRL works. Line 464 explicitly is focused on model-based RL methods, while there are model free approaches, even one that uses a MoE method [4, 5, 6]. There is also recent work showing that MTRL specific architectures (like Mixture of Experts architectures) are irrelevant and that making a model bigger in MTRL actually drives performance [7]. There's also working showing that sparsity can be a driver of scaling as well [8], alongside diverse data [7]. The authors should include a discussion about these works and how they differ from their own paradigm. \n\n(d) Hyperparameters: I applaud the authors for including hyperparameters, except they should also include the common parameters used in the UniZero model, even if they are the same for the ScaleZero. Additionally, could the authors report the number of parameters used in each of the models? \n\n3. Task Curation: this strategy requires apriori knowledge of the task reward/success thresholds. This doesn't transfer well to new tasks where this prior knowledge is not readily available. Recent works have attempted to characterize the training dynamics as a method for early stopping [2] \n\n4. Missing Baselines\n\n(a) A large portion of this work is about gradient conflicts -- there are no baselines for various optimization methods that attempt to solve this issue (i.e PCGrad, CAGrad, MGDA). \n\n(b) Alternatively, previous work has found that effective normalization can perform on par with these optimization methods [3] but that strategy is not experimented with either. \n\n5. Citations\n\nLine 195: Dohare et al do not propose feature effective rank, they might use it but that's not the work that should get cited. \n\nLine 185: Dormant Neurons needs to be cited when it is defined\n\nLine 103: \"... demonstrated remarkable success in domains with known rules\" -- need to provide some evidence of this through citations\n\n\nIn general, it feels like this paper is trying to do a lot of things which clouds the overall readability of the work. Doing an empirical study of the effects of the changes in the 5 highlighted axes could be a useful submission, and doing an empirical study of the effects of adding various parameter efficient methods (such as LoRA) would also be valuable. But the way that these things are presented in the current version of the manuscript make it very difficult to understand the value of each of the methods individually. \n\n1. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare \"Deep RL at the Edge of the Statistical Precipice\", NeurIPs 2021. \n\n2. Jiashun Liu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning, ICML 2025.\n\n3. Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, M. Pawan Kumar, In Defense of the Unitary Scalarization for Deep Multi-Task Learning. NeurIPS 2022\n\n4. Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In NeurIPS, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/32cfdce9631d8c7906e8e9d6e68b514b-Abstract.html\n\n5. Ahmed Hendawy, Jan Peters, and Carlo D’Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=aZH1dM3GOX.\n\n6. Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-compositional multi-task reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=LYXTPNWJLr.\n\n7. Reginald McLean, Evangelos Chatzaroulas, J K Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. \"Multi-Task Reinforcement Learning Enables Parameter Scaling.\" Reinforcement Learning Journal, vol. 6, 2025, pp. 1075–1093. \n\n8. Guozheng Ma, Lu Li, Zilin Wang, Li Shen, Pierre-Luc Bacon, Dacheng Tao, Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning. ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)