Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policies
Authors: Ce Hao
Xuanran Zhai
Yaohua Liu
Harold Soh
Venue: ICLR 2026 Poster (2026)
Keywords: Imitation Learning
Mixture of Experts
Primary area: applications to robotics, autonomy, planning

Abstract: Diffusion-based policies have recently shown strong results in robot manipulation, but their extension to multi-task scenarios is hindered by the high cost of scaling model size and demonstrations. We introduce Skill Mixture-of-Experts Policy (SMP), a diffusion-based mixture-of-experts policy that learns a compact orthogonal skill basis and uses sticky routing to compose actions from a small, task-relevant subset of experts at each step. A variational training objective supports this design, and adaptive expert activation at inference yields fast sampling without oversized backbones. We validate SMP in simulation and on a real dual-arm platform with multi-task learning and transfer learning tasks, where SMP achieves higher success rates and markedly lower inference cost than large diffusion baselines. These results indicate a practical path toward scalable, transferable multi-task manipulation: learn reusable skills once, activate only what is needed, and adapt quickly when tasks change.

Published as a conference paper at ICLR 2026
ABSTRACTING ROBOT MANIPULATION SKILLS VIA MIXTURE-OF-EXPEET DIFFUSION POLICIES
Ce Hao*t Xuanran Zhai" Yaohua Liu", Harold Soh
1 School of Computing, National University of Singapore: 2 Smart Systems Institute, NUS;
3 Guangdong Institute of Intelligence Science and Technology: Beijing Zhongguaneun Academy Equal contribution. Emails: cehacêu. nus edu and haroldênus edu. sg
ABSTRACT
Diffusion-based policies have recently shown strong results in robot manipula- tion, but their extension to multi-task scenarios is hindered by the high of scaling model size and demonstrations. We introduce Skill Mixture-of-Expert Policy (SMP). diffusion- orthogonal skill basis and uses sticky routing to compose actions from small, task-relevant subset of experts each step. A variational training objective sup- ports this design, and adaptive expert activation at inference yields fast sampling without oversized backbones. We validate SMP in simulation and on a real dual- platform with multi-task learning and transfer learning tasks, where SMP achieves higher success rates and markedly lower inference cost than large diffu- sion baselines. These results indicate practical path toward scalable, transferable multi-task manipulation: learn reusable skills once, activate only what is needed, and adapt quickly when tasks change.
mixture-of-experts policy that learns compact
Left Arm
Pick
Ad just
Reach Cabinet
Relcose
Samontic
Trans
Trans. d
Action
Right Arm
Rench Cabinet
Grosp
Pull
EE Action
Gate values
Figure 1: Overview SMP. Top: Bimanual rollout of "put card in drawer" with key steps (1)-(5). Middle: Skill decomposition by arm and phase: the state-adaptive orthonormal skill basis and sticky routing yields spatial specialization (left/right), and organizes behavior into pick, adjust, reach, and release with corresponding end-effector (EE) actions. Bottom: Gate values time show sparse, phase-consistent activation-on few experts are active per step for efficient sampling.
INTRODUCTION
Recent advances in diffusion-baasee policies have demonstrated remarkable success in solving com- plex robot manipulation tasks (Chi et al., 2023; Hao et al. 2025). By modeling action generation as denoising process, these approaches achieve high success rates and robustness in single-task set- tings. However, how to effectively generalize such policies to diverse multi-task scenarios remains crucial open challenge in robotics. A common strategy pursued in recent research is to scale up the size of policy networks (Liu et al. 2024), motivated by the scaling laws that enable large models to interpolate across unseen tasks. While promising, this paradigm comes steep cost: the inference speed of oversized models is often impractical for real-time manipulation (Mei et al., (024)) and the required demonstration datasets may grow exponentially with task diversity. Thus, enabling effec-

---

Published as a conference paper at ICLR 2026
tive multi-tast mullii-askgae generalization under moderate model size and sampling latency is vital for weal-world robotic applications.
An alternative line of work seeks generalization through skill-based policy learning (Lang et al, 2024] Wu et al. 2025), which abstracts task-invariant skills from demonstrations and reuses them across tasks. Prior approaches, such as information- diversity regularization (e.g., DI- AYN (Eysenbach et al. 2018) and goal-conditioned hierarchical RL (Liu et al. 2022). are designed primarily for exploration in sparse-reward environments rather than efficient skill abstraction for manipulation. More recently. mixture-of-experts (MoE) architectures have been applied to robot diffusion policies (Wang al., 2024). where large feedforward backbone is replaced by smaller expert modules. However, existing MoE formulations do not explicitly disentangle and represent reusable manipulation skills, limiting their interpretability and transferability (Yang et al. 2025).
In this paper, we introduce Skill Mixture-of-Experts Policy (SMP) (Fig. []. diffusion-based mixture-of-experts policy that performs skill abstraction in a locally whitened action space. Here, a skill denotes state-adaptive orthogonal action lonillye-locally disentangled in the current ac- tion geometry-whose consistent activation patterns across time and tasks form higher-Ievel roles. Rather than blending unconstrained experts, SMP learns these state-adaptive skills and employs slowly varying (sticky) gates. yielding disentangled, phase-consistent behaviors. We develop a prin- cipled variational objective that combines reconstruction in the whitened basis and gate regulariza- tion, and distill a state-only router. At inference, an adaptive expert activation mechanism selects compact subset of experts. Together, these choices produce compact, reusable, and transferable skills that improve multi-task generalization and sampling efficiency, enabling real-time bimanual manipulation.
We validate SMP in both simulation (Chen et al. 2025 Grotz et al. 2024) and real-world bimanual manipulation tasks (Fu et al. 2024). Across multi-takk evaluations. SMP achieves consistently good performance with lower inference cost than strong diffusion baselines. Analyses show that the orthonormal skill basis and sticky routing yield stable behavior with fewer gate switches and cross- task skill reuse. Adaptive expert activation maintains policy quality while substantially reducing active parameters and latency. In transfer learning, reusing the compact skill set enables effective few-shot adaptation to new tasks.
In summary, the contributions of this paper are as follows:
We propose SMP, a diffusion-based mixture-of-expent framework that explicitly abstracts reusable manipulation skills via a state-dependent orthonormal action basis with sticky routing, improving performance across multiple tasks.
We design an adaptive expert activation strategy that dynamically selects a compact subset of ex- perts at inference time, reducing computational cost while maintaining action sampling accuracy.
We validate SMP in bimanual manipulation tasks-including multi-task and transfer set- degonstrann high success rates and lower inference cost than strong diffusion baselines.
BACKGROUND AND RELATED WORK
Robot Manipulation Policies. Diffusion-based generative models have recently become strong paradigm for robot manipulation, where policies such as Diffusion Policy (Chi et al.) (003) and flow-machhing variants (Zhang al. (2025) generate actions via iterative denoising or continuous- time dynamics. These methods achieve strong single-task performance and stable training, and have been extended to longer horizons using trajectory or decision transformers (Chen et al. 202T). Vi- sion- lawg-langgagattiinn action (VLA) models (Zitkovich et al. 2023 Kim et al. (024) further improve gen- eralization by leveraging large-scale multimodal pretraining, while other works combine keypoint- conditioned representations (Sundaresan et al. (023)) or hybrid controllers such as MPC (Zhao et al.) 2024]. Although effective, these methods generally rely on scaling up data and model capacity to gain transfcrability, which introduces prohibitive computational costs for real-time control.
Generalizable Skill Abstraction. A broad line of work aims to reuse task-agnostic motion primi- tives across tasks. Unsupervised skill discovery methods maximize mutual information related di- versity criteria to induce distinct behaviors without rewards (e.g., DIAYN (Eysenbach et al. 2018 while hierarchicalloption frameworks learn temporally extended actions simplify long- morizoorion control (Liu et al., 2022). A complementary thread learns latent skill spaces from demonstrations via variational discrete codebook models, enabling a policy to condition on compact skill iden-
2

---

Published as a conference paper at ICLR 2026
tifiers during execution (Liang et al. 2024 Wu et al.
2025 More recently, mixture-of-experts architectures (Reuss et al. 2024) have been adopted to scale capacity and route computation across modules within diffusion- based policies (Huang et al. 2024 Wang al. 2024). Despite progress, most approaches do not explicitly decorrelate the action space or enforce phase stability-
often remain entangled, routing can switch rapidly, and reuse across bimanual roles is opaque.
3 PRELIMINARIES
Problem Formulation. We define set of robot manipulation tasks M {Mx, R= with sub- tasks Ma. All tasks share the same robot state space E S and action space a A. Note that the state space includes both observations and the task index which serves
task identifier to distinguish and indicate which sub-task the policy should execute. We collect trajectory demonstra- tions in each task Dx - {Tn..i and each trajectory contains series of consecutive state and action pairs Tr.j = {88, a;) J. In the multi-task robot manipulation problem, a manipulation policy p( p(] trained in all tasks M to imitate the demonstrations D. We evaluate the success of completing each task M, and computation cost (i.e. model size, computation time) in the policy training and sampling processes.
We adopt diffusion-based generative modeling for action generation, where a "clean" action ag viewed as the endpoint of latent Markov chain 01:3 of the same di- mensionality. The model defines po(as) = (o(os)))) da1-z. The forward (noising) process gradually perturbs data do q(ao) with variance schedule (1+) factorized as q(a1:z an) II q(ar ar-1) with q(a, ar-1) N((ai VI Br a1 3,1). Conditioned on the current state 8, the reverse (denoising) process is parameterized by a Gaussian chain pe(00:3 = p(0x) II== Po(ar-1 8) with standard normal prior p(ag) N 1) and (uo(ar. 8), (a T, 8)). Training maximizes the conditional data likeli- hood by minimizing diffusion ELBO: E[- log s)] S Eg -log po(aor= (s) Lour(a, au),
Diffusion Policy.
pe(ar-1 ar,
q(01:a/a0)
where Diif is implemented via standard DDPM (Ho et al. 2020) parameterizations of μa and Di (or equivalent noise-prediction losses) and supports efficient conditional sampling at test time.
4 METHOD: SKILL MIXTURE-OF-EXPER POLICY
Direct mixtures of unconstrained expert outputs overlap and become non-identifiable; many different combinations can reproduce the same action, leading to unstable routing and training. To address this, we introduce a method for skill disentanglement based on constructing state-adaptive orthogonal frame. At each state, every skill is mapped to a distinct, non-overlapping direction in the action space. This ensures non-overlapping contributions, yields well-conditioned demixing for supervising coefficients, and- combined with sticky gates promotes sparse, stable activation of only few skills per state. In the following, we describe the core ideas underlying our method; an overview of our framework and training pipeline is shown in Fig. Additional implementation details are provided in Appendix A
Generative model. At time t, s, R.d, denotes the state and a, E Rd the action. Let K < d be the number reusable skills. We write 9: A for the gate (simplex weights over skills, App. A. and E RK for the per-skill coefficients. Actions are decoded through an orthonormal skill basis B = [b.....bkkl d
a, (g: Oz)
where BTB B = Ix and @ is elementwise product. In this orthogonal decomposition, the i-th skill contributes the rank-1 vector b; (9,, in (b:), which ensures additive effects of each skill. While global skill basis B is appealing conceptually, in practice robot action geometry may depend on state (e.g., arm pose, contact). A fixed basis may fail to capture such variability. We therefore parameterize dependee basis (s)), which allows the skill frame to adjust with context while preserving orthogonality and local disentanglement.
QR factorization for state-adaptive skill basis. To construct the basis B(s) E R first generate an unconstrained matrix (s) E B K with a lightweight neural network, and project onto the Stiefel manifold using differentiable thin-QR retraction with sign stabilization (App.
3

---

Published as a conference paper at ICLR 2026
Skill Experts Policy Training
(b) State Adaptive Skill Basis
Step
Orthogonal basis ood (WGG)
Recon Action
Statie
Action
02
G all
Ron
Action- di
Leyocon
pricpptgie
state transsonn
Calten
S' a 5e
Stop
de
Canta,2)
Bockprop:
Food Forward
Gate posterior.
Drffusion experts piZzE)
Figure 2: Skill Mixture-of-Expere Policy (SMP) Training Framework. Left (a): During train- ing, raw observations are encoded into state features, which generate an unconstrained matrix (s).. QR retraction produces state-adaptive orthogonal basis B(s). Actions are reconstructed via (g @ 2), where trained with reconstruction, diffusion, gate regularization, and alignment losses. Right (b): Illus- tration of the state-adaptive basis across timesteps: the robot moves, the basis vectors adjust with the state, while sticky gates preserve consistent expert roles (e.g., translation and rotation).
ticky--gteed weights and
diffusion based coefficients. The model
Concretely, for each state s, we compute
((s) = BU, D =
B(s)= B IK, U upper triangular.
(2) Here, B is the orthonormal factor from the QR decomposition, and D is diagonal sign matrix that removes the column-sign ambiguity. This encourages B(s) to evolve continuously with by avoiding discontinuous flips in orientation. During training, treat (s)) as the forward map, update (s) via standard optimizers, and backpropagate through the QR retraction using automatic differentiation. This formulation yields moving orthogonal frame that adapts with task geometry, while the gating mechanism provides phase consistency. To avoid notational clutter, we omit the explicit dependence
in (s)) and W(s) when the context clear.
Sticky gates and global usage. Manipulation typically progresses through quasi-stationary phases, the skill gate Ot A K should change slowly rather than chatter every step, while still avoiding collapse to a small subset of skills. We formalize this intuition via "sticky" Dirichlet Markov dynamics (App. A.I).
I Dir(a 1), 91 Dir(ao 1), 9t Din( ao 0), t 2
Here E A a global usage vector capturing overall skill prevalence: the initial gate 9i is drawn around 0; and subsequent gates de blend persistence from the previous gate with a mild pull toward global usage. The hyperparameters have intuitive roles: к > controls temporal sticki- ness (larger values yield longer, phase-like segments), ag > anchors the process & to avoid degeneracy, and a > 0 sets the diffuseness the global prior (larger values encourage more bal- anced usage). This model yields piecewise-constant skill activations while maintaining broad yet non-uniform utilization across tasks.
4.1 VARIATIONAL LOWER- LOWER-BOUND AND LOSS FORMULATION
Variational inference for joint generation. We formalize SMP training via variational inference. We group the latents into gates and global usage (0,g).T) and skill coefficients 21.7. The trajectory- level model factorizes as:
Po (0, gI.T, 21:T, a1.T $1.T) = p(a1r g1. 21.T, 81.T B) p(0, 91.7 sL p(z1: $1T) (4)
recomsiruction: o;=B (g()=z)
gates
coefficients
Here the first term enforces reconstruction in the local whitened basis; the second term is the sticky Dirichlet prior with a global usage vector, p(1, p(.. $1.) = p()) 9) II p(g. 0t-1,0);
and the third term adopts diffusion priors for coefficients, p(=E:T 81:)) I[= p(z, s.). For
4

---

Published as a conference paper at ICLR 2026
inference, we adopt amortized, factorized posterior, q(),, g1:T 21:7 01:T, $1:T) 9(0.9.1 g1.7 01.r, $1.7) ((11:1
d1:T, $1:T)
(5)
Dirichlet amortizers
confficient posterions
Applying Jensen's inequality to Eqn. and yields: Pegpo(a (1.T) log a(.) q(0, 9I.T, 21.T
91.T 21:T, S1:T)
dil dg dz a1:T; S1:T)
(6)
2 F logı(ar: guT, 21,, SITT B)
DKL ((0, 9IT
OL., 81:) (lat))/((0,,000 gIr BLT))
reconstmation term: Cinor
gatefglobal-usagee regularization: Lyya
- -DL ((E1 o1., 81:1) p(z1er (1:))
coefficient regularization: Covett
In summary, Eqn. separates training into three components: (i) reconstruction term realized by synthesis in the local basis (implemented via the coefficient-space diffusion, described below), (ii) gate/global-usage regularizer Lgate that pulls the amortized gate posterior toward the sticky-gate prior, and (iii) a coefficient regularizer Leeef that aligns coefficient posteriors with their diffusion priors.
Reconstruction via coefficient-space diffusion. We decode actions à, = =B(ge C 20.1) and place a small-variance Gaussian around it, p(a, 20 It, &t, B) = Nia,, T)), yielding Precen =
Ei=1 ||ee àellă- To supervise coefficients while disentangling which gradients reach B, we form two targets and the reconstruction:
267
TTa a1
B at
T |
58 0.t
SIEC D0,
a.eh = B(ge t O
ARE 20,t), Lrecon
a,
1
1
(7)
Eyloo]
+
Eylge] +e"
20
1=1
stop-gradient diffusion
gradient flows
Here B sg[B] is a stop-gradient copy and the division is elementwise (e Re 10-3). We use -0,t for the diffusion surrogate, Lwvell Laa(2; 20.1:2), so gradient from Levelt updates B. In contrast, reconstruction uses the gradient-carrynng target 20,t- which propagates gradients into B (and gt) through both the projection B 0 and the decoder B(-). Together, Creson + Levef encourages action consistency while providing stable per-expert supervision in coefficient space, with only the reconstruction term updating the skill basis.
Gate regularization. We use Dirichlet distributions 4(0) = Dir(ā) and q( 8t, ax) Dir(â, ot)), and train coefficients with the DDPM loss. We also introduce state-only router for deployment, (9 84) (s))) whose distribution aligned to the training-time gate posterior. Plugging the sticky-gate prior (Eqn. B3 into the ELBO yields:
Cgate Dkz(g(U) Dir(a1)) Dxz(g(ς1) Dir(ap 1 ()))
(8)
global usage
initial gate
T Σ
E DKL(9(0))
Dir(nEyyg ag Ey(01)) Calign
DKilalgi s(, Dir (irg
sticky gates
router alignment
The three terms Cgate impose a global usage prior, anchor the initial gate, and encourage temporal stickiness via persistence from 9t-1- The auxiliary loss Calign aligns the state-only router to the training-time time gate posterior so deployment-time routing is consistent with inference.
ofxture-of-Expee Experts equivalence. Our decoder is exactly additive MoE in the local skill basis: a, Σ St. žt.i B (ge zz). Interpreting the i-th expert as the rank-1 map fe(s;; 2.,)) b; with coefficient distribution p(=,, 81) yields the pushforward expert conditional b,(a, st) 5(a, b,2) p(z s, dz, and the gate is w;(st) It.s- Because {b,} are orthonormal, each expert acts on the one- mee-imennional subspace }b,n giving orthogonal contributions and decoupled gradients; top-k routing makes the mixture sparse.
5

---

Published as a conference paper at ICLR 2026
4.2 ADAPTIVE EXPERT ACTIVATION
Activation with top-k or coverage selection. Evaluating all experts at every state is costly and unnecessary since only a few skill directions are typically important. At deployment we use the state-only router Po( 8t) = Dir(3s(()) and its mean Jr = Elge 8t to estimate the importance of each expert. In the orthonormal basis B = [bi....,bxr bx] with B I, we define the mass of expert i as m, = -
To decide which experts to activate, we score any candidate active set S C {1,.... K} by F(S) = Lies m, Because F is additive, the optimal set can be obtained by simply sorting experts by m, and selecting either (i)
Algorithm 1 SMP Training
1: Input: dataset D: basis params W (B = (r)((W))
gate posterior ((g 8, a); state-only router (o 8); diffusion experts for 2.
2: repeat
the top-k experts, (ii) the smallest prefix
3: Sample a trajectory (1:))~
of experts such that the selected set captures at least fraction Tm of the total mass, i.e.,
4:
Orthonormalize 4 qr"( (sign-stabilized)
5:
Compute amortized gates ((9: st, for t=1:7'
Ligs -5-1 my
2 Tm, with Tm E [0.9, 0.95].
6:
Build two coefficient targets 20, sTe
After choosing the active set S,, we denoise only the corresponding coefficients st,,, (all others set to zero) and decode the action as a, B (gt O ze). This simple ranking rule is equivalent to greedy maximization of the additive objective F(S) Eies mi- It yields sparse, state-dependent activation, re- ducing inference cost while preserving accu- racy.
7:
Compute diffusion loss Levett +- Lan (2; 2 2)))))
8:
Reconstruct àt B( O a0.e) and Lreeon
9:
Compute gate regularizers Lguee and router align- ment Lalign
10:
Total loss: LSHIMEE <- Lovel + Lreonn Lyue + Calign
11:
Update W. diffusion experts, amortizers, and router via gradient descent
12: until converged
4.3 ALGORITHM SUMMARY
Algorithm 2 SMP Inference (Sampling)
We summarize SMP in Algorithms and 2 Training alternates over trajectories by (i) or- thonormalizing the skill basis B (W). (ii) inferring gates with the amortized poste- rior, (iii) forming two coefficient targets in the whitened basis-aj (stop-gradient) for the diffusion surrogate and gnee (gradient-
1: Input: orthonormal basis B; state-only router psi diffusion experts for 2; budget coverage Ti while task not complete do
3: Observe 8, and get router mean 3r = BK[ge st]
4: Set masses m. = =g
5: Initialize active set S+ Mor = E]=1 my
6: while |S| < l: and Σ iES Ma/Mai < Tm do
carrying) for reconstruction-anue (iv) opti- mizing a compact objective that combines coefficient-space diffusion, light recon- struction term, sticky-gate regularization, and router alignment. At inference, given 8; we query the state-only router, select a small ac- tive set of experts via top-k or greedy cov- erage, denoise only the selected coefficients, and decode the action via a B(g. @ 2)), yielding sparse, low- low-latency control.
7: Define F(S) Σ Lies! mh
8:
Choose x" arg max (F(S (i))
F(S))
9:
Update S < SU()),
10:
end while
11:
Denoise only 21, (set zny = 0 for j & S)
12:
Decode and execute a, = B (g:
13:
end while
EXPERIMENTS
In this section, we evaluate SMP on multi-task bimanual manipulation using both simulation bench- marks and real-robot experiments. We investigate the following questions:
Does SMP improve multi-task success while reducing inference cost compared to strong diffu- sion baselines?
Do the orthonormal skill basis and sticky routing yield stable, phase-consistent behavior-wind fewer gate switches/oscilaations- and better cross-task skill reuse than unstructured mixtures?
Can adaptive expert activation maintain reconstruction quality and success while substantially reducing active parameters and latency versus dense activation and MON-MoE-stylee methods
After multi-task training, does the policy adapt new tasks with limited demonstrations more effectively than baselines?
'FFN-MoB is mixture-of-experts only on individual feed-forward neural network Shazeer et al. 2017

---

Published as a conference paper at ICLR 2026
RoboTwin-2
Basia direction
Pick bottles
bunger fries buy
4
Left
Right
trans.
Left
rotat.
Hand over (HHO) block plans target
Put bread into stillet
Right
rotat.
H
8
+ Exp
Exp
RLBench-2
Sweep dustt duxipan
Lift tray with block on
Exp
Exp 4
Exp
Exp 6
Exp 7
Figure 3: Multi-task learning in Robo and RLBench-2. SMP partitions bimanual control into an orthonormal skill basis and routes with sticky gates. Across tasks, the same experts are reused for left- and right-arm primitives and for pick-move-place phases, with few switches and long segments. Gate traces reveal sparse, phase-consistent activation, and cross-task skill reuse, indicating that actions are composed from a small, task-relevant subset of experts.
5.1 EXPERIMENT SETTING
Simulation Environments. We evaluate on two bimanual manipulation benchmarks RoboTTwin (Chen al. 2025) and RLBench-2 (Grotz et al., (024) which an agent controls two arms that may act independently or cooperatively. The enlarged action space and multimodal action distribu- tions challenge conventional policies, while the natural decomposition into skill-specific subspaces enables tross-task skill reuse. We study multi-task learning and two forms of transfer RoboTTwin- In the multi-task learning setting, we train separate policies: one jointly on Robo tasks (cross-arm skill reuse) and another jointly 4 tightly coordinated RLBench-2 tasks (spatiotem- poral collaboration). For transfer, we consider: (i) few-shot adaptation, where the RoboTTwin-2 multi-task policy is fully fine-tuned on 4 new RoboTwin-2 tasks with 10 demonstrations each; and (ii) skill composition, where experts (and the skill basis) are frozen and only the router is fine-tuned on new RoboTwin-2 tasks with 10 demonstrations each to test recomposition of learned skills. Additional environment details are provided in Appendix B
Baseline Methods and Evaluation Metrics. Baselines include vanilla diffusion policies (DP CChi et al. 2023). DP3 (Ze et al. 2024)), Sansforrer-baseee policy (ACT (Zhao et al.) 2023 ), a fine- tuned diffusion foundation model (RDT (Liu al. 2024)), an information-theorre
skill-abstraction policy (Discrete Policy (Wu et al., 2025 ). and an MoE diffusion policy with feed-forward experts (Sparse Diffusion Policy (Wang et al.J 2024)).
each environment, policies are trained jointly across multiple tasks, increasing data complexity and stressing model capacity under multimodal distributions. We report task success rate, measures of learned skill abstraction, and computa- tional cost and sampling efficiency. For woaldworld experiments, report and compare the progress scores.
5.2 MULTI-TASK LEARNING EXPERIMENTAL RESULTS
Baseline methods struggle to generalize in bimanual multi-task settings. Across both suites, DP, DP3, and ACT underfit the multimodal distributions induced by large bimanual action spaces and multiple tasks, yielding low success (Tab. D. Scaling via RDT increases parameters by 10x over DP (Tab. 2 but improves success by only 19%. indicating limited returns and the inefficiency of naive model scaling for multimodal multi-task control. Discrete Policy separates some behaviors yet routes into single backbone trained on heterogeneous data and quickly saturates without sub- stantially larger capacity. Sparse Diffusion Policy replaces the backbone with FFN-MoE experts but lacks explicit geometric skill disentanglement; during sampling its gates switch frequently (higher flip-rate. introducing action oscillations and hurting precision-tase success (Wang et al.) 2024)
SMP abstracts reusable manipulation skills across bimanual tasks and improves overall suc- cess. As reported in Tab. [ and visualized in Fig. 33] the policy partitions action generation across space and over time, routing experts with consistent semanties. In RoboTwin-2 and RL Benchh22,

---

Published as a conference paper at ICLR 2026
Table 1: Success Rates in Bimanual Multi-Tass Learning Tasks 1
RoborTwin-2! Friees
RLBench-21
Methods
Bottle
H/O
Pot
Skillet
Cab.
Avg.
Tray
Rope
Oven
Sweep
Avg.
DP
0.21
0.17
0.34
0.54
0.11
0.37
0.29
0.10
0.13
0.08
0.09
0.10
DP3
0.26
0.31
0.46
0.59
0.06
0.32
0.33
ACT
0.31
0.43
0.61
0.50
0.08
0.15
0.34
0.15
0.15
0.12
0.18
0.15
RDT
0.55
0.60
0.63
0.66
0.05
0.43
0.48
0.18
0.17
0.13
0.20
0.17
Disc. Poliey
0.29
0.23
0.46
0.74
0.13
0.52
0.40
0.13
0.14
0.07
0.17
0.13
Spurse DP
0.37
0.42
0.59
0.77
0.08
0.42
0.44
0.15
0.18
0.12
0.17
0.16
SMP (ours)
0.56
0.61
0.64
0.79
0.14
0.52
0.54
0.19
0.20
0.14
0.20
0.18
RoboTwin-2 Chen ct al.
2025
RLBench-2 (Grotz et al. 2022).
Tasks see Fig.
Results are averaged in 100
episodes. DP [Chi 2023]. DP3 (Ze 20024, ACT (Zhao al. 20_3 RDT (Liu al. [2024] Sparse
DP (Wang al. 2024). Discrete Policy (Wu et al. 2025). SMP denotes our Skill Mixture-ol- Expertsss Policy.
we observe two dominant patterns tied our design: (i) left- and right-arm behaviors are handled by different experts-for example, one expert group consistently produces left-arm translation/rotation for grasping, while other experts govern the right arm's 's task-specific motions and (ii) trajectories or- ganize into pick (pre-grasp, grasp), move, and place (pre-release, release) phases, where move/pre- release primarily invoke translation-focusee experts, grasp/release are captured by gripper-focuse experts, and gre-grasp combines translation and rotation experts for precise alignment. These struc- tured patterns indicate reusable skills across tasks and align with the consistently higher (or compa- rable) success rates observed in the bimanual multi-tase environments.
SMP reduces active computation and latency via adap- tive expert activation. Tab. 2 reports total parameters Ng- active parameters during sampling Apct, and inference time Tin Rather than evaluating all experts at every step, SMP ac- tivates a small, state-dependent subset (top-k/coverage), so most timesteps use only a fraction of the backbone. In prac- tice, it activates about 30% of its own parameters- roughly 7% of RDT's total-while maintaining high success. This selective routing yields consistently lower Tine than single- backbone diffusion baselines (DP, DP3, RDT, Discrete Pol- icy) because only the chosen experts are denoised and com- posed in the final action, and multiple small experts can run in parallel. By contrast, the FFN-MoE baseline (used in Sparse Diffusion Policy) offers limited speedup since only part of its computation is parallelizable and each denoising step must synchronize across experts, introducing overhead. We also observe the expected trade-off: increasing the activation budget improves reconstruction slightly but raises latency; our reported operating point is selected on vali- dation to balance success and Tir.
Table 2: Computation costs 4
Methods
N2
Ao
Tint
DP
132.5
132.5
120.3
DP3
128.5
128.5
122.1
ACT
83.9
83.9
94.8
RDT
1200
1200
183.1
Disc. Policy
162.9
162.9
153.7
Sparse DP
154.4
110.1
148.3
SMP (ours)
258.9
80.2
107.3
Ny: total network parameters (M), Nates expected active network parameters (M), Tar:' inference time (ms). Results av- craged over all tasks.
Real-robot case study. As shown in Fig. we conducted a real-robot multi-task case-study on four bimanual manipulation tasos-remove pen cap, pul cups into bowl, hand over screwdriver, and pour beans into bowl on the PiPER platform (AgileX 2025). For each task, we collect 50 demonstrations and train a single multi-task policy per method. Performance is reported as the progress score (progress of task completion) averaged over 10 trials. Details in Appendix B.12
pul
Across all four tasks, SMP attains the highest average progress score over 10 trials while using fewer active parameters and lower inference latency than diffusion baselines (Fig. (5). RDT reaches com- petitive progress but incurs the largest compute and slowest inference our setup. Discrete Policy and Sparse Diffusion Policy (SDP) operate at lower budgets yet trail SMP in progress, and their routing occasionally produces inconsistent phase selections. In contrast, SMP composes actions from a compact, task-relevant subset of experts with sticky routing, delivering steady progress and efficient execution.
5.3 TRANSFER LEARNING EXPERIMENTAL RESULTS
Few-shot transfer. We evaluated few-shot transfer on RoboTwin-2 by fully fine-tuning the multi- task policy on four new tasks with 10-shot adaptation. The results summarized in Tab, show that DP and RDT transfer poorly; prior behaviors remain latent in large backbones and the limited data
8

---

Published as a conference paper at ICLR 2026
Remove pen cep
RDT 0.55
Disc. 0.23
SDP 0.45
SMP 0.76
cups into bowl
RDT 0.78
Disc. 0.39
SDP 0.54
SMP 0.89
Harad oveer screwedriveer
RDT 0.47
Disc. 0.30
SDP 0.58
SMP 0.65
Pour beans bowl
Disc. 0.21
SDP 0.32
SMP 0.57
Figure 5: Real-robot experiments with four manipulation policies. Left: SMP executes bimanual manipulation tasks. Right: Progress score of each task averaged in 10 trials.
was insufficient to obtain strong performance. Discrete Policy was able but the limited samples did not reliably tune its policy, In contrast, SMP applies MoE such that only the relevant experts are activated and fine-tuned without adding new experts; concentrating the ten-shot data on this sparse subset yielded the best success.
some latent codes,
Skill composition. We study skill re- composition by transfer learning on two new RoboTwin-2 tasks that reuse left- and right-hand behaviors from the multi-task set (e.g., fries > skillet and bottle -> cabinet; see Fig. 4p. Each task provides 10 demonstrations; we freeze all experts (and the skill basis) and
Table 3: Success Rate in Few-shot Transfer
Methods
Div.
Mic
Roller"
Box
Avg.
DP
0.06
0.13
0.18
0.16
0.13
RDT
0.14
0.18
Disc. Policy
0.17
0.38
0.44
0.25
0.31
SMP (ours)
0.22
0.49
0.49
0.31
0.38
fine-tune only the router to test recombination without updating expert parameters.
Pick two diverse bottles. Hand over mic. cooking roller. two cans into box. Results averaged over 100 episodes.
Grab a
SMP achieves higher performance than Sparse Diffusion Policy (SDP) (success rates reported in Tab. 4. SDP's layer-wise gating couples experts across diffusion steps, so router tuning did not isolate skills; qualitatively, we observed that per-arm roles blur-e.g. hesitant grasps in friel-sskillet and misaligned placement in bot- ile->cabiner. In contrast, SMP encodes sep- arates skills cleanly, enabling router-only up- dates to recompose right/left pick-and-plans which yielded steadier grasps and placements.
AGUER
Put fries into skillet (L).
bottle cabinet
Figure Skill Composition Tasks
Table Success Rate in Skill Composition
Methods
Skillet-Pries'
Bottle-Cah. ? Avg.
CONCLUSION
SDP
0.11
0.25
We introduced SMP, a diffusion based mixture- of-experts policy that composes actions through compact state-adaptive orthogonal skill basis with sticky routing and adaptive activation. By routing sparsely and stably in this basis, SMP yields chase-consistent skills that are reused across tasks; in simulation and on a real dual-arm platform, it delivers higher multi-task and transfer success substantially lower inference cost than large diffusion baselines, enabling real-time control. Be- yond success rates, the learned experts align with left/right-arm roles and pick-move-plas phases, and the adaptive expert activation provides simple, effective knob trade off accuracy and latency.
SMP (ours)
0.15
0.44
0.30
Pick fries (right) and place skillet (left). Pick bottle (left) and place in cabinct drawer (right).
Limitations and future work. This study uses relatively small diffusion backbones and focuses on bimanual manipulation; next we will scale SMP to larger models and datasets, broaden evalu- ation to single-arm and mobile manipulation, and run more extensive real-robot studies. We will also conduct targeted ablations of sticky routing and adaptive activation to quantify success-latency trade-off's and assess robustness under sensing noise and domain shift.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research/ / project is supported by the National Research Foundation, Singapore, under its The- matic Competitive Research Programme 2025 (NRT-T-CRP-2025-0005) The authors would also like to acknowledge support from the Google Foundation.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics and follows principles of responsible stewardship. avoidance of harm, faimess, and respect for privacy. Human operators provided teleoperation demonstrations but no user study/human-subject experiment was conducted and no personal data was collected.
REPRODUCIBILITY STATEMENT
All algorithms, model architectures, and training procedures are described in the paper and Ap- pendix A Experimental details are provided in Appendix B Hyperparameters, dataset sources, and pre-processing steps are specified. We also provide robot hardware details to support the replicabil- ity of experiments.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
AgileX. PiPER: a lightweight robotic arm. http::/gggooll..ggeee.aaa//dddie ptt.ouuuxsttporuusss///
piper 2025. Accessed: 2025-09-20.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement Iearnin via sequence modeling. Advances in neural information processing systems, 14:15584-50977, 2021.
Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Qiwei Liang. Zixuan Li, Xi- anliang Lin, Yiheng Ge, Zhenyu Gu, et al. Robotwin 2.0: A scalable data generator and bench- mark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506. 18088, 2025.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273668, 2023.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprint
2018.
Zipeng Fu, Tony Z Zhao, and Chelsea Finn, Mobile aloha: Learning bimanual mobile manipulation with low-cost chole-body telcoperation. arXiv preprint MeXiv:2401.02117 2024.
Markus Grotz, Mohit Shridhar, Yu-Wei Chao, Tamim Asfour, and Dieter Fox. Peract2: Benchmark- ing and learning for robotic bimanual manipulation tasks. In CoRL 2024 Workshop Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Beyond, 2024.
Ce Hao, Kelvin Lin, Zhiwei Xue, Siyuan Luo, and Harold Soh. Disco: Language- guided manipu- lation with diffusion policies and constrained inpainting. IEEE Robotics and Automation Letters, 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 6851, 2020.
Suning Huang, Zheyu Zhang, Tianhai Liang, Yihan Xu, Zhehao Kou, Chenhao Lu, Guowei Xu, Zhengrong Xue, and Huazhe Xu. Mentor: -f network with task-oriented pertur- bation for visual reinforcement learning. arXiv preprint 2rXh::2410.14977 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-acee action model. arXiv preprint arXiv:2406.09244 09246, 2024.
Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldif- fuser: Interpretable hierarchical planning via skill abstractions in iiffusiin-bbsed task execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16467-16476, 2024.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.08200
2022.
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rb: diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
/aaxiv.org/abs///.
Kangfu Mei, Zhengzhong Tu, Mauricio Delbracio, Hossein Talebi, Vishal M Patel, and Peyman Milanfar. Bigger is not always better: Scaling properties of latent diffusion models. Transactions on Machine Learning Research, 2024.
Moritz Reuss, Jyothish Pari, Pulkit Agrawal, and Rudolf Lioutikov. Efficient diffusion transformer policies with mixture of expert denoisers for multitask learning. arXiv preprint arXiv:2412. 12953, 2024.
11

---

Published as a conference paper at ICLR 2026
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The spansely-gate mixture-of-experts layer. arXiv preprint arXiv:1701.065.0 2017.
Priya Sundaresan, Suneel Belkhale, Dorsa Sadigh, and Jeannette Bohg. Kite: Keypoint-conditioned policies for semantic manipulation. arXiv preprint arXiv:2306. 16605, 2023.
Yixiao Wang, Yifei Zhang, Mingxiao Huo, Ran Tian, Xiang Zhang, Yichen Xie, Chenfeng Xu, Pengliang Ji, Wei Zhan, Mingyu Ding, al. Sparse diffusion policy: sparse, reusable, and flexible policy for robot learning. arXiv preprint arXiv:2407.0153/ 2024.
Kun Wu, Yichen Zhu, Jinming Li, Junjie Wen, Ning Liu, Zhiyuan Xu, and Jian Tang. Discrete policy: Learning disentangled action space for multi-task robotic manipulation. 2025 IEEE International Conference Robotics and Automation (ICRA), pp- 8811 1 8818. IEEE, 2025.
Xingyi Yang, Constantin Venhoff, Ashkan Khakzar, Christian Schroeder de Witt, Puneet K Dokania, Adel Bibi, and Philip Torr. Mixture of experts made intrinsically interpretable. arXiv preprint arXiv:2503.07633, 2025.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. diffusion policy: Generalizable visuomotor policy learning via simple 3d representations, arXiv preprint arXiv:2403.03954, 2024.
Qinglun Zhang. Zhen Liu, Haoqiang Fan, Guanghui Liu, Bing Zeng, and Shuaicheng Liu. Flow- policy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14754-14762, 2025.
Tony 7. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 130.0 2023.
Wentao Zhao, Jiaming Chen, Ziyu Meng. Donghui Mao, Ran Song, and Wei Zhang. VImpe: Vision- language model predictive control for robotic manipulation. arXiv preprint arXiv:2407.09829, 2024.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: aoooo-llnguage-cttinn models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165-2183, PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
DISCLOSURE OF LLM USE
We used a large language model (LLM) mainly to check and correct grammar and language issues in the manuscript. We also used an LLM to validate ideas and check for errors. The authors take full responsibility for all content and affirm that all scientific ideas, analysis, and conclusions were conceived and written by the authors.
A SKILL MIXTURE-OF-EXPERE POLICY METHOD DETAILS
In Sec. we presented the SMP formulation and its training and inference procedures. This ap- pendix provides the accompanying mathematical and implementation details: (i) preliminaries simplex-valued gates, Dirichlet-Markov (sticky) dynamics, and thin-QR retraction for the state- dependent basis; (ii) the subspace-decompositoon assumption and the resulting variational objective for joint generation; (iii) the coefficient-spe sppcee reconstruction targets and gate regularization terms; (iv) adaptive expert activation during inference: and (v) a breakdown computation cost for training and sampling.
A.1 MATHEMATICAL PRELIMINARIES
In this subsection, we collect the basic mathematical notions used in Section 4.1: simplex-valus gates, Markoo ("sticky") dynamics, and the thin-QR retraction with sign stabilization for constructing the orthonormal skill basis B(s).
Simplex weights. We represent expert-selection weights as elements of the probability simplex
A K-1
{g E 1 g = 1]
where K is the number of skills and is the all-ones vector. A vector E K- can be interpreted a categorical distribution over experts as mixture weights in convex combination of expert contributions. Throughout, we write Ir E A K-1 for the gate at time in Eq.
Dirichlet distribution and Dirichlet-Markov dynamics. The Dirichlet distribution a standard prior over cimplex-valued random variables. For concentration parameter a RRK random vector g Dir(a)) satisfies g E K-1 and has density
K II
p(g a) = ((a)
9z 11-
DEAR-1
(10)
where ((x) is the normalizing constant. Larger ON encourages to be larger on average, while smaller og pushes probability mass away from the k-th corner of the simplex.
To describe the temporal evolution of gates gI.T, a first-order Markov process with Dirichlet conditionals, which we refer to as Dirichlet-Markov dynamics. In the main text (Eq, 3. this takes the form
1 ~ Dir(a
91 Dir(ao 0),
9 d) v). t 2,
(11)
where E K-1 global usage vector, ag > controls how strongly & influences the gates, and k 2 0 controls the dependence on the previous gate St-1. Intuitively, the term ag & encourages
overall usage patterns that match i, while the term K 9t-1 encourages the process to stay close to its previous value.
Sticky Dirichlet-Markov dynamics. In manipulation, we often expect phase-like behavior, where the active skills stay roughly constant over several timesteps. To capture this, we use sticky Dirichlet-Markov dynamics: in Eq. [1] a larger value of increases the self-reinforcing effect of 9t-1, making the conditional distribution of more concentrated around the previous gate. In the limit of large K, 9t changes only slowly over time. This induces smooth, piecewise-constar gate trajectories while still allowing occasional switches between experts, and is exactly the prior used in the gate regularization term Laste in Eq.
13

---

Published as a conference paper at ICLR 2026
Thin-QR retraction with sign stabilization. The skill basis B(s) E R in Eq. is required to have orthonormal columns, i.c., B(s) B(s)= IK.
(12)
for each state The set of such matrices is the Stiefel manifold
St(d, K) = R B B = [x]-
(13)
Rather than optimizing (s)) directly under this constraint, we maintain an unconstrained matrix (s)) E Rd (the output of small neural network) and map to St(d, K) via thin-QR factor- ization:
W(s) Uss)
(14)
where B(s) RdxK has orthonormal columns and (() E RKXK is upper triangular. naive choice (s)) = (s) suffers from a sign ambiguity: multiplying a column of B(s) by -1 and the corresponding row of (s) by -1 leaves W(s) unchanged, but flips the direction of the basis vector.
To avoid such discontinuous flips as s varies, we apply sign stabilization. Let
((s) diag (sign(diag(UU())))
(15)
and define
B(s) B(s) D(s).
(16)
This enforces consistent sign convention (e.g.. making the diagonal entries of U(s) positive) and yields an orthonormal basis (s) that evolves smoothly with the state. In practice, we treat (s)) as the learnable parameter, apply the QR-based map ((s) + B(x) in each forward pass, and
backpropagate through this retraction using automatic differentiation. This implements the state-
adaptive orthogonal frame described in Eq. and used throughout Section 4
A.2 SUBSPACE DECOMPOSITION AND VARIATIONAL INFERENCE
The decoder in Eq. assumes that, at each state 8t, the action d, E Rd can be synthesized from a K-dimensional skill subspace spanned by the columns of the orthonormal basis B(,,):
0g (sx: G 2)),
B(st) B(s:))((s) IK,
(17)
where K < and 9: A- 21 R.K are the gate and skill coefficients. We first make explicit the geometric assumption underlying this decomposition and how possible remainders are handled, and then derive the resulting variational formulation.
Subspace decomposition and remainder. In full generality, any a E RRd can be decomposed into a part lying in the skill subspace and an orthogonal remainder. Let B(st) E Rdxk be an orthonormal basis and R(st) E (dx(d-K) an orthonormal complement satisfying
B(st) T B(se) = Ik, R(s)) R(se) = Ia-k, B(xt) T R(sx) = 0.
(18)
Then every action admits an exact decomposition
a2 B(s:) + R(st) Tr,
(19)
for some coefficients u, K and r. E Æd-K In our model identify the structured component de with the gated skill coefficients,
4 9: 2t.
(20)
and treat the remainder R(se)) rr small residual. Rather than modeling ra explicitly, absorb it into a small-variance Gaussian likelihood around the reconstructed action:
Aa, gt: St, St, B) (B((s)) @ x)), )),,
1
(21)
Thus, in Eq. [ we explicitly model only the low- -in-eimesiioaa,, task-relevan component (g.))))))))
and regard any energy outside span ( (Bx))) as reconstruction noise.
The key modeling assumption is that, for a suitable choice of K, the demonstration actions lie approximately in a K--dimensional state-dep subspace:
[[s))r)lll is small for all 1,
(22)
14

---

Published as a conference paper at ICLR 2026
so that the residual term can be neglected without materially affecting the policy. This is analogous to low-rank models (c.g.. PCA), where most of the variance is captured by a small number of principal directions, while the remaining directions contribute only minor corrections. By constraining the decoder to (8t), we deliberately force actions to be expressed through a small set of reusable skill directions, which improves identifiability of skills, stabilizes gating, and makes coefficient supervision in the whitened basis well-conditioned, In practice, we found that choosing a modest K suffices reconstruct manipulation actions accurately, so we omit an explicit residual branch in the main formulation and let the Gaussian likelihood account for any small remainder.
Variational inference for joint generation. Given states $1:7 and actions 01.T, SkilIMoE synthe- sizes each action in this state-dependent orthonormal skill subspace,
at = B(sc) 3 zx),
B(st) B(sr) = Ik, St E 24 E R
(23)
and we write B = (sx) below for brevity. The likelihood is given by the Gaussian model above, and we place a sticky Dirichlet prior on the gates with global-usage variable (Eq. 3); coefficient priors follow the diffusion construction:
Mat Mt, 21: St, B) BB @ 2), )
a <1
(24)
T
p(d.g $1.T) = p(U) p 0) II
p(g. 9c-1, 9),
(25)
t=2
T
p(z (1:7) = II
p(z 8t);
(26)
t=1
where g 9I.T and z -1.T.
The trajectory joint conditioned on 81.7
T pla 21, St. st B) p(o.g S1:) p(z s1T))
po(0, g: a1.T 81:1) II
(27)
t=1
We adopt factorized amortized posterior
q0,, 0,,g.2 z (1:T,,8:T) q(0, g (1:1) q(z 01.1811:1)
(28)
with Dirichlet amortizers for 8 and gt. and diffusion posteriors for 2,. By Jensen's incquality,
logpo(a1:r) $1:1) = log po g. z, 0ir $1:1) d0 dg dz
(29)
po g, 2, aier (1:T) du dg dz q(0, g.
log
q(,,g, g.z
01.T.8I.7
(30)
an., S1.T)
T
Eg
log A/a 9t, 21 St B) + p(0 s14) + log / sir)
(31)
log ( (311.. log | [1T1)
(32)
Collecting terms,
LELBO Eg
Σ log pl at 9e, 24, 81, B)
DKL( (((, am, sur) || p(0.g (1:))
Cye
Cyccnn
(33)
Pku(q(e aI.T 81.)) || p(z (1.7))
Condt
Implementation notes. (i The Gaussian likelihood yields an MSE in the whitened space that directly supervises the (state-dependent basis B (Sec. A.3). (ii) The gate KL decomposes into global-usage, initial-gate, and sticky terms (See. A.4). (ii) The coefficient KL is realized by the standard DDPM surrogate on 2, using clean targets formed by projecting a, into the basis (Sec. A.3 Only freamm backpropagates through B: Levef does not.
15

---

Published as a conference paper at ICLR 2026
A.3 RECONSTRUCTION TARGET
To stabilize training while ensuring the (state-dependent) basis (sx,) captures action variability, we form two coefficient targets from the same projection, differing only whether gradients flow to B:
B(8t) 0,2 19 +E
((st) E g6 +e
B(s,) sgl B(s)), e 10-3,
(34)
0,,
where the division elementwise and [y[ge] is the mean the amortized gate posterior. The grg-gradient copy (..) prevents the diffusion loss from directly changing the basis, while the reconstruction path gradient-carrying projection that B(-) is updated toward capturing action energy.
Using the gradient-carryinn target, decode
T
rec = ((tt)(et O 20.1);
Lreco
20
dt
rea aj
1
(35)
1=1
In parallel, we train the coefficient denoisers by re-noising the stop-gradienl clean targets o0, and applying the standard DDPM objective (omitting constants),
T Levell Σ Er,e E 81) 11)
(T) 2t = Var
v1 - ā, E~ N(O,
(36)
Crucially, gradients from Levett do not affect B()) (they flow only to the denoisers and to the amortiz- ers that produced Eg,g. while Crecon updates B(.) (and g:) through both the projection B(ss) Taa at and the decoder B (-).
Projecting with (s;)) concentrates action energy into orthogonal directions; training with 20. makes the whitened decoder (g) sufficient to match or, pushing any unexplained variation toward zero. effect, B() is encouraged to span the task-relevant subspace (skills), and reconstruc- tion error vanishes when the basis captures all structure. This yields compact, interpretable skills and stabilizes routing, since non-overlapping directions reduce ambiguity in gr.
A.4 STICKY GATE REGULARIZATION
We parameterize the gate dynamics with global usage vector and first-order sticky process:
0 Dir(( 1);
91 Dir(as 0), ge Dir(K
9:11 0t + ag 9). 2,
(37)
with a 0 (diffuseness of global usage), ap > 0 (anchor strength toward 19), and バ > (tempo- ral stickiness). Increasing lengthens segments (fewer switches), larger balances usage across experts, and op prevents collapse.
Let (()) = Dir(ā) and ((x)) = Dir(3:) denote the amortized posteriors. The gate regularizer in the BL decomposes
Lgtae Dki(a(0) Dir(a1)) + DKI(a(s)) Dir(oo (,))
T
(38)
Σ DkL((((e)
Dir (x on E.(o1)
encouraging (i) plausible global usage, (ii) non-degenerate initial gate, and (iii) temporal persis- tence around 9:-1 with soft pull toward 0. We use the closed-form KL for Dirichlet distributions:
r 8.) ( | Dir(3)) log T( 8))
T( T(
log
X (8, ()(((()) + (E,),)) i
(39)
where v5(-) is the digamma function.
16

---

Published as a conference paper at ICLR 2026
At deployment, state-only router Po(9: | = Dir(Ge(s:)). To match the training time amortized gates g(9e $t, at), we add an auxiliary alignment loss
DrL(9 st, oz) [ (irr(B((s)))) Calign Σ
(40)
which improves test-time consistency and stabilizes sticky routing when actions unavailable.
Practical implementation. We find it helpful (i) anneal from small value its tar- get to avoid early over-stickiness; (ii) warm-start og to prevent expert collapse; (iii) optionally lemperature-sharpen 3, during alignment to encourage sparse, phase-consistent gates; and (iv) mon- itor a flip-rate (fraction of steps with arg max 9: o arg max 9t-1) and segment length as diagnostics of temporal smoothness.
A.5 ADAPTIVE EXPERT ACTIVATION
At test time, evaluating all K experts every step is unnecessary. We seek sparse, state-dependent active set S. S fl...., ......,K) }} that preserves policy quality while reducing latency. Using the state- only router ps (gt 8t) = (i)() we take its mean 9r = Elge st] and define the mass of expert
m; -
Given orthonormal skill basis B = ..,,..., bx] with B B I, the additive score
F(S) Σ m
(41)
SES
modular (additive). Therefore, sorting experts by m, is optimal for two selection regimes: (i) top-k, which chooses the & largest-m, experts, and (ii) coverage, which chooses the small- est prefix of the sorted list whose cumulative mass reaches a target fraction Tm of the total, i.c.,
Eies ma/ وj=1" my 2 Tm (we use E [0.9, 0.95)). After forming S,, we denoise only =t.s, (others set to zero) and decode at = B C 24), yielding state-wise sparsity and reduced inference cost.
Optimality and complexity. For the top-k constraint maxisisk Eies m, selecting the l largest masses is exactly optimal (by additivity). For the coverage rule, the smallest prefix that attains the threshold Tm is also exactly optimal for minimizing [S] under the modular objective. The per-step complexity is O(K log K) for sorting (or O(K) with partial selection when À is small).
Practical notes. We cap |& by small i (e.g., 2-4) and set Tm E [0.9, 0.95] to stabilize latency with negligible accuracy loss. Mass-only selection requires no task metrics or Jacobians, robust across embodiments, and integrates cleanly with sticky routing.
B EXPERIMENTS DETAILS
B. ENVIRONMENT SETTINGS
We evaluate multitask learning in both simulated and real-world settings. In simulation, addi- tionally conduct transfer-learning experiments to assess skill reusability and adaptation. Tasks are deliberately chosen or designed stress close bimanual coordination, enabling a thorough and in- formative evaluation. Detailed task definitions and protocols are provided in Secs. B.I.I. and B.1.2
B. SIMULATED EXPERIMENTS
Multitask Learning. For multitask learning experiments, conduct experiments 6 tasks from RoboTwin-2 (Chen et al. 2025) and 4 tasks from RLBench-2 (Grotz et al. (024), separately. Meth- ods trained for 3000 epochs Robotwin-2 and 300k iterations on RI Bench-2.
17

---

Published as a conference paper at ICLR 2026
Table 5: Selected bimanual tasks from Robotwin-2 (Chen al. 2025). RLBench-2 (Grotz al, 2024). and real-world evaluations. Each entry lists the task abbreviation, full name, and brief description.
Abbr.
Full name
Description
RoboTwin-2 (Chen al.
2025
(Multitask Learning)
Bottle
Pick Dual Bottles
Pick up one bottle with one arm, and pick up another bottle with the other arm.
H/O
Handover Block
Use the left arm to grasp the red block the table, handover it to the right arm and place on the blue pad.
Pot
Lift Pot
Use both arms lift the pot. Use dual arm pick the hamburg and frenchfries and put them onto the tray.
Burger
Place Burger Fries
Skillet
Place Bread Skillet
There is one bread on the table. Use one arm to grab the bread and the other arm to take the skillet. Put the bread into the skillet. Use arm open the cabinet's drawer, and another arm to put the object on the table to the drawer.
Cab.
Put Object Cabinet
RoboTwin-2 (Transfer Learning)
Div.
Pick Diverse Bottles
Pick up one bottle with one arm, and pick up another bottle with the other arm. Use one arm to grasp the microphone on the table and handover it to the other arm.
Mic.
Handover Mic
Roller
Grab Roller
Use both arms grab the roller on the table. Use dual arm to pick and place cans into plasticbox.
Box
Place Cans Plasticbox
RLBench-2 Grrttz et al. 2024)
Tray
Lift Tray
The robot's task is to lift tray that is placed holder. An item on top of the tray and must be balanced while both arms lift the tray.
Rope
Straighten Rope
The robot's 's task is straighten rope by manipulating it that both ends are placed into distinct target areas.
Oven
Take Tray Out of Oven
The robot's task is to remove a tray that located inside an oven. This involves opening the oven door and then grasping the tray. The robot's task is to sweep the dust into the dust pan using broom. This involves coordinating the sweeping motion to ensure the dust is effectively collected.
Sweep
Sweep Dust Pan
Real-World Tasks
Remove pen cap
On pen placed on the table. Use on arm to pick the pen and the other arm to remove the cap.
Put cups in bowl
Two cups are placed on the table. Sequentially pick them and place them into the bowl. A tool is placed handover to the other arm.
Handover screwdriver
to pick the tool and One bowl and a cup of beans are placed on the table. Use one arm to grab the cup and the other arm grab the bowl. Then pour the beans into the bowl.
the table. Use
Pouring Beans
select tasks prioritizing scenarios that demand tight, real-time cooperation between the two arms (e.g., coordinated grasp-and-handover, cooperative manipulation under constraints, dual-arm assem- bly). Focusing on such challenging, interdependent tasks intentionally enlarges the skill base: poli- cies must discover and reuse shared coordination primitives spatial alignment, contact-rich syn- ergies, role switching, and timing- across tasks, which in turn improves generalization and overall performance.
18

---

Published as a conference paper at ICLR 2026
Saliet
Ptt
Swap
Bage
The
RoboTwin-2
RL R.Bexhh
Figure 6: Examples of Simulated Experiments
Transfer Learning. For transfer-leaming study, we select 4 bimanual tasks from RoboTwin-2 We first pre-train single policy on the six tasks used in the multitask setting, then fine-tune that policy on each target task separately. The four targets are chosen to be skill-adjacent to the multi- task set set-riigg similar coordination primitives while introducing new features such as different object geometricss/attrials. spatial layouts, or action sequences. This design directly tests each method's ability to transfer by reusing previously acquired skills, evaluating both zero-shot general- ization and fine-tuning efficiency alongside final performance.
To further assess skill decomposition and cross-task re-composition, construct two composite tasks on top of Twin2::: Put Fries into Skillet and Put Bottle into Cabinet. Both are recombina- tions of base tasks used in our multitask training-Put Fries into Skiller combines Skillet and Fries, while Put Bottle into Cabinet combines Bottle and Cabinet. During training on these composites, freeze the learned experts and bases and only finetune the gating network, which enables a clean evaluation of each method's ' ability to decompose and reuse skills.
In the main part, their full names abbreviated for brevity; Tab. lists the abbreviation-full-nane mapping, brief descriptions, and the required level of bimanual coordination.
B.1.2 REAL- WORLD EXPERIMENTS
To broaden evaluation beyond simulation and stress-test robustness under sensing noise, calibration drift, and actuation latency, design and execute suite of 4 real-world bimanual tasks- including remove pen cap, put cups into bowl, hand over screwdriver, and pour beans into bowl. These tasks demand tight inter-arm cooperation (role allocation, handoff timing, compliant contact) and precise motion control, making them deliberately challenging.
For each task, we collect 50 expert demonstration trajectories via human teleoperation. Training and evaluation follow a multitask regime: methods are trained for 2000 epochs on the union of demonstrations, with an explicit task identifier provided the policy. All experiments are conducted on platform of 2 PiPER arms. Detailed descriptions are provided in Tab. [5] The progress score reported Fig. calculated based on the progress of task completion. And the progresses are defined as follows.
Remove pen cap
1. Progress 1/4: Left hand grasps pen.
2. Progress 2/4: Left hand lifts up pen.
3. Progress 3/4: Right hand grasps pen cap.
4. Progress 4/4: Pen cap is removed.
Put cups in bowl
1. Progress 1/4: Grasp and lift the left cup.
2. Progress 2/4: Place the left cup in the bowl.
3. Progress 3/4: Grasp and lift the right cup.
4. Progress 4/4: Place the right cup in the bowl.
Handover screwdriver
Progress 1/4: Left hand grasps the screwdriver.
2. Progress 2/4: Left hand lifts up the screwdriver.
19

---

Published as a conference paper at ICLR 2026
3. Progress 3/4: Right hand grasps the screwdriver.
4. Progress 4/4: Left hand releases the screwdriver.
Pouring beans
1. Progress 1/5: Left hand grasps the cup with beans.
2. Progress 2/5: Left hand lifts up the cup and moves to the central area.
3. Progress 3/5: Right hand grasps the bowl.
4. Progress 4/5: Right hand lifts up the bowl and moves the central area.
5. Progress 5/5: Left hand pours the beans into the bowl.
B.2 BASELINES
We select strong, representative baselines to ensure persuasive comparisons. To use them in our multi-task bimanual setting, we apply only minimal adaptations while preserving each method's core design.
DP and DP3. Diffusion Policy (DP) (Chi al. 2023): a conditional diffusion visuomotor policy that predicts action sequences from observations; strong supervised IL baseline. 3D Diffusion Policy (DP3) (Ze et al., 2024): DP extended with 3D/SE(3)-aware modeling from point clouds, representative for precise spatial reasoning.
ACT. Action Chunking with Transformers (ACT) (Zhao et al. 2023): learns variable-length action primitives ("chunks") with transformer policy representative temporal-abes -bbtrractoot baseline for long-horizon control.
RDT. Robotics Diffusion Transformer (RDT) (Liu et al. for sequence-leve action generation with a large size of 1.2B; representative modern baseline for multitask robot manipulation foundation model.
2024):
architecture
Discrete Policy. Discrete Policy (Wu et al. 2025) is an info-based skill-abstraction method that learns discrete latent codebook of action "skills" via VQ-VAE and uses conditional latent diffu- sion model to generate task-specifi codes; the discrete bottleneck encourages skill disentanglement and makes DP representative multi-tass baseline (including bimanual settings).
SDP. Sparse Diffusion Policy (SDP) (Wang et al. 2024 is representative MoE-based multitask imitation-leaming method that performs skill abstraction and reuse via sparse gating. Concretely, inserts layer-wise FFN experts inside the diffusion backbone and activates a small subset per step. Unlike our SMP, which decomposes the entire policy into separate experts (each a standalone pol- icy/generator over a skill-specific subspace), SDP applies MoE only within network blocks-i.e. the final action is produced by a single diffusion head rather than an ensemble of independent policies.
B.3 SMP IMPLEMENTATION DETAILS
Optimization. We optimize all models with AdamW (Loshchilov & Hutter 2019), using learn ing rate of x 10-5, The batch size is 128 for Robotwin-2 and real-world experiments, and 6 for RlBencc-22. We use an observation horizon of steps and a planning horizon of 8 action steps. The SMP employs K experts with an activation threshold = 0.95.
Architecture. SMP comprises: (i) a shared observation encoder; (ii) posterior and prior gating networks, g(g: Sc, ar) and p(g: st). that produce expert weights; (iii) basis generator W(x);; and (iv) K per-expert diffusion generators that output coefficients 2,. At inference, experts with cumulative posterior/prior weight above Tm are activated (with a top-1 fallback none exceed the threshold). and their outputs are combined via the learned basis.
Components. Observation encoder: Following the standard observation encoder used in ResNet-
diffusion- based visuomotor policies (e.g., Diffusion Policy Chi al. (2023)). 18 that processes the RGB inputs together with the robot state and outputs shared feature vector St, which is then used input to the gating network, the state-adaptive skill-basis network, and the
20

---

Published as a conference paper at ICLR 2026
diffusion experts. Gating and basis: both 4(9e | St. 0z). p(g: st), and ((s)) are implemented as lightweight MLPs for simplicity and speed: (se) maps the state to a task-adaptive set of bases used to synthesize the final action from expert coefficients. Experts: each expert diffusion generator following the CNN architecture Diffusion Policy (Chi et al. 2023). with reduced channel widths control model size.
Computation cost. All SMP models are trained end-to-end on a single NVIDIA A6000 GPU. For the multi-task experiments Robo-Tinn2.0 and RLBench-2, train for 3000 epochs, which corresponds to roughly 20-25 hours of wall-clock time. The state-dependent orthonormal skill basis learned jointly with the gating network and diffusion experts; no additional pre-training or separate optimization stage is required beyond this standard training procedure.
To make the sampling cost more transparent, Table o reports the number of parameters and the measured per-step inference time of each main component on the same A6000 GPU:
Table 6: Parameter count and per-step inference time of each SMP component.
Module
# Parameters
Inference time
Vision encoder
1 11.2M
23 ms
Gate
3.6M
10ms
Skill-basis network
12.5M
24 ms
Diffusion expert (single)
28.9M
74 ms
Obs Encoder
sw 717-
sw v2-
mm DL-
Basis
Gate
select
experts
Expert
Expert
parallel computations
Figure 7: Inference pipeline of SMP. The observation encoder first maps the input observation to shared feature, which is then fed to the state-dependent skill-basis network and the MoE module (gate and selected diffusion experts). The basis and experts are evaluated in parallel, with the gate adding only a small overhead. The annotated values indicate the measured per-step runtimes cach component NVIDIA A6000 GPU, which sum overall inference time of approximately 107 ms per control step.
As illustrated in Fig. 7 the observation encoder is evaluated per control step and its output feature is shared by all subsequent modules. Conditioned on this feature, the skill-basis network and the diffusion experts are evaluated in parallel, while the gating network selects the active experts with a relatively small additional cost. The resulting end-to-end inference time of SMP is approximately 107 ms per control step, and the overhead introduced by learning and using the state-dependent skill basis is moderate compared to the diffusion experts.
C ABLATION STUDIES
C.1 ABLATION OF STICKY GATE FUNCTION
We briefly recall that SMP regularizes the skill gates g, A-1 with sticky Dirichlet-Markov prior
1 2~ Dir(a 1), 91 Dir(an 0), gi Di((Kg-111 0) 1 2, (42)
21

---

Published as a conference paper at ICLR 2026
Table Success Rates of Ablation Studies
Multi-task Learning
Transfer Learning
Methods
RoboTwin-2
RLBench-2
Few-shot Learning
Skill Composition
SMP (ours)
0,54
0.18
0.38
0,30
W/o sticky gate
0,44
0.15
0.33
0.26
W/o adaptive expert
0.53
0.18
0.37
0.29
Linear mass adapt.
0.52
0.17
0.37
0,29
Fixed skill basis
0.40
0.14
0.31
0.24
PCA skill basis?
0.32
0.11
0.26
0.20
Results are averaged over all tasks. Use top-k = as default. Change the mass definition of in Directly use PCA to abstract the action
adaptive expert activation linear function m, 9t, space as fixed basis.
and train with closed-form Dirichlet KLs (Eq, 8 Intuitively, shapes the global usage vector 0, oo controls how strongly each gate is pulled toward 0, and x controls the temporal stickiness gr around 9t-1- Unless otherwise stated we use (a, Cku, k) (2.0, 0.5, 20.0).
To assess the importance of the sticky prior itself, we first remove it entirely and train a variant where a feedforward network predicts Jr at each timestep without any Dirichlet-Markov structure or global-usage variable (no KL. terms on or 1) As reported in Table (row "Woo sticky gate"), this leads to clear degradation in performance: on RoboTwin-2 multi-task learning, the success rate drops from 0.54 (SMP) to 0.44, with similar declines on RLBench-2, few-shot transfer, and skill composition. Qualitatively, this model exhibits high-frequency switching between experts and less interpretable gate trajectories, indicating that temporal stickiness and global usage regularization are both important.
We then study the effect of each hyperparameter in Eq. 42 For each of a, ag,, and K, we sweep over five values while keeping the other fixed at their default:
a E {0.1, 0.5, 2.0, 5.0, 10.0),
Or E (0.0, 0.1, 0.5, 1.0, 2.0).
A E {0, 5, 20, 50, 100).
The resulting RoboTwin-22 success rates are shown in Fig. 8 Across all three sweeps, performance peaks near the default values (red dots) and degrades when the prior is either too weak too strong. Very small a or Ogn encourage collapse onto a few experts, while very large values over-regularize the gates toward uniform global usage. For K, the worst is K 0 (no stickiness), corresponding to the "W/o sticky gate" variant, whereas moderate stickiness (ı s 20-50) yields the best results; extremely large & makes gates overly inertial and slightly reduces performance.
Sensitiity to (sticiness)
Sensitiiity (giobal usage grior)
Sensitiviy strength
defuu
412
dofbult
1.0
400
300 420
Figure 8: Sensitivity of sticky-gate hyperparameters on RoboTwin-2 multi-task success. Each curve varies single parameter while keeping the others fixed at (ox, og. x) = (2.0, 0.5, 20.0). Left: sweep over i (stickiness). Middle: sweep over a (global usage prior). Right: sweep over ag (anchor strength). Red dots mark the default setting used in all main experiments.
Overall, these ablations show that the sticky Dirichlet-Markov gate an essential but not overly fragile component of SMP: removing it significantly harms performance, while within the sticky family the method robust around the chosen defaults and performs best when global usage, an- choring, and stickiness all set to moderate levels.
22

---

Published as a conference paper at ICLR 2026
C.2 ABLATION OF ADAPTIVE EXPERT ACTIVATION
SMP does not activate all experts at every timestep. Given the router mean 9r E A we define a per-expert mass m, = gi, and greedily add experts in descending m, until the cumulative mass exceeds a coverage threshold Tm or a hard top-k budget is reached. This adaptive expert activation lets the policy use more experts on difficult states and fewer on simple ones; on RoboTwin-2, the default setting Tm 0.95 activates average R 2.3 experts per step.
We first remove this mechanism and always activate a fixed number of experts, setting top-k at all timesteps ("Woo adaptive expert" in Table 77. All other components (state-adaj) -ddativee basis, sticky gates, diffusion experts) are unchanged. This variant performs slightly worse than full SMP on both multi-task and transfer benchmarks (e.g., RoboTwwin22 success drops from 0.54 0.53), showing that allowing the number of active experts to vary with the state gives a small but consistent benefit.
Next we keep the adaptive selection rule but change the mass definition from the quadratic form my; = Zh. to a linear one m, 9r.s ("Linear mass adapt." in Table 7) This also underperforms SMP Twino success 0.52), suggesting that the quadratic mass helpful: it emphasizes high- confidence experts and suppresses mediocre ones, yielding sharper active sets, whereas the linear mass keeps medium gates relatively heavy and tends to recruit more partially relevant experts.
Thp-c abtet on: suocees ut number artive accerts
Theeshold stration ave age octive experts 45 Eo.
Thresho ui abilt on sU000s5
0.34
SMP (eraptivee
0.54
590
0.52
0.52
0.30
0.48
0.48
2.46
0.46
0.02 2.01
0.52
0.25
0.04
0.25
0.50
0.31
0.92
0.02
0.04
0.50
top k.0o2ct rumber octive exports)
Coverage threshes
Coverage
Figure 9: Ablation of adaptive expert activation on Robo- Left: success rate when fixed number of experts is used (hard top-k, no coverage rule). Middle: average number of active experts versus coverage threshold Tm under adaptive activation; the SMP default (Tm. 0.95) uses about 2.3 experts on average. Right: success rate versus Tm, showing performance improving as more experts are recruited and saturating near the SMP configuration.
Finally, we perform a more fine-grained sensitivity study over (i) the hard top-k and (ii) the coverage threshold Tm. In the first sweep we disable coverage and activate exactly k E 2,, 4, 5) experts per step. The left panel of Fig. shows that success on Robo increases monotonically from 0.45 (top-1) to 0.54 (top-5). with large gap between k 1,2 and h 2 3, and saturation once k 2 4, which matches the full SMP model.
In the second sweep we restore adaptive activation and vary the coverage threshold while keeping all other settings fixed. As shown in the middle and right panels of Fig. 9 increasing Tm from 0.90 to 0.95 increases the average number of active experts from about 1.4 to 2.3, while the success rate rises from roughly 0.46 and smoothly saturates around 0.54. When Tm is too small, the model under-activates experts and behaves similarly to low top-k; beyond the default, adding more experts yields little gain. Overall, these ablations indicate that SMP benefits from recruiting a small but state-dependent set of experts, and that our default choice of Tm lies in regime where performance is high while the number of active experts remains modest.
C.3 SKILL BASIS SELECTION
In SMP, actions represented in state sepeeddenn orthonormal skill basis B(s:) E Ra* that the decoded action is a, B(s:) (g. @ 24). This basis changes smoothly with the robot state and task context, and its columns typically align with meaningful motion primitives such as translations and rotations of the end-effector. In multi-task settings, this state adaptivity is important: the "same" high-level skill (e.g., pushing, lifting, rotating) must manifest differently depending on arm pose, contact configuration, and the current object.
To assess the importance of this state dependence, we first replace the state-based basis with a static skill basis that does not depend on 8t. Concretely, we learn single global orthonormal matrix B and
23

---

Published as a conference paper at ICLR 2026
use it for all states and tasks, while keeping the gates, diffusion experts, and activation mechanism unchanged. As shown in Table (row "Fixed skill basis"), this severely degrades performance: on RoboTwin-2 the multi-task success rate drops from 0.54 (SMP) 0.40, and the decline is more pronounced in transfer settings (few-shot learning and skill composition). This suggests that state-free basis cannot align skill directions with the local manipulation geometry and forces the MoE to work in poorly matched coordinate system.
We then consider PCA-based variant ("PCA skill basis" in Table where we compute K principal components of the action dataset and fix B to this global PCA basis. This also performs poorly and, in our experiments, can come close collapse: multi-task performance on RoboTwin-2 falls further to 0.32, with similar drops on RLBench-2 and transfer tasks. Although PCA provides an orthogonal low-rank decomposition, is global and task-agnostie; the resulting basis cannot adapt to specific tasks or phases, so different behaviors are entangled in the same components. Together, these ablations confirm that a state-dependent skill basis ((x,) crucial for disentangling skills and achieving strong performance in multi-task and transfer robot manipulation.
C.4 SCALING DP-STYLE BASELINES
remaining question is whether the lower performance of DP-style baselines in multi-task settings primarily due paddeeraraatteiizaaiion, and whether scaling them similar capacity would close the gap to SMP. To provide direct empirical evidence, we conduct controlled scaling experiments on DP, DP3, and ACT under the same RoboTwin-2 multi-task learning protocol, keeping the data, training schedule, and evaluation procedure fixed, and only increasing model capacity. In addition, we report inference time to quantify the accuracy-efficieng trade-off as capacity grows.
Scaling DP/DP3/ACT to ~300M parameters. We scale each baseline to approximately 300M parameters (roughly 2x the original size) and compare the average multi-task success rate (SR) and inference time 2obo'Twin-2.
Table Effect scaling DP-style baselines to ~300M parameters on RoboTwin-2 multi-task learn- ing. We report success rate (SR) and inference time 4.
Method
Original SR
Scaled (~300M) SR
Original Time
Scaled Time
DP
0.29
0.37
120 ms
160 ms
DP3
0.33
0.40
122 ms
167 ms
ACT
0.34
0.42
94.8
135 ms
Scaling improves all three baselines, confirming that capacity matters. However, even at ~300M parameters, these models remain below SMP (0.54 SR). Moreover, scaling increases inference time for all baselines. In contrast, SMP attains higher success with sparse activation (258M total parame- ters but only 80M activated at inference) and 107 ms inference time, yielding more favorable accuracy efficiencyy trade-off in multi-task multimodal manipulation.
24


Reviews:
[{"review_id": "7ekBYU8Lsq", "reviewer": "Reviewer_fTZq", "summary": "The paper proposes a mixture of experts framework for diffusion-based policy learning in multi-task training scenario. The main difference from a \"standard MoE model for score function\" approach is that the authors introduce a state-dependent orthonormal action basis, where each vector in the basis represents a skill. The weights per skill are predicted by skill-specific diffusion experts. A gating function is used to activate a set of experts and ensure that the gating weights do not change significantly for consecutive inferences. Strong results are shown on multi-task bimanual manipulation scenarios.", "questions": "See weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper presents an intuitive method for learning skills implicitly for multi-task scenarios. The engineering in choosing the suitable method to construct the orthonormal basis, training targets for skill-specific diffusion experts, and sticky gating function is well executed.", "weaknesses": "Overall the major concern I have is that the paper do not provide any ablation results for the specific design choices like:\n\n1. How does the results change without the sticky gating function? Since this is one of the major contributions, it will be worth looking at how this imapcts simpler MoEs like Sparse DP.\n2. How does the results change with k or the mass threshold? Linear mass vs quadratice mass?\n3. The practical implementation subsection in the appendix suggests that the proposed method requires significant tuning to avoid collapse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FNaYYpyKwW", "reviewer": "Reviewer_aj4r", "summary": "This paper introduces a diffusion-based MoE framework designed for reusable motion primitives across multiple tasks. It features in orthonormal skill basis and sticky routing that reduce gate switches and activated experts. Abundant experiments have been conducted both in simulation and real world with ablation studies supporting the design choices. Overall I think this is a solid submission for ICLR.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-organized and easy to follow.\n2. The idea of orthogonal skill basis seems very attractive, based on the ablation studies it does provide good cross-task skill reuse with few switches.\n3. The ablation studies are abundant and helpful in supporting the claimed contributions.\n4. Real-world experiments looks good.", "weaknesses": "1. I am curious about the training cost of learning good orthonormal skill basis, more details on this part will be appreciated.\n2. In Figure 3 task \"Put bread into skillet\" and \"Lift tray with block in it\", there are still considerable portion of the trajectories with multiple experts activated at the same time with similar gate values, does it mean in these cases the experts are still overlapping?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xPgRuyedAv", "reviewer": "Reviewer_kdN5", "summary": "This paper proposes Skill Mixture of Expert Policy (SMP), which learns a compact orthogonal skill basis, trains a mixture-of-experts diffusion policy on each basis, and employs a sticky routing mechanism. The main idea lies in decomposing the action space into orthogonal bases using differentiable QR factorizations to learn such bases, and establishing Dirichlet–Markov dynamics to control the gating. The results surpass the baselines and demonstrate a clear skill routing phase.", "questions": "1. In Appendix B.3, the vision feature is concatenated with the robot state and projected into a shared feature o. Do the gating and basis modules also use this shared feature, or only the robot state? Additionally, could the authors provide details on the number of parameters and inference time for each component—namely, the vision encoder, gating module, basis, and diffusion experts?\n2. Could the authors visualize the learned orthogonal basis throughout task execution? Figure 3 shows the basis directions, but they appear consistent within the same task execution, suggesting they may not vary with state.\n3. It is quite common to apply PCA to extract principal components in dexterous manipulation and vehicle trajectory prediction, where PCA also learns an orthogonal basis. What advantages does learning a state-dependent basis provide compared to PCA or other fixed orthogonal decompositions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes learning a lightweight basis in the action space and introduces a novel method for learning the gating mechanism. The experiments demonstrate the effectiveness of this approach in terms of success rate and inference computation (activated parameters). The paper also presents explainable and stable routing phase transitions, such as rotation and translation.\n2. This paper exhibits several appealing features resulting from its decoupled structure (action basis, routing, and coefficients on the basis), including flexible inference sampling strategies and skill composition capabilities.", "weaknesses": "1. The paper claims that “a fixed basis may fail to capture such variability.” However, there is no comparison provided between a fixed basis and a learned basis. Moreover, from Figures 2 and 3, the action bases appear to be fixed and simple—such as left/right translation and rotation—suggesting that a fixed basis might be sufficient. How does the learned basis actually change with state?\n2. In Equation (3), there are three hyperparameters. How do these parameters influence the final results?\n3. Section 4.1 lacks sufficient explanation. For example, the equation is correct only under certain conditions, and the variable B does not appear in the right hand side. It would be better to clearly state the underlying assumptions and derivations.\n4. Section 4.2 mentions top-k or coverage selection. What is the comparison between these two approaches? As the number of active experts increases, does the performance consistently improve?\n5. The paper introduces several mathematical notations—such as simplex weights and Dirichlet–Markov dynamics—but does not explain them clearly. Including more mathematical details or visual illustrations would help improve clarity.\n6. The paper would benefit from clearer writing and more detailed mathematical explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eViGd0jXMI", "reviewer": "Reviewer_zQHP", "summary": "The paper introduces SMP, a diffusion-based Mixture-of-Experts (MoE) policy designed to improve efficiency and generalization in multi-task manipulation. Unlike the existing MoE approaches, where the experts are entangled, the proposed method learns to route a subset of expert skills for each task, thereby reducing inference cost and enabling transferable skill learning across tasks. The authors provide theoretical analysis supporting the framework and conduct extensive experiments in both simulated and real-world environments to demonstrate its effectiveness. Overall, the paper contributes a novel integration of diffusion models and expert routing mechanisms for scalable and generalizable robotic control.", "questions": "- For the inference time results in Table 2, your approach, SMP, uses the fewest active parameters during inference (80.2M), but its inference time (107.3 ms) is not the smallest; the ACT baseline is faster (94.8 ms). Do you have any insight on this? Is this slight overhead caused by other parts of your model, such as the router, the state-adaptive basis generation, or the final action composition step?\n- The authors stated that the DP, DP3, and ACT baselines \"underfit the multimodal distributions\". In Table 2, the total number of parameters for these baselines (83.9M - 132.5M) is much smaller than the total parameters for your SMP method (258.9M). Is it possible that this \"underfitting\" is simply because the baseline models are under-parameterized in training space? What would you expect the success rate to be if these baselines were scaled up to have a similar total parameter count as SMP?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method is theoretically well-grounded, using a principled variational objective to learn a state-adaptive orthogonal skill basis.\n- The paper's core innovation is combining this orthogonal basis with \"sticky\" routing dynamics. This is a novel approach that directly addresses prior limitations of skill entanglement and unstable, \"chattering\" gates.\n- The approach shows significant empirical success, achieving higher multi-task and transfer learning performance than strong baselines.\n- The method demonstrates markedly lower inference cost and latency by adaptively activating only a small subset of experts, highlighting its practical value for real-time control.", "weaknesses": "- The paper claims that a fixed basis \"may fail to capture\" action variability, which is the main justification for using a more complex state-adaptive basis. It would be better to include an ablation study comparing the proposed method to a version with a fixed basis. This would provide stronger evidence for this key design choice.\n- The paper introduces some advanced mathematical concepts, like \"thin-QR retraction with sign stabilization\" and \"sticky Dirichlet Markov dynamics\". These might be a little unfamiliar to some readers. It would be helpful to add a brief intuitive explanation or a few more citations in the main text to help people understand the paper more smoothly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)