VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routing
Authors: Yixiao Wang
Mingxiao Huo
Zhixuan Liang
Yushi Du
Lingfeng Sun
Haotian Lin
Jinghuan Shang
Chensheng Peng
Mohit Bansal
Mingyu Ding
Masayoshi Tomizuka
Venue: ICLR 2026 Poster (2026)
Keywords: Robot Learning
Distillation
Vision Foundation Models
Primary area: applications to robotics, autonomy, planning

Abstract: Pretrained vision foundation models (VFMs) advance robotic learning via rich visual representations, yet individual VFMs typically excel only in specific domains, limiting generality across tasks. Distilling multiple VFMs into a unified representation can mitigate this limitation but often yields inflexible task-specific feature selection and requires costly full retraining to incorporate robot-domain knowledge.
We propose VER, a Vision Expert transformer for Robot learning. During pretraining, VER distills multiple VFMs into a vision expert library. We then fine-tune only a lightweight routing network (fewer than 0.4% of parameters) to dynamically select task-relevant experts from the pretrained library for downstream robot tasks. We further introduce Patchwise Expert Routing with Curriculum Top-K Annealing to improve both flexibility and precision of dynamic expert selection. Moreover, VER supports parameter-efficient finetuning for scalable expert utilization and robot-domain knowledge integration. Across 17 diverse robotic tasks and multiple policy heads, VER achieves state-of-the-art performance. We find that VER reduces large-norm outliers in task-irrelevant regions (e.g., background) and concentrates on task-critical regions. More visualizations and codes are available in https://yixiaowang7.github.io/ver_page/.

Published as a conference paper at ICLR 2026
VER: VISION EXPERT TRANSFORMER FOR ROBOT LEARNING VIA FOUNDATION
DISTILLATION AND DYNAMIC ROUTING
Yixiao Wang', Mingxiao Huo", Zhixuan Liang", Yushi Du', Lingfeng Sun', Haotian Lin', Jinghuan Shang". Chensheng Peng', Mohit Bansal Mingyu Ding Masayoshi Tomizuka' UC Berkeley CCaneeiie Mellon University 3 University of Hong Kong
*Peking University
*Stony Brook University
PUNC-Chapel Hill
ABSTRACT
Pretrained vision foundation models (VFMs) advance robotic learning via rich vi- sual representations, yet individual VFMs typically excel only in specific domains, limiting generality across tasks. Distilling multiple VFMs into unified represen- tation can mitigate this limitation but often yields inflexible task-specific feature selection and requires costly full retraining to incorporate robot-domain knowl- edge. We propose VER, a Vision Expert transformer for Robot learning. During pretraining, VER distills multiple VFMs into vision expert library. We then fine- tune only a lightweight routing network (fewer than 0.4% of parameters) to dy- namically select task-relevant experts from the pretrained library for downstream robot tasks. We further introduce Patchwise Expert Routing with Curriculum Top- K Annealing to improve both flexibility and precision of dynamic expert selection. Moreover, VER supports parameter- warameter-efficieet finetuning for scalable expert utiliza- tion and adaptive robot-domain knowledge integration. Across diverse robotic tasks and multiple policy heads, VER achieves state-of-the- art performance. We find that VER reduces large-norm outliers in task-irrelevan regions (e.g., back- ground) and concentrates on task-critical regions. More visualizations and codes are available in unthtps:: http::/yyxxxiwwaao.. www.hu.............. caruub.ioovveee io/ver_page/
INTRODUCTION
Developing robotic systems capable of perceiving and interacting with complex, unstructured en- vironments remains a fundamental challenge in embodied AI. Recently, visuomotor robot policy learning has emerged as promising approach, enabling robots to directly map visual observations to control actions. Pretrained vision foundation models (VFMs) such as DINOv2 (Oquab et al., 2024], CLIP (Radford et al.) [202), and ViT (Dosovitskiy et al. (020), provide transferable visual representations that support robotic perception and control with certain generalizability, improving the scalability of robotic systems (Huang et al., 2024 Wan et al. 2024).
However, executing even a single robotic task, and especially a diverse set of tasks, often requires multiple implicit visual competencies that single VFM cannot fully capture. Directly integrat- ing multiple VFMs for robot tasks increases computational and operational complexity. Previous works (Ranzinger al. 2024 Shang et al.) 2024 Chen et al. 2025) distill diverse foundation mod- els into a unified representation, but three key challenger remain. First, heterogeneous VFM features often misaligned, so unified representation tends to dilute discard model-specific capabili- ties. Second, the policy head must extract task-relevant information from the fused representation, which limits flexibility to leverage the most relevant VFMs across tasks and leads suboptimal results. Third, existing distilled models typically require full retraining to incorporate robot-domaa knowledge and it hard to scale computation (down for simple tasks and up for complex tasks).
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
Selection
Exprrt Library
Disillation
Distillttin
VFM
VPM
Learning
Policy Learaing
Vision
Pelicy
Robot
Visinn Transformer
Policy Head
Rebot tasks
Tranformner
(VTT)
Add
adaptation
adaptation
Figure 1: A comparison between our VER and previous distillation framework. Our method not only enhances knowledge distillation from vision foundation models (VFMs) into vision experts but also offers two key advantages over previous works (Ranzinger al. 2024 Shang al. 2024). First, VER trains a lightweight router that dynamically selects vision experts for downstream robot policies. Second, VER allows the integration of additional trainable experts, enabling the adaptation to robot-specific domain knowledge to further improve robotic performance.
To address these limitations, we propose VER, a Vision Expert transformer for Robot learning via foundation distillation and dynamic routing. VER distills knowledge from multiple vision founda- tion models into a unified representation library and uses a dynamic routing mechanism to selec- tively activate the most relevant experts for robot policy learning. Specifically, VER introduces Eexture-of-Expertt (MoE)-based Vision Expert Library (VEL), replacing traditional static vision transformer backbones with collection of specialized experts, each capturing distinct aspects of visual understanding. This design enables robots to selectively leverage the specialized experts best suited for task-aware policy learning.
Our method operates in three stages as shown in Fig. D First, during pretraining, we distill knowl- edge from multiple VFMs into a vision expert library using Teacher-Specific Routers with mutual- information regularization. This covers a broad spectrum of visual knowledge while maintaining efficiency via sparse expert activation. Second, in the robotic policy learning phase, we freeze all pretrained vision experts and fine-tune only a lightweight Robot Router that dynamically selects task-relevant experts, whose outputs are fed to policy head to generate actions. To expand selec- tion capacity across patches and layers, enhance exploration, and prevent premature convergence to suboptimal expert combinations, we employ Putchwise Expert Routing with Curriculum Top-K Annealing. leading to more robust policy learning. Third, we offer parameter-efficiee fine-tuning strategies that scale expert utilization and facilitate the integration of robot-domain knowledge.
Across different types of policy heads, such as diffusion and flow matching policies (Chi ct al. 2023 Zhang & Gienger 2024), extensive experiments on diverse robotic benchmarks show that VER achieves state-of-the-arr performance. With Patchwise Expert Routing and Curriculum Top-K An- nealing, VER suppresses high-norm background outliers and reduces information in task-irrelevant patches while preserving details in task-critical regions, yielding more compact and diseriminative visual features and robust policy learning.
RELATED WORKS
2.1 VISION FOUNDATION MODELS FOR REPRESENTATION
Vision Foundation Models (VFMs) have revolutionized computer vision through self-supervised and seakkly-upervised learning on large-scale datasets (Radford et al., 2021 Caron et al. [021]] Oquab et al., 2024). Notable examples include CLIP (Radford al.) 2021 which pioneered image- text joint embeddings. DINOv2 (Oquab et al. 2024 which advanced self-supervised learning, and SAM (Krrillov et al. 2023) specialized for segmentation tasks.
Knowledge distillation has emerged as powerful paradigm for transferring learned representations from large teacher models to more compact student architectures (Hinton et al.) 2015 Romero al., 2014). While traditional distillation approaches focus on compressing single teacher into smaller
student (Hinton et al.] 2013), recent advances have explored multi-teacher distillation (You et al.
2017 Shang et al. (2024) where complementary knowledge from multiple source models com- bined. Theia (Shang et al. 2024) demonstrated that careful fusion of representations from diverse vision foundation models can achieve superior performance for downstream robotic tasks. However,
2

---

Published as a conference paper at ICLR 2026
Pretraining
Expert Library
Unitted Reproratation
TeadherSpeciite Rester
Experts
Datilation
Large image detaeet
DINO
Base Yiiio
Transfeener (BVT)
vI
CLIP
Trainable
Prosm
Downstreum Robot Tasks
Previom robot detaset
Unified
Stractured
Vidon Expert Libeary
Rerreentation
Frove Fxperts
Artions
Trinnhle Bobal Rester
Pulicy
Head
experts
superts
Figure 2: Overall structure of VER. VER comprises two key components: the Base Vision Trans- former (BVT), which processes images into unified representations; the Vision Expert Library (VEL), which stores a diverse set of specialized vision experts and selectively utilizes the experts mimic teacher vision foundation models and enhance performance downstream robotic tasks. Our framework consists of two phases: (1) Pretraining, where we distill multiple foundation models
(DINOv2 (Oquab et al., 2024), ViT (Caron al. 2021), CLIP (Radford et al. 202T) into VER; (2) Downstream Robotic Tasks, where freeze the experts and train lightweight Robot Router (< 0.4% parameters) that dynamically selects task-relevant visual features to guide the policy head in generating appropriate robotic actions. This two-stage approach enables efficient knowledge dis- tillation from diverse vision foundation models and adaptive feature selection for robotic tasks.
these methods typically produce static representations with fixed weights, limiting their adaptability to specific downstream tasks.
Our work differs by distilling multiple VFMs into a specialized expert library rather than a sin- gle unified representation, preserving diverse knowledge from VFMs while enabling task-specific feature selection through learned routing mechanisms.
2.2
MIXTURE OF EXPERTS VISION AND POLICY
Mixture of Experts (MoE) architectures (Shazeer et al. 2017a Fedus et al. 2022 Riquelme et al,,
2021
Wang al. 2024) have gained popularity for their ability scale model capacity without proportional increases in computational costs by activating only subset of expert networks for each input.
Router design represents a critical component in MoE systems, determining which experts process specific inputs. Top-k routing (Shazeer al. (001aa) selects the k highest-scoring experts for each token, while Switch Transformers (Fedus et al. 2022) employ simpler top- routing for efficiency.
Recent work has explored learned routing mechanisms (Dai et al. 2022, Wang al.. 2024) that balance expert utilization while preserving specialization. Sparse MoE Router (Wang et al. 2024) introduced mutual information maximization between tasks and experts to encourage meaningful specialization while maintaining balanced utilization.
While MoE has been widely applied in language processing and general computer vision (Riquelme) ct al. 2021 Fedus et al. 2022). its application robotic learning remains relatively unexplored. Our work bridges this gap by adapting MoE principles for vision-based robotic policy learning, in- troducing specialized routers that dynamically select visual representations most relevant specific robotic tasks.
3

---

Published as a conference paper at ICLR 2026
2.3 VISUOMOTOR ROBOTIC POLICY LEARNING
Visuomotor robotic policy learning maps visual observations directly to robot actions
(Levine et al.,
2016 Brohan al. 2023 Liang al. 2024a Mu et al., 2024 Chen al.
2024), showing bet-
Ter generalization ability compared to state state-based methods (Janner et al.) 2022 Liang et al. 2023
Ajay et al.; Ni et al. 2023;
Liang et al. 2024b). Recent approaches have leveraged pre- vre-trained
sion models to improve sample efficiency and generalization (Nair ct al. 2023] Xiao et al. [022
Radosavovic et al., 2023] Chen et al., 2025 Kim al. Black et al. 2024). but typically use fixed visual encoders that may not capture optimal representations for specific tasks. A persistent chal-
lenge is identifying which visual features are most relevant for different robotic tasks (Xiao et al.)
Luo et al.) 2023). Current methods using attention mechanisms (Luo et al. 2023) or feature selection (Jiang et al. 2025) often lack the flexibility to incorporate diverse visual expertise from foundation models. Our work advances this field by introducing dynamic visual representation selection mechanism specifically for robotic tasks. Unlike fixed visual feature approaches, ours en- ables selective leveraging of different representations from diverse expert library based task requirements, leading to more robust policies across varied robotic scenarios.
2022
METHOD
3.1 OVERVIEW
In this section, present VER framework for visuomotor robot policy learning, illustrated in Fig. 2 Our approach begins with visual perception, where input images are processed by a Base Vision Transformer (BVT) to extract foundational visual features, referred to as unified representa- tions. These representations are then fed into a Vision Expert Library ( (VEL), a collection of special- ized neural network experts designed to capture diverse aspects of visual understanding. A dynamic routing mechanism determines which experts should be activated based on the specific task: dur- ing pretraining, mimic the teacher vision foundation models (VFMs); and for downstream robot tasks, to select experts that enhance performance. This mechanism enables selective attention to the most relevant visual features. Finally, the outputs of the selected experts are integrated generate a more structured representation, which is either used to replicate the teacher VFMs or passed to policy head that translates these representations into robot actions.
3.2 MODEL ARCHITECTURE
As illustrated in Fig. 2 our approach consists of two main components: a Base Vision Transformer (BVT) that generates unified feature representations from input images, and a Vision Expert Library (VEL) comprising specialized experts that capture diverse visual representations from various vision foundation models.
We design VER based a modified vision transformer (Dosovitskiy et al. 2020) architecture, where the FeedForward Network in the last N transformer layers is replaced with Mixture of Experts (MoE) (Shazeer et al. 2017b) modules. The initial unaltered ViT layers referred to BVT, while the MoE-enhanced later layers constitute the VEL.
In the n-th MoE layer (Shazeer al. 2017b) of VEL, where (1, 2,..., N). we incorporate L experts {Er) | each implemented as a multilayer perceptron (MLP). We then introduce Teacher- Specific (TS) Routers RT, where E 1, 2,2,... ..., corresponds to each teacher vision foundation model. TS Router R. takes in the input feature vector R X and learn MLP to determine the score for each expert, as well as score noise. During inference, only the top-K scoring expert networks activated, while the remaining experts remain inactive, ensuring computational efficiency. The MoF. output is computed as:
Ei= RT () ET (z),
R. (z.1) m. Pl. p= Softmax(2). == 1 6 [s1: MLP((). c~ NN,, Softu((s)))))))
(I)
where is the noisy logit, Pt is the routing probability for expert 1, and my {0, 1} is the Top-K indicator (which equals if p, is among the K largest probabilities, and 0 otherwise). This sparse gating mechanism enables efficient computation while maintaining representation quality.
4

---

Published as a conference paper at ICLR 2026
3.3 DISTILLATION TRAINING
Building on our network architecture, we now describe the distillation process that enables our model to acquire diverse visual representations from multiple VFMs, forming comprehensive li- brary of specialized vision experts for effective utilization in downstream robot tasks.
Given an input image x, the BVT f(-) first processes the image produce unified rich representa- tion 2 = f(x). Subsequently, the VEL g(+,)) -) generates teacher-specific representations y = g(2, i), where i denotes the index of the specific teacher model being mimicked. Note that g(, utilizes the corresponding TS Router to select the most appropriate experts for emulating the i-th VFM's representational characteristics.
Following (Shang et al. 2024), we formulate the distillation loss Edistall as weighted combination of cosine and smooth
losses:
Cdistall
(b
(1 - C). (z)) ()), ((x))),
where h;(.) represents projection head for the i-th teacher, & (x) is the representation from the i-th teacher model, a; 1/1 and 3 0.9.
To mitigate optimization conflicts and ensure balanced expert utilization when distilling multiple Vi- sion Foundation Models (VFMs), we introduce teacher-level mutual information loss (Wang al. 2024). Unlike standard token-level loal-balnn heuristics, this objective explicitly maximizes the mutual information between the categorical teacher variable I and the routed experts 8 across all MoE layers:
I EE
p(z,, E(r) log plI, Er)
Lmi I(I,E")= -
((I))p(EP)
n= 1=1
where we assume uniform prior over the teachers, setting p(L.) =
To further analyze the underlying mechanism of this objective, we examine its entropy decompo- sition, E(I,E") = H(E") HE I), which intrinsically couples two critical regularization effects. First, maximizing the marginal entropy H(E") acts as global load balancer; it encour- ages a uniform marginal expert-usage distribution, effectively preventing expert routing collapse perspective. Second, minimizing the conditional entropy H I)
from an information-theonee enforces teacher-specific specialization by minimizing the uncertainty of expert assignment given specific teacher I, This promotes sparse, highly predictable routing trajectories, encouraging different VFMs to activate disjoint expert subsets. Consequently, it preserves the fine-grained vi- sual semanties of individual teachers and minimizes gradient interference within the shared MoE pool. We defer further theoretical analysis and the formal definition of the routing probabilities to Appendix D.2
The overall pre-training objective is thus formulated as:
Ppretrain Laistill + > Cmii
(4)
where 7 is scaling hyperparameter
eepceparamtterrmppriialllll empirically set to 0.0005.
3.4 ROBOT POLICY TRAINING
After distilling diverse visual representations, we freeze Vision Expert Library (VEL) together with frozen base visual transformer (BVT) for downstream robot tasks. We introduce lightweight robot router R that selects task-relevant vision experts and feeds the resulting representations to a newly trained policy head to produce actions.
We consider two routing modes for robot tasks.
Teacher Routing (TR) Because pretrained vision foundation models perform strongly on robot tasks, option is to choose which VFM to by selecting among the Teacher-Speeific (TS)
5

---

Published as a conference paper at ICLR 2026
Table 1: Per-task performance comparison (success rate in %) across various robotic bench- marks. The same policy head as (Shang et al. (2224) is used for a fair comparison on vision encoder. The best result is in bold and the second best result underlined. Our approach (VER) outperforms previous state-of-the-at methods across 11 diverse tasks from Franka Kitchen (Gupta et al., 2020).
Meta-World (Yu et al., 2020), and Adroit [Rajeswaran et al. 2018) environments, achieving the highest average success rate (74.7%).
Model
[LigMOn
DoorOpen DoorSlide Knob/Trn Microwase
BinFick
ButtonPress
DrawerOpen
Hammer
Pcs
Relocate
Averagee
VC-I
Majuralar cl
2023
1.6
8.2
14.4
1.2
18
66.7
56.0
1060
95.3
68.0
24.0
42.6
MV
puse
13.6
5.3
17.8
188
4.0
73.3
62.2
106.0
97.3
77.7
26.7
45.7
RAM
Natrar ni pr233
67.3
31.2
13.1
55.4
15.8
91.0
68.0
196.
96.7
73.3
58.2
57.6
RADIO
55.2
19.7
69.3
24.4
25.3
62.7
90.0
1860
108.0
66.
45.3
61.3
61.3
25.2
83.0
44.6
$1.3
70.7
76.0
98.7
96.8
73.3
20.3
62.8
Theia
Sangg Mi
2024
58.8
34.1
$1.3
47.8
24.8
76.0
52.7
1900
08.7
28.7
46.7
67.1
VER-B (Curs)
67.2
38,0
85.8
55.3
35.2
93.3
94.7
1800
97.3
80.0
64.0
74.7
[RR)M. M routers {R. Ji=1 learned during distillation. Specifically, for each image/frame t and layer n, the robot router Roba produces a categorical distribution Tte E M over {πş J. The selected TS router at layer n then used to select among the experts {Ez = in that layer. During training. we optimize the discrete teacher choice with the Gumbel-Sofimax estimator:
log Ttn +g g Gumbel 1),
Zt,n softmax
using a straight-through estimator, while inference take arg max( We share teacher logits across layers within a frame, i.e., Ti, = ny, yielding a single teacher choice per frame; or allow per-layer logits Tzn so that shallow and deep layers route different teachers (e.g., DINOv2- like early, CLIP-like late) for finer control. We refer to the former as Framewise Teacher Routing (FTR) and the latter as Layerwise Teacher Routing (LTR).
Patchwise Expert Routing (PER) PER applies standard MoE routing per patch token (Eq. [. of- fering maximal adaptivity to local content with minimal overhead (< 0.4% additional parameters). Because the router anced utilization, we omit the mutual-usage regularizer Lmmmal (Fq. 3.. However, naive end-to-end training without explicit regularization leads premature convergence, which we formally identify as early collapse.
planning selector for task-relevant experts rather than enforcing bal-
Proposition 1 (Router Gradient in Top-K MoF). Let f. be the task loss. The gradient of the loss with respect to the router logit 21 is given by:
Oc
(min 4)
dz
where q = -2E) Er is the expert-specifer directional derivative and q = (96)
As shown above, for any inactive expert (m4 0), the gradient simplifies to 0z -p-- This update is strictly independent of the expert's output & potential contribution qi- Because the shallow router adapts faster than the downstream policy network, initial random fluctuations perma- nently lock experts into an inactive state. This deprives them of informative gradient signals, causing the routing distribution to prematurely collapse into suboptimal, seed-dependent configuration.
ac
To mitigate this early collapse and encourage exploration, we introduce Curriculum Top-K Anneal- ing (CTA). We initialize training with all experts active (Kn = 7.) and linearly anneal the number of active experts down to a target K Kmin over S training steps:
(ss) max (Kmin:
L+ - (L+ - Kmin) 5
By applying CTA to PER's 's token-wise dispatch, we promote broad exploration in early stages and stable, sparse routing later, preserving inference efficiency at the target Kmin- Further analysis of router collapse and CTA is provided in Appendix

---

Published as a conference paper at ICLR 2026
Table Per-task performance comparison (success rate in % across different policy heads. VER consistently outperforms Theia across ViLT (Kim et al. 2021 Flow-Matching (Zhang & Gienger 2024 and Diffusion heads (Chi al. 2023).
ViLT head
Flow-Matching head
Diffusion head
Model
LIBERO
LIBERO-OOD
>bin
cube >cup
cylinder >plate
Real-world pour
Theia-T
0.61
0.50
0.70
0.45
VER-T
0.70
0.71
0.95
0.75
0.85
0.90
Table 3: Ablation of robot routing strategies. Mean success rate + standard deviation over 10 seeds. Best per task in bold; second best underlined. DINOv2, ViT, and CLIP denote the cor-
responding Teacher-Specifiic Routers. We can see different VFMs suit different tasks, and VER improves performance by dynamically routing to the appropriate experts distilled from these VFMs.
Task
DINOv2
ViT
CLIP
FTR
LTR
PER
PER+CTA
pen 78.0=
72.8
8..0
81.2
79.2+6.2
7..0-6.3
80.815.3
relocate
38.4
41.6#6.6
41.2#3.8
41.2#6.0
36.4
47.6=5.1
56.416.9
4 EXPERIMENTS
4.1 NETWORK STRUCTURE
To address the limited computational resources of robotic systems, we use DeiT-Tiny Touvron jet al., (02T) for VER-T, DeiT-Small for VER-S, and ViT-Base (Dosovitskiy et al.) 2020) for VER- B. Distillation is performed on ImageNet-IK (Deng et al. 2009) from three foundation mod- els-DINOv2 (Oquab et al. 2024). ViT (Caron et al. 2021). and CLIP (Radford et al. 2021). This configuration, aligned with Theia (Shang et al. 2024), controls for pretraining variations and enables a fair comparison. We use total of trol complexity, VER replaces only the last three layers of 12-layer transformer with the Vision Expert Library, yielding 9 standard transformer layers for the Base Vision Transformer and N MoE. layers for the Vision Expert Library. Routing network is provided in Appendix C Details of the pretraining procedure are provided in Appendix
experts and activate K = 2 experts. To con-
4.2 PERFORMANCE ON ROBOT TASKS
With different policy heads such ViLT (Liu et al. 2023 Kim et al. 202T). flow-matching and diffusion policy, we evaluate VER against pretrained vision encoders, including VC-1 (Majumdar eet al. (023)) R3M (Nair et al.) 2023). MVP (Xiao et al., (022), RADIO (Ranzinger et al.) 2024), VIP (Ma et al.), and Theia (Shang et al. 2024). Among these baselines, Theia is particularly strong it distills multiple vision foundation models into unified representation, while VIP leverages large-scale human video datasets to learn transferable features for robotic control.
We first follow Theia (Shang al. 2024) and apply the same policy head across 11 diverse manipu- lation tasks spanning three benchmarks: tasks from Franka Kitchen (Gupta et al. 2020) (LightOn, DoorOpen, DoorSlide, KnobTurn, Microwave), 4 from Meta-World (Yu et al. 20200) (Binpick, But- tonpress, DrawerOpen, Hammer), and 2 from Adroit (Rajeswaran et al. 2018) (Pen, Relocate). Second, we adopt the ViLT head (Liu et al.) 2023 Kim a0.12 and evaluate VER on four LIBERO tasks (Liu et al. 2023). including LIBERO-OOD, where object colors are modified to test out-of-distribution generalization. Third, we apply a flow-matching head on three Pick and Place task in the Manddllka et al. 202I). Finally, we use diffusion policy head for the Pour task in real-world experiments.
As shown in Table I VER consistently outperforms prior approaches, achieving the highest average success rate of 74.7%. Furthermore, Table 2 shows that VER surpasses Theia across all policy heads both in simulation and real world experiments. Figure shows the performance of our VER. Moveover, our VER also surpassed finetuned Vision-Lann Langonn-nngggge-Aee
model GROOT NYIIIA al. (0255) shown in Table 10 More details and results can be found in Appendix B
7

---

Published as a conference paper at ICLR 2026
0.7
Thale VIR (Oun))
Leyer 0
Leyer 1
Layer 2
Cosite
DINOv2
0.3g
20
60 Acthee Parameters (M)
10
80
3
5
6
2
Figure Cosine loss for DI- NOv2 distillation. Circle size indicates total parameters (TP).
Figure 4: Expert utilization frequency across three MoE lay- ers. Heatmap shows how each teacher model activates experts
(1-6) during distillation on ImageNet-IK.
Figure 5: Visualization of real world experiments. We find with human interference (not in the training dataset), our VER can successfully complete the task.
4.3 DISTILLATION PERFORMANCE
Figure shows that our framework effectively distills more knowledge from diverse foundation models; additional results provided in Table 8 Figure further illustrates expert utilization ImageNet- 1K (Deng et al.) 2009). Instead of pre-assigning experts to teacher models, our Teacher- Specific Routers dynamically allocate them, with mutual information regularization encouraging di- verse expert usage. We observe that ViT activates fewer experts, whereas DINOv2 and CLIP engage more, suggesting that VIT easier mimic while DINOv2 and CLIP present greater challenges. This trend is confirmed in Table 8 where the cosine loss after pretraining is significantly lower for VIT than for DINOv2 and CLIP. Overall, these findings demonstrate that our method outperforms fixed expert assignments by adaptively allocating more experts to stronger foundation models and fewer to weaker ones, thereby improving both utilization and distillation performance.
4.4 ABLATION ROUTER DYNAMICS ROBOT TASKS
In this subsection, we investigate the functional role of routers in robotic policy learning through three key experiments: (1) evaluating the impact of noisy gating on performance, (2) analyzing feature entropy evolution during training, and (3) examining the relationship between feature entropy and task performance. These experiments provide insights into how routers function as implicit planning modules for robotic tasks.
Robot Routing Stratgies Performance. We compare: (1) select one frozen Teacher-Specingg (TS) Routers (DINOv2, ViT, CLIP); (2) Framewise Teacher Routing (FTR) and Layerwise Teacher Routing (LTR); and (3) Patchwise Expert Routing (PER), with and without Curriculum Top-K An- nealing (CTA). Results in T that relying on single VFM performs poorly across diverse tasks, whereas PER, when combined with CTA, adapts more effectively to local content across lay- and achieves superior performance.
Patch Feature Analysis. To investigate the mechanism of CTA, we compute the norm of the last- layer patch features in VER, comparing models trained with and without CTA. As shown in Figure CTA reduces high-norm outliers and concentrates attention on task-critical patches, whereas models without CTA exhibit large outliers in background regions. We further analyze patch features on 30% of the robot dataset by measuring entropy and mutual information before and after the Vision Expert Library (VEL). Patch features are first reduced to five dimensions via Principal Component Analysis (PCA), and then NPEET (Steeg 2022) is used to estimate entropy and mutual informa- tion. As shown in Figure 7 PER+CTA filters out iask-irrelevant background patches (lower mutual information before after expert selection) while preserving tesk-relevant information (e.g., the target pen pose in pen-vO, which consistently appears in the left region the image). Finally, Fig- ure 8 compares feature norms from Theia, from VER before VEL, and from VER after VEL in the
8

---

Published as a conference paper at ICLR 2026
wio CTA
with CTA
Figure 6: Feature visualization of PER with and without CTA (seed = 0). Row 1: pen; Row 2: relocate. Without CTA, the Robot Router attends broadly to the dexterous hand, objects, and task signals (e.g.. target pen pose, target ball region). With CTA, the Robot Router suppresses task- irrelevant patches and concentrates on task-related, object-centric regions throughout execution.
(1) DINOO, I-5.5
(3 VI,, 3-6
C1) CLR 1-1.3
(4) FTR, 1-5.14
(5) ETR, 1-1
(5) PIR, I-1
(6) PFSH-CTA, 1-4.00
(3 DINO,A,,
(2) NIT,
(3) 1-S5
(4), I-4.58
(T), 1-4.99
(6) PER, 1-4.44
(6) PER-CTA, 5-4.01
Figure 7: Mutual information between patch features before and after the Vision Expert Li- brary. Row 1: pen; Row 2: relocate. PER+CTA suppresses information in background patches while preserving information task-relevant regions, yielding more compact visual represen- tation (lower average per-patch mutual information). For example, in pen, the left-middle region containing the target pen pose exhibits higher mutual information.
Pick and Place task. This task consists of two stages: first pick up the cross, then place it into the bin. We supply patch features and robot proprioceptive state to the flow-matching policy head. We find that pretrained VFMs such as Theia, as well as VER before expert selection, broadly attend to all potentially important objects. After expert selection in VEL, however, the features focus exclu- sively on task-relevant objects (cross and bin) and suppress robot-related patches-consistee design choice to provide robot proprioceptive state directly to the policy, eliminating the need for robot-related information in patch features. More analysis can be found in Appendix E.3
with
Architectural Optimization and Depth. To balance represen- tational capacity with computational efficiency, conduct sys- tematic ablation on expert granularity and architecture depth (See more in Appendix [P). To ensure rigorous comparison under constant computational budget, we scale the MLP hidden dimen-
Theia
Succos 20
sion by 1/ maintaining a consistent active parameter count. As shown in Table II our analysis identifies the (K, L) = (2, 6) configuration as an optimal trade-off, providing sufficient spe- cialized knowledge without the routing overhead or optimization complexities associated with larger expert pools. Furthermore, while deeper MoE stacks yield lower pre-training distillation loss, they produce complex representations that are difficult for down- stream policies to navigate, ultimately hindering task success (as shown in Table [2]. Driven by these findings, VER employs 12-layer transformer backbone where only the last three layers replaced with the Vision Expert
Trainable Figure Trainable parame- ters average success rates. Performance is evaluated on pen and relocate tasks.

---

Published as a conference paper at ICLR 2026
Figure 8: Feature visualization compared with Theia (Shang et al., 2024) on place the cross into the bin. Both Theia and our features before the Robot Router tend to attend broadly to other objects, the robot itself, and background regions, resulting in noisy feature norm. After routing, VER concentrates on task-relevant objects and suppresses robot-related and background patches.
Table 4: Ablation study Top-K. More active Table 5:
Ablation on the mixture of (DFM) and (TFS) experts.
experts lead to better performance.
Train-
Model
TopK
AP(M)
Relocate
Pen
Avg
# DFM
TES
TopK
Relocate
Pen
Avg
Theia-Tiny
5.3
74.0
46.0
60.0
6
0
2
64.0
80.0
72.0
4.8
42.7
77.3
60.0
0
2
2
69.3
74.7
72.0
VER-Tiny
5.2
52.0
80.0
66.0
6
2
74.7
82.7
78.7
3
5.7
57.3
78.7
68.0
Library (VEL), establishing a configuration of M 9 standard layers and N 3 MoE layers that maximizes task performance.
Inference Efficiency and Overhead. Despite its enhanced capacity, VER maintains a minimal computational footprint. The inference time utilizing diffusion policy in Table on an RTX 4090 0.105s for both VER and Theia (Shang al. 2024). While VER introduces lightweight routing components, it achieves substantially better downstream performance than Theia while utilizing fewer active and total parameters (see Table S and Figure 9 Ultimately, this demonstrates that VER delivers significant performance gains over existing baselines with comparable or even lower computational complexity.
Scalability and Extensibility. We examine the effect of the Top-K hyperparameter by finetuning only the lightweight Robot Router to adjust how many experts are selected per patch. As shown in Table increasing the number of selected experts improves success rates but also increases computational cost. This demonstrates that VER enables controllable trade-off between accuracy and efficiency without retraining the backbone or the experts. Beyond scalability, VER also offers extensibility by adaptive robot-domain knowledge integration. While distilled experts from pre- trained VFMs encode strong general visual knowledge, they may miss information critical for spe- cific downstream tasks. Our framework allows seamless integration of trainable experts tailored to socilled-FFunaationn MMddel (DFM) experts (TFS) experts achieves the best performance, highlighting the complemen- tarity between generalist and sask-specialized experts in enhancing overall task success.
such tasks. As shown in [3] adaptively combining with
5 CONCLUSION
In this paper. we introduce VER, a Vision Expert transformer for Robot learning. Our approach distills knowledge from diverse vision foundation models into a vision expert library and employs a task-adaptive Robot Router to select task-relevant features for downstream control. To maximize selection capacity and prevent early collapse during router learning, we further propose Patchwise Expert Routing with Curriculum Top-K Annealing. Across multiple policy heads and a range of robotic benchmarks, VER achieves state-of-the-art performance. Patch-level analyses show that the Robot Router learns to selectively leverage pretrained experts, yielding increasingly structured representations that drive performance gains. In addition, VER is highly extensible: seamlessly incorporates new robot-domain knowledge through expert addition, and scales the number of active experts to meet task complexity through lightweight router fineuning. These results highlight the value of expert-driven visual representation distillation and selection for robust, generalizable robot learning.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research adheres to the ICLR 2026 Code of Ethics and upholds the principles of responsible research. Our experiments No human subjects or vulnerable groups were involved, and no personally identifiable, sensitive, or harmful data were used in any part of this work. We have carefully considered the potential societal impacts of our methods, including risks of misuse or unintended consequences. We believe that our contributions primarily advance scientific understanding and do not pose foreseeable harm.
conducted using publicly available datasets and self-collected data.
REPRODUCIBILITY STATEMENT
We follow the reproducibility guidelines outlined in the ICLR 2026 Author Instructions. To sup- port reproduciiility, we include detailed descriptions of dataset construction, model training, and evaluation in the main text and appendix. The main code and checkpoints provided in the sup- plementary materials, Furthermore, we will release the complete source code, configuration files, and scripts on public platforms (e.g., GitHub and Hugging Face) enable others fully reproduce our results.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? The Eleventh Interna- tional Conference on Learning Representations.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, et al. no. A aision-language-aee flow 2024.
model for general robot control. arXiv preprint arXiv:2410.2444
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision Assionlanguaae-actionn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. (00.:2337./888... 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 9650-9660, 2021.
Junting Chen, Yao Mu, Qiaojun Yu, Tianming Wei, Silang Wu, Zhecheng Yuan, Zhixuan Liang, Chao Yang. Kaipeng Zhang, Wenqi Shao, et al. Roboscript: Code generation for free-form ma- nipulation tasks across real and simulation. arXiv preprini arXiv: 14623, 2024.
Tianxing Chen, Yao Mu, Zhixuan Liang, Zanxin Chen, Shijia Peng, Qiangyu Chen, Mingkun Xu, Ruizhen Hu, Hongyuan Zhang, Xuelong Li, et G3flow: Generative 3d semantic flow for pose- aware and generalizable object manipulation. In Proceedings of the IEEE/CVF international conference on computer vision, 2025.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffussonn policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273668, 2023.
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. Stable- moe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7085-7095, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 255.255. Icve, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Leam- ing Representations, 2020.
11

---

Published as a conference paper at ICLR 2026
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- long-horizon tasks via imitation and reinforcement learning. In Conference Robot Learning, pp. 1025-1037. PMLR, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. stat, 1050:9, 2015.
Wenlong Huang, Chen Wang, Yunzhu Li. Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-tempoot reasoning relational keypoint constraints for robotic manipulation. In 8th Annual Conference on Robot Learning, 2024. URL
Michael Janner, Yilun Du Joshua Tenenbaum, and Sergey Levine Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, pp. 9902-9915 PMLR, 2022.
Guangqi Jiang, Yifei Sun, Tao Huang, Huanyu Li, Yongyuan Liang, and Huazhe Xu. Robots pre- train robots: Manipulation-centiz robotic representation from large-scale robot datasets. In In- ternational Conference on Learning Representations, 2025.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. In 8th Annual Conference on Robot Learning.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision Vision-and-lann and-language transformer without convo- lution or region supervision. In International conference on machine learning, pp. 5583-5594. PMLR, 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015 4015-4026, 2023.
Sergey Levine. Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Zhixuan Liang. Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. International Conference on Machine Learning, pp. 20725-20745. PMLR, 2023.
Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldif- fuser: Interpretable hierarchical planning via skill abstractions in diffusion- based task execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16467-16476, 2024a.
Zhixuan Liang, Yao Mu, Yixiao Wang, Fei Ni, Tianxing Chen, Wenqi Shao, Wei Zhan, Masayoshi Tomizuka, Ping Luo, and Mingyu Ding. Dexdiffuser: Interaction-awarc diffusion planning for adaptive dexterous manipulation. arXiv preprint arXiv:2411.18562 2024b.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023.
Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Learning visual affordance grounding from demonstration videos. IEEE Transactions Neural Networks and Learning Systems, 2023.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implich pre-training. In The Eleventh International Conference Learning Representations.
12

---

Published as a conference paper at ICLR 2026
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Piere Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems, 36:655-677, 2023..
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:23100 230.2339..7766 2023.
Yao Mu, Jumting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian GE, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, et al. Robocodex: multimodal code generation for robotic behavior synthesis. In Proceedings of the 41st International Conference Machine Learning. pp. 36434 36434-36454, 2024.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. In Conference on Robot Learning, pp. 892-909. PMLR, 2023.
Fei Ni. Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadif- Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pp. 26087-26105. PMLR, 2023.
fuser:
NVIDIA, :, Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang. Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. Gr00t n1: An open foundation model for generalist humanoid robots, 2025. URL https: /ttpxiv///xiv...// abs 2200..///2222... 14734
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp. 1-31, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PmLR, 2021.
Ilija Radosavovie, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pp- 416-426. PMLR, 2023.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Roboties: Science and Systems XIV, 2018.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12490-12500, 2024.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.65500 2014.
13

---

Published as a conference paper at ICLR 2026
Jinghuan Shang, Karl Schmeckpeper, Brandon B May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. In 8th Annual Conference Robot Learning, 2024.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations. 2017a.
Noam Shazeer, * Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017b. URL https:// openreview net Btte://ooommmddBlckkd....
Greg Ver Steeg. Npeet: Non-parametric entropy estimation toolbox. https: happs://githhln com/ 9Tegversteeeg/NPEET 2022. Commit 8b0d948, MIT License. Accessed: 2025-09-24.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Weikang Wan, Yifeng Zhu, Rutav Shah, and Yuke Zhu. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 537-544. IEEE, 2024.
Yixiao Wang, Yifei Zhang, Mingxiao Huo, Thomas Tian, Xiang Zhang, Yichen Xie, Chenfeng Xu, Pengliang Ji. Wei Zhan, Mingyu Ding, and Masayoshi Tomizuka. Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learning. In 8th Annual Conference on Robot Learning. 2024. httph:htttpsptttteee
net Eorum? Seerum?ld=zeYaas2tte
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203. 06173, 2022.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1285-1294, 2017.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-tast task and meta reinforcement learning. In Conference robot learning, pp. 1094-1100 PMIR, 2020.
Fan Zhang and Michael Gienger. Robot manipulation with flow matching. In CoRL 2024 Workshop on Mastering Robot Manipulation in a World of Abundant Data, 2024.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE DISCLOSURE
We employed ChatGPT assist with language refinement. All suggestions were reviewed and revised by the authors, who take full responsibility for the final manuscript.
B EARLY COLLAPSE OF ROUTER AND CTA
B.1 ROUTER GRADIENT DYNAMICS TOP-K MoE
In this section, analyze how the loss gradient with respect the router logits is determined by the alignment between the MoE output y and each active expert output (z). We show that, for active experts (m²- 1), the router updates compare the current mixture with each expert direction (x()) and increase the routing probability when moving towards that expert reduces the loss. In contrast, inactive experts (my 0) receive only weak, indirect updates that do not depend on their outputs, so their re-entry into the Top-K set is not guaranteed, even if they could potentially improve the loss.
Recall Equation here:
R (x, (x),
(8)
(x, = Top- TSo- + +c)1),
[21(2):22(z) = (x), e (10, (o
Let
<xx () = 1s (x); + es
exp(2; (x,
(9)
r (x, 1) == Softmaxi (x, Σ exp(2 ())
Define the Top=K(Softmax(<1(1) (x) + e),()) indicator m. (v,() as (x, 1, if is in Top-K for token at layer n, 0, otherwise.
(10)
Thus, the output y is
= Σ m (z. (1)
(11)
1=1
For brevity, temporarily suppress the indices (n, i, Φ) and write my, mi, &.. Denote L is the loss, then we compute the gradient with respect to the noisy logits 21, 2 (r. 1) of expert l.
Oc
JL
Oy
Oz1
(LL)
Σ mops&s
(12)
Oy
(DL) T
Opi
Σ
E,
Oy
Oz1
Op;
0z
0zl
e 8,11
Σ, et
(x
(13)
8y1
Σ, e+r
Σ, exr
Ps (5.z - -Pi).
15

---

Published as a conference paper at ICLR 2026
Thus,
Oc
(C T
Op, Ozi
Σ
m,e
Öz
(3C) T
E
(m3Csps(6g) - -pi))
Oy
(OC)
m&Eipi Pt Σ Esp
By
(14)
02
(m1 - -1)
Oy
dC
T
[C T δy
m4
dy
Pt (miq
①)
(oL) Where gi = By
E,, q
(ac) By
y.
When m. = 1, which is to say that expert is active, the gradient ac effectively compares the dz performance between the current mixture output and the single expert output E7(2), and determines the update direction for 24. The change in 2 will in turn shift the output y through the routing probabilities. Let us assume that
y = y (=) v),
i.e. we slightly move the mixture output towards expert I with a small step size a >> 0. Using first-order Taylor expansion,
OL By
0) - (oL) By (E(()) - v) = aa 4),
T
L(v') C(v) ~
(15)
When @ < q, moving y towards (1) reduces the loss, i.e., L(//) C(y) for small a. this case, the gradient ac becomes negative, so 2/ increases under gradient descent. Since expert is active, ds:
a larger 2; increases its routing probability and reinforces its selection. Thus, for active experts, the router automatically increases the weights of experts with smaller q during training, leading to better expert assignments.
When my = 0, which is to say that expert / is not active, there is no direct comparison between the current mixture output and the single expert output ((x) the gradient signal: the update of z does not involve qt. and (xx) does not effectively participate in the gradient descent for this token. As result, inactive experts receive only weak, indirect updates; consequently, their re-entry into the Top-K set is not guaranteed, even if they could potentially improve the loss.
B.2 EARLY COLLAPSE OF ROUTER
In the previous section, we showed that only active experts (my = 1) receive expert-specific gradi- ents based on the comparison between y and (x), whereas inactive experts (m4 = 0) receive only weak, indirect updates and their re-entry into the Top-K set is not guaranteed.
When training the robot router of VER for robot task, a large, randomly initialized policy network Fo is placed on top of y. Because the router is typically shallow, it can adapt much faster than this downstream network. to concentrate on an essentially arbitrary subset of experts, meaning that expert selection is highly sensitive to random initialization and training seeds. Once a small set of experts is consistently selected as Top-K, the inactive experts (m, 0) receive only weak, indirect updates that do not depend on (x),, as discussed above. This early router collapse creates a mismatch in convergence suboptimal routing pattern determined by the random seed, while the large downstream network has not yet learned a meaningful representation. After this point, it is difficult for the router to revise its expert selection, because inactive experts longer
the early phase, random fluctuations of qi - 9 cause the router logits 2
rates: the router quickly commits
16

---

Published as a conference paper at ICLR 2026
Without CTA
Export Activation Frequencies - Layer (All Seeds)
Seed
Seed
Soed
Sand
Seed
1007h
With CTA
Expert Activation Frequencies Layer Seeds)
See2
Seed
Swid 2
Sand
Sead
Figure 10: Expert activation frequency during the training with and without CTA on (10 seeds). Without CTA, expert activation frequencies converge prematurely in the early stage train- ing (for example, the activation frequencies of Seed and Seed 5 do not change after 10 epochs) and depend strongly on the training random seed, leading substantial variability across seeds. With CTA, the robot router explores more effectively and converges more consistently across different random seeds.
17

---

Published as a conference paper at ICLR 2026
Without CTA
Epoch
With CTA
Expert Frequencles Layer
Beat2
Read
Reael
Figure 11: Expert activation frequency during the training with and without CTA on relocate (10 seeds). Without CTA, expert activation frequencies converge prematurely in the early stage of training (for example, the activation frequencies Seed 5, 7 and do not change after around 20 epochs) and depend strongly the training random seed, leading to substantial variability across seeds. With CTA, the robot router explores more effectively and converges more consistently across different random seeds.
18

---

Published as a conference paper at ICLR 2026
participate in gradient descent in significant way, if their fixed representations would later become beneficial as Fg improves.
To address this issue, propose Curriculam Top-K Annealing (CTA) in Equation 7 which initially activates all experts and then gradually decreases the number of active experts. This curriculum encourages exploration converge, before enforcing a sparse Top-K routing pattern. As shown in Figure [10 and Figure [1] expert activation frequencies tend converge early in training. With CTA, however, the expert activations exhibit more consistent pattern across different random seeds. Figure 17 and Figure [8] present the final expert activation frequencies across all layers, further demonstrating that CTA leads to more consistent final activation pattern across random seeds.
expert combinations in the early stage and allows the policy network to
5=0
5m40
50:00
60
Success
25
50
75
100
125
150
175
200
Epoch
Figure 12: Ablation study on Sin CTA on pen (3 seeds. As increases, the success rate improves, and we also observe reduction in the variance of the success rate, indicating more robust policy training.
B.3 ABLATION STUDY ON
In Curriculum Top-K Annealing (CTA) in Equation 7 there hyperparameter S that controls the annealing schedule. A too small S leads to insufficient exploration, whereas a too large S results in slower convergence to the based on the total number of training epochs. We evaluate performance on the pen-v0 and relocate- v0 tasks by training for 200 epochs with three random seeds and averaging the best success rate. Specifically, we set S to 0, 40, 60 and 80 epochs, where S = O means that CTA is not applied. Table 6 shows that as increases, the success rate also increases. Moreover, Figure shows that a higher value of S reduces the variance of the success rate in the later stages of training, indicating more robust training with CTA.
Kmin and thus higher computational cost. practice,
choose S
Table Ablation study on S in CTA on pen.
Task
S=0
4=40
S=60
S=80
pen-v0
8.3
78. 13.7322.3
3..
81.3122.
19

---

Published as a conference paper at ICLR 2026
Theorem (Top-K router gradients: active inactive experts). Consider a Top-K MoE layer
Σ my P2 Ex),,
Py Softmax,(3),
my {0, 1]
and let g 0g
0C
Define
93 g g6 Ex)x,
q g y.
Then for each expert logit 24.
Qc
pt (miq q).
0z
In particular:
If mi = 1 (active), the router update depends on the expert-specific alignment qi and compares Ex()) against the current mixture y: under gradient descent, 2 increases iff qi < q, i.e., iff infinitesimally moving y toward Ei(z) decreases L.
If my 0 (inactive). Oc -Pi is independent of E(xx) (no expert-specific signal), so d2:
re-entry into Top-K not guaranteed even {z) could reduce the loss.
Theorem 2 (Early router collapse and curriculum Top-K annealing (CTA)). Let shallow Top- K router feed a much larger downstream network Fg during early training. Because the router adapts faster. stochastic fluctuations in (qi q) can rapidly concentrate probability mass onto an essentially arbitrary subset of experts, after which inactive experts receive no expert-specific gradi- ents (Theorem l). Consequently, the routing pattern becomes reed-sensitive and difficult to revise later, yielding early router collapse. A sufficient mitigation is Curriculum Top-K K Annealing (CTA); start training with all experts active (K = L) and gradually reduce to the target sparsity. This forces early exploration over expert combinations while Fe learns meaningful features, and only later enforces sparse Top-K routing, leading to more stable and consistent expert utilization across seeds.
C ROUTING NETWORK
For the Teacher-Specifee (TS) Router RJ, -> Linear) that produces per-patch logits over experts.
lightweight two-layer MLP (Linear -> GELU
For the Robot Router Ryohot Routing. For Patchwise Expert Routing, we adopt the same architecture as the TS Router. For Framewise Teacher Routing and Layerwise Teacher Routing, we first apply attention pooling over all patch features to obtain summary token, followed by a three-layer MLP head (SiLU activations. dropout) that outputs logits over the Seacher-Specifee Routers.
different network structure for Teacher Routing and Patch
Table 7 shows the details of router network.
Table 7: Network structure for routers.
Module
Input Granularity
Core Layers Pooling
Output
TS Router
Single patch
Linear, GELU, Linear
Expert logits (per patch)
PER
Single patch
Linear, GELU, Linear
Expert logits (per patch)
FTR/LTR
All patches
Attn pooling: 3-layer MLP (SiLU, dropout)
Teacher logits (per patch/layer)
D PRETRAIN EXPERIMENTS
D. PRETRAINING DETAILS
We adopt DeiT-Tiny (Touvron et al. 2021 the backbone for VER-T, DeiT-Small TTovrro al. 202T) for VER-S, and ViT-Base (Dosovitskiy et al., 2020) for VER-B. We choose the last N 3 layers Vision Expert Library (VEL). The projection head h₂(-) consists of shallow CNNs,
20

---

Published as a conference paper at ICLR 2026
following the same design of Theia (Shang et al. 2024). The training dataset is ImageNet- 1K (Deng jet al. 2009). We initialize the model weights using Theia and train VER on four A6000 GPUs for 50 epochs. The learning schedule consists of linear warmup for the first 10% of training steps, followed by constant learning rate 0.002 for the next 40%, and then Cosine Annealing LR for the remaining steps.
D.2 MUTUAL INFORMATION LOS$
In order to calculate Lms, we need to get p(I,), p(E(") and &f). Where Ij is the ith of teacher VFM, ETe the Ith expert at layer n. We assume each teacher model is equally important so p(T.) = where / the number of teacher models. For p(I,. Ei"), we have
p(E} I.) is the score of Teacher-Specifie router for expert I layer n. Then p(E)) can be caleu- lated by E(,,
Intuitively, this encourages the router to assign different experts to different VFMs while avoiding excessive overlap and promoting balanced expert utilization. Since
objective can be understood as follows. Maximizing [[Z En)
E(I,E") T(E" H(E") H(E" [],
(16)
maximizing (I, E") simultaneously increases ((E)) and decreases H( 2). A larger (()") drives the marginal expert-usage distribution toward being approximately uniform, similar in spirit to traditional MoE token load-balancing losses. A smaller (E I] implies that, given a specific teacher I, (and input r), the selected experts are more predictable and less uncertain. In practice, this encourages different teacher VFMs to rely on distinct subsets of experts, allowing them to preserve their fine-grained visual characteristics. Overall, the nutual-inforration loss both promotes task- agnostic load balancing across experts and preserves fine-grained vision features, thereby reducing conflicts when distilling multiple VFMs into shared expert pool.
D.3 PRETRAINING RESULTS
Table 8 demonstrates the distillation performance compared with Theia We can see that our VER can achieve better distillation performance with similar active parameters. And VER-S with less active and total parameters achieve comparable distillation performance compared to Theia-B.
Table 8: Distillation performance comparison between our VER and Theia across three foun- dation models (DINOv2, ViT, CLIP) using different loss metries. TP(M) denotes total parameters (Million), and AP(M) denotes active parameters (Million).
Cosine Loss
L1 Loss
MSE Loss
Model
(M)
AP
DINOv2
Theia-T
5,3
5.3
0.641
0.431
0.651
0.377
0.301
0.373
0.873
0.673
0.875
VER-T (Ouns)
0.829
Thela-S
20.7
20.7
0.554
0.335
0.587
0.351
0.255
0.356
0.800
0.556
0.826
VER-S (Ours)
27.7
20.8
0.453
0.299
0.517
0.311
0.235
0.332
0.695
0.507
0.762
Thela-B
0.216
0.688
0.462
0,767
VER-B (Ouns)
110.1
82.2
0.337
0.226
0.455
0.255
0.189
0.307
0.555
0.400
0.700
E ROBOT TASK EVALUATION
BENCHMARKS EVALUATION SETTINGS
Franka Kitchen We mainly follow R3M Nair et 2023) evaluation protocol. Specifically,
train the policy for 20,000 steps and evaluate success results every 1,000 steps throughout training. The final reported performance is based on the best average of three success rates observed during
21

---

Published as a conference paper at ICLR 2026
evaluation. To ensure robustness, our results in each environment averaged over different camera views (left and right) and different numbers of demonstrations (5, 10, and 25). We use the same policy network as Theia (Shang al. (024) for comparison. Specifically, we employ three-layer ML. for CNN-based models using vector-based representations. For transformer- -bsse models, we introduce three-layer CNN before the MLP process spatial inputs.
Adroit & Meta-World We primarily follow the original evaluation setup of Cortex (Majumdar et al. 2023), with mod- ifications the training epochs for the Adroit environment. Since VER introduces additional training parameters for the router, which functions as planning module requiring ex- tended training, we increase the training epochs for the pen task to 200 and for the relocate task to 400, ensuring full per-
Table 9: Performance vs, epoch. We report average highest success rates.
Epoch
relocate
pen
formance convergence. As shown in Tab. 9 training for 100
100
48.0/52.0
78. 78.7/80.0
200
50.7/52.0
80.0/84.0 -/-
epochs is insufficient for policy convergence. For VER-T, 200
300
56.0/60.0
epochs enough for relocate. In this paper, focus is
400
64.0/68.0
primarily on the functionality of router, while optimizing its
training efficiency is left for future work. We the same policy network as in Franka Kitchen.
LIBERO We select first four tasks from LIBERO OBJJET (Liu al. 2023 and train
ViLT (Kim et al., 2021) policy for 30 epochs, following the LIBERO evaluation protocol. In ad- dition, change colors for all the colors to evaluate performance an out-of-distribution setting.
Pick and Place We set up the Pick and Place task in robomimic (Mandlekar et al. 2021). The object is one of {cross, cube, cylinder). and the container is one of (bin, cup. plate}. Objects are ran- domly positioned and oriented on the left side of the desk; containers are randomly positioned and oriented the right side. We use a SpaceMMouse to teleoperate the robotic arm in robomimic, col- lecting 50 human demonstrations, and then use MimicGen (Mandlekar al., 2023) to generate 450 additional demonstrations, yielding 500 demonstrations per task. We train the flow-matching policy for 16,000 steps and evaluate over 40 trials. The flow-matching policy network is U-Net-based.
World Experiment We conduct real-robot experiments on a FANUC LR Mate 200iD/7L robotic arm equipped with an SMC gripper. The task is to pick up teapot and pour into cooking pot. Both the teapot and the cooking pot are randomly positioned and oriented. We collect 20 demonstrations, train diffusion policy for 120,000 steps, and evaluate over 20 trials. The diffusion policy network is based-
Comparison with Vision-Language-AAee flow-matching policy head, against two primary baselines: (1) Theia-based model equipped with a How-matching policy, and (2) the fine-tuned vision-languagc-action (VLA) model, GROOT N1.5 (NVIDIA et 2025).
Models We compare our VER model, utilizing a
For each task, the models are trained using 500 expert demonstrations. Following the officially recommended fine-tuning protocol for the GROOT architecture, we fine-tune GROOT N1.5 for 20k steps with a batch size of 16. As shown in Table I0, our method consistently outperforms the baselines across manipulation scenarios.
Table 10: Performance comparison between VER and baselines.
Model
cross -bin
cylinder-
Theia
0.65
0.50
0.70
GROOT N1.5 INVIDIA et
2025
0.75
0.73
0.70
VER (Ours)
0.95
0.75
0.85
22

---

Published as a conference paper at ICLR 2026
Pick cube and piace into
Pick cylinder and place into plate
Pick cross and place into bin
Figure 13: Performance on Pick and Place.. We find that when the first attempt fails, the VER- equipped policy can retry and complete the task, as shown the images with red boundaries.
E.2 EXPERIMENT VISUALIZATIONS
Figure 13 shows the performance of VER in Pick and Place tasks. We find that when the first attempt fails, the VER-equipped policy retry and complete the task, shown in the images with red boundaries.
E.3 PATCH FEATURE ANALYSIS
relocate
05-
Prorson -- -0351
p-vauue 0,0029
Poomon r -0333 p-vcuu 0,0045
Trend Curve (s-70)
90-
CLIP
DINO
Success Rtte (%6)
85-
Success Rate (96)
VIT
80-
45-
60-
PER+CTA
31-
65-
30-
60
25-
Mutual Information
Mutual Information
Figure 14: Mutual information success rate. For each method (CLIP, DINOv2, ViT, FTR, LTR, PER, PER+CTA), we report the mean + s.d. over 10 random seeds. The dashed line is an ordinary least-squares fit on 70 points (7 methods 10 seeds) summarizing the overall trend.
Figure 14 shows that PER+CTA occupies the region with the lowest mutual information and the highest success rate. We a linear model to all 70 training results (7 methods x 10 random seeds) and observe negative association: lower mutual information correlates with higher success. For the effect CTA, we find that adding CTA markedly reduces both mutual information and variance across seeds, leading more stable and robust training. Additional per seed visualizations with and without CTA (Figure 15) further support this observation: without CTA, the mutual information distribution varies substantially across different random seeds; with CTA, the Robot Router always
23

---

Published as a conference paper at ICLR 2026
concentrates the task relevant region (the left middle area with high mutual information corre- sponding to the target pen pose) and suppresses background regions with low mutual information.
w/o CTA
with CTA
Figure 15:
Mutual information between patch features before and after the Vision Expert Library on pen. We plot the image across 10 random seeds for training.
Figure 16 shows that without CTA, the Robot Router produces noisy patch embeddings with ex- tremely large feature norms in background regions. Although it sometimes attend to the correct regions and ignore task-irrelevant patches, its behavior is highly sensitive to the training random seed. This indicates that training lightweight router is prone early collapse and limited explo- ration. In contrast, with CTA, the Robot Router consistently focuses on task-relevant patches in more robust manner. To further analyze this phenomenon, we plot the expert utilization frequency over the entire robot dataset for 10 random seeds in Figures 17 and [8] With CTA, the utilization frequencies are noticeably more consistent across seeds, which indicates that CTA helps avoid early collapse and insufficient exploration, thereby leading to more robust Robot Router learning.
F ABLATION ANALYSIS: MoE CONFIGURATION AND ARCHITECTURE DEPTH
We systematically evaluate the architectural design of our framework by analyzing the trade-offs between expert granularity, total model depth, and inference efficiency. To ensure rigorous com- parison under constant computational budget, we scale the MLP hidden dimension by 1/K for all configurations, thereby maintaining consistent number of active parameters per forward pass.
F.1 EXPERT CONFIGURATION ANALYSIS OF K AND L
We first investigate the impact of the number of active experts (K) and total expert pool size (L). Models are pre-trained on 25% subset of ImageNet- and evaluated on the pen-vo task.
As summarized in Table 11 performance scales significantly from (K, L) (1,3) 3) to (2, 6). fol- lowed by marginal degradation at (4, 12). This suggests that (2,6) 6) represents "sweet spot" in rep- resentational capacity; while increasing L enhances the model's os specialized knowledge, excessive experts may introduce routing instabilities and optimization complexities that hinder downstream policy performance.
Furthermore, profile the inference latency of the Base Vision Transformer (BVT) against our MoE variants. We decouple the latency contributions of the router and the expert layers. As shown in Table TT] latency scales positively with K and L., primarily driven by the increased overhead in the routing mechanism and expert computation. The (2, 6) configuration is selected as primary backbone, as it maximizes success rate with only a marginal increase total latency.
Table 11: MoE Configuration Analysis. Success rates (%) averaged five random seeds; latency (ms) is measured with batch_size on a VER-Tim Tiny hardware platform.
Success Rate (%)
Total Latency
Router
Expert
44.0 + 11.9
1.4491
0.5620
0.5464
2
48.8 + 5.3
1.4873
0,5526
0.6021
2
69.6 = 4.8
1.6227
0.5855
0.6939
4
12
66.4 + 4.1
1.9383
0.5847
1.0089
24

---

Published as a conference paper at ICLR 2026
w/o CTA
with CTA
GAGA
Figure 16: Feature visualization of PER with and without CTA across seeds. Without CTA, the Robot Router attends broadly the background and generates extreme feature-norm outliers, and its behavior is strongly influenced by the training random seed. With CTA, the Robot Router robustly suppresses task-irrelevant patches and concentrates on task-related regions across all the seeds.
F.2 ARCHITECTURE DEPTH: BALANCING BVT AND VEL LAYERS
VER contains standard transformer layers (BVT) and N sparse MoE layers (VEL). We study on NM identify the optimal balance between general feature extraction and expert-driven specialization.
Results in Table 12 indicate that increasing the BVT depth from M to M improves both the distillation cosine loss and the final success rate. However, increasing the VEL depth to N presents notable divergence: while it achieves the lowest distillation loss, the downstream success rate drops significantly to 48.8%. This suggests that while deeper MoE stacks are superior mimicking pre- pre-training targets, they may produce sith-diinnsionall representations that are more difficult for the downstream policy to navigate. Consequently, we fix N to maintain robust bridge between pre-training quality and task execution.
Table 12: Ablation Study on Architecture Depth. Comparison of standard layers (M) versus MoE layers (N).
M
N
Distill Cosine Loss
Success Rate (%)
0.561
50.4 = 12.5
9
3
0,551
69.6 + 4.8
9
5
0.546
48.8 = 12.2
25

---

Published as a conference paper at ICLR 2026
w/o CTA
Lager Iote
Lage Pake
Lave Inken
bades
Expert
Exper ltdoo
L2ne 242
Laco MEC
24
Popert Batle
Expert Eedos
Dapert Tadk
Expert bales
Expert Eedes
with CTA
Seud
Seed 1
Sevd 2
Seed
Seed
Lawelaa
Lage Ladoa
Lape Inde
Lapetadd
Expert Seed
Seed
Seed
Seed
Seed
Layer BAR
Lope bnk
Cove
Figure 17: Expert utilization frequency with and without CTA pen (10 seeds). Without
CTA, expert utilization frequency varies substantially across random seeds. With CTA, is more consistent across seeds, indicating improved training robustness and stable Robot Router.
26

---

Published as a conference paper at ICLR 2026
w/o CTA
Indis
Intox
Indes
Eaposteda
Land Eodex
Lawe Toden
Land
Expurt Endes
Dxpert Tales
Expurt Eades
Expert Todi
Exportados Eade
with CTA
Seed 8
Seed 1
Seed 2
Seed
Seed
Laye Detc
TaperTadan
Laner
Lape
Seed
Seed
Seed
Seed
Sape
Figure 18: Expert utilization frequency with and without CTA on relocate (10 seeds). Without CTA, expert utilization frequency varies substantially across random seeds. With CTA, more consistent across seeds, indicating improved training robustness and more stable Robot Router.
27


Reviews:
[{"review_id": "dWjLilTC7e", "reviewer": "Reviewer_dKGG", "summary": "This paper addresses the limitation that single vision foundation models (VFMs) cannot meet the diverse requirements of robotic tasks, and existing multi-model distillation methods dilute model-specific capabilities while lacking task adaptability. The proposed VER framework distills multiple VFMs into a Vision Expert Library and employs a dynamic routing mechanism to select the most relevant visual experts for different robotic tasks. The first core innovation is Patchwise Expert Routing combined with Curriculum Top-K Annealing (PER+CTA), which achieves fine-grained patch-level expert selection and prevents early training collapse through curriculum-based K-value annealing. The second core innovation involves fine-tuning only a lightweight Router (<0.4% parameters) while freezing all pretrained experts, enabling parameter-efficient task adaptation and flexible integration of trainable experts to incorporate robot-domain knowledge. Experimental validation across 17 robotic tasks achieves state-of-the-art performance (74.7% average success rate) and outperforms existing methods across multiple policy heads (ViLT, Flow-Matching, Diffusion). A key finding is that dynamic routing suppresses large-norm outliers in task-irrelevant regions and concentrates attention on task-critical areas (such as target object poses), generating more compact and discriminative visual features.", "questions": "Can you explore Inference Speed and Efficiency?\n\nI recommend acceptance, but cannot confirm yet whether a higher score can be given.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality and Problem Formulation: The paper presents a novel paradigm shift from static unified representations to dynamic expert selection for robot learning, introducing the innovative combination of vision foundation model distillation with Mixture-of-Experts (MoE) architecture and the creative Curriculum Top-K Annealing mechanism that addresses the early collapse problem in lightweight router training.\n\nQuality and Experimental Rigor: The work demonstrates exceptional experimental rigor with comprehensive evaluation across 17 diverse robotic tasks spanning multiple benchmarks (Franka Kitchen, Meta-World, Adroit, LIBERO), different policy heads (ViLT, Flow-Matching, Diffusion), and real-world experiments, complemented by thorough ablation studies and insightful patch-level feature analysis (entropy, mutual information) that reveal the underlying mechanism of how dynamic routing concentrates attention on task-critical regions.\n\nClarity and Significance: The paper is well-structured with clear motivation and methodology, providing strong evidence that VER achieves state-of-the-art performance (74.7% average success rate) while maintaining computational efficiency (only <0.4% trainable parameters during downstream tasks, identical inference time to baselines), and the framework's extensibility—enabling flexible Top-K adjustment for compute scaling and seamless integration of task-specific trainable experts—offers significant practical value for scalable robot learning systems.\n\n\nIn addition, the experimental results in Figures 6 and 8 (1) are one of the reasons I gave you a high score.", "weaknesses": "Arbitrary choices lacking design principles: The paper fixes L=6 experts and K=2 active experts without providing any ablation studies to justify these hyperparameter choices (e.g., why not L=3,K=1 or L=9,K=3), and the \"coincidence\" that L=6 is exactly twice the number of three teachers suggests a potential pre-assigned expert-teacher correspondence, contradicting the paper's claim of \"dynamic allocation.\"\nMissing critical architecture search: The paper fails to explore the joint impact of L and K (e.g., performance comparison of L=3,K=1 vs L=6,K=2 vs L=12,K=4 under the same computational budget), making it impossible to determine whether the performance gains stem from the inherent advantages of the MoE architecture or merely from accidentally selecting a configuration that works for this specific task set, severely undermining the method's generalizability and reproducibility. But these are not big issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xCm8zUdXfe", "reviewer": "Reviewer_KMyG", "summary": "This paper investigates how to distill a vision encoder from existing vision foundation models for downstream robotic tasks. Unlike existing works that distill a monolithic encoder, this paper utilizes MoE to enable a more diverse, flexible and adaptive feature extraction process. Experimental results show that the proposed method outperforms baseline methods, especially Theia, on a broad range of tasks.", "questions": "1. It would be good to add a qualitative analysis on the failure reasons of different methods. \n2. Do you fine-tune the vision backbone of the baselines when learning the policy? If yes, do you fine-tune all their parameters? Or use some parameter-efficient methods like LoRA?\n3. The problem setting and training pipeline are different, but how does your method compare with VLA methods that first pretrain on large-scale robotic data, then fine-tune on specific downstream tasks? Can your method be extended to replace the visual encoders used in VLA? If yes, does it perform better or not?\n4. What is the size of your model at the three different parameter scale? Do you use a smaller MLP dimension in MoE so that the total parameter number is similar to that of Theia? As from Figure 9 it seems that your method has a smilar number of total parameters as Theia, but from Table 4 it seems that your method has a similar number of activated parameters (K=2) as Theia. \n5. What will happen if you put more layers in your ViT as MoE layers?\n6. As Theia is an important baseline you compare to, I think it's better to give a more detailed introduction on how Theia works for consistency of the paper, so we better know how your method differs from it.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly motivated and presented. \n2. The proposed method is evaluated on a broad range of tasks and shows clear advantage compared to baseline methods. \n3. The ablation analysis and visualization are thorough and illustrative to demonstrate how the visual representation differs across different methods and the importance of different design choices.", "weaknesses": "My main concern about this paper is that its novely seems not very significant, more like just replacing existing distillation models for visual encoder in robotic models with MoE. The empirical gain is significant, but I think if we care about absolute performance, it may be better to compare with a broader range of methods for robotic control, such as VLAs that are fine-tuned on the same datasets. For other points, see my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HpXXXYhanb", "reviewer": "Reviewer_We1f", "summary": "The paper introduces VER (Vision Expert Transformer), a novel architecture for robot learning. It addresses the limited generalization of single Vision Foundation Models and the inflexibility of distilling multiple VFMs into a static representation. VER's core contribution is a two-stage framework: 1. Pretraining: Knowledge from multiple VFMs is distilled into a \"Vision Expert Library\" (VEL) based on a Mixture-of-Experts (MoE) architecture. 2. Fine-tuning: The pretrained experts are frozen, and only a lightweight \"Robot Router\" is trained to dynamically select task-relevant experts. To enhance expert selection, the authors propose \"Patchwise Expert Routing\" (PER) and \"Curriculum Top-K Annealing\" (CTA). CTA starts training with all experts active and gradually anneals the number of active experts to prevent premature convergence. Experiments show that VER achieves state-of-the-art performance.", "questions": "1. The CTA schedule depends on the hyperparameter $S$. How was $S$ determined in your experiments? Is the method sensitive to this choice? I may have missed the relevant ablation studies. Additionally, are there any empirical conclusions that help in selecting this hyperparameter?\n2. Could you explain the main differences between VER and the baseline Theia? Table 7 shows that Theia does not use the MoE architecture, while VER has 135% more parameters than Theia. Would it be feasible to replace Theia's network architecture with the MoE architecture for comparison?\n3. Could you please elaborate on the training procedure for the \"Train-from-Scratch\" (TFS) experts in Table 5? Specifically, for the (#DFM=6, #TFS=1) setting, how is this 7th expert initialized and trained? Additionally, would fine-tuning all parameters across 6 DFMs yield better performance? Or introduce a LoRA module for fine-tuning?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This work addresses a practical and important problem: efficiently leveraging multiple powerful VFMs for robotics. The ability to adapt to new tasks by fine-tuning only <0.4% of parameters is highly valuable for resource-constrained robotic systems.\n- The work creatively combines multi-teacher VFM distillation with an MoE architecture.\n- The paper is well-organized. The experimental evaluation is thorough and robust.", "weaknesses": "I'm sorry, but I'm not very familiar with this field. Please see the Questions section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTlH31YI00", "reviewer": "Reviewer_b4k1", "summary": "The paper introduces VER (Vision Expert Transformer for Robot Learning), an approach that consolidates Mixture-of-Experts (MoE) for downstream control tasks. The authors introduced a two stage approach, a routing network to select task relevant experts and a curriculum top-k annealing to improve the dynamic expert selection. The paper reports state-of-the-art performance on multiple robotics benchmarks.", "questions": "- What are the theoretical guarantees or formal analyses regarding how the annealing schedule in CTA impacts expert load balancing or the stability/convergence of the MoE router?\n\n- Beyond empirical performance, what is the formal algorithmic advantage of PER over a standard token-wise or layer-wise routing scheme in the context of dense visual representations?\n\n- The distillation loss uses mutual-information regularization. Please elaborate on the theoretical motivation for this specific regularization in this context.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper demonstrates strong empirical results across a large and diverse set of benchmarks\n- VER is an important practical contribution to real-world robot deployment and scalability", "weaknesses": "- The proposed mechanism is an important contribution to robotic learning, nevertheless the papers lacks theoretical justification and several design choices seem heuristic engineered. Further principled analysis and justification are needed. \n \n- The ablation study is insufficient to justify several hyper-parameter choices and to justify the complexity of the proposed technique. My major concern here is it fails to isolate the performance gains attributed to each novel mechanism versus the underlying MoE architecture or increased model capacity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)