VER: Vision Expert Transformer for Robot Learning via Foundation
  Distillation and Dynamic Routing
Authors: name: Yixiao Wang; name: Mingxiao Huo; name: Zhixuan Liang; name: Yushi Du; name: Lingfeng Sun; name: Haotian Lin; name: Jinghuan Shang; name: Chensheng Peng; name: Mohit Bansal; name: Mingyu Ding; name: Masayoshi Tomizuka
Published: 2025-10-06 18:00:43

Abstract: Pretrained vision foundation models (VFMs) advance robotic learning via rich
visual representations, yet individual VFMs typically excel only in specific
domains, limiting generality across tasks. Distilling multiple VFMs into a
unified representation for policy can mitigate this limitation but often yields
inflexible task-specific feature selection and requires costly full re-training
to incorporate robot-domain knowledge. We propose VER, a Vision Expert
transformer for Robot learning. During pretraining, VER distills multiple VFMs
into a vision expert library. It then fine-tunes only a lightweight routing
network (fewer than 0.4% of parameters) to dynamically select task-relevant
experts from the pretrained library for downstream robot tasks. We further
introduce Patchwise Expert Routing with Curriculum Top-K Annealing to improve
both flexibility and precision of dynamic expert selection. Moreover, VER
supports parameter-efficient finetuning for scalable expert utilization and
adaptive robot-domain knowledge integration. Across 17 diverse robotic tasks
and multiple policy heads, VER achieves state-of-the-art performance. We find
that VER reduces large-norm outliers in task-irrelevant regions (e.g.,
background) and concentrates on task-critical regions. Visualizations and codes
can be found in https://yixiaowang7.github.io/ver_page/.