VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing,
  Speaking, and Acting
Authors: name: Xiaoyu Liu; name: Chaoyou Fu; name: Chi Yan; name: Chu Wu; name: Haihan Gao; name: Yi-Fan Zhang; name: Shaoqi Dong; name: Cheng Qian; name: Bin Luo; name: Xiuyong Yang; name: Guanwu Li; name: Yusheng Cai; name: Yunhang Shen; name: Deqiang Jiang; name: Haoyu Cao; name: Xing Sun; name: Caifeng Shan; name: Ran He
Published: 2025-10-21 17:59:56

Abstract: Current Vision-Language-Action (VLA) models are often constrained by a rigid,
static interaction paradigm, which lacks the ability to see, hear, speak, and
act concurrently as well as handle real-time user interruptions dynamically.
This hinders seamless embodied collaboration, resulting in an inflexible and
unresponsive user experience. To address these limitations, we introduce
VITA-E, a novel embodied interaction framework designed for both behavioral
concurrency and nearly real-time interruption. The core of our approach is a
dual-model architecture where two parallel VLA instances operate as an ``Active
Model'' and a ``Standby Model'', allowing the embodied agent to observe its
environment, listen to user speech, provide verbal responses, and execute
actions, all concurrently and interruptibly, mimicking human-like multitasking
capabilities. We further propose a ``model-as-controller'' paradigm, where we
fine-tune the VLM to generate special tokens that serve as direct system-level
commands, coupling the model's reasoning with the system's behavior.
Experiments conducted on a physical humanoid platform demonstrate that VITA-E
can reliably handle complex interactive scenarios. Our framework is compatible
with various dual-system VLA models, achieving an extremely high success rate
on emergency stops and speech interruptions while also successfully performing
concurrent speech and action. This represents a significant step towards more
natural and capable embodied assistants.