VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing,
  Speaking, and Acting
Authors: name: Xiaoyu Liu, name: Chaoyou Fu, name: Chi Yan, name: Chu Wu, name: Haihan Gao, name: Yi-Fan Zhang, name: Shaoqi Dong, name: Cheng Qian, name: Bin Luo, name: Xiuyong Yang, name: Guanwu Li, name: Yusheng Cai, name: Yunhang Shen, name: Deqiang Jiang, name: Haoyu Cao, name: Xing Sun, name: Caifeng Shan, name: Ran He

Current Vision-Language-Action (VLA) models are often constrained by a rigid,
static interaction paradigm, which lacks the ability to see, hear, speak, and
act concurrently as well as handle real-time user interruptions dynamically.
This hinders seamless embodied collaboration, resulting in an inflexible and
unresponsive user experience. To address these limitations, we introduce
VITA-E, a novel embodied interaction framework designed for both behavioral
concurrency and nearly real-time interruption. The core of our approach is a
dual-model architecture where two parallel VLA instances operate as an ``Active
Model'' and a ``Standby Model'', allowing the embodied agent to observe its
environment, listen to user speech, provide verbal responses, and execute
actions, all concurrently and interruptibly, mimicking human-like multitasking
capabilities. We further propose a ``model-as-controller'' paradigm, where we
fine-tune the VLM to generate special tokens that serve as direct system-level
commands, coupling the model's reasoning with the system's behavior.
Experiments conducted on a physical humanoid platform demonstrate that VITA-E
can reliably handle complex interactive scenarios. Our framework is compatible
with various dual-system VLA models, achieving an extremely high success rate
on emergency stops and speech interruptions while also successfully performing
concurrent speech and action. This represents a significant step towards more
natural and capable embodied assistants.