RoboOmni: Proactive Robot Manipulation in Omni-modal Context
Authors: name: Siyin Wang, name: Jinlan Fu, name: Feihong Liu, name: Xinzhe He, name: Huangxuan Wu, name: Junhao Shi, name: Kexin Huang, name: Zhaoye Fei, name: Jingjing Gong, name: Zuxuan Wu, name: Yugang Jiang, name: See-Kiong Ng, name: Tat-Seng Chua, name: Xipeng Qiu
Organization: OpenMOSS

Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid
progress in Vision-Language-Action (VLA) models for robotic manipulation.
Although effective in many scenarios, current approaches largely rely on
explicit instructions, whereas in real-world interactions, humans rarely issue
instructions directly. Effective collaboration requires robots to infer user
intentions proactively. In this work, we introduce cross-modal contextual
instructions, a new setting where intent is derived from spoken dialogue,
environmental sounds, and visual cues rather than explicit commands. To address
this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor
framework based on end-to-end omni-modal LLMs that unifies intention
recognition, interaction confirmation, and action execution. RoboOmni fuses
auditory and visual signals spatiotemporally for robust intention recognition,
while supporting direct speech interaction. To address the absence of training
data for proactive intention recognition in robotic manipulation, we build
OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640
backgrounds, and six contextual instruction types. Experiments in simulation
and real-world settings show that RoboOmni surpasses text- and ASR-based
baselines in success rate, inference speed, intention recognition, and
proactive assistance.