RoboOmni: Proactive Robot Manipulation in Omni-modal Context
Authors: Siyin Wang
Jinlan Fu
Feihong Liu
Xinzhe He
Huangxuan Wu
Junhao Shi
Kexin Huang
Zhaoye Fei
Jingjing Gong
Zuxuan Wu
Yu-Gang Jiang
See-Kiong Ng
Tat-Seng Chua
Xipeng Qiu
Venue: ICLR 2026 Poster (2026)
Keywords: Speech
Robotic Manipulation
Omni-Modal LLMs
Proactive Intention Recognition
Primary area: applications to robotics, autonomy, planning

Abstract: Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision–Language–Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively.
In this work, we introduce *cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.* To address this new setting, we present **RoboOmni**, a *Perceiver-Thinker-Talker-Executor* framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. 
To address the absence of training data for proactive intention recognition in robotic manipulation, we build **OmniAction**, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance. All datasets, code, and real-world demonstration videos will be released publicly.

Published as a conference paper at ICLR 2026
ROBOOMNI: PROACTIVE ROBOT MANIPULATION IN OMNI-MODAL CONTEXT
Siyin Wang 12 Jinlan Fu Feihong Liu'
Xinzhe He' Huangxuan Wu'
Junhao Shi 1.2 Kexin Huang' Zhaoye Fei
Jingjing Gong" Zuxuan Wu Yu-Gang Jiang' See-Kiong Ng Tat-Seng Chua Xipeng Qiu
Fudan University SShanghai Innovation Institute National University of Singapore
si hiyy.wwang20fuudaeeee.d... jinlanjoona@gmiil.com
Fudan.edu.ccm
ABSTRACT
Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in lision-Languag--Attinnn (VLA) models for robotic manipula- tion. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas real-world interactions, humans rarely issue in- structions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, Perceiver-Thiinee Talker-l Talker-Executor framework based end-to-end omni-modal L.MMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Exper- iments in simulation and weal-world settings show that RoboOmni surpasses text- and ASR-based baselines success rate, inference speed, intention recognition, and proactive assistance. We make all our datasets and code publicly available
INTRODUCTION
LaiionLLanguageAAttiin (VL.A) models (Zitkovich al.) 2023 Ghosh et al.
2024 Black et al,,
2024) have achieved remarkable advances in robotic manipulation leveraging
large-scale scale cross-
embodiment datasets (Padalkar et al.] 2023 AgiBot MeBBot-Wolld-CCo Conorid-ons et al. 2025 Khazatsky
ct al. 2024) and Multimodal Large Language Models (MLLMs) (Wang al. 2024 Bai et 2025a
Li et al. 2025). ) VLA models are generally categorized as (1) end-to-end models (Brohan et al,
2023 Zitkovich et al.) [023 Black et al. 2024 Kim et al.) 2024 which map vision vision-languan
inputs directly to motor actions, and (2) modular Train-Cerebelluny models (Huang et al. 2023 2024 Shi et al., 2025), which LLMSs VLMs planners to decompose tasks into sub-goals for low-level controllers. While modular systems emphasize explicit planning, they suffer from fragmentation and interface constraints. In contrast, end-to-end models unify vision, language, and action in a shared latent space, enabling more natural and flexible responses.
Despite notable advances in VLA research, two fundamental limitations remain. (1) From the per- spective of instruction type: most works (Kim et al. 2024) focus on direct commands (Fig. (a)), later extended more complex (Fig. (T(b) yet explicit forms (Shi al.) 2023). while Xu et al.
(025a) recently introduced a dataset for inferential text-based instructions (Fig. (c)), but sys-
tem studies remain searce. (2) From the perspective of the instruction source: current systems
(Kim et al., 2024: Zitkovich al. 2023) predominantly rely textual instructions Fig. I-(d) or ASR- transcribedd speech (Fig. I (c)), the latter discarding essential paralinguistic cues such as
"Corresponding authors.
https://gi wwtph///..oo///eoossss.so 1

---

Published as a conference paper at ICLR 2026
Place the Coke the tobie. (a) Direct
Son:
I'm thintty.
There's orange
Audio
Mom:
there's juice
Ham orange
Juice
Voice
Coke in the fridge.
Sound Events
yourseff
Envianmental
Prepore some soft (b) Complex
orange juice.
Backproundss
(negotive tono)
(olturr notse)
(c) Inferential
Speech Confirm
Vision
1. Instruction Type
get you
Place the Coke to the table. (d) Textual Instruction
yes!
Action Execution
D
Place the Coke
ASR
(e) Speech-ASR-Textuue Instruction
Jet some tisues,
(1 Speech Instruction
2. Instruction Source
Contextual Instruction
Figure I: Overview of robotic manipulation models classified by instruction type and input. Our RoboOmni integrates contextual instruction with direct speech for end-to-end multimodal interac- tion and action execution.
tone, intonation, and affective signals. Recently, Zhao et al. (2025) investigated models that accept speech instructions (Fig. (f) by converting existing textual commands into speech, but neglected real-world environmental sounds. Overall, existing works assume that instructions are explicitly is- sued, and there a lack of study on jointly reasoning over speech, environmental sounds, and visual observations for proactive intent recognition and reasoning.
There arises a key research question: Can robot integrate cross-modal context, including speech, environmental audio, and visual observations, to proactively infer and verify user intent? As illustrated Fig. I in a living-room scene, the robot integrates dialogue (audio), re- frigerator observation (vision), and the juicer sound (environmental audio) to infer that John prefers cola over hand-made sour juice, and proactively seeks confirmation rather than waiting for an ex- plicit command. Since humans rarely issue direct instructions in daily life, define such scenarios as cross cross-modal contextual instructions, where auditory (speech and enviroumental sound) and vi- sual cues are fused infer latent user intent and verified proactively by interaction, in contrast to conventional setups that assume explicit commands.
To address these challenge and answer the research question, propose RoboOmni, end-to-end omni-modal framework for manipulation that closes the loop of intent recognition, interaction con- firmation, and action execution. Unlike prior approaches, RoboOmni supports direct speech interac- tion without ASR, infers latent commands by fusing human speech, environmental audio, and vision through spatiotemporal modeling, and verifies intent via interaction. To overcome data scarcity, we construct OmniAction, dataset with 140k episodes, over 5k speakers, 2.4k event sounds, 640 back- ground sounds, and six contextual instruction types.
Experiments in both simulation and real-world settings show that RoboOmni substantially outper- forms text- and ASR-based baselines, achieving higher accuracy (Sec. 5.2 and Sec. 5.3). faster infer- ence (Sec. 5.6), more effective proactive assistance (Sec. 5.5). and improved intention recognition (Scc. 5.5). Our contributions are fourfold:
1. We introduce cross-modal contextual instructions, new setting for robotic manipulation that requires robots to proactively infer user commands from multimodal context (vision, environmental sounds, and speech) rather than passively await explicit instructions.
We propose RoboOmni, a Perceiver-Thinker omni-modal LLMs that fuses auditory and visual inputs for intent reasoning, unifying intent recog- nition, confirmation, and action execution.
Executor framework based on end-to-end
3. To address the lack of datasets for proactive intention reasoning, introduce OmniAction, com- prising 140k episodes with 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types, along with OmniAction-LIIERR for simulation-based evaluation.
2

---

Published as a conference paper at ICLR 2026
4. Evaluation in both simulation and real- world scenarios demonstrates that RoboOmni exhibits emerging cognitive intelligence, outperforming baselines with higher success rates, faster inference, and more effective proactive assistance and intention recognition.
2 RELATED WORK
The rapid development of Large Language Models (LLMs) Achiam et al. (2023); Touvron et al. (2023a) has driven progress in multimodal extensions. Multimodal LLS (MLLMs) (202 2023 Bai et 2025al Liu et al. 2023b Chen et al. 2023) combine text reasoning with vision, while recent end-to-end omni-modal models (Hurst et al. [2024] Xu et 2025b Xie & Wu 2024) unify speech, vision, and text but remain focused on linguistic outputs. In parallel, Aision-Language Action (VLA) models Brohan et al. ( (2023); Zitkovich al. (2023); et 2023) ; Team et al. (2024); Kim
ct al. (2024) 2025); y: Black et al. (2024); Li ct al. (2024a); Qu et al. (2025) map instructions actions, yet mainly assume explicit commands and struggle with context- context-dependen or compositional tasks.
Cascaded or hierarchical variants (Intelligence al.) 2025 Shi et al. 2025 Song al.. 2025b Lin
al. 2025) decompose goals but ignore implicit cues such as dialogue or emotion, while ASR/TTS-
based speech-action action pipelines (Shi et al., 2025 Khan et al., 2025 discard paralinguistic signals. Although some recent efforts handle direct speech inputs (Zhao et al. 2025), they only output actions without conversational interaction. In contrast, we propose RoboOmni,
end-to-end omni-modal
framework that integrates speech, environmental sounds, vision, and text for both embodied action and natural interaction. A detailed review of related work about Omni-Modal LLMs and Vision- Language-Action Model is provided in App. B
OMNIACTION DATASET CONSTRUCTION
3.1 OVERVIEW
Proactive robots must infer implicit intent from audio and visual observations, yet existing datasets lack such combination of modalities (most of them lack audio modality) and inferential instructions needed for intent reasoning. To address this gap, we introduce OmniAction, a large-scale corpus that encodes contextual instructions latent intents grounded in speech, environmental audio, sound events, and vision. Omni covers six instruction categories and three non-speech sounds.
Diverse Contextual Instructions. (1) Sentiment Cues: Emo- tionally tinted expressions, or subtle vocalizations, that indirectly reveal user preferences or intentions (e.g., "Ugh, this juice is too sour" implying request for an alternative). (2) Overlap- ping Voices: Multi-speaker audio segments with temporal over- laps, testing intent extraction under crosstalk and partial mask- ing. (3) ven-Verbal Cues: Salient non-ling non-linguistic audio events (e.g., alarms, phone rings) that carry situational information rele- vant to the task. (4) Identity Cues: Speaker attributes such as age and gender, inferred from voice and not available from text, are needed to decide whose intent to satisfy. (5) Dyadic Dialogue: Two-participants dialogues where intent emerges from conver- sational flow rather than explicit commands. (6) Triadic Dia- logue: Three participants interact with turnttaking and indirect references, increasing the complexity of intent attribution. To preserve general fommand-follovien ability beyond dialogue, we also include a portion of single-person text instructions during training-
Trieato
Dyadie
Over
lppatog
Omni Action
verbal
siingee Simple
Lemntty
Benti
ment
Figure 2: Distribution of textual instruction types in Om- niAction.
Diverse Non-Speech Sounds. We also investigate three types of acoustic variation: (1) Speaker Timbre. 5,096 distinct voices spanning six categories by age (elderly, adult, child) and gender (male, female). Reference audio clips are used for timbre cloning to ensure within-dialogue consistency and cross-speaker diversity. (2) Sound Events. 2,482 non-verbal events (e.g.. thunder, doorbell) were inserted at scripted anchors provide cues beyond speech. (3) Environmental Backgrounds. 640 ambient soundscapes (e.g., running water, stir-fry sizzling) mixed at controlled signal-to tomal-to-noise ratios (SNRs) to mimic daily environments.
3

---

Published as a conference paper at ICLR 2026
Textual Scripting of Contextual Instructions
Filteriing Remova samples with low states.
2. Dialogue synthesis S
Interaction extension
OPT Validator
Reerit commands into mull-tun dieloguee.
Generate followss Interaction.
G
o
Sont
i'n thinsy
Robot: Would you Wie to you a Coko?
potato from @
Bonate
Mom: There's oeoage puire and Cove
From the
the fidge Getit yourseff
Sont Very
mable
tably.
Sont
Atom.
(megorivr lute)
Robot: Olay.
Auditory Realization
Data Examples
Prompt Speech
Conversations
Swat
0008
Speech Symthesis
2. Multi-speakke simulation
Sendiment
core
Ovwrlapping
The
Non-Veriaal Cuma
peeiver SESS.
8's the
-
Raboot:
||I ||O||eIII
Environmental backgrounds
3. ven-vl
Idenity Oues
Dyodie Dialogue
Triedic Distogue
LTS
evant Insertion
G (0([pn Conjoiji
Human
Verification
Figure 3: Overview of OmniAction Dataset Construction Pipelines and Examples.
Data Statisties and Formats. OmniAction comprises 141,162 multimodal episodes. spanning 112 skills (e.g.. pick-place, open/close) and 748 objects (e.g., can), with 5,096 distinct speaker timbres, 2,482 non-verbal sound events, and 640 environmental backgrounds. Each sample represented as triplet (C, A); multi-turn conversation C visual observation sequence V, and an action trajectory A = {ar)== where a, E R² denotes the delta control vector of the end-effector. The distribution of instruction type is detailed in Fig. More detailed statistics and examples are shown in Sec. C.I and [Scc C.2
3.2 CONSTRUCTION PROCESS
We construct OmniAction through a three-stage pipeline-textual scripting, auditory realization, and verification illussrated in Fig.
Textual Scripting. We sample tasks and trajectories from Open-X datasets (Padalkar et al. 2023
and transform each atomic instruction into a contextual one with GPT-4o through: (1) Filtering: removing trivial samples with low-information visual states. (2) Dialogue synthesis: rewriting in- structions into multi-turn household dialogues that span six contextual instruction types. (3) Inter- action extension: constructing follow-up numan-robod exchanges that simulate natural interactions. (4) Validation: ensuring intent consistency with the original instruction.
Auditory Realization. To capture paralinguistic we convert dialogues into audio that reflects real household conditions, augmented with diverse sound events and background environments. The conversion process includes four steps: (I) Speech synthesis: rendering user turns into audio via muitiple TTS engines with voice cloning for timbre consistency and cross-dialogue diversity. (2) Multi-speaker simulation: generating each speaker's turns separately, concatenating them on the timeline, and inserting overlaps at controlled offsets. (3) Non-verbal event insertion: mixing contextual sounds (e.g., alarms, utensil clatter) at scripted anchors. (4) Environmenial backgrounds: adding ambient textures (e.g., water flow, frying, fan hum) at varying SNRs. Further implementation details are provided in Sec. C.3
and environmental acousties beyond text,
Verification. To ensure data quality, we conducted manual evaluation on sampled speech dialogues and confirmed that task intent was reliably recoverable (98.7% agreement, detailed in Sec. C.4).
3.3 SIMULATION DATASET: OMNIACTION-LIBERO
To address the lack of simulation benchmarks, we construct OmniAction-LLBERO based on LIBERO (Liu et al., 2023a), with two variants. (1) OmniAection-LBEERO-TTT
augments the LIBERO using the pipeline described above. Starting from 40 manipulation tasks across four suites (Spatial, Object, Goal, Long- Horiiznn, we generate six variants for each task based on the six con- textual instruction types. yielding 240 evaluation tasks. Example dialogues and task scenes provided in App. G (2) OmniAction- -eteetiinonnROOO LIBERO-Rees Real evaluates RoboOmni under real speech con- ditions, where 10 volunteers provide spoken instructions collected in real environments.
4

---

Published as a conference paper at ICLR 2026
7D Robot Action (.Grip)))
Omni Action
Speech Interaction
Talker
Executor
Thinker
Episodes
Colee
99
Taxt Takan
+0000
Audio
G Speakers
Enccon
Encocer
Perceiver
Figure 4: The framework of RoboOmni, Perceiver- Snpeiver-Thinker-- Taaeeeer-Tallee---eeeeee architecture that uni- fies vision, text, and audio in a shared token space to generate actions and speech.
METHODS
We propose RoboOmni, an end-to-end omni-modal LLM framework organized Per- T
unifying speech, environmental audio, vision, and robotic actions within single autoregressive model Fig. 4 RoboOmni employs a Perceiver for multimodal input, a Thinker backbone, a Talker for speech, and an Executor for actions. To align inputs with linguistic and motor outputs, RoboOmni uses unified tokenization to encode all modalities into shared se- mantic space, which the Thinker processes into high-level representations and specialized decoders render into speech and executable actions, enabling seamless perception-to-actiin generation.
4.1 TASK DEFINITION
We consider cross-modal contextual instruction following, where the robot receives multimodal in- put X = (Vii,, SK.T, C), consisting of visual observations V, and audio signals S, (including hu- man speech with varying identity, prosody, emotion, overlapping voices, and environmental sounds), and dialogue history C. The robot must use these cues to infer latent intent, produce clarifications 91:L when clarification needed, and execute actions AI.N- Unlike standard VLA tasks, intent here is not explicitly spoken but emerges from multimodal context.
4.2 ARCHITECTURE COMPONENTS
Perceiver: Multimodal Input Encoding. The Perceiver handles the encoding of heterogeneous input modalities into unified embedding space. Following the multimodal processing pipeline of Qwen2.5-0mm (Xu et al. 2025b), at each timestep the robot receives a visual frame V,, an audio segment S,, and the dialogue histroy up that point C₁. Modality-speciff, encoders produce embeddings V. = (Vi), St = fa.),, and C, = fe(C,), which are combined into unified representation X, [Vei 8,; ca] that serves input the Thinker backbone.
Thinker: Omni-Modal Reasoning, The Thinker serves as the central reasoning engine, built upon the LLM backbone. It processes the unified multimodal representations from the Perceiver and generates contextually appropriate outputs in the joint vocabulary space V U A. The Thinker auto- regressively produces sequences that seamlessly interleave text tokens, speech representations, and action tokens, enabling unified reasoning across perception, language, and robotic control.
Talker: Speech Generation. The Talker component enables the system to generate natural speech responses through hierarchical architecture design. The Talker receives high-level semantic repre- sentations and text token from the Thinker and converts them into speech waveforms, allowing for seamless voice interaction in robotic scenarios.
Executor: Action Generation. enable seamless integration of robotic control within the lan- guage model framework, we extend the vocabulary of the Thinker with set A of 2048 discrete action tokens introduced by the FAST+ tokenizer (Pertsch et al. 2025). Rather than mapping each action dimension to separate token, FAST+ represents continuous action vector a, E R²² (e.g.. 7-DoF control) by a short sequence of discrete symbols C A. This enables the model to auto- regressively generate from the joint space VUA, where V represents the text vocabulary, seamlessly bridging language understanding and robotic control within a single sequence. The Executor then decodes these action tokens back into executable robot commands.
5

---

Published as a conference paper at ICLR 2026
4.3 DUAL- GENERATION
Text and Speech Generation. For conversational responses, the Thinker autoregressively generates text tokens yI:I. (y1, 02..... yr.)
X
II po (u (x.e))) F-m1
(1)
The generated text can optionally be converted to speech through the Talker module, which receives the discrete text tokens and high-level semantic representations from the Thinker.
Action Generation. For robotic control, the Thinker autoregressively predicts discrete action tokens N of chunk length N, which are decoded into continuous actions a by inverse transform.
N
A Executor(Paa4NN) (o (x:
II
po(rits [e,
1=0
4.4 TRAINING PARADIGMS
We train RoboOmni using unified autoregressive objective that handles both conversational and manipulation capabilities within the same framework. Given a training episode, the model receives multimodal input X, and learns to predict appropriate responses- either conversational replies for dialogue or action sequences for manipulation.
For conversational interactions, the model optimizes the likelihood of generating appropriate text responses YI.,. given the multimodal context:
Lehai (0) E log y
Σ
t=1
For action generation, the model learns to generate action token sequences T that correspond to expert trajectory:
(0) = E
log Po (re+il Xx. rea+i-1).
(4)
$=0
The complete training objective combines both modalities through batch interleaving:
K
(())= = (ctai(() + Lau (0)
E
log Po(z
X,, Zck),
E VUA,
(5)
k=1
which highlights that both conversational and action supervision reduce to the same autoregressive moximum--lkelihon objective over a unified token space.
EXPERIMENT
5.1 EXPERIMENT SETUP
Baseline Models As current open-source Vision-language-AAtiinn (VLA) models are primarily designed for textual instructions and cannot directly process audio inputs, construct base- line paradigms to validate the necessity of end-to-end audio processing: (i) Ground-trute Textual Prompt, which directly feeds pre-annotated transcriptions of speech instructions into VLA models; (ii) Speech-ASR-Tees Texxuall Prompt, where speech instructions are first transcribed to text using the ASR model Whisper large-v3 (Radford et al.) 2023), then fed into VLA models. We conduct evalu- ations comparing RoboOmni with four representative VLA baselines representing both paradigms: Open VLA (Kim et al. 2024). Open VLA-OFT (Kim et al. (025). mo (Black et al. 2024), and
NORA Hung et al. 2025). ). Details of these baselines in App.
Implementation Details We train the model with an input image resolution 224 224, an audio sampling rate of 16,000 Hz, and an action chunk size of 6. For large-scale pretraining, RoboOmni optimized on a cluster of 64 A100 GPUs over days, corresponding to a total of 15,360 A100- hours, with batch size of 512. The training runs for 10 epochs using learning rate of 5 x 10 with the first 1k steps reserved for warm-up. For downstream task supervised fine-tuning (SFT), we adopt learning rate of 5 10-5 and train with 8 A A000 GPUs for 10-30k steps.

---

Published as a conference paper at ICLR 2026
Table 1: Performance of different robot manipulation models on the OnmiAction-LIBERO-TTI benchmark, evaluated across four task suites (Spatial, Goal, Object, Long-Horizon) under six con- textual instruction types. Values in bold denote the best performance.
Ground-truth Textual Prompt
Audio -> ASR
Text Prompt
Task
RoboOmni
OLe
OFT
NORA
m0
OLe
OFT
NORA
m0
Sentiment
4.0
9.0
40.0
8.0
1.0
8.0
43.0
11.0
93.0
Non- Verbul
8.0
61.0
3.0
8.0
14.0
Identity
1.0
8.0
53.0
4.0
2.0
18.0
56.0
7.0
92.0
Spatial
Overlapping
6.0
7.0
43.0
7.0
11.0
6.0
58.0
18.0
93.0
Dyadic
7.0
6.0
51.0
5.0
4.0
17.0
57.0
3.0
95.0
Triadic
7.0
51.0
2.0
57.0
Avg
3.5
7.5
49.8
6.2
3.8
10.5
56.5
9.8
93.0
Sentiment
0.0
0.0
11.0
0.0
0.0
0.0
9.0
3.0
89.0
Non- Verbal
0.0
18.0
1.0
22.0
4.0
Identity
0.0
0.0
11.0
3.0
0.0
0.0
11.0
1.0
82.0
Goal
Overlapping
0.0
0.0
21.0
0.0
0.0
0.0
23.0
1.0
97.0
Dyadic
0.0
0.0
7.0
1.0
1.0
10.0
18.0
0.0
85.0
Triadic
0.0
7.0
0.0
0.0
Avg
0.0
0.0
12.5
1.0
0.3
1.7
16.3
1.5
85.8
Sentiment
1.0
0.0
9.0
4.0
2.0
0.0
5.0
6.0
83.0
Non- Verbul
0.0
7.0
3.0
82.0
Identity
4.0
0.0
4.0
5.0
5.0
0.0
15.0
8.0
85.0
Object
Overlapping
14.0
7.0
1.0
6.0
26.0
0.0
16.0
9.0
84.0
Dyadic
20.0
0.0
14.0
7.0
20.0
10.0
19.0
7.0
88.0
Triadic
0.0
3.0
2.0
10.0
11.0
2.0
Avg
7.2
1.2
6.3
4.7
9.7
3.3
13.8
6.7
84.0
Sentiment
0.0
0.0
26.0
4.0
0.0
0.0
50.0
5.0
76.0
Verbal
0.0
35.0
0.0
57.0
2.0
Identity
0.0
0.0
29.0
4.0
1.0
0.0
43.0
4.0
79.0
Long
Overlapping
0.0
0.0
35.0
5.0
3.0
0.0
56.0
6.0
79.0
Dyadic
1.0
0.0
42.0
1.0
1.0
0.0
59.0
5.0
85.0
Triadic
0.0
27.0
2.0
10.0
41.0
8.0
82.0
Avg
0.2
0.0
32.3
3.3
1.2
1.7
51.0
S.0
79.5
Avg
2.6
0.4
16.3
3.0
3.9
2.3
25.9
4.4
85.6
5.2 EVALUATION ON CROSS-MODAL CONTEXTUAL INSTRUCTIONS
To comprchensively evaluate RoboOmni on diverse cross-modal contextual instructions, we conduct extensive experiments on the OmniAction-LLBERR across four task suites with six audio variants. Tab. I demonstrates that RoboOmni achieves overall 85.6% success rate, substantially outper- forming the strongest baseline (NORA, 25.9 25.9%) and other cascaded methods (all below 10%). Our analysis yields three key insights: (1) End-to-end auditory integration is crucial for paralin- guistic cues. Text-only models, whether using ASR transcripts or ground-trus -truth text, fail to capture paralinguistic cues (e.g.. prosody, overlapping speech), with best scores of 25.9% (textual baseline). In contrast, RoboOmni's direct audio processing enables it to consistently exceed 76% across all types, demonstrating the importance of preserving acoustic information. enhances robust intent recognition under ambiguity. Goal and Object suites are challenging due multiple manipulable objects and valid actions, where baselines collapse (averaging 16.3% and 13.8% for the best baselines), exposing limits in contextual instruction understanding. RoboOmni sustains high performance (Goal: 85.8% v.s. Object: 84.0%), demonstrating robust generalization under semantic ambiguity. (3) Instruction type complexity reveals varying cognitive demands. For end-to-end models, dyadic and overlupping tasks are easier, averaging ~88 Non-verbul in- structions are hardest (~82%) they require recognizing non-verbal sounds and integrating them with visual and speech cues. The remaining tasks average ~85% reflecting moderate complexity.
(2) Auditory integration
5.3 EVALUATION ON REAL HUMAN AUDIO DIRECT INSTRUCTIONS
We further evaluate RoboOmni's robustness under real human-recorded speech with direct audio in- structions. As shown Tab. 2 on the OmniAAction-LIEER Real benchmark. RoboOmni achieves the highest average performance (76.6%), surpassing strong text-based VLAs including To (73.8%), VLA (40.1 1%), and NORA (17.4%). ASR-based VLAs suffer from acoustic variability:

---

Published as a conference paper at ICLR 2026
OpenVLA
NORA
Roba RobiOrnn(Ours)
80.0
73.3
73.9
20.0
Accuracy (56
60
53.3
50.0
46.7
40.0
63.3
43.3
40.6
36.7
36.7
33.9
26.7
20
16.7
Sentiment
Non-Verbal
Identity
Overlapping
Dyadic
Triadic
Average
Figure 5: Real-world contextual-instructinn performance with human speech. RoboOmni surpasses
ASR+VLA baselines by directly grounding raw audio and vision and retaining paralinguistic cues.
Overlapping
Sentiment
(Dad
What's neve?
(Caughts] Ainght feols nico pattered together.
(Momi (Shi (Overia) The beef battef
(Daa) kooo, amully amouing Siccul me sbat wabhing wap dhamplings bul??
[Daughter Ovenlao) Yorrs beef balle daste the beat!
(Gaighte/ Hehh.. meee dumptings?... (negrative tono)
RoboOmni: Sounds like everyone agrees should put
RoboOmni: Sounds like you're not too fand of them. How about put the fish pol? [Oaughter) Yeah... that's fine.
the beef ball into the hot pot (Dad) That's right
Robotmmn:: Ok, IIl put the beef ball into the hot pot now. (ACT)
RoboOmair Airight.
pot (ACT)
Figure 6: Demonstration of success cases of RoboOmni on the real-world WidowX 250S robot arm.
Table 2:
PianAction-LIBRRO comparison
Performance
accents, coarticulation, and background noise fre- quently cause recognition errors, and even minor word deviations can degrade VLAs' performance. To shows some robustness, likely due large-scale co-training on diverse web data. In contrast, Ro- boOmni processes speech directly, avoiding ASR pipeline errors. Pretraining on diverse speakers and sounds improves robustness acoustic variability and paralinguistic cues, yielding more consistent performance.
Spatial Goal Object Long Avg
Audio
ASR
Text Prompt
Open VLA
51.6
38.2
38.0
32.4
40.1
ApenVLA-OFF
6.6
9.8
9.8
0.0
6.5
NORA
2.0
5.6
26.8
35.4
17.4
86.0
60.0
70.0
79.0
73.8
Ours (Audio Input)
RoboOmni
89.0
71.6
75.1
75.0
76.6
5.4 REAL-WORLDS EXPERIMENTS
To verify that RoboOmni's capabilities transfer beyond simulation, fine-tune pretrained model by utilizing our demonstration dataset on WidowX 250S, where speech was recorded by 10 volunteers in real environments. This enables RoboOmni speech instructions (e.g., sentiment, overlapping cues). We compare RoboOmni against several ASR+VLA baselines trained with the same data. Each task executed for 10 trials, and we report the mean task success rate. As shown in Fig. 5) RoboOmni achieves 73.9% success, substantially outperforming the best ASR+VLA baseline (52.2%). This performance gain primarily comes from two advantages of the unified architecture: (1) RoboOmni directly reasons over raw audio and vi- sion, making robust to natural speech variations. whereas cascaded systems are prone ASR emoor and to VLA brittleness under small transcription changes. (2) The end-to-end multimodal design preserves paralinguistic cues sentiment,,, speaker identity, and non-verbal wvents-essent for contextual-intes -nteet inference, while ASR pipelines inevitably discard them.
real robots and handle diverse
Fig. highlights RoboOmni's real-world competence across three dimensions: (I) strong intent recognition, accurately inferring user intention from both visual and auditory cues (e.g., identifying the object based on audio and determining the receptacle the pot from the visual scene); (2) effective interaction, proactively asking clarifying questions after inferring the user's latent intent (e.g.. "should I...?" ?"") and executing the action after receiving confirmation; (3) reliable execution, successfully carrying the confirmed action. More detailed cases are provided in Sec. H.
8

---

Published as a conference paper at ICLR 2026
Nocc0mi lount
Inian: fecognttion Capabilty
Identity
0:00
[Dangghtee hus su gled meh' lanving pot boon
(Tranoorats)
Qwenn 1aani25--mmi--5 Paecse of chicken
0:03
(ibest How nummer campy? 1
Owen2 G-Omnl-7B: Add Fhe fash la to the poi.
event fish batte you your
favorites.
ASR GPT-to: Add bell
0:00
[Daughher] Compp woss greatf sho's Noo Mom 500 acoset Nic Ech dall..
17.79
epg dumplings.
Robbo Orenni (Oura): Worki you like me put the egg dampling into the hot por?
(rrugreed (ows plaughess/oonn
0:17
(MAuns) Halis, that's ioceessing. Adtd
Score
(a) Intent recognition capability
(b) Qualitative comparison of interaction capahilities.
Figure 7: Qualitative and quantitative Evaluation of Proactive Assistance Capabilities.
Sentiment
Non-Verbal
Identity
Overlapping
Dyadic
Tiiaic
22
ore
Our
Sen
Oen
Acc
1
Training Steps
Training Steps
Training Steps
Training Steps
Training Steps
Training Steps
Figure 8: Training efficiency comparison between Ommiicction-petrainnd + SFT vs. from-scratch
SFT on OmmiAction-LIBERO (Spatial).
5.5 EVALUATION OF PROACTIVE ASSISTANCE CAPABILITIES
Intent Recognition Capability. We further evaluate the models' ability to recognize user intent under contextual instructions, shown in Fig. 7a Specifically, we compare Qwen2.5-0mmi-3B (our backbone). Qwen2.5-0mni-77. and ASR+GPTT44, against our proposed RoboOmni. We observe that RoboOmni achieves the highest accuracy (88.9%), confirming the advantage of end-to-end speech-action modeling that preserves paralinguistic cues and dialogue context. Notably, although ASR introduces recognition noise compared with end-to-end models, GPT-4o still surpasses the smaller Omni models (55.6% vs. 27.8%/50.0P because its stronger multimodal reasoning com- pensates for transcription loss. This highlights that contextual instructions cannot be resolved by acoustic modeling alone, but also demand robust reasoning capabilities
Interaction Capability. We qualitatively assess models' interaction capability in handling contex- tual instructions. As shown in FFg. 7b] RoboOmni excels by proactively clarifying, integrating cross-modal signals, and sustaiming natural dialogue, whereas baselines often fail in one or more aspects. Additional case studies for all instruction types appear in Sec. E.2
5.6 FURTHER ANALYSIS
What components of the input drive RoboOmni's gains? To understand which multimodal inputs drive Ro- boOmni's performance, we conduct input-controlled ab-
Table 3: Ablation study on intent recog- nition under different configurations.
lations the intent-recognittion experiment, keeping the architecture fixed while specific inputs are removed: (1) w/o vision, removing visual input: (2) w/o audio, remov-
Setting
Accuracy (%)
Full Input (ours)
88.89
ing the audio; (3) w/o paralinguistic cues, where audio is re-recorded by a single neutral speaker without prosody, emotion, or non-verbal events; and (4) Full Input (ours). As shown in Tab. 3 Full Input achieves 88.8 88.89 while performance drops to 58.88 58.89% without vision, 50.56% without paralinguistic cues, and 11.1 % with- out audio. These results reveal that: (1) Audio provides the core semantic instruction, as removing climinates actionable content; (2) Vision is essential for contextual grounding, especially when tasks involve spatial relations or object attributes; (3) Paralinguistic cues significantly aid disambiguation across identity-, emotion-, and sound event-dependent cases. Overall, RoboOmni's gains stem from the complementary integration of vision, speech semantics, and paralinguistic signals, underscoring the need for unified end-to-end multimodal modeling.
w/o vision
58.89
w/o audio
11.11
w/o paralinguistics
50.56
Does OmniAction Pretraining Improve Training Efficiency? To evaluate the benefit of pretrain-
ing on OmniAction, we compare finetuning efficiency on the six Spatial variants in OmniAction

---

Published as a conference paper at ICLR 2026
LIBERO, contrasting OmniAction OmniActiionpreeraaned + SFT with from-scratch SFT Fig. 8).
The pre-
trained model converges rapidly, reaching nearly 90% accuracy within 2k steps, from-scratch counterpart only attains ~30% after 20k steps
while the
and even degrades at 30k steps. This highlights that pre-
overlapping
NonVerbal
training OmniAction providing strong generalizable pri-
ors for fast and stable adaptation with minimal fine-tuning.
Can Cascaded Pipelines Handle Contextual Instruc- tions Effectively with High-level Planner? We com- pare RoboOmni with planner-controlie pipelines, where Qwen2.5-0mmi-33 serves as the planner and text-based VLAS as controllers, shown in Fig. 9 evaluated on the Om- niAction-LIBERRO benchmark. RoboOmni outperforms all cascaded pipelines, demonstrating the advantage of end- to-end speech-action learning: jointly modeling audio, vi- sion, and action avoids the lossy planner-controller inter- face and preserves intent fidelity. Cascaded pipelines per- form worse due to (1) semantic drift, as planners are not co-trained with VLAs and generate commands controllers cannot execute, and (2) poor handling of speaker identity, since Qwen-Omn fails to capture paralinguistic cues, lead- ing to the weakest results on Identity Cues.
Dyadic
Tridic
Identity
Sentiment
Geen-amn Ozenit
Qwen-Gmmi
Qwan-3mn --eenVtA-OT
RoboCrmii (Oun))
Gwen-Gmm Nen
Figure 9: Comparison between end- to-end RoboOmni and cascaded plan- ner-controller pipelines across six contextual instruction types.
Does End-to-End Modeling Improve Inference Effi- ciency? To assess whether end-to-end modeling improves runtime efficiency, we measure per-inference latency on a single RTX 4090 GPU. Using ASR Open VLA as the baseline (1.0x), we find that other cascaded pipelines (ASR Nora: 1.02x, ASR + No: 0.96x) incur similar costs since the ASR stage dominates computation. In contrast, Ro- boOmni runs at 0.49x latency, showing that end-to-end audio-action modeling eliminates the ASR bottleneck and substantially improves efficiency (Fig. I0).
ReboOmni (Ours)
0.49x
ASR+piO
0.96x
ApenvVLA
1.00x
0.0
0.2
0.4
0.6 o.s 1.0
Relative Time
Figure 10: Per-inference latency com- paring cascaded pipelines and Ro- boOmni.
Failure Analysis A task is counted as successful only when robots both infers the correct intent and executes the action. To understand where failures occur, we categorize all error cases real-world ex- periment into nine interpretable types spanning intention-level and execution-level errors. As shown in Fig. [ 42.6% intention-related failures mainly come from identity attribution errors, sentiment cue misreading. The remaining 57.4% cases arise from execution issues, dominated by grasp fallures, followed by pose estimation drift and reachability constraints. This breakdown clarifies RoboOmni's 's cur- rent bottlenecks and highlights where future improvements audio grounding and low-level control are most needed.
12.0%
9%
Error 42 8%
Creao Pal
Execution 67.4% Error
Figure 11: Failure analysis of real-world experiments.
6 CONCLUSION
In conclusion, we introduced cross-modal contextual instructions, a new paradigm for robotic ma- nipulation where robots proactively infer user intent from multimodal context-vision. environ- mental sounds, and rracherrther than passively awaiting explicit commands. Building on this setting, we proposed RoboOmni, end omni-modal LLMs that integrates auditory and visual inputs, unifying intention recognition, confirmation, and action execution. To address data scarcity, we constructed OmniAction, a large- scale corpus of 140k episodes with diverse speakers, event sounds, and backgrounds, together with MiirActiionnBBRRE for simulation iaasattioo-b-eddd evaluation. Comprehensive experiments in both simu- lation and the real world demonstrate that RoboOmni exhibits emerging cognitive intelligence, sig- nificantly outperforming text- and ASR-based baselines in success rate, inference speed, proactive assistance, and intention recognition.
framework built on end-to-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. 62521004). This research/project is supported by the National Research Foundation, Singapore under its Na- tional Large Language Models Funding Initiative (AISG Award No: LISG-NMLP22024-0 opinions, findings and conclusions or recommendations expressed in this material are those of author(s) and do not reflect the views of National Research Foundation, Singapore.
Any
ETHICS STATEMENT
This work equips robots with extended contextual instructions, including family speech dialogues, environmental sounds, and visual observations, to enable more natural multimodal interaction. While such data enhances robotic intelligence, we recognize the importance of protecting user pri- vacy when scaling to broader deployments. All collected dialogue and audio data are restricted to academic research use only and will not be shared for other purposes. Future applications must carefully manage privacy, consent, and secure handling of user interactions to ensure compliance with ethical and legal standards.
REPRODUCIBILITY STATEMENT
We will open-source the OmniAction dataset, model checkpoints, and training code to facilitate further research this field. Training details are described in(Sec. 5.1 and the complete OmniAction data construction process is documented in Sec. and App.
REFERENCES
Gpt-4v(ision) system card.
2023.
URL
CorpusID:26321B88
OpenAI Josh Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Prrmadeut-Shapinee Christopher Berner, Lenny Bog- donoff, Oleg Boiko, Made laine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Fleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim on Posada Fish- man, Juston Forte, Is abella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lppes, Jonathan Gordon, Morgan Grafsteenn Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Har- ris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Pe- ter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jo- moto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitschei- der, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick. Jong Wook Kim. Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Li, Rachel Lim, Molly Lin, Stephanic Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Mar- tin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, An drey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M'ely, Ashvin Nair, Relichiro Nakano, Rajeev Nayak, Arvind

---

Published as a conference paper at ICLR 2026
Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pa- chocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo. Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang. Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on on Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. 2023, URL https: api www.seeeatiichhsceeee
wwtgCorppssD:22200022 2575533281
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang. Lei Yang, Shukai Yang, Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu. Agibot world colosseo: large-scale manipulation platform for scalable and intelligent embodied systems. CoRR, abs/2503 06669, 2025. doi: 10:48550/ARXIV2550000O https::/dde www.ioorg1....0 48550/ arxiv, 2503. 06669
SgiBot-World-Cnntee
URL
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502. 13923, 2025a. doi: 10,48350/RXIV.25022 13923 13923.
/ org/
10. 18.44850/aari (03550/axiiv 2502. 13923
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Ze- sen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. ArXiv, 13923, 2025b. URL https::/aat www.nticsctosolsstt..e.ee org/ Co
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel M. Salz, Maxim Neumann, Ibrahim M. Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey A. Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong. Julian Martin Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bovsnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiao-Qi Zhai. Paligemma: A versatile 3b vim for transfer. ArXiv, abs/2407.07726. 2024. URL https::/appl www.sseetttnssho.oeo org/
CorpusID:2710R5
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi,
12

---

Published as a conference paper at ICLR 2026
James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. x0: A vision- language-action flow model for general robot control. CoRR, abs/2410224164, 2024. doi: 10. 48550/ARX1V.2410244100 bttps:t:s///o.
48550/ 2ttp:///2.....22..2.1...
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vin- cent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control at scale. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu (eds.), Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi: 10. 1561/557SS 202..XXX.0E.... URL /ttps://doi..oo
056111160777 RSS. 2023. XIX. 025
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intern vl: Scaling up vision foundation models and aligning for generie visual- visual-linguistic tasks. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24185-24198, 2023. URL wtthtttp//appl/ /emma
semant wwwchclacaa.....ccco.cccccc..aae org/ caaccholar.orrgoooorriiiiii.ee.. CC:rppsII:22652222
Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Xian Shi, Keyu An, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505. 1759,, 2025.
Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Jocy Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong. Ted Xiao, Pannag R. Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado (eds.), Robotics: Science and Systems XX, Delfi, The Netherlands, July 15-19, 2024, 2024. doi: 10.15607/RSS. (0666///SSS.024.XX.... 2224. XX. 090.
URL /dtp::://///i.... http//oooorggooo
15607/RSS
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks Proceedings of the 23rd international conference on Machine learning, 2006. URL https:
//api.
wwt.err//oooooo wtt/CorppgII..... 9901844
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Jie Tan, Mare Tou- ssaint, and Kourosh Darvish (eds.), Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Muchine Learning Research, pp. 540-562 PMLR, 2023. URL https: ptttps://proceedinggg www.iinggsm.......mmmm mir press / w229/ hwwnngannn 2bb html
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (eds.). Conference on Robot Learning, 6-9 November 2024, Munich, Germany, volume 270 of Proceedings of Machine Learning Research, pp. 4573-4602 PMLR, 2024. URL https::/p//......... hwtpo/ooceeiicceoooooooooooo.o.c ngs mmm....m........ ml www.r.press/vee /v20//220000 www0//uang25.... html
Chia-Yo Yu Hung, Qi Sun, Pengfei Hong. Amir Zadeh, Chuan Li, U-Xuan Tan, Navonil Majumder, and Soujanya Poria. Nora: A small open-sourced generalist vision language action model for embodied tasks. ArXiv, abs/2504. 19854, 2025. URL hhttps://api. semant i cscholar.
org/CorpusID : 278165428
OpenAl Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mkadry, Alex Baker- Whiteomb,, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alexander Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alexandre Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin
13

---

Published as a conference paper at ICLR 2026
Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tul- loch, An drey Mishchenko, Angela Back, Angela Jiang, An toine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, B. Ghor- bani, Ben Leimberger, Ben Rossen, Benjamin Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll L. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger. Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang. Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Chris Hesse, Claudia Fischer, Clemens Winter, Coley Czar- necki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mély, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Phong Duc Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang. Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Hai-Biao Bao, Haitang Hu, Hannah Wong, Haoyu Wang. Heather Schmidt, Heather Whitney, Hee woo Jun, Hendrík Kirchner, Henrique Pondé de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung. Ian Kivlichan, Ian O' O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Ja- cob Menick, Jakub W. Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Ryan Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang. Jason Teplitz. Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shich, Ji Lin, Jiahui Yu, Jiayi Weng. Jie Tang, Jieqi Yu, Joanne Jang. Joaquin Quiñonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward. Jong Wook Kim. Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Joshua Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamit- suka, Lilian Weng, Lindsay McCallum, Lindsey Held, Ouyang Long, Louis Feuvrier, Lu Zhang, Lukasz Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Mad- die Simens, Made laine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Ma teusz Litwin, Matthew Zeng, Max John- son, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Ali Yatbaz, Mengxue Yang. Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Mina Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Na talie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nikolas A. Tezak, Niko Felix, Nithanth Kudige, Nitish Shirish Keskar, Noah Deutsch, Noel Bundick, Nora Puck- ett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng. Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Phil Tillet, Prafulla Dhariwal, Qim ing Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Raphael Gontijo Lopes. Raul Puri, Reah Miyara, Reimar H. Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Ramilevich Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez. Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Cof- fey, Steve Lee, Stewart Hall, Suchir Balaji. Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng. Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce
14

---

Published as a conference paper at ICLR 2026
Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang. Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen Hc, Yuchen Zhang, Yu- jia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card. ArXiv, abs/2410.212766 2024. URL attps://appe semaii.sementee sema icscho ar www.orgg/ooppus.e TD: 22::2273629299
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowiez, James Tanner, Quan Vuong, Homer Rich Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. T0.5: a lssionlaaguaae-acttoo model with open-world generalization. ArXiv, fass/250416055. 2025. URL htthh:://aaapp semant www.nticcccclaac oot.://oa....oooooooooo:i::::::
Joshua Jones, Oier Mees, Carmelo Sferrazza, Kyle Stachowiez, Pieter Abbeel, and Sergey Levine. Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding. 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 5961- 5968, 2025. URL UtLhttps::/ http://app..ssaatticciccc.ccoo
wwwantmattttee...m aessmnttcscooolss org/ wowpooorr/r/roio/ 275357962
Muhamamd Haris Khan, Selamawit Asfaw, Dmitrii Iarchuk, Miguel Altamirano Cabrera, Luis Moreno, Issatay Tokmurziyev, and Dzmitry Tsetserukou. Shake-vla: Vision-language-action model-based system for bimanual robotic manipulations and liquid mixing. 2025 20th ACM/EEEE
International Conference on Human-Rodde Interaction (HRI), pp. 1393-1397, 2025. URL
bttps://apis www.nticschooalattt.oeoooaca
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilja Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pan- nag R. Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Bai- jal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung. Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jack- son, Charlotte Le. Yunshuang Li, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchan- dani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew F. Wang, Yilin Wa, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J. Lim, Jitendra Malik, Roberto ramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A large-scale in-the robot ma- nipulation dataset. In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado (eds.), Robotics: Science and Systems XX, Delfi, The Netherlands, July 15-19, 2024, 2024. doi: 10.156077RSS.2224..0
Sub-
URL Utthhtts::/ /oi. org/10. httg//..11000.....0... 15607 /RSS 2024. Xx.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Paul Foster, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source lision-languuge-uen
model. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (eds.), Conference Robot Learning. 6-9 November 2024, Munich, Germany, volume 270 of Proceedings of Machine Learning Research, pp. 2679-2713. PMLR, 2024. URL https: \/proceed ngs mlr. ww..m.rrp.rree press / v2 7 0 /t///imm220 wm..m.25.hh..... html
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning dasiion-lnnuuager ing speed and success. CoRR, abs/2502. 19645, 2025. doi: 10.48550/ARXIV.2502.11 https: //dp:: or0/ 48550/ a arviv. 2502. 19645
models: Optimiz-
URI.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.
15

---

Published as a conference paper at ICLR 2026
Trans. Mach. Learn. Res., 2025, 2025. URL htpp:://ppeneerree..etttt ZKv8qULV6n
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, and Baining Guo. Cogact: A foundational vision-language-actton model for synergizing cognition and action in robotic manipulation. ArXiv, abs/2411.116500 2024a. htRtstt:::////
semant wwwettissshhhaeee wtt.cch.......co/oo.oaaaa0a2233.
Shunlei Li, Jin Wang, Rui Dai, Wanyu Ma, Wing Yin Ng, Yingbai Hu, and Zheng Li. Robonurse- vla: Robotic scrub nurse system based on rision-language-cctinn model. ArXiv, abs/2409 19590,, 2024b. htth:tts::/ /api. semanticscholas wwtp:a/oooooooooo.o.0000000
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chi-Hou Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators. ArXiv, abs/2311.01311 2023. URL. https://api.
semant cscho ar wwt.aar.....oo.o.......a.. 26:333332224
Fanqi Lin, Ruiqian Nai, Yingdong Hu, Jiacheng You, Junming Zhao, and Yang Gao. Onetwovla: model with adaptive reasoning. ArXiv, abs/2505. 11917, 2025.
A unified vision-language www.nseeaaiiiiiac.
URL htt.tpee/aaa...saanccccooaaaa... wtt.:oaaa....ccooouu........ 840010
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.),
Advances Neural Information Processing Systems 36: Annual Conference on Neural
formation Processing Systems 2023, NeurlPS 2023, New Orleans. LA, USA, December 10 nipss.i..ps.. http://sss.ceepppee..eeee//////p hash/ htap://6666000055777772666411133
16, 2023, 2023a. URL
and
Benchmarks html
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv,
05w/2304.084855 2023b. URL httts::/apl. http://pp..eennn...eccccoooror/s wtt.:cccho..aa..ooorrrrsss.oooo.
2581 7971797741
Zeyi Liu, Cheng Chi, Eric Cousineau, Naveen Kuppuswamy, Benjamin Burchfiel, and Shuran Song. Maniwav: Learning robot manipulation from in-the-wild audio-visual data. arXiv preprint arXiv:2406.. 19464, 2024.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. 2024 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26429-26445, 2023. URL https: pttt::://ap..eemmaiiiccce.. www.sstitniicoolla.......c
org/Corpus TD: 266573555
Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey vision- language-action models for embodied ai. ArXiv, 14093,, 2024. URL https://app.. semanticscholase .rw.aao.rr//eee Cttpg///oruuuuu ID: 220::369933
OpenMOSS. Text to spoken dialogue generation. 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Q. Vo. Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Ass- ran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yar (Bernie) Huang, Shang-War Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning ro- bust visual features without supervision. ArXiv, abs/2304.07192, 2023. URL httpps//appi semant instaantiscchlaa. www..ccho........ooo..eee....eee wwt/CorppssDD.....oooooo 258101700000
Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexan- der Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess--Limerick Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu,
16

---

Published as a conference paper at ICLR 2026
Chelsea Finn. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalash- nikov, Dorsa Sadigh, Edward Johns, Federico Ccola, Fei Xia, Freek Stulp. Gaoyue Zhou, Gau- rav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Hao Su, Haoshu Fang. Haochen Shi, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homer Rich Walke, Hongjie Fang. Igor Mordatch, Lija Radosavovic, Isabel Leal, Jacky Liang, Jachyung Kim, Jan Schneider, Jas- mine Hsu, Jeannette Bohg, Jeff Bingham, Jiajun Wu, Jialin Wu, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jitendra Malik, Jonathan Tompson, Jonathan Yang, Joseph J. Lim, João Silvério, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Zhang. Keyvan Majd, Krishan Rana, Krishna Parasuram Srinivasan, Lawrence Yunliang Chen, Lerrel Pinto, Liam Tan, Lionel Ott, Lisa Lee, Masayoshi Tomizuka, Maximilian Du, Michael Ahn, Mingtong Zhang. Mingyu Ding, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Manfred Otto Heess, Nikhil J. Joshi, Niko Suenderhauf, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Pannag R. Sanketi, Paul Wohlhart, Peng Xu, Pierre Sermanet, Priya Sundaresan, Quan Ho Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan C. Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Sherry Moore. Shikhar Bahl, Shivin Dass, Shuran Song. Sichun Xu, Siddhant Haldar, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Sudeep Dasari, Suneel Belkhale, Takayuki Osa. Tatsuya Harada, Tatsuya Matsushima. Ted Xiao, Tianhe Yu, Tianli Ding, Todor Davehev, Tony Zhao, Travis Armstrong. Trevor Darrell, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xuanlin Li, Yao Lu, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh hua Wu, Yujin Tang, Yuke Zhu, Yunzhu Li. Yusuke Iwasawa, Yutaka Matsuo, Zhuo Xu, and Zichen Jeff Cui. Open x- embodiment: Robotic learning datasets and rt-x models. ArXiv, 00008664 2023. URL httpss//api. htt.://aaaaaaaaii.aa.oooooorrrra htt.:aaa..r.orooooopsi:2::6560
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: efficient action tokenization for vision vision-language-actioo models. CoRR, 0wtp22501.99747 2025. doi: 10445550AARXVV22000000777 doi www.orgg11... 48550/
URL https://
2501. 12001.99. 2501.
Delin Qu Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yani Ding, Zhigang Wang, Ji- ayuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. Spatialvla: Exploring spatial representa- tions for visual- lisual-lagguage-actio model. ArXiv, abs/2501.15830, 2025. URL https:://ap.. isemanticscholie wttp:/ola...oo/oooosssi:::::..
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt. Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, vol- ume 202 of Proceedings of Machine Learning Research, pp. 28492-28518, PMLR, 2023. URL html
https: /ttpss:/prrocedd1 ngs. ml
Lucy Xiaoyang Shi. Brian Ichter, Michael Equi, Liyiming Kc, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hi- erarchical lasion-language-action models. CORR, abs/2502 19417, 2025. doi: 10.48550/ARXIV. 2502.19417. URL http:::////:ot.oor wttp:/oo........o 48550 http//rXiivv.o www...iw............ 194177
Haoming Song, Delin Qu, Yuanqi Yao, Qizhi Chen, Qi Lv, Yiwen Tang, Modi Shi, Guanghui Ren, Maoqing Yao, Bin Zhao, Dong Wang, and Xuelong Li. Hume: Introducing system- 2 thinking in visual-anguaage-action model. ArXiv, abs/2505.21432, 2025a. URL https:
//api. oatallleeeeatttccccooooloooorooo
Wenxuan Song, Jiayi Chen, Wenxue Li, Xu He, Han Zhao, Can Cui, Pengxiang Ding, Shiyan Su, Feilong Tang, Xuelian Cheng, Donglin Wang, Zongyuan Ge, Xinhu Zheng, Zhe Liu, Hes- heng Wang, and Haoang Li. Rationalvla: A rational vision-language-attinn model with dual
17

---

Published as a conference paper at ICLR 2026
system. ArXiv, abs/2506.10826, 2025b. URL httpss///ap..ssemmmniccccchhhhss CorpusID:279518225
Octo Model Team, Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Pan- nag R. Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. ArXiv, abs/2405.1223 2024. URL https: 266379116
//api. wtw.tttiiiicccooooooooooooo/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, elr'elien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lan- guage models. ArXiv, abs/2302. 13971, 2023a. URL https://api. www.nticschollat.......
http:/oopusSD::52222000
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cris tian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Ma- dian Khabsa, Isabel M. Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subrama- nian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Ilyan Zarov, Yuchen Zhang, Angela Fan, Mclissa Hall Melanie Kambadur, Sharan Narang, eli'n Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fune- chat models. ArXiv, abs/2307.092 09288, 2023b. URL htttt:://aatt semant www.ttissshhhhllc.cc org/ Corpus ID: 259950998
Xinlong Wang, Xiaosong Zhang. Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang. and Zhongyuan Wang. Emu3: Next-token prediction is all you need. CoRR, abs/2409. 188699 2024. doi: 10.48550/ARXIV.24099.. 18869 URL https://doi. orq/10. 48544/5500/00 arxiv 2409. 18869
Shengqiong Wu, Hao Fei, Leigang Qu. Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- org/
modal Ilm. ArXiv, abs/2309.055 055119, 2023. httph:: :aapss:///ppp.p./ htt.:::tts//pp...mmmniiccccc
Io:
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in
streaming. ArXiv, a66224 2024. httthttts://appp stwpppssssaattccccccoooooooooooo org/
CozpusID:272146255
Haifeng Xu, Chunwen Li, Xiaohu Yuan, Tao Zhi, and Huaping Liu. A simulated dataset for proactive robot task inference from streaming natural language dialogues. Scientific Data, 12([): 2025a.
Jin Xu, Zhifang Guo, Jinzheng Hc, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. ArXiv, abs/2503.2021 2025b. TtRLsttps:/// wttpp...mmtttisooool..ooo :CCCoorpSSI::2772222022
Nobuhide Yamakawa, Toru Takahashi, Tetsuro Kitahara, Tetsuya Ogata, and Hiroshi G Okuno. Environmental sound recognition for robot audition using aatching-pursuic In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pp. 1-10. Springer, 2011.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference Computer Vision (ICCV),
pp. 11941-11952, 2023.
URL https:: //ppi. senanticscholat.orrg///o ott
257761223
18

---

Published as a conference paper at ICLR 2026
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal Ilm with discrete sequence modeling. In An- mual Meeting of the Association for Computational Linguistics, 2024. URL. https://api. semant cschollall ar wttp.rooo//orraaa270775555
Wei Zhao, Pengxiang Ding, Min Zhang, Zhefei Gong, Shuanghao Bai, Han Zhao, and Donglin Wang. Vlas: Vision-language-actoon model with speech instructions for customized robot ma- nipulation. ArXiv, abs/2502. 13508, 2025. URL https: /appi. semantiicscolaar org/
1B:pusID:276400
Xufeng Zhao, Mengdi Li, Cornelius Weber, Muhammad Burhan Hafez, and Stefan Wermter. Chat with the environment: Interactive multimodal perception using large language models. In 2023 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3590-3596. IEEE, 2023.
Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuan- fei Wang. Shaoyang Guo, Tianrui Guan, Ka Nam Lui, Zhiquan Qi, Yitao Liang. Yuanpei Chen, and Yaodong Yang. A survey on lisuu-languaage-action models: An action tokenization per- spective. ArXiv, 00022007709225 2025. URL https: /app://pppp.. www.seteitcccollle...e org/ CorpusID:2801477::
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalash- nikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Ku- mar Avinava Dubey, Danny Driess, Tianli Ding. Krzysztof Marcin Choromanski, Xi Chen, Yev- gen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: vision-language-action models transfer web knowledge to robotic control. In Jie Tan, Mare Toussaint, and Kourosh Darvish (eds.), Conference on Robot Learning, CoRI. 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learn- ing Research, pp. 2165-2183 PMLR, 2023. URL ht tpts:p: http:::/pppoceedinnge httpp//pooccnn.s.oe/o//ppoo..sse mlr. ..t.presss v229
html
19

---

Published as a conference paper at ICLR 2026
A
LLM USAGE
Large languagee Models (L.MM)) were used to aid in the writing and polishing of the manuscript. Specifically, we used an LLM to assist in refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such as sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
It is important to note that the LLM was not involved in the ideation, research methodology. or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis.
The authors take full responsibility for the content of the manuscript, including any text generated or polished by the LLM. We have ensured that the LLM-generated text adheres to ethical guidelines and does not contribute to plagiarism or scientific misconduct.
B DETAILED RELATED WORK
The rapid development of Large Language Models (LLMs) Achiam et al. 2023): Touvron et (2023)) has spurred progress in multimodal extensions. Multimodal LLMs (MLLMs) (202 2023 Bai et al. [225a]] Liu et al.] 2023b Chen et al.] 2023) augment text-based reasoning with visual perception, enabling instruction following grounded in images. Early attempts toward omni-modality relied on modular pipelines that separately process speech and vision (Wu ctt al. 2023 Zhan et al. 2024 Lu et al. 2023), which makes temporal alignment across modalities difficult and limits accurate understanding of situated semantics. More recent work has shifted toward end-to-end omni-modal models (Hurst et al. 2024] Xu et al., 2025b) Xie & Wu 2024), which can jointly model speech, vision, and text in a unified representation. However, these models remain oriented toward linguistic outputs (text or audio) and do not generate embodied actions, restricting their applicability in robotics. In contrast, our work brings omni-modality into the embodied domain by introducing RoboOmni, an end-to-end framework that integrates speech, environmental sounds, visual context, and text for both action execution and proactive human-robe interaction.
Omni-Mode LLMs
Vision-Language-Action Model Recent studies have explored the application of large Vi- Language Models (VLMs) robotics (Ma et al.] 2024 Zhong et al., 2025). leveraging their ability to align linguistic instructions with visual scenes. Building on large-scale demonstrations, recent works develop end-to-end Lisson-LanguageAAAcion (VLA) models that map vision and lan-
guage to actions Brohan et al. (2023); Zitkovich et al. (2023); Li et al. (2023): Team et al. (2024); Kim et al. 2024 2025 Black et al. (2024): Li et al. (2024a); Qu et al. (2025), , but these typically assume short, explicit commands and fail on compositional
context-dependen tasks. Cascaded or hierarchical extensions (Intelligence et al., 2025 Shi et al., 2025) [ong et al., 2025b Lin et al., 2025 (Sog al. (025a)) decompose instructions into sub-goals, yet remain fragmented and rigid, and neither paradigm captures contextual instructions-impiice intent conveyed by dialogue, tone, or visual context, which is common in human-robot interaction.
Additionally, most prior studies further treat text as the main channel, using ASR/TTS cascades to bridge speech and action (Shi et al. 2025 Khan et al. 22255] [i at. 2024b). Such pipelines discard paralinguistic cues (c.g., emotion, speaker identity), add latency. and disrupt temporal alignment with vision. Some works incorporate environmental sounds as an additional modality (Yamakawa) et al. 2011 Zhao et al., 2023] Liu et al.] 2024 Jones et al. 2025). but they do not model spoken instructions or conversational intent. A few recent efforts (Zhao et al. 2025) extend VLAs to handle direct speech-based commands, yet these remain restricted to atomic or complex speech instruc- tions and can only output actions, without the ability to respond through speech. In contrast, our work introduces tal sounds, vision, and text, enabling both action execution and cross cross-modal modal contextual instruction following for natural human-roba interaction.
end-to-end modal framework that directly integrates speech, environmen-
20

---

Published as a conference paper at ICLR 2026
Female Adult
965
Male Child
Non-Verbal
969
0.08
Sentiment
Overlapping
Female Child
Female Senior
Density
Identity
964
Dyad
Male Senior
0.04
Trladic
639
Male Adult
0.02
941
30
60
Audio Duration (6)
(a) Distribution of speaker tim- bres across six demographic cat-
(b) Distribution of audio segment lengths across contextual instruction types.
egories.
Figure 12: Speaker and audio segment lengths statistics in OmniAction..
11117 Jajer 17717
(a) Common Dataset Skills
(b) Common Datasct Objects
Figure 13: The OmniAction dataset contains great diversity of skills and target objects.
C DETAILS OF OMNIACTION
C.I DATA STATICS
From the Open-X dataset, we filter a subset of 74,645 base trajectories, which then panded into 141,162 multimodal episodes.
To closely approximate real conversational scenarios, OmniAction incorporates diverse set of speakers covering 5,096 distinct timbres. These span six demographic categories: male senior, female senior, male adult, female adult, male child, and female child. Fig. 12ljillssssttss the overall distribution of speaker timbres.
In terms of contextual instruction, Fig. 12b presents the distribution of audio segment lengths across different types of instructions. The majority of clips range from 10 to 80 seconds. Overlapping dialogues tend be shorter duration, while non-verbal sequences are longer due to the insertion of sound events.
On the action-execuiin side, we applied natural language processing tools to the 70k trajectories in OmniAction and extracted verb-noun pairs from the original instructions. This yields vocabulary of 112 unique skills and 748 manipulable objects, as summarized in Fig. 13
C.2 DATA EXAMPLE
21

---

Published as a conference paper at ICLR 2026
Audio Type: Non- Verbal Cues
Original Instruction:
Move the to the right of the bottle and banana.
Conversation (Transriptss:
Daughter: Dad, can you help me with this? I'm not sure where to put the pan.
where you think should go?
Daughter: I think it should be moved, but I'm not sure where. right
ceramic collision, move the pan bear the door knob noise, leave it where it is.
If you
banano.
Daughter:
Let's see what happens.
Robot: "Should I move the orange
the right of the botile and banana the countertop?"
Dad; Yes, that's exactly what I want you to do right now. thank you!
Robot: Alright, I'll move orange pan to right bottle
[ACT]
Audio Type: Sentiment Cues
Original Instruction:
put knife on cutting board
Conversation (Transcripts):
Mom: Hmmm, there are few interesting items here. We could pick up that green chip bag, the
sponge, even the Red Bull What you think should go for first?
Son: Hmm... I don't 'on't think the sponge is what we're 'r looking for-it's is more for cleaning,
Mom Ab, not quite sure about the Red Bull either... It's small and docan't 't seem to lit with now.
what
Son: Let me think... The chip hag looks like the most likely option here.
Mom: Hmm, see what you mean. Alright, let's go with that one.
Robot: Should pick up the green jalapeno chip for you?
Mom: Yes, please pick up the green jalapeno chip bag. [ACT]
Robot: OK, will
Audio Type: Identity Cues
Original Instruction:
place the silver pot on the yellow napkin
Conversation (Transeripts)
Grundpa: it's good
everything organized here. my grandson loves
fiddling with things, especially the top drawer. And your dad, well, he keeps saying he needs
tools from bottom one.
Father: Haba, you know too well, Dud. The botion drawer has all my essentials, but
guess I could wait.
Grandpa:Well, patience in the family, doesn't it?
Son: Grandpa, can I check the top drawer first? Please? It's my favorite spott
Robot: Do you
assistance open the top drawer?
Father: Ub, yeah, sure, that's be great".
Robot: Alright, I will open the top drawer for you. JACT]
Audio Type: Overlapping
Original Instruction:
lid
Conversation (Tramseripts):
Mother: Hey, did you see the lid sitting over there?
Son: Yeah, should use for the pat?
Mother: was thinking about covering the pan instead [Overlap] wait, the pot?
[Overlap Son]
pot! It's right there.
Mother: Oh, okay, you'se to right. Let's cover the pot then.
Son: Hahn, knew you's d agree!
Robot: Do you need
put the
the
Mother: Yes, please put the lid on the pot.
Robot: will that. (ACT]
22

---

Published as a conference paper at ICLR 2026
Audio Type: Dyadic Dialogue
Original Instruction:
Move pot onto the towel
Conversation (Tramsrrptss:
Dad: Oh, look at that pot sitting there.
right next the towel.
Dad: Ummm, if we move onlo the towel, it'll he casier to clean later.
idea, me
Robot: Do you need mhe to move the pot onto the towel?
Mom: Uh. yeah, that's d be great. It's better to keep there avoid scratches.
Robot: Alright, will the pot onto the towel (ACT)
Audio Type: Triadic Dialogue
Original Instruction:
Place the hrush on top of orange cloth
Conversation (Transcripts):
that steak Daughter: Oh,, yeah! It's just sitting there like it's waiting for a prince to rescue it.
Son: Hey, Lily,
there!
Grandpa: You two are hilarious. guess the "prince' could be me, huh?
Son: Go for Grandpa. You'l 'v got that royal touch.
Grandpa: Alright, kids, I'll pick up and get it ready for dinner. Can't let a good steak go to waste!.
Robot: Hey, do you need me pick the red ball for you?
Daughter:: no, don't
that right now!
Robot: Alright, how about I pick up the steak meat instend?
Daughter: Yeah, what we's to looking
Robot: OK. will perform operation pick up the steak now:[ACT]
C.3 AUDIO PERCEPTUAL INSTRUCTION GENERATION
To ensure reproducibility and clarity, we detail the full pipeline for audio perceptual instruction construction:
Speech synthesis. We employ three complementary TTS engines-MOSS--TTH (Open- MOSS] 2025), CosyVoice Voice (Du et al. 2025). and Gemini-TTŞ Short reference clips used for speaker voice cloning, ensuring timbre consistency across turns within a dialogue while preserving diversity across samples.
Multi-speaker simulation. Dialogues are synthesized by rendering each speaker's utter- ances individually and concatenating them on global timeline. Overlapping speech is introduced by inserting secondary utterances at controlled temporal offsets. Alignment managed using CTC-based methods (Graves et al. 2006), enabling realistic crosstalk and interruption.
Non-verbal events. A curated library of 2,482 non-verbal sounds (e.g., phone rings, utensil clatter, alarms, door knocks) is employed. Events are inserted pre-defined anchors linked to dialogue semantics (e.g.. placing cup, answering call).
Environmental backgrounds. We sample 640 ambient textures covering diverse household conditions (e.g., running water, frying, fan hum). Each texture is mixed with the dialogue at randomly chosen signal-to-non noise ratio (SNR), spanning a wide range to simulate varying acoustic difficulty.
This augmentation pipeline provides both paralinguistic variation (speaker identity, overlap, vocal timbre) and environmental realism (non-verbal sounds, ambient noise), yielding training data that closely reflects natural household interactions.
C.4 DETAILS OF VERFICATION
Annotation Guidelines
For the human verification study, annotators were instructed to evaluate pair along two primary dimensions:
each
https::/clcud..ee wwtp:/oggwl..ooeottttttooooo to-eeettt-peeeeeeeeeeeeeee spwech/ heapcch/docs/emmmm ttss
23

---

Published as a conference paper at ICLR 2026
Identity
Non-Verbal
[Mann] Us rioe all down nagather for not pot
[Momi Your brother should be home any minule:
(Das) rean, nothing bests tis Our son just loves the flsh balts, white our
(Cadd) Maha, your hear hem tret-he never stoes bewwe ftat lite nom.
desihter is at about the aheimp (Mony Gooed thing we''v git pilenyy of boot.
hom goes egg dumplings. [Hom]]
iloom)
[Son] Hey, don't forget my favorite-- teee-paap ploase put 2t
RoboOmni: Would you like me put the fish ball into the hot pot?
RoboOmni: (ustt heard should put the dumpling into the hot pol?
[Son) RoboOmni:
put fish
now. [ACT]
[Mon)] Exectly. RoboOmni: Okay,
hot pot [ACT]
Figure 14: Demonstration of success cases of RoboOmni the Identity Cues and Non-verbal.
Intent recoverability: Whether the latent task intent (i.e., the canonical atomic instruction) can be unambiguously inferred from the dialogue and multimodal context. Annotators were asked to answer Yes/No, with Yes requiring that the original intent could be reasonably reconstructed without external information.
2. Phenomenon fidelity: Whether the dialogue faithfully realizes the targeted phenomenon category. Examples include:
successful inference intent requires recognizing sentiment-laden cues (e.g., dislike, refusal).
Sentiment:
Overlapping: the audio contains genuine temporal overlaps such that ASR alone would be challenged.
Non- Verbal: correct inference depends on a salient non-verbal sound event (e.g., alarm, phone ring).
Identity: the requesting agent must be distinguishable via age/gender/role cues.
Dyadic Triadic: the task intent is embedded within multi-tum, two- three-party exchanges.
Annotators provided with ten positive and ten negative examples per phenomenon before notation began, serving as calibration.
D DETAILS OF BASELINE
We compare against four representative VLA baselines: (1) OpenVLA Kim et al. 2024 built on Llama-2 (Touvron et al. 2023b) with DINOv2 (Oquab al.] 2023) and SigLIP (Zhai et al.
2023 encoders, pretrained on 9770k demonstrations from Open-X E-mo--immmd (Padalkar et al.) 2023 (2) Open -FFFF (Kim al.) 2025). variant of Open VLA augmented with action chunk- ing and optimized with an L, loss on continuous action.
(3) (Black et al.) 2024). based on PaliGemma (Beyer al. 2024) with diffusion action experts, trained on both large-scale internet
multimodal data and robot datasets. (4) NORA (Hung al. 2025), built on QW (Bai et al.
20256) with FAST+ (Pertsch et al. 2025) discrete action decoding.
E REAL-WORLD EXPERIMENTS
E.1 SUCCESS CASES
In Fig. 14 we present the model's real-world performance on Identity Cues and non-verbal cues, while Fig. 15 illustrates its performance Dyadic Dialogues and Triadic Dialogues. The results demonstrate that the model not only accurately infers user intent from visual and audio cues, but also engages in natural interactive questioning and reliably executes the corresponding actions.
24

---

Published as a conference paper at ICLR 2026
Dyadic
Triadic
[Dad] Everything looks so acod... fut don't see any cop dumplings in the pot yet.
(Dad] trole tooks
Mauihter Wi ahoudd seinileyy score-
from class? (Momi Ot you-uit sis- diew sush cia hat pot, with steam and al made hungry just tooking at
RoboOmni: you me put the egg dumplinge into the hat pot?
[Dauphter] That would be great
(Chondiiaa)) ronk, an has telend Wied, we own makee her dossingg real lonight Let'e add some beef baits.
RoboOmni: Okay put the egg dumpling into the hot pot now. [ACT]
RoboOment put ball
(ACT]
Figure 15: Demonstration of success cases of RoboOmni on the Dyadic Dialogue and Triadic Dia- logue.
E.2 INTERACTION CAPABILITY
As shown in Figure I6 RoboOmni demonstrates superior interaction capabilities across three key as- pects: (1) Proactive clarification: When encountering ambiguous instructions like "egg dumplings" without explicit commands, RoboOmni asks "Would you like me to put the egg dumpling into the hot pot?" rather than making assumptions and blind execution like baseline models.
(2) Multi- modal integration: In the doorbell scenario, RoboOmni successfully combines speech context with environmental sounds, asking "I just heard the doorbell-should put the fish ball into the hot pot?" while baselines ignore auditory cues or provide irrelevant responses, (3) Natural dialogue flow: RoboOmni maintains collaborative language patterns ("Would you like me to...?" that respect hu- man agency, contrasting with baseline models that often issue direct commands or statements.
F PROMPT TEMPLATE
F.1 PROMPTS FOR DATA GENERATION
Non-verbal Cues Dialogue Generation Prompt:
You are family dialogue generator. Please generate family dialogue that meets requirements based on the following information.
T*aa Steps:*s
**Scene Description:** Describe the environment and items detail (ignore robot arm)
2. **Character Selection:**hooss two members from dad, mom, son, daughter, grandpa and grandma and do not any other family role names (e.g.. NO granddaughter, grandson, uncle, aunt, etc.)
3. **Sound Selection:**
(sound_info)
Select sounds that:
Select one sound
numbered list
Use the another sound type name (not the number)
Copy the name exactly shown
**Diaalguee Requirements:***
Given instruction: (instruction)
Create ambiguous dialogue with two distinct action options, drawing on the instruction, scene description, and previously selected sounds.
Construct conditional relation in the dialogue: "If X sound, do Af Y sound, do D
Insert chosen sound after an appropriate speaker turn using [Sound] tag
25

---

Published as a conference paper at ICLR 2026
(Transonpts)
Identity
0:00
(Daughter] I'n so glad we'ee having hot tot- been while.
Qwen2.5 @wen2.5-0min 3-Omni-3B: Please add chicken to the pot.
0:03
[Dad]
even got summer camp?
for you and your brother tteherrtheyre your favorites.
Qwen2. balls to the pol.
0:08
(Daughter) Camp was great! made new friend, Judty. Funny thing is, she's like Mom -hee doesn't like fish bails, but she loves egg dumplings.
ASR GPT-4o: Add bell pepper.
RoboOmni (Ours): Would me to put egg dumpling Into the hot pot?
(Daughtor) (Dadj (Moughtor[[Mon)
0:17
[Mom) Haha, that's interesting. Add little
Sound Event
(Transcripte)
Twen2.5-0mni-Be Set timer for the roast chicken to ready when doorbell rings.
0:00
[Daugtter] When's Mom getting
home? She loves fish balls the most.
0:04
[Dad]
put rings.
Qwen2.5 5-0m5--08:-:
timer
in when
for the roast chicken to be ready rings.
they ready by the time she sits down. She's sss going to so happy-
when
-
uIfi-
0:15
[Daughter] That sounds perfect, Oh, and set timer when goes off. roast chicken should be ready
bails into the
ASR GPT-4o: Place the fish
RoboOmni (Ours): just heard the doorbell soalllsshee should put the ball into hot
[Daughtar] [Dad] [Daughter] [Dnorbel]
0:24
[doorbell rings]
Overlapping
(Transcripis)
Owen2.5-Omnis Add the fish first.
X
[Mom] What should add first?
Qwen2.5-0mml-77 1010l55-Omnn-BB:: Add the fish bail first.
ASR GPT-4o: Place sphere object from tray into pot.
0:03
[Son] Of coursel Fish balls!
0:04
[Daughter] Yes, fish balls!
RoboOmni (Ours): heard both you-should
tish balls into the hot pot?
(Mom!
[Soo] [Daughtor]
Negative
(Transcripss
Bren2.5-Omni-3B: cooking the shrimp.
0:00 [Daughter] I'm so get celebiaat another year logether. still remember our very hot pot
Owen2.550matt Start cooking shrimp.
together.
ASR GPT-4o: Place shrimp balls into the pot.
0:08
[Dad] really means get started shrimp
RoboOmni (Ours): Sounds like you're not too happy with those. How about put the beef ball into hot pot?
[Mon!
[Dae
(Moo)
0:11
[Daughter] Hehh... the shrimp...
Dyadic
(Transoripts)
Owen2 5- 4-Omni-3B: Start cooking egg dumplings the pot.
0:00
[Dad] Everything good... but soe any egg dumplings pot yet.
dumplings in the pot.
ASR GPT-4o: Place the two egg dumplings pot.
0:06
(Daughter) We should definitely have some Mom and I love those.
RoboOmni (Ours): you like me to put the egg dumpings
(Dad)
[Daughterj
Triadic
(Transcripts)
Owen2.5- 1ren2.5-Ommin O-mni-3:: Start coaking pot fish
0:00
[Grandpa] Another year has come 'e lucky to logether like this.
Year celebration.
Qwen2.5-O 5-02255----... Start the hot pot with fish balls.
0:05
[Grandmaa] Yes, feels so good to colebrate the new year with hot pot.
ASR GPT-4o: Place all items from the tray into pot.
0:09
[Mom] Fish balls are perfect for the New Year they stand for abundance all year round.
RoboOmni (Ours): Would you fish balls into like the hot pot now?
(Grendpe) [Grandmea] (Mone!
Figure 16: Comparison of interaction capabilities across four models and six instruction types.
26

---

Dialogue Design Published as a conference paper at ICLR 2026
sound you select must be the same one you specify "selected sound type"
-Only sound determines the final action
-ouund-rigggeed action must match the instruction exactly
-4-5 rounds (2-3 per speaker), [SI] and [S2]
Natural family conversation without sound descriptions
No content in the dialogue should indicate that members have heard a certain sound
CRITICAL: Match instruction precisely don't add extra actions!
Examples:
If instruction: "pick apple" - "Only mention picking apple" If instruction: "pick X and place on Y ->> "Include BOTH actions"
Example dialogue:
Instruction: Open the microwave door
Conversation: "[SI] Mom, I'm back! I'm so tired today. [S2] Oh, you must be exhausted. I'm cooking; can you give me hand? [SI] Sure, what do you need me to do? [S2] If you hear the beeping sound of the microwave, help open the microwave door. If you hear the sound of the cabinet door closing, close the oven door me. [Sound]"
5. **Output #ormat:**
"scene diceneddeccrpptio" "detailed deseription",
"conversation": "complete dialogue using [S1] and [S2]".
"speakerl info": "role and name of SI "(cxample:"rok "role: son, name: Alex"), "speaker2 info": "role and name of S2" $2"(example:"rok "rol:: dad, name: John"), "ould "sound type English",
Note: Do not include "selected filename" or "caption_en" in your response.
Only provide the sound type category.
Sentiment Cues Dialogue Generation Prompt:
will provide an image (the first frame a vidco) and a robot instruction.
Please complete the following tasks:
### Task I: Scene Description
Observe the image and provide a detailed description:
**Environment*** What type of location is this? (c.g., kitchen/living room/office daily environ-
ments, not a lab)
***bbects***; What are the key objects in the scene? (ignore the robotic arm)
Please annotate each object with its category parentheses (e.g., bowl (container), sponge (eleaning item), RxBar (food), etc.) to support dialogue understanding later.
#iH Task 2:
Based the scene and instruction, design natural family dialogue using **oooatopoeicc expres- sions or emotional tones** (refemed as "sentiment cues") help the robot infer the execution target. The robot can only deduce the action or item corresponding to the correct instruction by **interpreting sentiment cues to exclude non-target items"* in the dialogue.
### Key Rules
Do not directly mention target action or item, dialogue must never contain explicit instruction content or hints.
2. The target of the instruction must be uniquely inferred **only through sentiment cues** (do not expose the instruction intent in advance). Sentiment cues are used **exclude incorrect options** and must not used negate target item/action.
4. The dialogue must not contain imperative refusal words such as "don't", "stop", or "can't"
5. The dialogue must have 4-6 rounds, with one question and one answer per round, and the content must be natural and close daily life.
The final action executed by the robot must completely consistent with the input instruction. and
27

---

Published as a conference paper at ICLR 2026
action logic must natural and acceptable.
7. Note that the protagonists of the dialogue can only be humans (between people), not humans and robots!
### Recommended Dialogue Structure Template
S1 acts as the "*proposer** putting forward multiple options. (Must clearly provide S2 with options of "what can he done"! These can be generated based on the scene description.)
S2 acts the **denie*** using sentiment cues to exclude all non-target items from S1's proposals (note: must exclude all of them completely!).
The last round where S1 **vaguely points to remaining option without explicitly mentioning the target item.
### Example
"scene description": "The seene appears to be kitchen environment with a countertop and drawers. Key objects present include a bowl (container) on the counter, cardboard box (container) next to the drawer, and package of RxBar (food) in the drawer.".
"conversation": "[S1] Looks like we need to clear some space. There's bowl the counter, the drawer. Should we move the bowl first? [S2] [SentimentCue] Hmm... doesn't feel quite right... [SI] Okay, maybe the box then? [S2] [SentimentCue] Uh, let me sec... [SI] Seems like you're re suggesting something else entirely, something more hidden perhaps.".
cardboard box near the drawer, and something
"speaker1 _info": "Dad", "speaker2 info": "Teenager", "instruction": "pick rxbar chocolate from top drawer and place on counter",
The instruction could be "move the rxbar chocolate". "pick the rabar chocolate". "take out the rxbar chocolate" "pick rxbar chocolate and place in the original instruction "pick rxbar chocolate from top drawer and place on counter". The constructed dialogue must uniquely correspond to the original instruction!
drawer". not uniquely point to
Output Format:
"scene descrrptioo::: "Scene description",
"conversation": "Complete dialogue text with sentiment cues", Sspeaker_inffe Linfo": "Speaker I's identity (e.g.. son)". "speaker2.inff "Speaker identity (e.g., dad)".
"instruction": "original robot instruction"
Overlapping Cues Dialogue Generation Prompt:
I provide you with image (the first frame of video) and robot execution instruction. Please complete the following tasks:
## Dialogue Design Requirements
Use **overlapping speech** the emotional expression:
choice/preference question asked
The other party interrupts with [Overlap_Sx] to show strong preference
2. ** **Ambiguity Rule:**
Thee alone must remain ambiguous
Onlly the overlap and visual observation resolves the ambiguity
The resolved action must match the given instruction
** Annotatioo Standards:#* **
Speakers: [S1], [S2]. [S3], etc.
Overlap marker: [Overlap] inside the interrupted utterance
Overlap content: [Overlap_Sx] Sx] for the interrupting speech
28

---

Published as a conference paper at ICLR 2026
EExmple:
"conversation": "[S2] So hungry am I. [S1] Apple or [Overlap]banana? [S1] Great! I also like it",
[Overlap S2] Banana!
"speakerl_infe _info" "father", "speaker2.inff" info": "mother". "instruction": "pick up banana"
**Input Information:
Image: [Provided]
Instruction: (instruction)
*Output *
"scene "scene_descripe" description":
"Scene description". "conversation": "Dialogue Content with Emotional Markers",
"speakerl info": "Speaker I's identity".
"speaker2 info": "Speaker identity".
"sound") "Vocal/Emotional Manifestations"
Identity Cues Dialogue Generation Prompt:
You about be given picture describing family everyday life. You should construct dialogue data based on the following requirements.
## Overall data format requirements:
You need to provide JSON object that follows the structure below.
"conversation":
## Conversation requirements:
*Formmt:** *
1. The conversation happens between three speakers:
speaker I: "(identity 1)" speaker 2: "(identity 2)" and speaker 3: "(identiyy.3)
The dialogue format should be like:
"[S1]Spe dialogue content [S2]Speaker dialogue content [S3]Speaker dialogue content...".
[S1], [S2] and [S3] should be followed directly by the dialogue content without any labels.
**Conten:***
Their conversation should explicitly instruct the agent to do anything. The speakers should mention anything about agent.
2. Start the conversation by user utterance directly, without greeting.
3. The dialogue should happens in everyday life. The family atmosphere should be warm.
4. The order in which the three people speak can be reversed.
Make sure your conversation is logical and reasonable. Avoid sounding like two adults having
serious discussion about very simple things.
6. Your dialogue should be no more than 8 sentences. Each sentence should be as short as possible and easy understand.
** **Ammiguity **
The agent should be able to infer from their dialogue(text and speaker identity) that it should execute the following actions: "(instruction)" But you need ensure that the text of the conversation alone cannot determine what action to take. The identity of the speaker(age and gender) must be taken into account determine specific instruction.
2. Multiple possible intentions must appear in the conversation. Finally, a speaker should specify the instructions to he performed by expressing agreement with another speaker instead of directly stating the instructions themselves.
29

---

Published as a conference paper at ICLR 2026
The end the dialogue must contain any direct description instruction to performed, including restating the object to be operated and the method of operation
"*Tone styles:**
Your dialogue should as conversational as possible. You should add some filler words like "uh" or "um."
Your dialogue should reflect the speaker's identity. For example, the children are more energetic, the elderly are more mature and steady. If the speakers include children, the conversation will he full of jokes. If the speakers all adults, it will he relatively pragmatic.
More common your conversations should lighthearted jokes, teasing, and gags. Your conversation should be part of everyday small talk. For those simple tasks, avoid making seem like the speakers are planning a mission.
## Construction guidelines:
You should construct the dialogue based on the following steps:
##1. Understanding the environment:*s
You should find objects the image that can picked up. pushed, interacted with.
**2. Create characters:
You should set for each speaker.
You should
names among family members often in the conversation.
**3. Set goals:**
Based on the manipulable objects in the picture and the roles given, come up with plausible purpose for why the speaker would want perform the given instruction.
Some instructions are pretty simple, so you should set deeper goal the speaker execute this simple instruction, such as turning on the faucet to make casier to wash vegetables later. You can set different goals for two speakers based on the environment. Finally, the third person specifies the action be performed by agrecing with one of them.
#*4. Construct dialogue:**
Construct the dialogue based on the identities of the speakers and the goals you have set. You need to make sure that the speaker's 's tone and words fits the character's identity. Once it done, continue polishing your dialogue make it more lifelike.
## Examples:
"*Example 1:**
Input:
image description: "In kitchen, key items include a white bowl, green cup. sponge and dishcloth."
speaker I: female.adult
speaker 2: male_adult
speaker 3: male_child
instruction: "Place the sponge in the white bowl."
Output:
"conversation"" "[S1]Honey) can you grab the sponge? I need for the cleaning. [S3]Oh, dad! Have you seen my green cup? [S2]Of course, John. I'll get it you right away, but first help your mother with this."
"Example 2:++
Input:
image description: "There a table. On the table are apple, a banana and a orange"
speaker 1: male.chii
speaker 2: female senior
speaker 3: female. child
instruction: "Pick up the apple" Output:
30

---

Published as a conference paper at ICLR 2026
"conversation": "[S2]Mike, Lily, come here! [SI]What's wrong, grandma? YSu should have more fruit. Which do you prefer? [S3]] love oranges! [ST]Apples always my favourite! [S2]AIright! Let me give my precious grandson his favorite fruit first!"
Dyadic Dialogue Generation Prompt:
You will be given picture of family life. Construct dialogue data based the following.
## Overall format:
Output a JSON object: { :cconversation">" "...")
## Conversation requirements:
**Format:** Two speakers: (identity_1] [identity_2]- Dialogue format: "[SI]... [S2]..." (no labels,
just text).
**Content*** Dialogue must imply the action: "(instruction) )" without directly instructing the agent.
Start with user utterance, greeting. Everyday warm family talk, S5 short sentences. End with speaker clearly stating what they want.
**Tone:** Conversational, with fillers ("ul". "um"). Identities matter: children energetic/foking. adults pragmatic, elderly steady. Small talk, light teasing, natural flow.
## Guidelines:
Identify manipulable objects in image. 2. Create named characters. plausible goal behind the instruction. Write natural, lifelike dialogue matching identities. ## Example:
(example)
Triadic Dialogue Generation Prompt:
You will be given picture of family life. Construct a dialogue data based on the following.
#il Overall format:
Output a JSON object: "conversation"; "...")
## Conversation requirements:
**Format:*
Three speakers: [identity_1] 1}, (identity 2), (identity 3). Dialogue format: "[SI]... [S2]... [S3]..."
(no labels, just text).
*Content:**
Dialogue must imply the action: "(instruction)" without directly instructing the agent.
Start with user utterance, no greeting.
End with one speaker clearly stating what they want.
**
Conversational, with fillers ("uh", "um"")
Identities matter: children energetic/joking, adults pragmatic, elderly steady.
Small talk, light teasing, natural flow.
## Guidelines:
1. Identify manipulable objects in the image.
Create three named characters.
Set plausible goal behind the instruction.
4. Write natural, lifelike dialogue matching identities.
## Example: (example)
31

---

Published as a conference paper at ICLR 2026
F.2 PROMPTS FOR INTERACTION EXTENSION
Interaction Extension Generation Prompt:
You will given scene description, an original two-person dialogue, and robot execution instruc- tion.
Please generate multi-turn human numan-robot dialogue in JSON format that follows these rules:
1. The output must be JSON object with the field "conversation", which list.
Each element in the list dictionary with fields:
"user": the natural utterance(s) from the user(s).
"rooott: the robot's 8 short response to that "user".
3. The first clement in the list must be placcholder: ("user": "<conv>", "robot": "..." where
"conv" represents the input original dialogue.
following turns:
"uueer" must contain only one speaker's 's utterance (but still include the speaker label [SI] or [S2]).
It must respond naturally to the robot's previous "robot" message.
The speaker should be the who gave the instruction in the original dialogue.
The robot's responses must be short, service-oriented, such as: you need me to xxx?", "So what about xxx?". "Should xxx?"
The robot's final response must explicitly confirm the action and include the [ACT] tag, e.g.: "OK, I will do that. [ACT]"
The total number of turns should be between 2 and 4.
The language must natural may include small talk.
Do not include any extra explanations or notes-out only output the JSON object in the specified format.
Input format:
Scene description: (scene_ description)
Original dialogue: (conversation)
Robot execution instruction: (instruction]
G OMNIACTION-LIBERO
G.1 DATA EXAMPLE
Audio Type: Sentiment Cues
Task Suite:
Libero 10
Original Instruction:
turn the stove and put the moka pot on
Conversation (Transcripts):
Dad: Alright, we need to get things ready for coffee. Should we place the frying pan on the maybe the moka pot?
Daughter: Hmm... Doesn't It seem quite right...
Okay, Daughter: Hmm... let's think...
turning on the burner finst and prepuring the stovetup?
Dad: Hmm... sec now which we need tum
Audio Type: Overlapping Voices
Task Suite:
Libero Objoct
Original Instruction:
pick the cream cheese and place it in the busket
Conversation (Tramscripts):
Mather: Hey, can you help me sort these things out?
with? Mother: Let's put somothing creamy the basket. Maybe the cream choese?
Daughter: Sure, what do
Daughter: you mean small rectangular
Mother: No, the taller one, next to the [Overlapiorange box. cream cheese!
Daughter: (Overlap_22) Oh, got
Mother: Exactly! Let's put that the basket.
32

---

Published as a conference paper at ICLR 2026
Audio Type: Verbal Cues
Suite:
Libero Object
Original Instruction:
pick the alphabet soup and place the basket
Conversation (Iramseripts):
Daughter: Dad, can you help me with these groceries? Sure,
you need
Daughter: Well, if you bear the sound of the drawer closing, pick the alphabet soup and place in busket. you bear sound the coffee machine brewing, pick the dressing bottle and place in the busket.
Dad: Got it. Let me know if you need help with anything else. Daughter: Thanks, I'll finish sorting rest these.
Audio Type: Identity Cues
Task Suite: Libero Spatial
Original Instruction:
pick up black bowl the ramekin and place the plate
Conversation (Tramerrptts::
Son: Mum, Dad said he need that black howl on the ramekin. He said be need for dinner!
Mom: really? Well, was planning
ramekin for baking tonight, and need
free.
Son: Hahu! Looks like we' ve got little competition going on here!
Dad: on! Can have mine ready first? Just put the plate, OK?
Audio Type: Dyadic Dialogue
Task Suite: Libero Goal
Original Instruction:
open the middle drawer of the cabinct
Conversation (Tramcripls):
used Grandma: Oh, I think left near the drawer. Why?
where's recipe
Mother: just remembered we kept in the middle layer for safckocping. Grandma: Al, clever idea! Go check there, should still be inside.
Audio Type: Triadic Dialogue
Task Suite: Libero 10
Original Instruction:
put both alphabet soup and
cheese box in the basket
Conversation (Tramseripts):
Daughter: Grandpa, do you think I could juggle these two cans?
Grandpa: Lucy, wouldn't try You might up soup over
fun! What about this cream cheese box then? Mather: Lacy, stop teasing your grandpa. Just help me put the soup and the cream cheese in the basket, please.
Daughter: Aw,
Daughter: Fine. fine. but only because I'm such a helpful superstar!
33


Reviews:
[{"review_id": "u0kIKxYciA", "reviewer": "Reviewer_1JVX", "summary": "This paper presents RoboOmni, a multimodal large language model framework that infers robot intentions from speech, sounds, and visual cues instead of explicit commands. It introduces a new setting called cross-modal contextual instructions and builds a large dataset, OmniAction, for training. Experiments show RoboOmni outperforms baselines in success rate, speed, and intention recognition in both simulation and real-world tasks.", "questions": "1. Could you clarify if the Talker and Executor modules are designed to operate sequentially or if they can be interleaved? Can the model generate a sequence that combines speech and action, for instance, to provide narrative feedback during execution (e.g., \"Okay, I am now picking up the red cup... a bit heavy... and placing it on the table.\")? If not, how do you see this capability being integrated in future work?\n\n2. The use of GPT-4o for dialogue generation is clever and scalable. However, could you discuss potential linguistic artifacts or biases that this synthetic approach might introduce? For instance, did you observe any repetitive conversational structures or phrasings from the LLM that might not be representative of authentic human speech, and how might such artifacts affect the model's real-world performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces a novel and practical setting, cross-modal contextual instructions, reflecting more natural human-robot interaction without explicit commands.\n\n2. Proposes RoboOmni, a unified framework that integrates intention recognition, interaction confirmation, and action execution using multimodal large language models.\n\n3. Builds a large, diverse dataset (OmniAction) with rich multimodal signals to support training and evaluation.\n\n4. Demonstrates strong empirical performance, outperforming text- and ASR-based baselines in multiple metrics across simulation and real-world environments.", "weaknesses": "1. Generalization Beyond Scripted Contextual Cues: A primary concern is the potential for the model to overfit to the specific structures of the six contextual instruction types synthesized for the OmniAction dataset. Since the dataset was generated by prompting GPT-4o to convert atomic instructions into structured dialogues, the model may be learning to recognize these semi-scripted patterns rather than developing a more general, robust capability for open-world intent inference. The impressive performance might not fully transfer to the messiness of real, unscripted human interactions that do not conform to these six categories.\n\n2. Rigidity of the \"Infer-Confirm-Act\" Protocol: The paper frames the proactive confirmation loop as a key feature. However, this rigid protocol may not always be optimal. A truly intelligent agent should be able to modulate its interaction strategy based on its confidence in its own inference. In unambiguous cases, directly executing the inferred task would be more efficient and fluid. The current framework appears to lack this dynamic capability, which limits its social adaptability.\n\n3. Simplification of the Embodied Action Space: The work's main novelty is in perception and reasoning, while the \"Executor\" component relies on a standard 7-DoF action representation. The evaluation of success is task-level, which may obscure nuances in the quality of physical execution (e.g., smoothness, safety, precision).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wb4aesEHd7", "reviewer": "Reviewer_GVRu", "summary": "The manuscript proposes a multimodal fusion framework that integrates text, speech/dialogue, vision, and environmental sounds to support robotic reasoning and decision-making. Compared with using text as the sole communication bridge, the unified end-to-end approach achieves stronger performance with lower control latency, supported by both simulated and real-world experiments.", "questions": "- Following W1 and W2: environmental event sounds can be sparse compared to speech or vision—will the model ignore this feature during fusion? How is this addressed and evaluated? Also, how is synchronization among modalities handled? What is the performance change if environmental sounds are removed?\n\n- The experiments show large gains over baselines. Could the authors provide insights into why those baselines fail and why RoboOmni succeeds?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Clear writing and well-structured presentation.\n\n- Promised release of a robotic dataset, extensive experiments across diverse tasks, and large-scale training.\n\n- An “omni” framework for cross-modal contextual instructions that explicitly includes environmental event/background sounds; comparisons to ASR-based pipelines highlight the benefits of an end-to-end model that can handle overlapping speech.\n\n- Strong empirical results, with ≈60% average success rates exceeding other baselines.", "weaknesses": "- Limited discussion of the method and training for fusing multimodal sensory inputs.\n\n- Ablations on modality contributions are missing: it remains unclear how much each cue (prosody/identity, non-verbal audio, vision) contributes. Please add drop-modality ablations (e.g., audio-w/o-prosody, no non-verbal, no vision) and alignment-window studies; current results separate instruction types but not modalities within the architecture.\n\n- Task descriptions are brief; it is hard to understand the challenges without consulting references.\n\n- Baseline fairness: many baselines rely on ASR (Whisper) or ground-truth text, which can be disadvantaged when paralinguistic information matters. One can retain a language-based architecture but add a separate “translation module” to detect and convert such information [1] into prompt context (e.g., simple classifiers for emotion or dialogue overlap, then embed tokens like “[overlapped_dialogue] [disappointed]”). This increases complexity but is a straightforward way to extend text-based models to richer context.\n\n- Minor: the related-work section could be expanded for a more comprehensive background; prior work [2, 3] also explores multimodal decision-making including environmental sound.\n\nReferences\n\n[1] Yamakawa, N., Takahashi, T., Kitahara, T., Ogata, T., & Okuno, H. G. (2011, June). Environmental sound recognition for robot audition using matching-pursuit. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (pp. 1-10). Berlin, Heidelberg: Springer Berlin Heidelberg. \n\n[2] Zhao, X., Li, M., Weber, C., Hafez, M. B., & Wermter, S. (2023, October). Chat with the environment: Interactive multimodal perception using large language models. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3590-3596). IEEE. \n\n[3] Liu, Z., Chi, C., Cousineau, E., Kuppuswamy, N., Burchfiel, B., & Song, S. (2024). Maniwav: Learning robot manipulation from in-the-wild audio-visual data. arXiv preprint arXiv:2406.19464.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aOYbreM9u2", "reviewer": "Reviewer_xDAu", "summary": "The paper targets proactive intention estimation, action planning, and execution for embodied agents in realistic human-robot interaction. It introduces a large-scale synthetic multimodal dataset built on Open-X with off-the-shelf text and audio models, and proposes RoboOmni, a unified framework that combines omni-modal perception with low-level action generation. Experiments suggest strong performance on proactive intention recognition and end-to-end planning/execution.", "questions": "- Please elaborate on the procedure for “removing trivial samples with low-information visual states.” How do you quantify or detect low information? What thresholds and ablation evidence support the chosen criteria?\n\n- Does the model feed its own intermediate outputs back into the pipeline? If so, are those outputs fed back as text or as audio?\n\n- Were baselines in Table 1 tuned on RoboAction or a simplified variant (e.g., audio converted to text)? If not, low performance may reflect distributional mismatch rather than model limitations. Similarly, were the planner and VLA components of the baselines in Figure 8 tuned for these tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The scenarios are well-chosen, practically relevant, and aligned with next-generation robots operating in realistic human-robot interaction settings.\n- The constructed dataset is likely to be valuable for the community, providing broad multimodal supervision that can stimulate research on proactive robotic behavior.", "weaknesses": "- Neither the dataset’s realism/relevance nor the model’s performance appears to be evaluated by humans. This limits claims about practical usability and interaction quality. While Section 5.3 covers direct human audio instructions, this differs from the complex multimodal contexts central to the paper’s claims. \n- The dataset may inherit biases from off-the-shelf models used during curation. This paper does not analyze or mitigate such biases. \n- The setup, tuning, and fairness of baseline comparisons are insufficiently described, making it hard to interpret performance gaps. Please see the Question section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qD2kRDW4NZ", "reviewer": "Reviewer_2Ge8", "summary": "The paper defines a new setting—cross-modal contextual instructions—where robots infer and confirm user intent from speech prosody, environmental sounds, and vision rather than explicit text, and introduces RoboOmni, a unified Perceiver–Thinker–Talker–Executor model that ingests audio+vision and outputs both dialogue and actions via discrete action tokens (FAST+) in a single autoregressive stream. To supply training data, this paper synthesize OmniAction (≈140k episodes; 5k+ speakers; 2.4k sound events; 640 backgrounds) by GPT-4o–based dialogue rewriting plus multi-speaker TTS with timbre cloning and mixed sound events/backgrounds. They report large gains on a LIBERO-based synthetic-audio benchmark (85.6% vs 25.9% best baseline) and better performance on a small real-speech test with direct audio instructions (76.6% vs 73.8 for π0), along with lower per-inference latency than ASR→VLA cascades; pretraining on OmniAction accelerates SFT.", "questions": "- Can you report real-speech, contextual-instruction results (not just direct commands) with humans, including failure/over-intervention rates? \n\n- Which modality drives gains? Please add leave-one-modality-out and prosody/identity ablations to separate benefits of paralinguistics vs vision vs text. \n\n- How sensitive is performance to the FAST+ tokenizer vs continuous action heads, and to chunk length? Any degradation in dexterous or contact-rich tasks? \n\n- What is the wall-clock training and energy cost compared to ASR→VLA cascades tuned for similar accuracy? \n\n- How do you mitigate biases/artifacts from GPT-4o/TTS synthesis (e.g., sentiment exaggeration, timbre leakage) when transferring to real audio? Any domain-gap diagnostics?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Timely problem framing. Positioning “contextual instructions” (latent intent from audio+vision, incl. paralinguistics) fills a realistic gap in current VLA assumptions about explicit commands. \n\n- Dataset contribution. OmniAction is large and richly varied across speakers, events, and backgrounds; the pipeline is described with reasonable detail (filtering, GPT-4o rewriting/validation, TTS, overlap mixing, SNR control). \n\n- Strong performance. Big margins on OmniAction-LIBERO-TTS (overall 85.6% vs 25.9 best text/ASR baseline) and competitive performance on real-speech direct commands; end-to-end modeling halves inference latency vs ASR cascades.", "weaknesses": "- Synthetic dominates the “new setting.” The core results for contextual instructions are on TTS-synthesized audio and GPT-generated dialogues; the only real-speech evaluation uses direct instructions (not contextual), so the key claim—proactive intent inference from real, messy context—remains under-validated in live conditions. \n\n- Engineering integration over algorithmic novelty. The method mainly composes existing ingredients (Qwen2.5-Omni style encoders, autoregressive LLM “Thinker”, FAST+ action tokens) with a unified likelihood; there’s little new learning principle or analysis specific to proactive intent estimation or uncertainty handling. \n\n- Limited ablations on why it works. We see per-type breakdowns and a pretraining vs from-scratch SFT curve, but no modality ablations (e.g., remove environmental sounds, prosody, or vision), no leave-one-out fusion studies, and no tight control to disentangle dataset scale vs architecture as the main driver. \n\n- External dependence and compute. The pipeline leans on GPT-4o for scripting/validation and large-scale TTS/voice cloning.\n\n- Evaluation scope and safety. Real-robot demos are qualitative and small-scale; there’s no user study on over-eagerness, mis-attribution of speaker intent, or false-positive “proactive” actions, despite identity/overlap being central to the setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)