OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoning
Authors: Yuecheng Liu
DaFeng Chi
Shiguang Wu
Zhanguang Zhang
Yuzheng Zhuang
Bowen Yang
He Zhu
Lingfeng Zhang
Pengwei Xie
David Gamaliel Arcos Bravo
Yingxue Zhang
Jianye HAO
Xingyue Quan
Venue: ICLR 2026 Poster (2026)
Keywords: Embodied AI
Embodied Reasoning
Spatial Reasoning
Multimodal Large Language Models
3D Large Language Models
Primary area: applications to robotics, autonomy, planning

Abstract: Recent advances in multimodal large language models (MLLMs) have opened new opportunities for embodied intelligence, enabling multimodal understanding, reasoning, and interaction, as well as continuous spatial decision-making. Nevertheless, current MLLM-based embodied systems face two critical limitations. First, **Geometric Adaptability Gap:** models trained solely on 2D inputs or with hard-coded 3D geometry injection suffer from either insufficient spatial information or restricted 2D generalization, leading to poor adaptability across tasks with diverse spatial demands. Second, **Embodiment Constraint Gap**: prior work often neglects the physical constraints of real robots, resulting in task plans that are theoretically valid but practically infeasible.To address these gaps, we introduce **OmniEVA** -- an embodied versatile planner that enables advanced embodied reasoning and task planning through two pivotal innovations: (1) a **Task-Adaptive 3D Grounding** mechanism, which uses a gated router to dynamically inject 3D features based on task context, enabling selective geometric reasoning. (2) an **Embodiment-Aware Reasoning** framework that incorporates task goals and physical constraints into the reasoning loop, ensuring executable plans. Extensive experiments show that OmniEVA achieves state-of-the-art performance on 7 of 8 embodied reasoning benchmarks and excels in downstream tasks such as object navigation and mobile manipulation. Evaluations on proposed primitive and composite benchmarks confirm its robust and versatile planning capabilities.

Published as a conference paper at ICLR 2026
OMNIEVA: EMBODIED VERSATILE PLANNER VIA TASK-ADA ADAPTIVE 3D-GROUNDE AND EMBODIMENT- AWARE REASONING
Yuecheng Liu", Dafeng Chi", Shiguang Wu', Zhanguang Zhang", Yuzheng Zhuang?, Bowen Yang, He Zhu, Lingfeng Zhang, Pengwei Xie, David Gamaliel Arcos Bravo, Yingxue Zhang, Jianye Hao, Xingyue Quan
Huawei Noah's ' Ark Lab
ABSTRACT
Recent advances multimodal large language models (MLLMs) have opened opportunities for embodied intelligence, enabling multimodal understanding, reasoning, and interaction, as well as continuous spatial decision-making. Nev- ertheless, current MLLM-based embodied systems face two critical limitations. First, Geometric Adaptability Gap: models trained solely on 2D inputs or with hard-coded 3D geometry injection suffer from either insufficient spatial informa- tion or restricted 2D generalization, leading to poor adaptability across tasks with diverse spatial demands. Second, Embodiment Constraint Gap: prior work of- ten neglects the physical constraints of real robots, resulting in task plans that are theoretically valid but practically infeasible. To address these gaps, we introduce OmniEVA an embodied versatile planner that enables advanced embodied rea- soning and task planning through two pivotal innovations: (1) Task Adaptive 3D Grounding mechanism, which uses gated router to dynamically inject 3D features based on task context, enabling selective geometric reasoning. (2) an Embodiment-AAaree Reasoning framework that incorporates task goals and phys- ical constraints into the reasoning loop, ensuring executable plans. Extensive ex- periments show that OmniEVA achieves state-of-the-ar performance on 7 of 8 embodied reasoning benchmarks and excels in downstream tasks such as object navigation and mobile manipulation. Evaluations on proposed benchmarks con- firm its robust and versatile planning capabilities, Project page.
INTRODUCTION
The rapid progress in multimodal large language models (MLLMs) has significantly enhanced Al's ability to interpret and reason across text, images, and video. This shift has opened new avenues to embodied intelligence (Reed et al. 2022 Ahn et al., 2022 Driess et al. (023), capable of per- ceiving, reasoning, and acting in physical environments. Spatial reasoning serves as a core bridge between perception and action, transforming sensory inputs into structured representations that sup- port loni-horizon planning and rational decision-makingg making.
Early vision language models addressed spatial reasoning primarily in two dimensions. Spa- tialVI.M (Chen et al. 2024a) introduced large-scale synthetic VQA grounded in real imagery. while RoboPoint Yuan al. 20244), RoboSpatial (Song al. 2025), and RoboRefer (Zhou et al.) 2025) incorporated fine-grained spatial grounding by predicting coordinates or bounding boxes from lan- guage prompts. RoboBrain (Ji al. 2025 Team et al.) 2025a) further unified high-level planning with low-level spatial pointing, outperforming general MLLMs on embodied benchmarks. More recently, 3D LLMs have extended reasoning beyond 2D perception by incorporating point clouds, voxel grids, or 3D position embeddings into MLLMs (Huang et al., 2023c Zhu et al. 2024a Hong
ct al. 2023 Zheng al. 2025 Huang et al. 2025).
*: Equal contribution. (liuyuechengl, chidafeng!, wushiguang, thanguang.zhang/ @huawei.com; t: Cor- responding author. zhuangyuzheng huawei.com

---

Published as a conference paper at ICLR 2026
Reasoning Benchmarks
3D Reusoning
148
30
S61
156
Perfrmnnc (5)
X29
+2.7
654
35.0
261 363 3MA
18.4
21,0
11.4
Whendffaer
VSI-bench
PACO-NIS
Rebalkeft
SQA30
ScanQA
ScantCap
Scaniluter
GPT4
Ganiai
Bubohrrii
OumiEEA
Figure 1: Performance Comparison across 2D and 3D Embodied Reasoning Benchmarks.
Despite recent progress, two core challenges remain. First, the geometric adaptability gap: mod- els trained solely on inputs struggle with tasks that require strong spatial reasoning. such as object stacking. occlusion handling, navigation in cluttered 3D scenes. This limitation arises from the absence explicit 3D structural encoding, which restricts generalization in geometry-rice environments. Existing 3D-LLM approaches (Zhu et al., 2024a Zheng et al. 2025 Huang al.) 2025) often depend on hard-coded 3D injection strategies that ignore task relevance, resulting in unnecessary computation and noisy embeddings when 3D inputs
incomplete or nonessential. Second, the embodiment constraint gap: current methods often rely on labeled web-scale image video datasets, or on rule-based synthetic simulations. Models trained on such data frequently overlook the constraints and capabilities of real robots, producing plans that may appear valid in theory but are infeasible in practice. In particular, neglecting object affordances, workspace lim- itations, and kinematic feasibility leads to action sequences that cannot be executed on physical platforms. Furthermore, the absence of embodied long-horizon planning benchmarks that explicitly incorporate embodiment constraints makes it difficuult systematically evaluate the unique chal- lenges they pose. To address these limitations, we introduce OmniEVA (Embodied Versatile Plan- ner), novel architecture that pioneers Task-Adaptive 3D Grounding and Embodiment-avaree Reasoning, OmniEVA the first framework to dynamically integrate 2D and 3D inputs via task- conditioned feature selection, enabling versatile and executable embodied reasoning through two key innovations:
Task-Adaptive 3D Grounding: We introduce gated routing mechanism that dynamically modulates the infusion of 3D features into the visual-language backbone based contextual task requirements. This allows for explicit, selective geometric grounding only when spatially essential, avoiding the drawbacks of static 3D fusion and enabling robust performance across both 2D and 3D reasoning tasks.
Embodiment-Awarr Reasoning: Moving beyond passive scene understanding. OmniEVA jointly incorporates task goals, environmental context, and physical constraints into its reasoning process. Through post-training with our proposed Task- and Embodiment-awaree GRPO (TE- GRPO) algorithm, the model learns to generate plans that respect object affordances, workspace boundaries, and kinematic limits, significantly improving executability on real robots.
We evaluate OmniEVA on public benchmarks spanning 2D, 3D, and video-basee reasoning. As shown in Figure OmniEVV achieves SOTA performance on 7 benchmarks and tops leaderboards in object navigation (HM3D, MP3D). To further probe embodiment awareness, we contribute four primitive benchmarks- Saanmarrkk-hhrr2GGG planning. OmniE VA outperforms existing models across all primi- tives, demonstrating strong generalization downstream tasks like mobile manipulation.
ing a core skill for Where2Go, Where2Grasp, Where2Approach, and Where2Fit-ents target-
RELATED WORK
MLLMs for Embodied Reasoning Multimodal large language models (MLLMs) have recently improved spatial reasoning via synthetic datasets and spatially grounded visual question answering (VQA). SpatialVLM (Chen et al. 2024a) introduced large-scale spatial QA on real-world images. Building this, models such RoboPoint (uaan al. 2024a). Robospatial (Song al. 2025)
2

---

Published as a conference paper at ICLR 2026
and RoboRefer (Zhou et al. 2025 introduced fine-grained spatial outputs, including coordinate prediction and bounding box localization. RoboBrain (Ji et al. 2025 Team et al., (025a) further advanced this line by integrating high-level planning with low-level spatial pointing, outperforming peneral-purpose MI.MM on embodied reasoning benchmarks. To assess reasoning and planning in dynamic large-scale environments, several video-based benchmarks have also emerged, such as VSI-Bench (Yang et al. 2025b) and EgoPlan (Chen et al., 2023). However, despite these develop- ments, most embodied reasoning models remain limited by their reliance on 2D inputs, lacking the capacity to fully interpret environments with complex 3D geometric structures.
3D Large Language Models Efforts to extend LLMs to 3D modalities have explored represen-
tations such as point clouds (Huang et al.] 2023c Zhu et al. 2024b Chen et al. 2024d) and voxel
grids (Hong et al., 2023 Zhang et al. 2025). More rencent approaches inject 3D positional in- formation into visual tokens, enabling pretrained MILMs to perform spatial reasoning in three di- mensions (Zhu et al.) 2024a; Zheng et al. 2025 Huang et al. (025). While these methods have achieved state-of-the-art results on several 3D benchmarks, the hard-coded 3D injection methods can be problematic when 3D inputs are noisy, incomplete, or irrelevant to the task.
3 METHODOLOGY
3.1 OVERVIEW
OmniEVA builds on pretrained MILMS which typically comprises three principal components: 1) A vision transformer encoder Eing that converts each RGB image into sequence of discrete visual tokens, 2) a lightweight network that maps visual embeddings into the language model's latent space for seamless cross-modal interaction and 3) autoregressive text decoder 'I that generates output tokens. The model accepts a natural language instruction T, a sequence of RGB images or video frames (I,, J2..... IN). ), and optionally, depth maps (D1, D2. .2..... Dv) for each view. To support cross-view spatial understanding, the model also ingests camera intrinsic parameters K and extrinsic poses M, corresponding to each frame.
Conventional MILMS such as QwenVI. and InternVL. split each frame into Hp x Wp patches, augment them with 2D positional encodings, and feed the flattened sequence into Eimg. For N frames, the encoder outputs V N
where dy denodes the embedding dimension. While effective for many vision-languag tasks, this approach omits direct 3D information-des values or world coordinates which is critical for complex geometric reasoning. Recent 3D LLMs, such as 3DRS(Huang et al., 2025), rely static architecture to integrate 3D features, limiting their flexibility in tasks where such features are unnecessary. OminiEVA introduces a Task-Adaptive Gated Router (TAGR) to dynamically fuse 3D features and two-stage training paradigm to enable Embodiment-avee aavae Planning.
3.2 TASK-ADAPTIN GATED ROUTER
The module of Task-Adaptive Gated Router (TAGR) illustrated in Figure 2
TAGR serves dynamic mediator between task demands and spatial complexity, selectively regulating the injection of 3D positional encodings. We will introduce the details of the framework the following sections.
Patch-Level 3D Pusitional Encoding Each depth image D, IR HWW is projected into world coordinate matrix P, E H x3 using the camera parameters. The 3D coordinates P; is then partitioned into patches aligned with the patch size of the RGB image processed by the ViT Encoder. For each patch, the 3D coordinates of all pixels are averaged, producing patched coordinate matrix P. H,, Finally, sinusoidal encoding is applied to the 3D coordinates of each patch, mapping them into vectors of dimension d.. For frames, this process yields the 3D positional encoding features denoted as /P E N
Dynamic 3D Injection via Gated Routing Rather than applying 3D positional encoding uni- formly for all tasks, propose Task-Adaptive Gated Router (TAGR) that selectively integrates 3D information based on task-specific requirements. TAGR determines whether to inject 3D posi- tional priors based
conditioning signals: 1) the task condition, reflecting the nature of the
3

---

Published as a conference paper at ICLR 2026
Overall Architecture
Task Gated Router
Examples Activation Gated Router
Text Tokens
Large Language Model
Dened the touedin whie sabter roan
Plus
Hard Samping
grankkyg tew 10 d01 to
Aaske-dappivee Gated Router
Gumbee Softmax
Tert Tokens
00Q00
Network
raor toble bushoves
Con cateraate
Text Tokenizer
VIT Encoder
3D Position Encoding
Sceree Sandition
Task Condition
Vounl slree sdfece desembtee abjent coio:
100a
164
Iow alasiing frund
Aug Pooling
Sentence Transformer
Serrise Aidden bekind box robir en
ara explr and jocing porter liner kok? Instruction
I: The 20 dato it requlied obies and aoliats depti
Image (s)
Depth (Optional)
Vision Tokeens
Text
Tokens
Figure 2: Model Architecture of OmniEVA. Left: The overall architccture of OmmiEVA, featuring novel lask-adaptive gated router that dynamically incorporates 3D positional embeddings. Middle: Detailed imple- mentation of the gated router module. Right: Illustrative examples of the gated router's activation state across different tasks.
task to be performed, and 2) the scene condition, reflecting the structural complexity of the visual input. For task conditioning, a lightweight sentence transformer (Reimers & Gurevych| 2019) en- codes the instruction T into a latent vector V 14, For scene conditioning, the vision encoder output V e RRNx HRNxH, is aggregated via average pooling to obtain a global scene descriptor Vil Rd.
VT (ransoneeTranssfome((())
VI = AvgPoolingg diim = 0, 2,
(2)
avg
The concatenated vector [VT, avg is passed through a multi-laye perceptron (MLP) module to produce gate logits // E TR2, which represent the probabilities corresponding to the activation and deactivation of the gate module.
V =
V)) E R
To preserve the amplitude of sinusoidal encodings VP wP-whose distortion can lead significant performance degradation- -e avoid soft-weighting and instead employ hard gating via Gumbel- Softmax (Jang et al. 2016) to enable end-to-end gradient propagation, where is the temperature. When g = 1 the model augments 3D features with 3D cues; when g O the model uses only 2D visual features. This is equivalent to a Mixture-of-Exxee -xpeerts (MoE) between pure visual tokens V and the fused tokens (V' + VP), shown in Equation 5 The resulting hybrid visual tokens and the text tokens are then passed to the LLM backbone F (-) to generate the response tokens o.
= GumbelSoftmax (V3,aaxVV..,, (V9,T) E 13
(4)
hybrid V' + y VP = (1 v) ' + g(V' + VT)
(5)
0 Fllm mm (T, Vinbnia)
(6)
The TAGR module is optimized using cross-entropy loss to align the predicted output o with the ground truth label glaeel, To stabilize and regularize gating behavior, we add KL divergence term between the predicted gate distribution and Bernoull((O.)) prior,
Cooal
CCB (obbell 0) + a C (V9][
3.3 EMBODIMENT- AWARE TRAINING STRATEGY
To unify perception, reasoning, and execution across heterogeneous embodied tasks, we introduce clear, sequential three-stage training pipeline: (1) Task-Adaptive Gated Router (TAGR) Pretraining (detailed in Section 3.2), (2) Supervised Fine-Tuning (SFT), and (3) Task- and Embodiment-awee Reinforced Finetuning (RFT). This cascaded approach progressively enables the model to evolve from foundational spatial understanding to sophisticated, executable plan generation.
4

---

Published as a conference paper at ICLR 2026
General Embodied Reasoning
Omni Embodied Reasoning
Spotial Relation Referring
Temporal Understadding
Versatile Embodied Tasks
A:A1 the cuees jucby the poster whet'
@ Alt's me Fee
Composite Embodled Tosks
exverrater oddy
pllter hamte?
Large Space ObjSeeking
Local Mobile Manipulation
Indoor Delivery
lght A oobinet
doskec bettl, piitte
dv ccpan!
labls moor porii
marrreee and Jice Kee
Grounding due 30aku ohe wnder act pekring A:
Scene Captonning
I'L @ Decrile hoo bettroct Senuues buttishinlt...
Primiitiv Embodiee Tassk
Where2Go
Where2Appreach
Where2Grasp
Whe
whoee F gra
How prosa the ou
Where No chet kn
2/30 Object Captioning
Scene Imagination
to por the bea?
Bourle the osject
O At (26897.45) appoar? inscin
12150
cadie lable ats ia funt
Embodliment-wwar Reasoning
please
Supervised Finetuning
Format Reword
SFT GRPO Update
OmniEVA
l.fforer atheom feree C
Task and Embodiment Reward
Embodiment
Large Language Model
conskanng nēst ants marispuc fol iobe suootl saunnee
Pantt 1200 200 <atwer
Action
Env
TAGR Module
(1 6) BwdTackig. #1 probelf 3) bodt ol
DOO0
(*((aa) (1 O (1 (1))
Resalts
Text
Image
Depth
Figure Training Paradigm of OmniEVA. The two-stage cascade progressively enhances embodied intelli- gence: Stage builds a broad reasoning foundation, while Stage grounds it in physical reality- culminating in robust task execution across diverse real-world scenarios.
3.3.1 STAGE I:
GATED ROUTER PRETRAINING
In this initial stage, the task-adaptive gated router is pretrained learn dynamic 31) feature inte- gration gating behavior, utilizing the optimization objective defined in Equation 7 The training incorporates combination of depth-aware spatial reasoning datasets, including scene data from ScanNet, Matterport3D, 3RScan, and ArkitScenes (detailed in Appendix D). To facilitate effective learning of the TAGR module, we apply differentiated learning rate strategy: small rate (5e-7) for the LLM backbone to preserve pretrained knowledge, and a larger rate (1e-t) for the TAGR pa- rameters to accelerate adaptation. Upon completion of pretraining, the TAGR parameters are frozen for use in subsequent stages, while the fine-tuned LLM backbone parameters are discarded to avoid interference.
3.3.2 STAGE 2: SUPERVISED FINE-TUNING FOR GENERAL EMBODIED REASONING
This stage takes the PrGR-preraad model its input to establish a robust foundation for gen- eral embodied reasoning. We curate hybrid dataset comprising: (1) General Embodied Reasoning Data: Multimodal sources (2D images. video. 3D) for tasks like spatial relation referring and scene captioning, building omni-dimensional spatial cognition (holistic scene understanding).
(2) Custom Embodied Task Data: Navigation and manipulation tasks that foster initial embodiment- soning (basic physical interaction awareness). The model resulting from the SFT stage is designated as OmniEVA-Bal -ase. Details of the dataset usage in this stage are provided in Appendix D
rea-
3.3.3 STAGE 3: TASK- AND EMBODIMENT-AWARD REINFORCED FINETUNING
This final stage uses OmniEVA- -Bsse as its starting point to ground reasoning in physical reality. While prior methods have largely focused on improving semantic fidelity, they often neglect the physical feasibility of generated plans. GRPO seeks to bridge this gap by promoting outputs that are not only semantically aligned with task objectives but also executable within the constraints of robotic embodiment. Building on GRPO (Shao et al. 2024), we keep the format reward format to encourage the "think-answer" reasoning structure. To further guide the model toward generating both task-directed and physically feasible plans, we introduce two additional reward components:
r: (g, o:) EvalTask(g,, )) [0, 1], r gnbod (q, oz) EvalExec 0:) E [0,
(8)
5

---

Published as a conference paper at ICLR 2026
where 9 is the user prompt and o, the i-th model response. EvalTaskk()) measures semantic task satisfaction independent embodiment (e.g., fraction of points placed inside a target region for pointing task). In contrast, EvalExec(-) verifies embodiment feasibility by checking kinematics, reachability, and environment constraints in simulation. These two reward components reflect dis- tinct optimization objectives: r!ack emphasizes performance on offline evaluation benchmarks, while rembod targets end-to-end execution success in real-world robotic deployments.
Progressive Embodiment Curricutum To accelerate convergence and promote physically grounded reasoning, we employ a curriculum approach gradually transitions the model's optimization focus from semantic correctness embod- iment feasibility. At training step I, the composite accuracy reward r (q, 0;) is defined as,
reward scheduling strategy. This
ace (q. 0:) r!ask (g. 0i) (Ae (mmbod (g. 0z) + ())))
where d E [0. 1] is scheduling coefficient that increases over time, gradually shifting the model's focus from task completion embodiment feasibility. Early in training, X s O, allowing the model to receive positive reward even when embodiment constraints are only partially satisfied. As training progresses, d + 1, enforcing stricter compliance with physical constraints. The final reward for the i-th response is then computed as:
(g, = = f. - -.t. (q. 0i)
(10)
Policy updates follow the original GRPO objective. See Appendix A.3 for detail. The model result- ing from this RFT stage is designated as OmniEVA-ER (ER for Embodiement-awar Reasoning).
4 EXPERIMENTAL RESULTS
This section presents a comprehensive evaluation of OmniEVA to validate its effectiveness and an- swer two pivotal questions: (1) How does the proposed dynamic 3DD-grunding mechanism enhance multimodal reasoning? (2) To what extent does the embodiment-awar reasoning framework im- prove task success rates under physical constraints?
To ensure clarity, we evaluate two distinct model checkpoints: OmniEVA- (after Stage SFT), which is equipped with strong general reasoning capabilities, and OmniEVA-ER (after Stage 3 RFT), which is optimized for physical fcasibility. OmniEVA VA-Bae''s results are in Tables 2-4; OmniEVA-ER are in Figure 5. Implementation details are provided in Appendix A.
4.1 BENCHMARKS FOR EVALUATION
Embodied Reasoning Benchmarks with 2D Inputs To assess the model's embodied reasoning capabilities across visual modalities, we employ four established benchmarks: Where2Place (uuan jet al. 2024a). VSI-bench (Yang et al. 2025b), PACO-LVIS (Ramanathan et al., 2023). and Ro- boRehi (Lu al. 2023). These datasets span both static images and dynamic video inputs, enabling comprehensive evaluation of spatial and temporal understanding and multimodal reason- ing. Furthermore, we introduce four specialized benchmarks targeting fundamental embodied rea- soning skills: Where2Go for next-best view selection under partial observability, Where2Fit for free-space prediction with collision constraints, Where2Approach for occlusion- -warre navigation, and Where2Grasp for object-centric recognition. These primitive benchmarks form the founda- tion for complex downstream tasks like mobile manipulation. Detailed descriptions are provided in Appendix D
Embodied Reasoning Benchmarks with 3D Inputs To evaluate 3D spatial reasoning we use SQA3D Ma et al.) 2022). ScanQA (Azuma et al. 2022). Scan2Cap (Chen et al. 2021). and Scan- Refer (Chen al. 2020). These benchmarks probe open-ended QA, captioning, and 3D visual
grounding in richly structured environments, stressing depth and geometry beyond planar views.
'Since the annotations in RoboRefit and PACO-LVIS lack VQA pairs, we huilt VL.M evaluation set based image distribution, object categories, and part categories. The evaluation code aligns with Where2Place.

---

Published as a conference paper at ICLR 2026
End-to-End Online Evaluation within Simulators To bridge the gap between high-level plan- ning and low-level robotic execution, we run end-to-end experiments in simulators on composite tasks: Large-Space Object Seeking and Mobile Manipulation. These tasks combine the primitive skills above to measure holistic embodied reasoning and execution.
4.2 TASK-ADAPTIN 3D-GGOUNDDNN: VALIDATION ACROSS MULTIMODAL BENCHMARKS
Effectiveness of the Task-Adaptive Gated Router To rigorously evaluate the efficacy of our ap- proach, we compared it against several baselines: (1) Hard- 3D integration, which statically fuses 3D features into visual tokens across all tasks-a a common strategy prior 3D I.L,s; (2) Without 3D integration, effectively reducing the model to a traditional 2D) MILM; and (3) Cross- attention-based fusion, implemented using either separate or interleaved visual and 3D tokens (see Appendix and Figure 10 for detail), to assess implicit feature weighting. As summarized in Ta- [ the proposed dynamic 3D integration approach not only surpasses cross-attention variants but also outperforms oftt-e--rtttttt 2D/3D fusion baselines, achieving an average performance gain of 1.2%. Notably, cross-attention methods result in a substantial performance drop approximately 6 points SQA3D and a striking 50-point decrease on Scan2Cap. This pronounced degradation highlights the limitations of naive cross-attention for 3D fusion, which doubles the input sequence length and compels the model to learn cross-modal interactions from scratch, thereby compromis- ing both efficiency and generalization. We further analyzed a soft-gating alternative that injects 3D positions via continuous sigmoid weighting. This method consistently underperformed benchmarks, as it compromises the numerical stability of the positional encodings crucial for spatial reasoning. The inferior performance of soft-gatt soft-gating validates the superiority of our strategy, which offers more stable and adaptive solution for geometric integration.
all
Results of Different ID-Integration Methods. To ensure a fair comparison and isolate the impact of 3D integration, models were trained exclusively on the training splits of the SQA3D, ScanQA, Scan2Cap, and ScanRefer datascts. This experimental setup is consistent with prior work like Vidco-3D-LLM and 3DRS.
Table 1:
Benchmark Results
Methods
SQA3D
ScanQA
Scan2Cap
ScanRefer
Average
Indivisual 3D Input
Cross-Attcution (scperate vision 3d tokens)
27.5
43,3
32.6
Cross-tttenioon (interleaved visson & 3d tokens)
55.8
27.5
42.0
3.6
32.2
Direct 2D/3D Fusion Methods
Hard-coded 3D Integration
61.2
31.5
95.5
41.2
57.3
Without 3D Intogration
61.2
30.7
75.5
4.3
42.9
Dynamic Integration, Gating
30.7
85.6
26.9
$1.0
Dynamic 3D Intogration, Hard Gating (Ours)
62.6
30.8
97.9
43.1
58.7
When Is the TAGR Module Activated? We analyze the activation behavior of the task-adaptive gated router (TAGR) by clustering prompts based on semantic categories. As shown in Figure 4 prompts are embedded using lightweight sentence transformer and grouped by semantic similar- ity. The activation probabilities vary considerably across categories: shape-related prompts trigger the highest activation (76.9%). followed by action/activity (50.9%) and visibility occlusion (33.0%), indicating that 3D reasoning is strongly invoked by geometric, dynamic, and occlusion- cues. In contrast, categories such as state condition (6.3%), counting (9.0%), and material texture (19.4%) result in much lower activation, suggesting limited reliance on 3D features for descriptive or numer- ical tasks. A detailed case study included in Appendix B.
Comparison Between OmniEVA and State-of-the-AAt Models on 2D/3D Benchmarks Ta- reporss OmniEVA-Base results on four 2D embodied reasoning benchmarks (Where2Place, VSI-Bench, PACO-LYIS, RoboRefit) spanning images and video. Despite its compact 8B pa- rameter size, OmniEVA-Bos passing much larger models and yielding average gain of +10.45 over the previous SOTA (Robobrain2.0-25 3-0322BBBBB (Team et al. 2025a)). Furthermore, OmniEVA-BBBB alization across four additional primitive benchmarks (Where2Go, Where2Fit, Where2Approach, Where2Grasp), showing its versatility and superiority in diverse embodied tasks. See Appendix F for qualitative examples.
attains state-of-the-art performance across all benchmarks, sur-
demonstrates robust gener-
7

---

Published as conference paper at ICLR 2026
Shape
76.4%
Activity
50.4%
Visibillty Occlusion
Semantic Category
Direction Orientation
30.5%
Identification
27.5%
Appearance Description
26.8%
Category
Color
24.2%
Comparison
21.45%
Other
20.4%
19.4%
Arangeeent Pattern
Counting 9.0%
State Condition
5.6%
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Activation Rate
Figure 4: 3D Activation Analysis by Prompt Clustering: Prompts embedded using lightweight sentence transforeer and clustered into semantic groups. The chart shows the 3D activation probability per category.
Table 2: 2D General Reasoning Benchmarks and In-house Benchmarks. [1] Hurst al. (2024 .21 Team et al.
[325bb[3 Zhang el [9] L et al. [10] Yang et al. 2025m Team 2025a
2024 Zhu
Bai al. 2025 [7] al.
Azzulaa
Public Embodied Benchmarks
In house Embodied Benchmarks
Models/ / Benchmarks
Where2Place VSI-bench PACO-LVIS ReboRedit Where2Go Where2Hit Where2Approsch
Where2Girasp
General Models
GPT-46 [I]
20.41
43.60
2.09
9.96
50.72
37.15
0.17
6.38
Gemini-2.5-Pro
28
48.83
3.14
17.91
55.07
41.82
3,50
27.00
Llava-Next-Vi Video 7B [3]
4,76
35.62
1.44
1.18
31.88
61.34
0.10
0.89
Llava-OneVision
MiemVLL1-8B [5]
12.68
42.89
4.57
13.76
41.06
33.07
2,08
8.63
I1tem [5]
21.74
21.48
51.69
41.16
11.80
Qwen2.5.VL. 7B [6]
10.99
37.51
3.21
1.21
38.16
38.59
1.50
12.75
Qwen2.5-VI--22 [6]
39.92
39.41
4.06
32.58
49.76
41.49
0.00
30.50
Embodied Models
RoboPoint
Cosmos-Reaste Rcason1-78 (8)
5,51
25.64
2.58
14.42
40.10
38.86
0.00
6.70
VeBrain-8B [9]
11.34
26.30
0.89
4,00
28.98
28.47
0.00
0.00
Magmu SB [10]
10.89
12.65
3.23
4,95
0.00
28.45
0.00
13.50
RoboBrain2.00 7. [1T]
63.59
36.
11.38
62.74
38.64
32.99
2.85
63.24
RoboBrain 2.0-2BB [11]
73.59
OmniE VA-Base
74.95
$7.17
21.01
91.19
86.96
78.14
7.37
73,91
Extending to 3D embodied reasoning, we evaluated OmniEVA-Base on SQA3D, ScanQA, Scan2Cap, and ScanRefer, which encompass 3D question answering, captioning, and 3D visual grounding tasks. As shown in Table 3 OmniEVA-Bau leads on three out of four benchmarks. outperforming state-of-the-ar specialized 3D LLMs such Video-3D-LLM (Zheng al. 2025) and 3DRS (Huang et al. 2025) with notable improvements of +2.3, +0.3, and +8.5, respectively. While it slightly trails 3D visual grounding (ScanRefer). OmniEVA AmniEVA-Basse attains 55.8 accuracy (IoU@0.25) using only text I/O, without external detectors or task-specific heads, substantially ex- ceeding the previous best of 44.4 and showcasing strong end-to-end grounding ability.
In addition to general embodied reasoning, OmniEVA- -ass demonstrates strong performance in downstream tasks such as Object Navigation, evaluated on the HM3D (Ramakrishnan et al. 2021] and MP3D (Chang et al. 2017) datasets. Here, the model is tasked to predict 3D subgoal location to guide exploration toward a target object. As shown in Table 4 OmniEVA Base outperforms the state-of-tt arrt navigation model UniNavid (Zhang al. 2024a in both success rate (SR) and path efficiency (SPL), achieving a notable +5.4 improvement in SPL. Qualitative examples of exploration trajectories provided in Appendix F2
8

---

Published as a conference paper at ICLR 2026
Table 3: 3D Reasoning Benchmarks. [1 Hong al. 2023
[2] Zhu
Table
ObjNay Benchmarks
[3]
et
al.
(2024b1131
[uung al.] (2023c [Chen ul. (20224))15)
Zhang
2019)
[2) Zhoo et [X22]_1 Wu et al.
Yokovama d a
< al.
20251]1
Wang et at. (2025)111 [11] Huang et al (2023b).8) Huang
[Huang ol L1124 Jol Yin
2024
Yo ab
20225
sl.
Deng Deng et w 2025
2023a1191
Chen al. 2024c |014ck/10 Ziu
Zheng et al, 20251. Tuang et 2025
2024a Yu
2025
( (adaa 22030
Yin 12025 Kamnakaya
(Kammkkyya et (2023 (15
Zrong et 20134a
Models
SQA3D
ScunQA
Scan2Cup
ScanRefer
Methods
HM3D
MP3D
EM
EM
CIDEr
SR
SPL
SR
SPL
Models
Baseline Methods
3D-LLM(Flam) [I]
20.3
21.2
DD-PPO [1]
27.9
14.2
3D-LLM(blip2) [1]
20.5
30.3
ESC [2]
39.2
22.3
28.7
14.2
PQ3D [2]
47.1
57.0
VoroNav
42.0
26.0
LEO 131
50.0
21.5
72.4
VLFM [4]
52.5
30.4
36.4
17.5
[4]
GAMap [S]
53.1
26.0
Scenel.I. [S]
53.6
27.2
SG-Nav [6]
54.0
24.9
40.2
16.0
S-3D-LLM
72.2
L3MVN [7]
54.2
25.5
ChatSeene [7]
54.6
21.6
77.1
35.5
UniGoal [8]
54.5
25.1
41,0
16.4
Chat-3D
54.7
63.9
42.5
Habital Web [9]
57.6
23.8
31.6
8.5
LI.3DA
62.9
Instract [10]
58.0
20.9
LLaVA-3D [[0]
55.6
27.0
79.2
54.1
OVRL(1I]
62.0
26.8
28.6
7,4
Inst3D-LMM
OVRL-v2
62.8
3D-LLaVA [12]
54.5
62.9
LFG
68.9
36.0
V-3D LLM[13]
58.6
30.1
58.
PIRLNav [14]
70.4
34.1
3DRS [14]
60.6
30.3
86.1
62.9
UniNavid [15]
73.7
37.1
OmniEVA
62.9
30.6
55.8
OmniEVA
74.2
42.5
59.1
26.2
4.3
EMBODIMENT- REASONING: PERFORMANCE UNDER PHYSICAL CONSTRAINTS
Effectiveness Across Diverse Embodied Tasks We evaluate the impact of task- and embodiment- aware reasoning by comparing models trained with and without task and pomood Experiments span primitive skill benchmarks (Where2Approach, Where2Fit, Where2Grasp) and downstream tasks requiring physical execution (Mobile Placement and Mobile Pickup).
Model Performance on Different Benchmarks
RoboBrain2.0-78
OmniEVA-Baan
OmniEVA-ER w/o
OmniEVA-ER
90.50
82.50
80
71.00
Scores
60
55.56
63.24
$4,04
43,50
47.50
$2,00
57.3668667
40
29.86
38.07
20
4,00
Where2Fit
Where2Approach
Where2Grasp
Mobile Placement Mobile Placement (Hard)
Mobile Pickup
(Easy)
Figure 5: Ablation Results the proposed TE-GRPO Method on Local Mobile Mabill-Manipplatiion Tasks
As shown in Figure 5, OmniEVA-ER jointly optimized with pask and yemmhod demonstrates sub- stantial performance gains over OmniEVA-Bas -Bss and naive RL training across all evaluated tasks. Where2Approach and Where2Fit exhibit accuracy surges 28 28.95% and 34.28% respectively, while Mobile Placement achieves notable success rate improvements of 43% (Fasy) and 50% (Hard). Al- though each reward component provides individual benefits, their synergistic combination yields optimal results. These consistent advancements across primitive and composite tasks confirm that embodiment-awwer reasoning significantly strengthens performance robustness in diverse embodied scenarios.
To assess the adaptability of embodiment-aware reasoning, we conduct experiments varying a key robot parameter-arm length-as a representa- tive embodiment factor. While other dynamic parameters warrant further exploration, arm length directly impacts workspace constraints and thus serves as a suitable initial testbed for evaluating physical generalization. We generate different embodiments by scaling the same base robot arm model, producing lengths from 72cm to 110c.. Models are trained on arm lengths 75cm, 88cm, and 10cm, and tested both and unseen configurations (72cm, 80em, 84cm, 92cm, 96cm,
Generalization Across Diverse Embodiments

---

Published as a conference paper at ICLR 2026
Table 5: Results of Different Embodiment Execution Success Rate.
Seen Length (cm)
Unseen Length (cm)
Models Embudinents
Average (SR)
75
110
Seen Avg.
105
Unseen Avg
RohoBrain2 0-7B
18.37
7.69
12.82
38.88
19.29
0
7.69
12.8
17.94
20.51 33.33 38.88
17.97
OuniEEV-BBase
43.30
51.28
41.02
45.61
35.89
43.58
51.28
36.11
42.32
OmmiEVA-ER w/o Timbed
65.35
69.23
71.79
69.44
70.17
41.02
64.10
66.66
61,53
69.23
72.22
69.44
63.29
OmmiEVA-ER
81,89
87.17
82.05
86.11
85,08
56.41
87.17
84.61
79.48
82.05
88.88
86.11
80.52
105em). The model prompted with embodiment specifications (e.g.. "The robot arm length is 72cm") to enable dynamic adaptation.
OmniEVA-ER consistently outperforms OmniEVA- and the ablated variant (without emmod achieving average improvement of 15% on seen arm lengths. Moreover, it generalizes robustly to unseen lengths, attaining an 80.5% success rate-a 17% gain over the baseline. These results confirm that embodiment-
reasoning facilitates not only task-level adaptation but also effee- tive generalization across previously unseen physical configurations, highlighting its utility for real- world deployment in diverse robotic systems.
Real World Deployment To emphasize the generalizability of our approach, we conduct real- world experiments on two off- sffeth wheeled mobile robots with inherently different dual 6-DoF arm designs: one platform has 75 em arm span, while the other (Galaxea R1) has a more con- strained 70 cm arm span. As shown in Table OmniEVA- demonstrates consistent performance improvements across both embodiments, particularly excelling in manipulation tasks. achieves near-perfect performance (9/10) in Cluttered Place tasks and maintains robust navigation capabil- ities. The results confirm that embodiment-awas reasoning enhances real-world robustness, with OmniEVA-ER showing better adaptation to the more constrained 70cm span configuration.
Table Real World Performance on Different Tasks and Embodiments. Both embodiments are wheeled mobile robots with dual 6-DoF arms. Embodiment has 75cm arm span, Embodiment 2 has 70cm arm span.
Cluttered Pick
Cluttered Place
Constrained Navigation
Models/ / Tusks
Embodiment
Emhodiment 2 Emhudiment Embodiment Embodiment
Emnhodiment 2
OmmiLVA Base
60
5710 7/10
6/
$/10 9/10
9/ 8/
8/10 8/10
OmniEVA ER
80
9/
LIMITATION AND FUTURE WORK
While OmniEVA demonstrates strong performance across various benchmarks, we acknowledge two key limitations. First, the scene-level gating mechanism in TAGR may lead to suboptimal 3D feature integration heterogeneous environments, as evidenced by unexpected low activation for tasks involving spatial relationships in activation analysis. Second, embodiment-awann planning may face challenges with unmodeled physical constraints, such as arm degrees of freedom or instal- lation height, which could impact robustness in dynamic scenarios. For future work, will explore patch-level gating strategies to enable finer-grained 3D adaptation and integrate broader physical parameters enhance overall adaptability.
6 CONCLUSION
This work presents OmniEVA, a versatile embodied planner that delivers robust cross-dimensional reasoning across a wide range of tasks. By introducing a dynamic routing mechanism for 3D ground- ing, OmniEVA adapts effectively to varying spatial demands and improves reasoning performance. Its embodiment-awart fine-tuning strategy closes the gap between high-level reasoning and low- level robotic execution, enabling plans that are both logically sound and physically executable. Taken together, OmniEVA takes a strong step toward building general purpose embodied agents that can reason, plan, and act in complex environments.
general-purpose
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
including model architecture and training con- figurations. Source code and benchmark datasets will be released upon acceptance.
We provide implementation details in Appendix A
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0166 2022.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question an- swering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129-19999 19139, 2022.
Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay. Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reason From physical common sense embodied reasoning. arXiv preprint arXiv:2503. /5558, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe. Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint wtt.xx::111108850 2021,
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068/7 2022.
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: /70./v//79..00 070..76189..0.. 2017.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain lchter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14455-14465,
2024a.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pp. 202-221. Springer, 2020.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Confer- ence on Computer Vision, pp. 370-387. Springer, 2024b.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reason- ing and planning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 26428 20228826438,. 2024c.
Yi Chen, Yuying Gc, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models. CORR, 2023.
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-lIm with referent tokens. arXiv preprint arXiv:2405. 10370, 2024d.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang, Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3193-3203, 2021.

---

Published as a conference paper at ICLR 2026
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2017.
Jiajun Deng, Tianyu He, Li Jiang. Tianyu Wang. Feras Dayoub, and Ian Reid. 3d-llava: Towards generalist 3d imms with omni superpoint transformer. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 3772-3782, 2025.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied mul- timodal language model. arXiv preprint arXiv:2303.0333 2023.
Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, and Stan Birchfield. Handal: A dataset of real-world manipulable object categories with pose annota- tions, affordances, and reconstructions, In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11 11428-1144 1435. IEEE, 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494, 2023.
Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. CoRR, 2023a.
Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. CoRR, 2023b.
Hao Huang, Yu Hao, Congcong Wen, Anthony Tzes, Yi Fang, et al. Gamap: Zero-shot object goal navigation with multi-scale geometric-affordane guidance. Advances Neural Information Processing Systems, 11:39386-39400 2024.
Jiangyong Huang, Silong Yong, Xiaojian Ma. Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.1288 2023c.
Xiaohu Huang, Jingjing Wu, Qunyi Xie, and Kai Han. Milms need 3d-aware representation super- vision for scene understanding. arXiv preprint arXiv:2506.00 1..../9944 0/946, 2025.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzation with gumbel-softmax. arXiv preprint arXiv:1611.00 10111.01144 144, 2016
Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, al. Robobrain: A unified brain model for robotic manipu- lation from abstract to concrete. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1724 1734, 2025.
12

---

Published as a conference paper at ICLR 2026
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International journal of computer vision, 123(1))32-73, 2017.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning, Advances in neural information processing systems, 36:34892-34916, 2023.
Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero- shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406. 04882, 2024.
Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. Vl-grasp: a 6-dof in- teractive grasp policy for language-oriented objects cluttered indoor scenes. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 976-983. IEEE, 2023.
Gen Luo, Ganlin Yang. Ziyang Gong, Guanzhou Chen, Haonan Duan, Erfei Cui, Ronglei Tong, Zhi Hou, Tianyi Zhang, Zhe Chen, et al. Visual embodied brain: Let multimodal large language models see, think, and control in spaces. arXiv preprint arXiv:2506.00123, 2025.
Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Learning affordance grounding from exocentric images. In CVPR, 2022.
Ruiyuan Lyu, Jingli Lin, Tai Wang. Shuai Yang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chenming Zhu, Dahua Lin, et al. Mmscan: A multi-modal 3d scene dataset with hierar- chical grounded language annotations. Advances in Neural Information Processing Systems, 37: 2024.
Xiaojian Ma. Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474 2022.
Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Movay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16498, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring externa knowledge. In Proceedings of the IEEEScvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. 20/9 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEFF, 2019.
Abby O' Neill, Abdul Rehman, Abhiram Maddukuri. Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 6892-6903, IEEE, 2024.
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Part- sey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint 10 2023.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-mateen matterort 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv: 109, 08.38. 2021.
13

---

Published as a conference paper at ICLR 2026
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7141-7151, 2023.
Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5173-5183, 2022.
Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Abhishek Das. Pirlnav: Pretraining with imita-
tion and rl finetuning for objectnav. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17896-17906, 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky. Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06/75. 2022.
Nils Reimers and Hugging Face. seneene-rransfommevvlallmmmmn huggingface ww//eentnnceccccce -
https:// 2021. Ac-
-6-v2
cessed: 2025-09-08.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: //axxiv.
prg/abs/1903 10084
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289-301, 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Dhruv Shah, Michael Robert Equi, Btažej Osiński, Fei Xia, Brian Ichter, and Sergey Levine. Navi- gation with large language models: Semantic guesswork a heuristic for planning. In Conference on Robot Learning. pp. 2683-2699. PMLR, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/VVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Ro- bospatial: Teaching spatial understanding to 2d and 3d lanionee models for robotics. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15768-15780, 2025.
BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, et al. Robobrain 2.0 technical report. arXiv preprint 0rX2022507.02029, 2025a.
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020. 2025b.
Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re- re-localization in changing indoor environments. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7658-7667, 2019.
14

---

Published as a conference paper at ICLR 2026
Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pp. 1723-1736. PMLR, 2023.
Xiaoyan Wang, Zeju Li, Yifan Xu, Jiaxing Qi, Zhifei Yang, Ruifei Ma, Xiangde Liu, and Chao Zhang. Spatial 3d-lIm: Exploring spatial awareness in 3d rision-languee models. arXiv preprint arXiv:2507. 16524, 2025.
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near- near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: 1ete::9111000510 2019.
Pengying Wu, Yao Mu, Bingxian Wu, Yi Hou, Ji Ma, Shanghang Zhang, and Chang Liu. Voronav: Voronoi-based zero-shot object navigation with large language model. arXiv preprint arXiv:2401.02695, 2024.
Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra. Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav. arXiv preprini arXiv:2303.07798 2023a.
Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for em- bodied navigation. In Workshop on Reincaraating Reinforcement Learning at ICLR 2023, 2023b.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, al. Magma: foundation model for multimodal ai agents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14203-14214, 2025a.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 16632-10633, 2025b.
Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompt- ing for Ilmed zero-shot object navigation. Advances in neural information processing systems, 37:5285-5300 2024.
Hang Yin, Xiuwei Xu, Linqing Zhao, Ziwei Wang, Jie Zhou, and Jiwen Lu. Unigoal: Towards universal zero-shot goal-oriented navigation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19057-19066 2025.
Naoki Yokoyama, Schoon Ha, Dhruv Batra, Jiuguang Wang. and Bernadette Bucher. Vlfm: Vision- language frontier maps for zero-shot semantic navigation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 42 48. IEEE, 2024.
Bangguo Yu, Hamidreza Kasaci, and Ming Cao. L3mvn: Leveraging large language models for visual target navigation. In 2023 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pp. 3554-3560, IEEE, 2023.
Hanxun Yu. Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14147-14157, 2025.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European conference on computer vision, pp. 69-85. Springer, 2016.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: Dision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406. 10721, 2024a.
Yugian Yuan, Wentong Li. Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10202-282111 2024b.
15

---

Published as a conference paper at ICLR 2026
Hang Zhang, Zhuoling Li, and Jun Liu. Scenellm: Implicit language reasoning in Ilm for dynamic scene graph generation. Pattern Recognition, pp. 111992, 2025.
Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: video-based rision-languaee-cttin model for unifying embodied navigation tasks. arXiv preprint arXi::2412.06224 2024a.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL
io wttpll/g//224-00000000 04 30-vvava next video/
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position-aware video represen- tation for 3d scene understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025.
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. arXiv preprint arXiv:2506. 254.:2500000408,, 2025.
Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation, In International Conference on Machine Learning, pp. 42829-42844 PMLR, 2023.
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering Imms with 3d-awareness. arXiv preprint 2rXiv:2409.18122 2024a.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
Ziyu Zhu, Zhuofan Zhang. Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via promptable queries. In European Conference Computer Vision, pp. 188-206. Springer, 2024b.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A IMPLEMENTATION DETAILS
A.1 MODEL ARCHITECTURE AND TRAINING CONFIGURATIONS
Our experiments built upon the pretrained Intern VL3-8B model (Zhu et al. 2025). which serves as the foundational backbone for our multimodal large language models (MLLMs). To encode user instructions within the task-adaptive gated routing (TAGR) module, utilize the all-MiniLMM-L6 v2 (Reimers Face 2021), chosen for its efficiency and semantic fidelity. Instruction embeddings further processed through lightweight two-layer multilayer perceptron (MLP) with hidden dimension of 256, enabling compact yet expressive representation learning.
For video-based inputs, uniformly sample 16 frames during training and 32 frames during in- ference, striking balance between temporal granularity and computational efficiency. To handle 3D spatial information, we voxelize both point clouds for positioning and 3D bounding boxes using a fixed voxel size of 0.1 meters. This discretization facilitates consistent spatial reasoning across diverse environments and tasks. Detailed sepee-paraeeters as given in Table
TAGR Pretraining During TAGR pretraining, we freeze the sentence transformer and train the MLP encoder with a learning rate of le The ViT encoder is frozen, while the LLM backbone is updated with reduced learning rate of 5e - 7 to prioritize learning within the TAGR module. We apply cosine decay for learning rate scheduling and use exponential decay control the Gumbel- -1.5-steps
softmax temperature т, defined as Tini exp(
max stops
Supervised Finetuning For For supervised finetuning, the ViT encoder remains frozen. We fine- tune the LLM parameters using learning rate of le with cosine decay.
Reinforced Finetuning In the reinforcement stage, continue finetuning the LLM with learn- ing rate of le - 5. Each training group generates 8 candidate outputs.
Complete hyperparameter settings for all training stages provided in Table
Table 7: OmmiEVA Training prrperpprramter Configuration
Hyper-parameters
TAGR Pretraining
Supervised Finetuning
Reinforced Finetuning
epochs
batch size
256
256
128
leaming (LLM)
5o-7
1o:5
lo-5
learning rate (TAGR)
le-4
learning rate (VITs)
Ir schedular
cosine
cosine
Thit (gumbel softmax)
1.0
le-6
1e
Tasa (gumbel softmax)
0.05
le-6
le-6
schedulur
exp( 4.5
exp
weight decay
0.1
0.1
0.1
gradient clipping
1.0
1.0
1.0
bf16
use fp16
false
false
false
warmuup
le:3
optimizer
AdamW
Adam
AdamW
image resolution
448x4488
448x448
448x4488
video frames (training)
16
16
16
video trames finlerence)
32
32
32
3D voxel size
0.1m
0.1m
0.1m
A.2 INPUT MODALITIES AND OUTPUT REPRESENTATIONS
OmniEVA is designed to accommodate wide range of input modalities and output formats, abling versatile interaction across visual and textual domains. Below, we detail the supported con- figurations.
17

---

Published as a conference paper at ICLR 2026
A.2.1 VISUAL INPUT MODALITIES
Single Image Ideal for static minimally dynamic environments, single-frame inputs support 2D spatial reasoning tasks such as object recognition, scene description, and basic grounding. This modality is particularly effective when temporal context unnecessary and spatial relationships are confined to single viewpoint.
Multi- View Images or Video By aggregating information across multiple viewpoints or temporal frames, this modality facilitates both spatial and temporal reasoning. It is well-suited for dynamic or scage-scale environments where understanding motion, continuity, or tross-framp object relation- ships is essential such as navigation, tracking, or multi-step manipulation tasks.
RGB-D Video This modality integrates RGB visual data with depth information reconstruct full 3D scene geometry. It is indispensable for tasks requiring occlusion-aware reasoning, volumet- ric understanding, or precise spatial manipulation. To enable accurate position embedding in world coordinates, users must provide the intrinsic camera matrix and corresponding extrinsic poses for each frame. These parameters allow the model transform depth maps into structured 3D repre- sentations, forming the foundation for geometry-awane decision-making
A.2.2 TEXTUAL AND GOORDINATE-BASEE OUTPUTS
OmniEVA accommodates range of textual and spatial formats for both input queries and output responses, enabling flexible interaction across semantic and geometric dimensions.
Natural Language Queries and Responses Natural language serves the primary interface for user interaction, supporting expressive queries and interpretable model responses. This format aligns with standard benchmarks in VQA and facilitates rich semantic engagement, allowing users to spec- ify tasks in intuitive, human-readable form.
2D Spatial Annotations For tasks such as 2D visual grounding and image captioning, inputs and outputs can be expressed using normalized pixel coordinates within the range [0, 1000]. This format enables precise object localization and descriptive annotation within a single image frame. Examples:
Question: Describe the object located <poinl>(24, 312)</poin1>.
Answer: is a brown book next to a pencil.
Question: Locate the apple on the left side of the book.
Answer: (po2nl>>122, </ponn/
3D Spatial Annotations For tasks involving 3D spatial reasoning- such as object captioning, grounding, and navigation- may specify coordinates manually allow the model to infer them from RGB-D inputs. Coordinates are discretized using a 0. 0m1-reter grid to ensure consistency and precision across scenes.
Question: What is the object located at (6.bo:x>6....1.266..577
Answer: is a brown wooden chair located the center of the room.
Question: Locate the second chair next to the table.
Answer: <3dbb>><74213333331.0888
This format empowers OmniEVA reason about partially occluded objects and those outside the current frame, enabling robust interpretation and interaction within complex 3D hhunnmenttsshuohh
A.3
IMPLEMENTATION DETAIL OF EMRODIMENT-AWARE REASONING
Given reward for the i-th response:
(i, 0:) =r format (
r.. (9, (g)
(11)
18

---

Published as a conference paper at ICLR 2026
For group of responses with group size G, the normalized advantages of the i-th response training step t is calculated as:
A Fia mean( (ro,,, fi, ,r,.)"
(12)
std(fru,e, f1, rG,.))
The final policy update objective is,
J.(0) G
Σ
min
To (e(l)))) Tona (o:g) Ar,, clip To(oilg) Teai (0:14)
e, +2) dees -T (Te Toau)
(13)
where /3 is a regularization coefficient that restricts the deviation degree between the current policy wo and the reference policy *aay during optimization; is positive coefficient that limits the mag- nitude of policy updates, preventing training instability caused by excessive updates. Through this embodiment-aware training pipeline, OmniEVA evolves from perceptual understanding to physi- cally grounded execution, enabling generalizable planning and reliable performance across diverse real-world scenarios.
A.4 MULTI-DIMENSIONS DATA MIXED TRAINING
To enable mixed training of text-only, image-text, and RGB-D data within mini-batches, we pre- process the data to ensure uniform tensor dimensions (as described Listing [ For image-only samples, add 3D data placeholder. A binary depth flag associated with each sample indicates the presence valid 3D data; during training, this flag acts as zero mask to ignore the 3D compo- nent for samples that lack See Listing 2 for detail.
def preprocess_sanee
(sample: dict) -> dict:
1) Load and transform frames -> pixel_ values: (T, C, H, W) frames Load_frames (sample)
pixei_values stack (Itranstcrm) (ing) for ing frames]]
pixe1 val ues. to (bfloat16)
T, C, H, w pixel_va val_vallees ues. shape wew..hape..eeeeeeeeeeeeeeeeeeeee
Process depth data based sample type if sample. ("video-rgod"):
RGB-D video Obtain 3D world coordinates from preprocessor world_coords = preprocessor get_ (sampll (sample) dcpth_flags tensor ([True] T, dtype=bool)
else: Pure (image or video without depth)
Pil1 with zeros and mark as no-depth world_coords = zeros ( (T, H, W, 3), dtype=bflcat16) depth_flags = tensor ([False] Т, dtype-bool)
3) Return dicticnary for model input return
"pixel values:: Lixel_values "world_coords"; world_coerds,
"depth_flags"; depth_flags,
Listing 1: RGB-D and RGB Data Processing
class OmniEVAModel (onaaaeeennnnMMuuiiii Module)
def ieiii_nnsee____
super () init_
self. self.visiio_ensoder.
self.llm self.mlp_proj
the following modules are introduced by CmniEVA
1.f.posiition__B_eccde
self. falg,lang_encodrr rentencel (...)
19

---

Published as a conference paper at ICLR 2026
self gate module gadlllatte_meeeeeee
def (self,, pixcl_values, input_ids, world_ooords,
depth_flags, lang_instr, **wwargs);
1) Extract vision features and project LLM space
vit_feats welf.vision_enceee (pixe__aaei $ (B+T, N, D_vit) vit vit_embeds self p
$ (B+T, N, D_llm)
2) Compute 3D positional encoding from world coordinates if world coords is not Nene:
pe3d self. pwssiiionneee _d__dd_encdder (world_coordd)
else:
pe3d - 0
Hard gating: pool vision features for gate control h w int tttt_embeds.shappee [1] 0.5)
Reshape: (BI, N, D) -> (B, T, h, W, D)
vit_pooled vit_embeds (etbbch)) dctach()
view (B, T, h, -1)
(B, vit_poolee vit_ppolld.. flattenn Average pool: (B, Tahaw, D) -> (8, D) vit_pooled vit_ mean meanmmimaddm (dim=1, keepdim=False)
Flatten spatial-temporal
h, W, D) (B, Thh end_dim=3)
Compute gate mask from Janguage and pooled vision features
lang_ embeds self.lang enccderecceerrreeeee encode (lang instr) # (B, D lang) gate_logits - self (ww.:gate_mmdullelagg___eeeeeeee
(B, 2) gate_mask gumbel_softmen ggate_logits, tau-tau, hard-True) 1;, 11 gate_mask gate_mask. fite_mask.viewtte (B, 1, 1) expand 1) view (BT, 1, 1)
vit_pooled)
4) Apply depth flag mask dcpth_mask = depth flags, view 1, 1) vit_embeds vit_embeds + pe3d gate_mask depth_mask
5) Merge vision tokens into language embeddings input_embeds =
ngs () (nnput_ids)
img_token_mase _ask (input_ids -- self. img_context_tooenn input_embeds [img_token_maske
vit_embeds. flatten 1)
6) Forward through LIM
outputs = self. (wwpttsseeneedeeeeeeee eebeees-inptttt_mmeeeee _mbedd, **kwargs)
Compute auxiliary gating loss (KL divergence with soft labols)
if self.training:
soft_labels = get_soft_labels/
aux_loss (1cg_sfaooo oog__ooofftaagggaa_elootttttt (gate_logits)) soft_labels) return outputs, less
8) Replace placeholders in input_ tokens with visual tokens.
Listing 2: Forward Pass of OMniEVA
A.5 IMPLEMENTATION DETAIL. OF EMRODIMENT-AWARE REINFORCED FINETUNING
B CASE STUDY
B. CASE STUDY OF THE ACTIVATION STATE OF THE TAGR MODULE
We illustrate this behavior through qualitative case studies (Figure 6). In the first two examples, querying the shape of table and desk activates the 3D gate with differing probabilities: 0.73 for
20

---

Published as a conference paper at ICLR 2026
Algorithm Embodied Task Learning with Embodiment Constraints
Input: Task parameters Embodiment constraints 3
2: Output: Optimized policy parameters l"
3: Initialize policy network Te
4: Initialize task reward model Rlask
5: Initialize embodiment constraint handler Rembod
6: for episode 1 to N do
7: Reset environment and get initial observation Op
8: Initialize episode trajectory
0
9: for 0 to 7 do
10:
Sample action a, (T(-)or)
11:
Execute action and get next observation o++1
Task Reward Computation:
13:
Given instruction and observation
Fusk & CVModel(q, oe) Rule BuuleBasedMethod(( 0z)
15:
Get corresponding mask M, for RGB image
Trusk + [|MaskinniiatteeSuuussss/// M,]] (1 if success, 0 otherwise]
17:
Embodiment Constraint Reward:
18:
Inflate objects r "usk" Mintaaed + InflateObjeets(MMi)
Compute intersection r "imerr' Mintersett f- M, Mintttted
20:
Random sample points from Minterseer' P < RandomSample((Mnens
21:
Initialize y embod -- Timeer (Start with original task reward)
22:
for each point py E do
23:
Execute MoP at p; if execution failed then
24:
25:
Fembod <- 0 (Set to if any point fails) break
26:
27:
end if end for
28:
29:
Accuracy Reward:
Taee <- Tusk (de Tembod + (1 Xc))
31:
Store transition (oe, at, Tace; 0t++1 in
32:
end for
33: Policy Optimization:
34: Update 9 using GRPO with trajectory т
35: end for
36: return Optimized parameters l"
21

---

Published as a conference paper at ICLR 2026
the rectangular table, indicating ambiguity between "square" and "rectangular" and thus reliance on 3D cues; and 0.52 for the round table, suggesting sufficient 2D visual information. In contrast, object counting and color identification in the two night-hand examples leave the 3D gate inactive, demonstrating the TAGR module's ability to omit 3D features when spatial reasoning is unnecessary.
Images
sitting on ... What shape I am shape of desk behind me Prompt the talle am sitting at?...
How many monitors are round. roctangular? the desk frent of me?
armehnirs on Is all the scating color?
Answer
roctangular.
round.
three.
yes.
Gate
Activaled
Activated
Dectivated
Dectivated
Prob.
0.73
0.52
0.39
0.38
Figure 6: Case Study of Gate Activation State. Selected examples from the validation dataset illustrate the most prominently activated deactivated words within the input prompts, highlighting model's sensitivity to specific language cues.
B.2 CASE STUDY OF EMBODIMENT-AWARE REASONING
We compare outputs from models trained with and without it for the same task, as shown in Figure Results show that both models effectively identify unoccupied areas the tabletop, indicating sat- isfactory task-level performance. However, the placement locations proposed by the model without TE-GRPO training frequently fall outside the operational range of the robotic optimal execution efficiency. In contrast, OmniEVA trained with TE-GRPO consistently identifies vacant areas that both feasible and executable, demonstrating enhanced alignment with physical and task constraints.
exhibit sub-
w/o. TE-GRPO.
w. TE-GRPO.
w/o, TE-GRPO.
TE-GRPO,
Figure Comparison of responses w/o and embodiment-awarr reasoning.
B.3 ROBUSTNESS IN DYNAMIC ENVIRONMENTS
While compatible with existing temporal embedding techniques like MRoPE used in Qwen2.5VLL OmniEVA VA's architecture exhibits strong emergent robustness in dynamic environments, despite not being specifically designed for them. This capability is evaluated through scenarios involving mov- ing pedestrians, as shown in Figure S In the first experiment, the model is tasked with locating table opposite to the fridge that is initially occluded by pedestrian. After a change in the agent's viewpoint causes the pedestrian to move away, OmniEVA successfully identifies the newly revealed table. In a more challenging second experiment, the model accurately detects a microwave despite a pedestrian continuously moving within the field of view. These results demonstrate that OmniEVA's design inherently handles the visual uncertainty posed by both transient and persistent dynamic obstructions.
22

---

Published as a conference paper at ICLR 2026
Dynamis Poolect
Prompt: identify nearest opposite refrigerator.
3D Visualization
Dynamic
Prompt: Identify the 3d box of the object according to the following description. A white
3D Visualization
microwave the cabinet below the countertop.
Figure 8: OmniEVA's Performance with Dynamic Obstructions. (Top) Sequence showing successful detec- of table after an occluding pedestrian moves away. (Bottom) Accurate detection microwave despite continuous pedestrian motion.
B.4 How DOES OMNIEVA ADAPT TO PHYSICAL AND EMBODIMENT CONSTRAINTS?
To further understand the model's embodiment awareness, we analyze the output of OmniEVA-ER trained with TE- TE-GRPO, As illustrated in Figure 9 OmniEVA-ER initiates reasoning with task-level analysis, including scenario interpretation, commonsense spatial analysis, and textual localization of vacant areas. It then incorporates physical constraints- as reachability and workspace bound- aries-to refine its predictions. Through such embodiment-aware slow thinking processes and re- ward reinforcement, the final output is placement region that satisfies all constraints. A compar- ative study (Appendix B.2) highlights the impact of embodiment-awar reasoning. For real-world validation, OmniEVA is deployed on wheeled dual-arm robot, demonstrating reliable performance converting user instructions into executable plans across diverse physical settings (Appendix G
Prompt: Locate some space for me, please.
Response: <think>1. J. Describing the Image: table with various objects, including cup 2. Common-Sense Consideration:: Free space refers
the table that not Embod- iment Awareness: For optimal performance, we should place the robot near the bottom of the table to ensure it can access its entire intended work area. S. Answering the Question: free space on the right side of the table near the edge... <Ithink> caannnt--ccaeee-oatiiiio are <point>(450, 346), (455, 348), (467, 352) </point>. <lanswer>
occupied... ... 3. Relevant Factors: Object Dimensions:
Table Dimensions:
Figure 9: Case study illustrating OmniEVA's reasoning process under embodiment aabbodiment-wwareee aware constraints.
C ABLATION STUDY IMPLEMENTATION DETAILS
C.1 IMPLEMENTATION OF PROSS-ATTENTION BASED 3D FUSION
To rigorously evaluate the necessity of proposed gated router, implemented two variants of cross-attenis mooss-attentio-baae based 3D fusion as strong baselines. The objective was to assess whether standard attention mechanisms could achieve adaptive feature fusion without an explicit gating module. The architectural details are illustrated in Figure TO
Separate Tokens Arrangement: In this variant, the sequences visual tokens (V') and 3D positional encoding tokens (VP) are kept entirely separate. They concatenated
23

---

Published as a conference paper at ICLR 2026
along the sequence dimension, effectively doubling the imput length presented to the trans- former backbone. standard cross- cooss-attention layer is then employed to enable interaction between these two modalities.
Interleaved Tokens Arrangement: In this variant, tokens are grouped and sequenced by the source image. Specifically, for batch of N input images, the token sequence is structured as: IVP. V.. Vg2 V22 ... V V- That is, for each image, all its 3D tokens are placed first, followed by all its visual tokens, and this pattern repeats for the next image. This arrangement maintains the total sequence length but structures it to preserve intra-image modality groupings, relying on the self-attention mechanism to learn the cross- modal dependencies.
As discussed in Section 4.2, both cross-attention variants led to significant performance drops com- pared our gated fusion. We attribute this the inefficient learning burden placed on the attention mechanism- must either handle doubled sequence length (Separate) or learn to associate inter- leaved condality-speciffe groups (Interleaved) from scratch. In contrast, our TAGR module performs explicit, lightweight routing before the backbone, leading to more stable and efficient integration.
Large Language Model
Large Language Model
Large Language Model
TAPC Module
leel
Depth
Depth?
Imaget
Image?
leel
Depth1
Image!
Death?
Image?
lect
magel
Image?
Depth3
Depth?
Baselinel:
separate 3d tokens
Baseline2: interleaved 3d and visual tokens
Ours
Figure 10: Architectural diagrams of the nosss-ateentionbeee bassd 3D fusion baselines. (a) Separate Tokens: Visual and 3D tokens are processed as separate sequenees. (b) Interleaved Tokens: Tokens are sequenced by grouping all 3D tokens and all visual tokens for each image consecutively. Both approaches introduce learning inefficiencies absent in our gated fusion.
C.2 GATE ACTIVATION ANALYSIS BY SEMANTIC CLUSTERING
We introduce a more robust analysis based on semantic clustering of the entire prompt. This method leverages a language model encoder (e.g., a lightweight Sentence Transformer) to capture the overall meaning of a prompt before categorizing it. The detailed procedure is outlined in Algorithm 2 The example prompts used for clustering are listed in Table
D TRAINING DATASET
D.1 DATASET OVERVIEW
Figure [1 presents a comprehensive breakdown of the training dataset utilized by OmniEVA. De- signed to support anni-multimodal and cross-dimensional reasoning, the dataset integrates three major categories: general data, image-based reasoning data, and 3D reasoning data. This diverse composition enables OmniEVA to develop robust capabilities across wide spectrum of tasks, from basic object recognition to complex spatial and semantic understanding.
In total, the dataset comprises approximately 5.2 million samples. Detailed descriptions and distri- butions of each data category are provided in the subsequent sections.
D.2 EMBODIED REASONING DATA
General Visual Question Answering To maintain the foundational visual reasoning capabilities and generalization strength of the vision-language model (VLM), we integrated diverse set of gen- eral visual question answering (VQA) datasets. Notably, VLaVA-665K (Liu et al. 2023) contributes a broad spectrum of tasks, including VQA, optical character recognition (OCR), region localization, and instruction instruction-follovingg To further enrich the dataset, we incorporated academic datasets such as GQA (Hudson& & Manning 2019), OKVQA (Marino al. 2019), and A-OKVQA (Schwenk al.,
24

---

Published as a conference paper at ICLR 2026
Algorithm 2 Semantic-based Gate Activation Analysis
Require: Prompts P {pi. P2: .... p. Gate logits G fgi 92...., In
Require: Semantic templates 7 = (7i, 72, ... Ta } for k categories
Require: Similarity thresholds Thigh 0.4, Noww 0.25
1: // Step I: Encode templates
2: for each category e {L...., k} do
3: E. + panteenceTransformeert
(Encode templates]
4: end for
5: // Step 2: Batch encode all prompts
6: Ep t- Sentence TraienneTransfommerr(. P) (Shape: n x d3
7: // Step 3: Vectorized similarity computation
8: for each category с E {l.... k} do
9: S. + CCosineSimilarity(ee (Ep, Ee) (Shape:
[Te]]
10: M. + m axis = 1) (Max similarity per prompt)
11: end for
12: M +- [M,, M2, .... M&] (Similarity matrix: n x k)
// Step 4: Assign categories
14: for each prompt i E [l.....n n) do
C. - fcl Mi. c| Thigh (High confidence match) }
16:
C then
< arg max, M cl
18: C - (ct) if M(i,c") Tioe else {other}
end if
20: end for
21: // Step 5: Compute activation statistics
22: for each prompt i with categories C, do
23: is activated, + [9.[1] ] > 9:(0]) (Binary gate decision)
24: for each category E C, do
25:
if is activivaee,, then
Count [c] + Countactive [c]
27: else
Countiaactiv [c] Count [c] 29: end if
28:
end for
31: end for
32: // Step Compute activation rates
33: for each category do
Rate, Countyatve (n] Countanstive [
Count [c)
35: end for
36: return [Rate.]
1
2022 which emphasize compositional reasoning and external knowledge grounding. OCR-focused datasets like OCR-VQA (Mishra et al.) 2019), TextVQA (Singh et al. 2019), and ScienceQA (Saikh) ct al. 2022) enhance the model's ability to interpret embedded textual content. Additionally. region- level vision-language understanding is supported through RefCOCO (Yu al. (016)) and Visual Genome (Krishna et al. 2017). which provide fine-grained spatial and semantic annotations. To bol- ster language fluency and multimodal dialogue coherence, also included approximately 40,000 pure text instruction samples from ShareGPT (Chen et al. 2024b).
2D Visual Grounding To endow OmniEVA with robust object detection and geometric localiza- tion capabilities, we incorporated the LVIS (Gupta et al. 2019) dataset-a comprehensive bench- mark for large-vocabulary instance segmentation. LVIS offers approximately 164,000 images anno- tated with nearly million high-quality segmentation masks, spanning over 1,000 entry-level object categories such as "chair" and "plate." Its rich diversity and precise annotations make it ideal foundation for training models in spatially grounded object recognition. We utilize 140K samples from LVIS train the 2D visual grounding module.
25

---

Published as a conference paper at ICLR 2026
Table 8: Semantic Category Templates for Gate Activation Analysis
Category
Template Examples
Category
Template Examples
Shape
What shape is it?
Spatial Location
Where is the object?
Square, rectangular, round Cylindrical, spherical Triangular, oval
Position and location
Left, right, above, below
State Condition
Is it open closed?
Appearance
What does it look like? Describe appearance Visual characteristics
The current Empty, full, broken
Counting
How many objects? Count the number Several items present
Identification
What this object?
Identify the item What type of thing
Spatial Relation
What next to it? Between which objects
Direction
Which direction facing?
The orientation Pointing towards
Closest to what
Material Texture
What is it made of? Wood, metal, plastic Smooth, rough surface
Visibility
Can you see clearly?
Visible or hidden? Partially blocked
Color
What color is it?
Existence
Is there any object?
The color object Red, blue, green
Does exist?
Something present or not
Semantic Cat.
What category?
Distance Depth
How far is it?
Type of furniture
Distance from camera Near far away
Classification of objects
Action Activity
What it doing?
Other
Uncategorized or low-similarity
The action or activity Walk, run, throw, placing
prompts (sim. < 0.25)
General Data
12.7%
Components
Object Raforrroz 511K
Free Space Local zallon: 230
Olt Par Reoog ikon 408K
20 Vuai Grounding 140K
5,2M
51.3%
Spadel Undenbeniing 2811
S33K
Data
Activo 18K
Image 30.3%
Subqual
Scame Imaginaton:
3D Captioning 37K
5.7%
Video Ren.
Data
Figure 11: Overview of the Training Data used by OmniEVA
Object Reference Object reference data plays a crucial role in enabling OmniEVA to associate linguistic instructions with specific visual regions. We curated three complementary datasets to sup- port this capability. (1) Osprey-724K (Yuan al.) 2024b) is a large-scale instruction tuning dataset specifically constructed achieve pixel-level Diston-language alignment. Designed to overcome the limitations of traditional multimodal models which primarily operate at the image-level or bound- ing box-level incorporates regional masks linked with precise language descriptions to enhance
26

---

Published as a conference paper at ICLR 2026
model performance in fine ine-grrined visual understanding tasks. (2) Robopoint Object Reference- 347K Yuan et al.) 2024a) involves locating keypointss within a given reference object in an image. For example: "In the image, object is marked with red box. Please indicate several points located in the area below this object." The model's response would be in the form of normalized coordinates such [00.66, 0.69), --]. Such data helps the model learn to accurately identify target positions that have spatial relationships with reference objects, making it suitable for applications like robotic grasping or object association scenarios. (3) RoboRefIt (Lu et al. 2023) specifically de- signed for visual grounding tasks in robot interaction. It aims to enhance robots' ability to recognize and locate target objects based on language instructions in real-world scenarios. The dataset com- prises 10,872 real RGB-D images collected from cluttered indoor environments in daily life. Each image is annotated with referring expressions (instruction sentences), totaling 50,758 entries, which describe object features or locations in robot-oriented language style. Approximately half of the images contain similar or distracting objects, increasing recognition difficulty to simulate challenges in real-world grasping scenarios. Together, these datasets provide 511K training samples for object reference grounding.
Object Part Recognition Part recognition data is primarily used to visually highlight specific functional parts of objects in images, as referred to by corresponding linguistic instructions. These datasets include: (1) AGD20K (Luo al. 2022 is constructed by collecting and labeling over 20K images from 36 affordance categories, such as sit on, type on, and drink etc. Affordance grounding aims to locate objects' "action possibilities" regions, an essential step toward embodied intelliiencee. (2) HANDAL (Guo et al. 2023) is used for category-leve object pose estimation and affordance prediction. Unlike previous datasets, it is focused on robotics-ready manipulable objects that are of the proper size and shape for functional grasping by robot manipulators. such as pliers, utensils, and screwdrivers. The dataset consists of 308k annotated image frames from 2.2k videos of 212 real-world objects in 17 categories. It focus on hardware and kitchen tool objects to facilitate research in practical scenarios which a robot manipulator needs to interact with the environment beyond simple pushing or indiscriminate grasping. (3) PACO (Ramanathan et al.) 2023) is a large-scale dataset constructed for fine- ane-grained image understanding tasks, designed to support object- and part-level instance segmentation as well attribute recognition. It contains 57,643 images with 1,644,461 annotated object instances spanning 270 distinct categories (e.g., "body", "rim", "handle"). The dataset can be used for object detection, semantic segmentation, and instance segmentation tasks, and supports conversion from instance masks to semantic masks or bounding boxes for diverse downstream applications. In total, we utilize 408K samples from these datasets.
Free Space Location Free space location data is primarily used to visually mark vacant place- ment areas in images, as indicated by corresponding linguistic instructions. These datasets include: (1) Robopoint Free Space Reference-320K (Yuan et al., 2024a) In this dataset, the language in- structions require the model to identify keypoints in free space near a reference object-despiee the absence of clear visual cues. For example: "Indicate several points in the empty space to the left of the pizza box." This type of data enables the model to understand "where is a suitable re- gion to perform an action, even if no object is visually present," " making it highly applicable to robotic navigation or assisted placement tasks. (2) RelSpatial 3D Vacant (Zhou et al. (2025) Ref- Spatial is a large-scale dataset created to support Visual Language Models (VLMs) in performing 3D multi-step reasoning for spatial referencing tasks. It aims to enhance models' ability to under- stand complex spatial instructions in real-world environments. The dataset contains approximately 20 million (20M) crestion-answer (QA) pairs, covering 31 spatial relation categories, and supports multi-step reasoning of up to 5 steps. It includes locate empty space: define a point in an empty area on surface based on its spatial relationships with surrounding objects, and ask to confirm this empty location (e.g., "Please provide a point in the vacant area on the desktop that simultancously satisfies the following spatial conditions: ..."). (3) Open-X-Embodimg current datasets involving actions performed by robots/robotic arms, skills typically include "pick," "pick and place," " and more. We extract keyframes and manipulated objects from the correspond- ing trajectory data to construct object placement data. For example: RT-1 (Brohan et al. 2022) dataset comprises over 130,000 real-world robotic demonstrations (episodes), covering more than 700 different tasks. These were collected by 13 robots over a period of 17 months. The trained actions include diverse skills such as grasping/placing objects, opening/closing drawers, extracting items, standing objects upright, knocking them over, pulling tissues, and opening jars. BridgeData
(O Neileil et af. 2024) In
27

---

Published as a conference paper at ICLR 2026
V2 (Walke et al. 2023) dataset contains 60,096 trajectories spanning 24 different environments, including toy kitchens, sinks, microwaves, desktops, washing machines, toolboxes, and other di- verse settings. It encompasses 13 types of manipulation skills, ranging from basic socc-nd-place, pushing/pulling, and sweeping, to more complex tasks such as stacking blocks, folding cloth, and manipulating granular media. Together, these datasets provide 530K training samples.
Video-based Spatial Reasoning Despite the emergence of benchmarks such as OpenEQA Ma-- jumdar et al. (024)) and VSI-Bench (Yang et al. | 2025b), large-scale training datasets for video-based spatial reasoning remain limited. To bridge this gap, we construct comprehensive dataset by har- nessing high-fidelity indoor scene sources from ScanNet (Dai et al. (017). Matterport3D (Chang ct al, 2017). and 3RScan (Wald et al. 2019). From these sources, we extract egocentric video sequences and generate question answer pairs aligned with the task taxonomy defined in VSI- Bench (Yang et al.] 2025b), encompassing. (1) object count, (2) relative distance, (3) appearance order, (4) relative direction, (5) object size, (6) absolute distance, (7) room size, and (8) route plan- ning. Each QA pair is produced through a hybrid pipeline combining automated template genera- tion with manual verification to ensure spatial coherence and semantic precision. For route planning tasks, we first convert point clouds into x-y navigation mesh maps. Navigable waypoints are selected based three independently defined anchors: the start object, its orientation, and the end object. Using the A# algorithm, we compute the shortest path while merging trajectory points of adjacent objects belonging to the same entity. Steering directions are then derived from angular changes along the path, enabling fine-grained spatial reasoning. We use 281K samples for training.
Active Exploration Prior approaches typically assume fully observable environments, limiting their applicability real-world scenarios. We propose novel task that enhances spatial reasoning under partial observability. Given multiple images from an indoor scene, the model must select the most informative view to locate specified object. For instance: "From the provided visual input, identify the most informative image frame that offers the best chance of locating the bed. Format your response as: Frame ID: [Selected Frame IDJ." This task strengthens the model's decision- making in incomplete environments and is crucial for downstream embodied tasks such as object navigation. To support this, we curated 18K training samples from HM3D) (Ramakrishnan et al.) 2021 and MP3D (Chang et al., 2017), covering 6 and 21 object categories respectively.
3D Visual Question Answering To advance spatial reasoning in 3D environments, we inte- grate three complementary datasets, each contributing unique challenges and perspectives. (1) SQA3D (Azuma al. 2022): This dataset emphasizes situational awareness, requiring agents to interpret their position, orientation, and context within a 3D scene before answering questions. simulates real-world embodied cognition, where understanding one's spatial state is prerequisite for reasoning. We collected approximately 79K samples, covering diverse indoor layouts and object configurations. (2) ScanQA (Ma et al. 2022): Focused on general spatial understanding, ScanQA includes questions about object alignment, relative direction, and localization. It challenges models to parse nuanced spatial relationships from textual queries and visual cues. Our training set includes 23K samples, offering a rich variety of spatial scenarios. (3) MMMScan-Q (Lyu et al.) [2024): As
the largest and most comprehensive resource, MMScan provides over 1.28M QA samples built on ScanNet (Dai et al., 2017). Matterport3D (Chang et al. 2017). 3RScan (Wald et al. 2019), and Arkitscenes (Baruch et al. (02T). features hierarchical grounded language annotations spanning object-level and region-level semantics, enabling multi-granular reasoning. In total, we utilize ap- proximately 1.4M samples for 3D VQA training.
3D Captioning In this task, the model generates descriptive captions for objects given 3D po- sition or bounding box, detailing attributes such color, shape, and spatial relations. For instance, "Question: Describe the object located at (155,72.23, 15, 13,3). Answer: It is a light brown, wooden chair, located in front of a white table in the room." This task bridges geometric localization with natural language generation. We train on the Scan2Cap dataset (Chen et al. 2021). which comprises 37K annotated samples across diverse indoor scenes.
3D Visual Grounding As the inverse of 3D captioning, this task requires the model to localize objects in 3D space based on natural language descriptions. It poses a significant challenge for MLLMs,, which often struggle generate accurate 3D bounding boxes without priors from off-the-
28

---

Published as a conference paper at ICLR 2026
shelf 2D 3D detectors. For example, "Question: Detect the bounding box of a chair in the cornrr of the room, opposite to a brown desk. Answer: (78, 23, 135, 5, 5, 7)". We leverage ScanRefer (Chen) et al. 2020 and MMScan- VG (Lyu al. 2024 totaling 1.1M samples.
3D Scene Imagination To push the boundaries of spatial reasoning, we introduce task set in partially observable environments-whert some objects are occluded or outside the agent's field of view. The model must infer the contents of unobserved regions based contextual cues and spatial layout, given a 3D location within the scenario. For example, "Question: Based on the currently observed environment, when the agent walks to position (384, 42, 15), what new objects might become visible? Only consider objects not currently seen. Answer: You may see various cookers, cabinets, kitchen counters, kettles, ...". This task probes the model's understanding of object co- occurrence and spatial regularities. We collect 45K samples using the Habitat simulator (Puig et al.) 2023), drawing from MP3D (Chang et al.) 2017) and HM3D (Ramakrishnan et al. 2021) with randomized walk policy to ensure diverse and unbiased scene coverage.
3D Subgoal Prediction Existing spatial reasoning methods for large-scale navigation-image based pointing Yuan et al. (2024a), marker selection, or direct command outputs (e.g.. "move for- ward" or "turn left") often struggle in complex, occlusion-heavy scenes under partial observ- ability. To overcome these limitations, we introduce a 3D-aware planning framework that ingests sequential RGB-D observations and directly generates subgoals in continuous 3D coordinate space. By formulating intermediate objectives as 3D waypoints, the model leverages the geometric struc- ture of the environment, avoids local optima caused by relying on single image views, and supports explicit long-term trajectory planning rather than making only myopic action predictions. More- over, accounts for occluded or unseen regions, enabling the agent to propose subgoals that guide exploration around obstacles and through partially observed areas. Our training set includes approx- imately 113K samples, supporting robust learning of spatial planning under uncertainty.
E BENCHMARKS FOR EVALUATION
E. EMBODIED REASONING BENCHMARKS
Embodied Reasoning Benchmarks with 2D Inputs To assess the model's embodied reasoning capabilities across visual modalities, we employ four established benchmarks: Where2Place (Yuan) et al. 2024a), VSI-bench (Yang et al. 2025b). PACO-LVIS (Ramanathan et al. (022), and RoboR- efit (Lu al. 2023). These datasets span both static images and dynamic video inputs, enabling comprehensive evaluation of spatial and temporal understanding and multimodal reasoning.
To further evaluate the model's 's capacities in versatile embodied tasks with physical constraints, we introduce four benchmarks that connect the primitive embodied capabilities with composite down- streamed tasks: Where2Go, Where2Fit, and Where2Approach, Where2Grasp. Compared to simulator-based online evaluation, this VQA-style approach substantially reduces evaluation over- head. Detailed examples and description can be found in Appendix E.3
The agent must select the most informative next view from multiple images to locate a target object in partially observable environment. The setting closely aligns with the Large Space Object Seeking tasks, where agents must infer spatial layouts and make decisions under uncertainty.
Where2Go:
The agent must identify the free space on the table by predicting a set of 2D points. Physical constraints, including object location, size, collision potential, must be considered, making this task highly relevant the Mobile Placement (Easy) tasks.
Where2Fit:
Where2Approach:
The agent must identify free space on the table that is not obstructed by any chairs. This task demands reasoning under occlusion as well as handling locomotion and manipulation constraints, making it closely aligned with the Mobile Placement (Hard) tasks in geometrically challenging scenarios.
"Since the original data annotations of RoboRefit and PACO- LVIS lack VQA pairs, we constructed min- imal evaluation set suitable for VLM based on image distribution, object category, part category, etc. The evaluation code is consistent with Where2Place.
29

---

Published as a conference paper at ICLR 2026
The agent must identify objects based on their color, size, location, and cate- gory. This task emphasizes object-centric recognition and directly aligns with the requirements of the Mobile Pick-up tasks.
Where2Grasp:
The relationship between primitive embodied capabilities and composite downstream tasks demonstrated in Section
To extend evaluation into three- dimensionnl spatial contexts, we adopt four 3D benchmarks: (Ma et al. 2022), ScanQA (Azuma et al.) 2022), Scan2Cap (Chen et al. (02T) and ScanRefer (Chen et al., 2020). These datasets challenge the model's capacity for open-ended question answering, scene captioning, and 3D visual grounding within richly structured 3D environments. By incorporating depth and geometry, they serve as critical tests of the model's ability to reason beyond planar representations.
Embodied Reasoning Benchmarks with 3D Inputs
E.2 END-TO-END ONLINE EVALUATION WITHIN SIMULATORS
While previous works often evaluate the performance of the MLLMs on offline dataset, we also perform end-to-end evaluation to bridge the gap between planning and robot execution within sim- ulators, on the following 3 introduced benchmarks. The benchmark is built based on a 3000m² office environment containing 8 core operation scenarios and 95 object categories representative of common workplace items. We categorize the benchmark into three progressive evaluation stages:
Large-Space Object Seeking: It is also referred object navigation in prior work. This task evaluates the agent's capability to locate given object in large space.
This evaluation set comprises over 30 representative scenarios featuring diverse background configurations, varing initial robot poses, and range of object types, sizes, and locations. The Mobile Pick-up task involves grasphing various objects across diverse scenes and tabletop configurations. The Moile Placement divided into two difficulty tiers based on environment complexity. In the easy tier, the robot only needs to consider the immediate table surface condition (e.g., object occlusion) to determine the optimal placement location, as done in Where2Fit, before placing the object. For the hard tier tasks, the robot must fist determine the optimal chassis poses while accounting for environmental constraints imposed by the spatial arrangements of tabletop objects and surrounding chairs (same setting as Approach) The evaluation involves navigating to target poses, followed by assessing trajectory planning for safe mug placement the table, with success rates calculated based on task completion accuracy. A comprehensive description of scenario design and task categoriza- tion is provided in AppendixE.4
Local Mobile Manipulation:
End-to-End Delivery: This task evaluates the integration embodied skills by requiring the robot complete end-to-end object-delivery tasks across the entire office environment. We select two metrics, the overall success rates and the average task completion times, to evaluate the effectiveness of the pipeline.
E.3 EXAMPLES OF THE IN-HOUSE PRIMITIVE EMBODIED BENCHMARKS
E.3.1 WHERE2GO
The Where2Go benchmark is constructed using the validation splits of the HM3D (Chang et al., 2017) and MP3D (Chang al.) 2017) datasets. Each sample presents partially observable en- vironment in which the model must select the most informative view to locate a specified target object. A frame designated as the ground truth if contains visible segment of the shortest nav- igable path from the agent's 's current position to the target object. In total, the benchmark comprises 207 samples, forming diverse and challenging validation set for evaluating view selection under uncertainty.
30

---

Published as a conference paper at ICLR 2026
Prompt: From the provided visual input, identify the most informative image frame (with IDs starting from 1) that offers the best chance of locating the sofa. Format your response as: Frame ID: (Selected Frame ID]
Ground Truth: Frame ID: 5,
2
3
Prompt: From the provided visual input, identify the most informative image frame (with IDs starting from 1) that offers the best chance of locating the monitor. Format your response as: Frame ID: [Selected Frame ID] Ground Truth: Frame ID: 4
31

---

Published as a conference paper at ICLR 2026
2
5
Prompt: From the provided visual input, identify the most informative image frame (with IDs starting from 1) that offers the best chance of locating the plant. Format your response as: Frame ID: |Selected Frame ID] Ground Truth: Frame ID: 2,
E.3.2 WHERE2FIT
The Where2Fit Benchmark addresses the task of identifying free space on tables by predicting a set of 2D points. These tables are drawn from variety of real-world scenes-such ence rooms, pantries, and workstations-ann and exhibit different levels of clutter, ranging from blank and sparse densely occupied. The benehmark systematically increases the number of objects across these clutter conditions, presenting a progressive challenge. In addition, it incorporates crit- ical physical constraints, including object dimensions, fit within the available space, and collision avoidance with other objects. The entire benchmark consists of 464 samples, including 200 genera- tion tasks that require the model to output corresponding points, and 264 judgment tasks where the model must determine whether given point would cause collision.
offices, confer-
Prompt: Locate some free space for me on the table.
32

---

Published as a conference paper at ICLR 2026
Prompt: Find me an empty spot on the table, thanks!
Prompt: Would you be able to place the red plug on the conveyor belt?
Prompt: Could you help me find vacant area on the table?
E.3.3 WHERE2APPROACH
The Where2Approach benchmark is required to identify unobstructed free space on a table while accounting for potential occlusions caused by surrounding chairs. The testbed features long table cluttered with objects and encircled by randomly arranged chairs, simulating geometrically complex and occlusion-rich environments. This task necessitates advanced spatial reasoning under substan- tial visual occlusion, as well as the integration of locomotion and manipulation constraints. Specif- ically, the agent must determine feasible chassis positions that offer sufficient unobstructed area for successful placement operations. These requirements closely align with the challenges posed by Mobile Placement (Hard) tasks, which emphasize operation in highly constrained and visually disordered scenarios. The entire test set consists of 200 samples, each covering completely different perspectives, robot positions, tabletop object configurations, and chair arrangements.
33

---

Published as a conference paper at ICLR 2026
Prompt: Find the nearest free space on the table with no chairs around.
Prompt: Locate the closest empty spot on the table that isn't surrounded by chairs.
Prompt: Look for the nearest available area on the table where no chairs are placed nearby.
Prompt:
Search for a nearby open space on the table that has no chairs in its vicinity.
E.3.4 WHERE22GRSSP
The Where2Grasp benchmark requires the identification of objects based on key altributes including color, size, location, and category. The evaluation set consist 200 samples and encompasses over 40 common object categories sourced from variety of household and office environments, with diverse backgrounds. This task emphasizes object-centric cognitive capabilities, focusing on the perception and interpretation of object characteristies under real-world conditions.
34

---

Published as a conference paper at ICLR 2026
Prompt: Locate the orange on the counter.
Prompt: Please locate the glasses.
Prompt: Locate the cola bottle on the table.
Prompt: Find the junction box the conveyor belt.
E.4 DOWNSTREAM TASK DESCRIPTION
E.4.1 MOBILE PLACEMENT EASY
For the Mobile Placement Easy benchmark, constructed scenes with 8 tables in office envi- ronment, with various items randomly scattered on the tabletops. There are total of 40 types of items enhance the diversity of the scenes. The robot's initial position is to 1.5 meters away from
35

---

Published as a conference paper at ICLR 2026
the edge of the table, with angular deviation -15 to 15 degrees, observe the environment and objects. The scenes are divided into three levels based on the number randomly scattered items on the tabletop: no-objects, sparse, and dense, with 0, and 8 objects the tabletop. respectively. The model's performance is evaluated in 200 simulation scenes, with 50 no-object scenes, 50 sparse scenes, and 100 dense scenes. We use the success rate of placing objects the evaluation metric.
E.4.2 MOBILE PLACEMENT HARD
In the Mobile Placement Hard benchmark, we construct long-table environment with chairs posi- tioned around the perimeter and varied objects distributed the table surface. The robot is required to identify approachable regions unobstructed by chairs or densely placed tabletop items. The table periphery systematically divided into 12 candidate zones (three per long side, one per short side, and four corners), each classified as either occupied (by chairs or dense objects, excluding corners) or vacant (clear spaces or sparse-object areas permitting approach). We additionally distribute ran- dom objects in the central table area, which improves visual realism and increases scene diversity.
Each simulation scenario involves randomized assignment occupancy states to candidate regions, with objects sampled from 70-category office inventory (e.g., monitors, books, mugs). The robot initialized 0.9 meters from the table edge (+15 deg angular deviation) observe the table and surroundings. The evaluation requires the model to predict feasible placement points, followed by robot execution attempts. Performance is assessed 100 simulated scenarios, where we measure the success rates of planning placement trajectories.
36

---

Published as a conference paper at ICLR 2026
E.4.3 MOBILE PICKUP
For the Mobile Pick-up benchmark, we constructed scenes with tables in an office environment, with various items randomly scattered on the tabletops. The objects to be picked up include: pen holders, cans, bottles, coffee cups, oranges, pears, apples, and lemons. There are total of 40 background objects to enhance the diversity of the scenes. The robot's initial position is to meters away from the edge of the table, with an angular deviation of -15 to 15 degrees, to observe the environment and objects. The scenes are divided into three levels based on the number of randomly scattered items the tabletop: easy, medium, and hard. In the easy level, only the objects to be picked up are on the tabletop. while in the medium and hard levels, there are 3 and additional objects besides the ones to be picked up. The model's performance is evaluated in 150 simulation scenes, with 60 easy scenes, 50 medium scenes, and 40 hard scenes. We use the success rate of picking up objects the evaluation metric.
F QUALITATIVE RESULTS OF OMNIEVA
Here present sample inference results generated by OmniEVA, including 2D visual grounding. object reference, object part recognition, free space location, activate exploration, 3d questionn answering, 3d captioning, 3d visual grounding and 3d scene imagination. All examples shown were not included the training dataset.
F.1 QUALITATIVE RESULTS ON 2D SPATIAL REASONING TASKS
37

---

Published as a conference paper at ICLR 2026
2D Visual Grounding
Prompt: the photo, where can I find the shirt?
Prompt: Highlight the location of the tarp. Response: Highlighted region:
Response:
You can find the shirt at <point>(457, 117), (428, 125)</point>.
<point>(331, 219), (248, 215) </point>.
Prompt: Locate the scarf in this image.
Prompt: Please specify the coordinates for the handbag.
Response: The searf found
(30int>(336,4233),331,,, </point>.
Response: Specifying coordinates:
<point>(331, 409), (256, 490) </point>.
Object Reference
Prompt: Please hand me the orange on the left.
Prompt: would like to grab the teddy bear in shirt.
Points are <point>(302 271), (343, 260)</poin1
Response:
Response: Center points are <point>(34) 241), (374, 245) </point>.
38

---

Published as a conference paper at ICLR 2026
The image features item encased within a red rectilinear shape and another item surrounded by green rectangular boundary. Pinpoint several spot son any item situated between the objects framed by these reclangles. Response: Locations are <point>(250, 212), (199, 214), (179, 226) </point>.
Prompt:
Prompt: What object in <point>(213, 261), (218, 239), (216, 220)</point>? Response: Scissors.
Object Part Recognition
Can you locate the frame on this mirror?
Prompt:
Prompt: How do you interact with the strainers?
Response: Points are <point>(301, 140), (318, 140)</point..
Response: By interacting with the <point>(1310, 880)</p
Kotes
Prompt: If I wanted to interact with the base of this glass, where would I find it?
Prompt: Could you point out the page of the book? Response: <point>(556, 2/4)</p
Response: You should interact with on the <249, 458), (272,
456)</point
39

---

Published as a conference paper at ICLR 2026
Free Space Location
Prompt: In the image, there's region outlined by red rectangle. Locate sev- eral points within the vacant area on this marked surface.
Prompt: Point free spot behind white textured radiator at lower right.
Response: Points are <point>(394, 345)
</point>.
Response; Locations are <point>(396, 435), (380, 440)</point>.
Point to free spot in front of white painted wood chest of drawers at center.
Prompt:
Prompt: What is the location of a free spot to the right of wooden trolley at left? Please provide its 2D coordinates.
Response: Points are <point>(196, 439)</point>.
Response: Points are <point>(173, 258) </point.
Active Exploration
2
3
5
Prompt: From the provided visual imput, identify the most informative image frame (with IDs starting from 1) that offers the best chance of locating the fireplace, Format your response as: Frame ID: (Selected Frame ID]
40

---

Published as a conference paper at ICLR 2026
Ground Truth: Frame ID:
Prediction: Frame ID: 5
1
2
3
Prompt: From the provided visual input, idenify the most informative image frame (with IDs starting from 1) that offers the best chance of locating the sofa. Format your response as: Frame ID: [Selected Frame ID]
Ground Truth: Frame ID: 3, 5
Prediction: Frame ID: 5
5
6
Prompt: From the provided visual input, identify the most informative image frame (with IDs starting from from 1) that offers the best chance of locating the sink. Format your response as: Frame ID: (Selected Frame ID]
Ground Truth: Frame ID:
Prediction: Frame ID: 1
41

---

Published as a conference paper at ICLR 2026
3
5
6
Prompt:
From the provided visual imput, identify the most informative image frame (with IDs starting from 1) that offers the best chance of locating the gym equipment. Format your response as: Frame ID: [Selected Frame ID]
Ground Truth: Frame ID: 5
Prediction: Frame ID: 5
F.2 QUALITATIVE RESULTS ON 3D SPATIAL REASONING TASKS
3D Question Answering
Prompt: am using the clothes dryer under another clothes dryer. What object is beneath the table to my left?
Prompt: am sitting on a chair while facing the whiteboard and there is chair the same side of the table on my left. What is the shape of the table in front of me?
Ground Truth: box Prediction: box
Ground Truth: rectangular Prediction: rectangular
42

---

Published as a conference paper at ICLR 2026
Prompt: Where is the bathroom vanity
Prompt: What color is the tool box the left of the piano? Answer the question sim- ply.
cated? Answer the question simply.
Ground Truth: to right of bathroom door below window
Ground Truth: red green
Prediction: left of toilet
Prediction: red
Prompt: I
making some copies and there are some pictures on my left. What is brown and square my left?
Prompt: throwing trash with chair on my right within reach. What is the color of the chair that is within reach on my right?
Ground Truth: table Prediction: table
Ground Truth: black
Prediction: black
Prompt: 1 am opening the window with the shoe rack on my right. What is on top of shelf behind me that I can place my back?
Prompt: I am standing by the poster printer and there is table behind me. Can see the box if turn around?
Ground Truth: no Prediction: yes
Ground Truth: bag Prediction: pillow
43

---

Published as a conference paper at ICLR 2026
3D Captioning (red box: 3D box given in prompt)
Given an object located at (ddbox2(15115555.11155ee33e8 describe...
Prompt:
Prompt: Given an object located at (3tbox<(12215..57777)33333 scribe
de-
Ground Truth: The coffee table is white. The coffee table is front of the orange couch.
The chair is the color black. It is facing the table along with other chairs. It is spaced beside other chairs. Prediction: This is a black chair. is at the corner of the table.
Ground Truth:
Prediction: The coffee table is in the middle of the room. The coffee table white rectangle.
Prompt: Given an object located at <3dbox>(140)
Prompt: Given an object located at (1b3x><153,152.555 7,))5,,8,7,)///bbbbx>>>>.
Ground Truth: black computer with wooden desk with a wooden chair in front of Prediction: This is black laptop. It is
Ground Truth: The chair is red and green. is to the right of the black chair.
blue screen. It's located
Prediction: This is chair with arms. It is in front of a desk.
desk.
44

---

Published as a conference paper at ICLR 2026
Prompt:
Given an object located <3dbox> 15,,5,,,,,111,,, 1,4,4,455 3.44444)5333dloo ///.oo//d
Prompt:
Given object located at (3,5,,1,,,,3,,,,6..,),))
Ground Truth: The black backpack is di- rectly in front of the nightstand. The back- pack also directly to the left of the bed. Prediction: The backpack is black. It is the floor, to the left of the bed.
Ground Truth: This sturdy chair, made of wood and with green cushioning. It is in contact with a wall surface and a brown ble. It is oriented about 120 degrees in rela- tion to the viewer.
Prediction: The chuir is the one closest to the door. the chair has a green seat and four legs.
3D Visual Grounding (red Box: ground truth; green box: prediction)
Prompt: There is black office chair. placed in the side of the wall.
Prompt: It is next to the tall floor lamp. it is snack machine and it is full of tasty snacks. it is over by the door Response: 3ddboxx(13, 135, 59, 10, 8. 17)</3dbox>.
Response: <3dbox>(130, 155, 55, 6, 5, 4)</3dbox>.
Result: Success
Result: Success
45

---

Published as a conference paper at ICLR 2026
Prompt: In the corner of the are boxes. the first two book shelves in the cor- ner to the right of the boxes are the book- shelves we are looking for
Prompt: .. The keyboard sitting top of the desk. it is in front of the left-side moni- tor.
Response: 20)</3dbox>
<3dbox> (141, 125, 59, 9, 1,
Response: <3dbox><127, 139, 61, 4, 21,
20)</3dbox>
Result: Success
Result: Success
Prompt: This is a black trash can. it is
Prompt: ... This is a green tool box. the green tool box is in front of red tool box on the floor next a piano.
under the counter top.
Response: <3dbox>(159, 143, 51, 3, 4. 4)</3dbox>.
Response: <3dbox>(147, 127, 52, 5, 3.
Result: Success
3) 3)</ddbox>.
Result: Success
Prompt: ... This is a black leather loveseat in a library. if you were sitting in it, the long. short bookshelf would be on your right and the tall book shelf, opposite you across the room.
Prompt:
The black chair; in front of the chair with a brown table, on the right has the left is black chuir
black chairs. with the left to right it is the first chair.
Response: <3dbox>(154, 159, 54, 5, 6. 10)</ddbox>.
Response: <3dbox> 145, 168, 57, 15, 9, 7)</3dbox..
Result: Fail
Result: Fail
46

---

Published as a conference paper at ICLR 2026
3D Scene Imagination (from HM3D and MP3D validation set)
Prompt: when the agent walks to position <3dbox>>122, 129,21, 10, 10, 10)
Prompt:
when the agent walks to position <3dbox>(153, 164,26, 10, 10, 10) </dbox>, what new objects might become visible? ...
</3dbox>, what new objects might become visible?...
Ground Truth: You may see various pictures, chairs, vases, trays, plates, coffee mugs, washing machines, lamps, curtains, rugs, vents, balustrades, sensors, tables, books, candlesticks, and pillows. Prediction: You may see various lamps, ventilations, and cabinets.
Ground Truth: You may see a few motion detectors, blinds, wall toilet papers, bath- room towels, trashcans, toilets, decorations, decorative plants, and bathroom shelves. Prediction: You may see various lamps, pic- tures, and beds.
Prompt:
when the agent walks to position <3dbox> 155,49, 10, 10, 10) </dbox>, what new objects might become visible?...
Prompt: when the agent walks to position <3dbox>(146, 157,21, 1, 10, 10, 10) </3dbox>, what objects might become visible?
Ground Truth: You may see various boxes, pictures. lamps. ceiling lamps, drawers, stools, appliances, beds, fire alarms, pil- lows, fireplace walls, racks, and tables. Prediction: You may see various pictures, chairs, tables, and plants.
Ground Truth: You may various bath- room cabinets, pillows, benches, handbags, seats, and pictures.
Prediction: You may see various lamps, cabinets, pillows, and beds.
47

---

Published as a conference paper at ICLR 2026
when the agent walks position <3dbox><130, 14,,00,0,,,,,0000
Prompt:
Prompt: when the agent walks to
position (3dbox>(16,1166.1 1186,2,10010.11
</3dbox>, what new objects might become visible?
</3dbox>, what new objects might become visible?
Ground Truth: You may see various shelves, tables, mirrors, decorations, chairs, sinks, faucets, liquid soaps, door windows, TVs, curtains, curpets, wine racks, closet areas for hanging clothes, clothes, bath- tubs, bath towels, shower soap shelves, soap bottles, shower walls, shower floors, shower glasses, shower mats, towel bars, towels, shower handles, ceiling lamps, and shower doors.
Ground Truth: You may see various ceiling lamps, smoke alarms, doorposts, headboards, wall lamps, nightstands, radios, lounge chairs, pillows, dog beds, window curtains, bathtubs, shower doors, bathroom cabinets, baskets, stair steps, and towels.
Prediction: You may see various shelves, cabinets, tables, chairs, flower vases, lamps, pictures, curtain rails, curtains, and carpets.
Prediction: You may see various pictures, tables, chairs. plants, cabinets, counters, sinks, and towels.
Object Navigation in Partially Observable Environments (examples sampled from HM3D validation set): green box-predicted 3D subgoal at each planning step; red line: past trajectory; red box: target object to be found
Prompt: Based on the currently observed environment, determine the best position go to find the toilet.
Exploration Trajectory: OmniEVA first explored the doorway of the room in the upper right corner. After mistakenly entering the kitchen area, it quickly adjusted its path, selected a new area to explore, and directly located the sofa the living room.
48

---

Published as a conference paper at ICLR 2026
Prompt: Based on the currently observed environment, determine the best position to go to find bed.
Exploration Trajectory: The model first moves to the hallway at the intersection of multiple rooms gather more comprehensive environmental information. It then determines that the room in the lower right corner is more likely to be bedroom, proceeds to that location, and ultimately finds the bed.
Prompt: Based the currently observed environment, determine the best position go find the chair.
Response: The model first walks the center the corridor, sees the sofa in the upper left. then heads to that room area and finds the chair.
F.3 EXAMPLES OF EMBODIMENT-AWARE REASONING
Embodiment-aware Reasoning
Prompt: Could you help me find a vacant area on the table?
Response: <think>l. ** **Deccribing the Image**; The image shows wooden table with various objects placed on it, including mug, a highlighter, small fruit, a notebook, white object resembling a bottle, and a chair positioned nearby. The table has some visible
49

---

Published as a conference paper at ICLR 2026
empty spaces, but the objects occupy certain areas.2. **Comnon-Sense Considerations The goal is to identify free space on the table where additional objects can be placed with- out causing collisions or instability. Free space should be large enough to accommodate typical objects like the ones present on the table. The placement should avoid overlap- ping with existing objects or causing them tip over.3. **Relevant Factors**: **Object Dimensions**: = The size of the objects already on the table should be considered when de- termining free spuce. **Table Dimensions **; The overall dimensions of the table should be taken into account lo ensure thut the identified free space is realistic. ***Colissonn Avoid- ance**, **: Any suggested free space should not interfere with the existing objects or lead to potential collisions with the chair or other furniture.4. **Free Space Identification**: The bottom-left corner of the table appears to have some free space, as there are no objects oc- cupying that area. The top-right corner also seems to have some free space, though it is farther than the bottom-left. 'The middle-right section of the table has limited free space due the presence of the notebook and the white bottle.5. ** *=Emoddmment Awareness The operational range of the robot necessitates that it be positioned as close as practicable to the table's bottom edge.6. * **Answer Formulation **: Based on the analysis, the bottom-led corner of the table is the most suitable for placing additional objects, as it offers the largest and most unobstructed free space. </think> <answer> Locations are <point>(66,, 359), (56, 360), (76, 357)</point>, </answer>
Prompt: Find the nearest free space the table with chairs around. Response: <think>Tb answer the question, we need to analyze the image comprehen- sively:l. **Describing the Image*s; The image shows large wooden table in room with multiple chairs placed around it. The table is cluttered with various objects, including books, containers, tools, and other miscellaneous items. There are also some empty spaces on the table.2. **Conmon Ceosonn--ensee Considerations **: The question asks for "nearest free space" on the table, meaning we need to identify areas that are not occupied by objects and are close to the center of the table. Free space should be large enough to place something without causing a collision with existing objects. The table's 's edges and corners are potential candidates for free space, but they may not be ideal due to limited usable area.3. **Relevant Factors**; -Identify all objects currently occupying the table. Determine the table's layout and identify any visible gaps open areas. Assess the proximity of these gaps to the center of the table. Ensure the identified space is large enough to accommodate new object.4. **Analysis center-left and top-right sections of the table. These gaps appear to have enough space to place small medium-sized objects without causing a collision with existing items. The center area of the table is less cluttered than the top-right edges, but is obscured by chairs in front.5. **Embodiment Awareness **; To ensure the robot can cover its entire potential work area, placement near the top-right edge of the table is critical.6. **Conclusion **; The nearest free space on the table can be identified the central-left section of the table, specifically around the coordinates (460, 250) to (470, 250), which is relatively clear area with no objects obstructing </think> canswer> Points are <point>(467 249), (458, 249), (478, 249) </point>. </answer>
The table has several visible gaps, particularly
the
50

---

Published as a conference paper at ICLR 2026
Prompt: Could you help me find a vacant area the table?
Response: <think> .... 1. Describing the Image: The image shows table with 2. Common-Sense Considerations: A vacant area must not overlap any existing objects large enough to accommodate
avoid edges or corners where objects might fall off. 3. Vacant Area Identification: The left side of the table appears have some empty space, particularly around the yellow cup and the green plate. 4. Embodiment Awareness: Consid- ering the range of the robot's executable area, it should be placed as close to the bottom edge of the table as possible. 5. Output: Based on the analysis: </think> <answer> Points <point><319, 416), (317, 404), (328, 398), (329, 410)</point>. (10)</point..//nsweee </answerr
G REAL WORLD DEPLOYMENT EXAMPLES
Figure 12: Example for meployment of OmniEVA on Real World Robots. Prompt: Place the paper cup in the empty space the table at the back right.
51

---

Published as a conference paper at ICLR 2026
Figure 13: Example for Deployment of OmniEVA on Real World Robots. Prompt: Place the cup the long table next to the mecting room.
H USE OF LARGE LANGUAGE MODELS
In accordance with the ICLR 2026 policy on LLM usage, we disclose that Large Language Models (specifically, DeepSeek-RI (Guo et al.) 2025) were employed during the writing process of this paper. Their usage was limited to aiding in the correction of grammatical errors of the manuscript. All scientific ideas, experiments, and analyses were conducted solely by the authors.
52


Reviews:
[{"review_id": "c2tjLyeOSg", "reviewer": "Reviewer_BSqi", "summary": "This paper introduces OmniEVA, an embodied planning framework that dynamically fuses 2D and 3D information via a task-adaptive gating mechanism (TAGR) and trains with embodiment-aware rewards (TE-GRPO). The model generates physically feasible plans for robotic manipulation and navigation. Extensive experiments on 8 benchmarks (2D, 3D, video, navigation, simulation, and real robot) show SOTA results with only 8B parameters.", "questions": "No", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors identify two critical yet often-ignored issues in embodied reasoning (geometric adaptability and embodiment constraints) and propose well-defined mechanisms to address both.\n\n2. Explicitly integrating robot kinematic limits and workspace awareness into the planner is both novel and useful for real-world applications.\n\n3. The model outperforms state-of-the-art baselines on several benchmarks and demonstrates credible real-robot deployment results.", "weaknesses": "1. The overall pipeline still largely follows prior MLLM-based planners, with incremental contributions (the gating and embodiment modules) rather than a fundamentally new paradigm. In addition, the claimed innovation of the gated router seems somewhat redundant, since modern attention layers already allow for selective fusion and weighting of modality-specific features. The paper does not convincingly justify why a separate gating mechanism is needed beyond what cross-attention or learned query routing could provide.\n\n2. While results cover 2D and 3D reasoning, most experiments focus on tabletop or indoor manipulation. It remains unclear how well the approach scales to more complex, dynamic environments.\n\n3. The real-world experiments are promising but relatively limited in scale. Only a few scenarios and robots are tested; cross-platform generalization is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03dGa8mvRz", "reviewer": "Reviewer_WCjS", "summary": "This paper aims to address two limitations of current multi-modal large language models (MLLMs): insufficient 3D geometric understanding and the embodiment constraint gap. The authors propose OmniEVA, featuring task-adaptive 3D grounding and embodiment-aware reasoning. OmniEVA achieves state-of-the-art performances across 7 out of 8 public benchmarks, including VQA and navigation tasks. Furthermore, this paper introduces 4 primitive benchmarks to demonstrate the strong embodied capabilities of OmniEVA.", "questions": "- Figure 4 shows the difference in gate activation across some object categories that involve only semantics, e.g., armchair, towel, copier, box, etc. How to explain this? Does this indicate a bias stemming from data distribution?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good motivation. Insufficient geometric understanding is a long-standing problem for MLLMs. And this paper takes one step further to address the gap between MLLMs and embodied AI applications by proposing embodiment-aware learning.\n- Thoughtful model design for the utilization of geometric features. Task-adaptive gated router (TAGR) effectively addresses the information bias between 2D semantics and 3D geometry. And the ablation study demonstrates the advantages of TAGR.\n- Embodiment-aware reasoning (TE-GRPO) is a novel solution to address the embodiment constraint gap, which properly integrates embodiment-specific rewards and reinforcement learning.\n- The experimental results are comprehensive and the authors propose four extra embodiment-oriented benchmarks that can take considerable efforts.", "weaknesses": "- The whole training pipeline is not delivered clearly, especially the data front. I cannot find an introduction to how the OmniEVA is trained. I only know that it goes through three stages: TAGR pretraining, SFT, and RFT. But for each stage, what datasets are involved? The terminologies are also confusing, such as general embodied reasoning, omnibodied reasoning, and embodiment-aware reasoning. Moreover, this results in my confusion about which model (stage) is reported in various experiment tables.\n- Limited insights beyond the overall good performance. The underlying factors include base model capability and data scale, which can make the comparisons unfair. The results turn out to be a pretty good performance with extensive training data and three training stages. However, we have no idea what data is significant and how much each stage contributes to the final performance.\n- Despite the efforts regarding embodiment-aware reasoning, I think the claims (such as Line 425) about real-world robotics tasks are not appropriate since there are no real-robot tasks and results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hcd99uDVAW", "reviewer": "Reviewer_5j5y", "summary": "This paper identifies two main limitations of current embodied planners (Geometric Adaptability Gap and Embodiment Constraint Gap) and presents a \"embodied versatile planner\" that uses MLLMs for embodied tasks by introducing two innovations: 1. Task-Adaptive 3D Grounding mechanism using a gated router for explicit, selective 3D feature integration, and 2. embodiment-aware reasoning and training framework that explicitly factors in real-world physical constraints during task planning and learning. Experiments on a range of embodied reasoning benchmarks—including 2D, 3D, and physically grounded settings—demonstrate OmniEVA's performance advantages and improved executability in simulated and, to some degree, physical tasks.", "questions": "1. Can the authors clarify whether the gate variable $g$ in TAGR is continuous (soft) or strictly discrete (hard) during inference? If it is sometimes fractional, how does this affect feature fusion, and are there cases where blended (i.e., partially injected) 3D cues yield improved or degraded performance? \n2. Please provide detailed breakdowns of training data composition (custom vs. public), any pre-processing steps, and how 2D and 3D modalities are mixed during training.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. the key analysis of current embodied planner limitations (Geometric Adaptability Gap and Embodiment Constraint Gap) is insightful and the two corresponding solutions make sense\n2.The introduction of the TE-GRPO curriculum addresses the persistent gap between theoretical plans and practical robot feasibility.\n3. OmniEVA consistently achieves or surpasses existing state-of-the-art performance on several challenging embodied tasks\n4. The work is highly relevant to current efforts to bridge the gap between vision-language models and physically interactive, multimodal embodied AI.", "weaknesses": "1. While some results for physical robots are reported and select qualitative examples shown (Figures 7–11), the main quantitative results are from simulation and offline metrics. There is mention of real-world deployment in the conclusion and some visuals, but the scalability and generalization of the framework to a variety of robot morphologies or environments is not systematically confirmed.\n2. While OmniEVA delivers strong results, the paper does not sufficiently discuss scenarios or limitations where the dynamic 3D gating may erroneously activate/deactivate or where the model’s embodiment-aware planning might fail or produce brittle outcomes. More failure cases and a nuanced discussion would add value. It would be great make clear the capability boundary of this approach and shed light on areas of improvement.\n\nThese two points lead to to question the claimed generalizability and applicability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eLPBDa19vi", "reviewer": "Reviewer_BYjf", "summary": "This paper primarily focuses on improving spatial reasoning, task planning, and embodiment feasibility within vision-language models (VLMs). Specifically, it introduces a gating mechanism to selectively fuse token-wise 2D visual features with their corresponding 3D representations, enhancing the model’s ability to absorb and reason over 3D information. Moreover, it incorporates task reward and embodiment feasibility objectives to strengthen the framework’s awareness of both task completion effectiveness and compliance with embodied physical constraints.", "questions": "Could you elaborate on how the task reward and embodied feasibility rewards are specifically designed and obtained?\n\nIn Table 3, did the baseline models also utilize the training datasets from these benchmarks?\n\nIn the real-world experiments, how does the model’s pick-and-place reasoning interface with and control the low-level policy?\n\nSince embodiment awareness depends on hardware characteristics, and different hardware platforms have varying physical constraints, how does the proposed method generalize or adapt to these differences?\n\nCan you provide a comparison of the reasoning time across models or settings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Enhancing the VLM’s understanding of 3D spatial structures is highly necessary, and enabling it to be aware of the robot’s embodiment feasibility is an important and well-motivated goal.\n\nThe introduction of the gating mechanism, task reward, and embodiment feasibility reward is conceptually sound and appears technically reasonable.\n\nThe experimental evaluation is comprehensive—covering diverse benchmarks that include 3D spatial reasoning, sub-goal localization in 3D environments, and 2D embodied reasoning tasks.", "weaknesses": "The overall writing quality is poor. Although the structure is generally acceptable, many paragraphs lack clear focus, and several sentences are unnecessarily verbose, leading to poor readability and coherence.\n\nSome key sections lack sufficient detail. For instance, several tasks are introduced without intuitive explanations — while I understand that detailed descriptions are provided in the appendix, a concise and intuitive introduction in the main text is still necessary. Moreover, the design and acquisition process of the newly introduced rewards are not clearly explained.\n\nThe baselines in Table 1 resemble an ablation study, and the effectiveness of the proposed gating mechanism is not convincingly demonstrated by the presented results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)