MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Models for Embodied Task Planning
Authors: Yuanchen Ju
Yongyuan Liang
Yen-Jen Wang
Gireesh Nandiraju
Yuanliang Ju
Seungjae Lee
Qiao Gu
Elvis Hsieh
Furong Huang
Koushil Sreenath
Venue: ICLR 2026 Oral (2026)
Keywords: Scene Graph
Task Planning
Spatial Understanding
Mobile Manipulation
Primary area: applications to robotics, autonomy, planning
TL;DR: We present MomaGraph, a unified scene representation for task-oriented understanding, along with a dataset and benchmark built upon it, and MomaGraph-R1, a 7B model that constructs MomaGraph representations and generates task plans.

Abstract: Mobile manipulators in households must both navigate and manipulate. This requires a compact, semantically rich scene representation that captures where objects are, how they function, and which parts are actionable. Scene graphs are a natural choice, yet prior work often separates spatial and functional relations, treats scenes as static snapshots without object states or temporal updates, and overlooks information most relevant for accomplishing the current task. To overcome these shortcomings, we introduce MomaGraph, a unified scene representation for embodied agents that integrates spatial-functional relationships and part-level interactive elements. However, advancing such a representation requires both suitable data and rigorous evaluation, which have been largely missing. To address this, we construct MomaGraph-Scenes, the first large-scale dataset of richly annotated, task-driven scene graphs in household environments, and design MomaGraph-Bench, a systematic evaluation suite spanning six reasoning capabilities from high-level planning to fine-grained scene understanding. Built upon this foundation, we further develop MomaGraph-R1, a 7B vision–language model trained with reinforcement learning on MomaGraph-Scenes. MomaGraph-R1 predicts task-oriented scene graphs and serves as a zero-shot task planner under a Graph-then-Plan framework. Extensive experiments show that our model achieves state-of-the-art results among open-source models, reaching 71.6% accuracy on the benchmark (+11.4% over the best baseline), while generalizing across public benchmarks and transferring effectively to real-robot experiments. More visualizations and robot demonstrations are available at https://hybridrobotics.github.io/MomaGraph/.

Published as a conference paper at ICLR 2026
MOMAGRAPH STATE-AWARE UNIFIED SCENE GRAPHS WITH VISION-LANGUAGE MODEL FOR EM- BODIED TASK PLANNING
Yuanchen Ju Yongyuan Liang"?, Yen-Jen Wang"), Nandiraju Gireesh, Yuanliang Ju
Seungjae Lee Qiao Gu Elvis Hsieh', Furong Huang Koushil Sreenath' '1 University of California, Berkeley University of Maryland, College Park
3 University Toronto
Project website: https: https://Rtyrridoobtttis.s.s
10/ 1o/MomaGraph/
"Power the laptup."
MomaGraph
"Open the window." "
"power by" "right of "lower than"
"open, or clase "close'
"in front_of
"Tum the TV."
"Turn off the light."
"cantrol" "right_of" 'in front_of
ccnnroll" right of*
'lower than
Figure 1: Overview of the MomaGraph. Given a task instruction, MomaGraph constructs task- specific scene graph that highlights relevant objects and parts along with their spatial- funttionnl relationships, enabling the robot to perform spatial understanding and task planning.
ABSTRACT
Mobile manipulators in households must both navigate and manipulate. This requires a compact, semantically rich scene representation that captures where objects are, how they function, and which purts are actionable. Scene graphs are a natural choice, yet prior work often separates spatial and functional re- lations, treats scenes static snapshots without object states temporal up- dates, and overlooks information most relevant for accomplishing the cur- rent task. To address these limitations, introduce MomaGraph, uni- fied scene representation for embodied agents that integrates spatial-functional relationships and part-level interactive elements. However, advancing such a representation requires both suitable data and rigorous evaluation, which have been largely missing. We thus contribute MomaGraph-Scent first large-scale dataset of richly annotated, task-driven scene graphs in house- hold environments, along with MomaGraph-Bench. tion suite spanning six reasoning capabilities from high-level planning to fine- grained scene understanding. Built upon this foundation, we further develop MomaGraph-R1, 7B vasion-language model trained with reinforcement learn- ing on MomaGraph-Scenes graphs and serves zero-shot task planner under Graph-then- Eraph-then-Plan framework. Extensive experiments demonstrate that our model achieves ofate-of-the-ar results among open-source models, reaching 71.6% % accuracy the benchmark 11.44%%% % over the best baseline), while generalizing across public benchmarks and transfer- ring effectively real-robot experiments.
the
systematic evalua-
MomaGraph-BP predicts task-oriented scene
Equal Contribution, Equal Advising

---

Published as a conference paper at ICLR 2026
INTRODUCTION
When mobile manipulators (Qiu et al., 2024; Honerkamp al., 2024a; Wu et al., 2023; Zhang al., 2024a) enter houschold environments, they face the fundamental challenge of understanding how the environment works, which objects are interactive, and how they can be used. In other words, such robots must not only be capable of navigating through the home, but also manipulate objects within it. While navigation requires modeling the overall spatial layout, manipulation demands capturing more fine-grained object affordances (Ju et al., 2024; Zhu et al., 2025). This naturally raises central question: What the most effective, compact, and semantically rich representation of an indoor scene? An intuitive answer is the scene graph, which organizes objects and their relationships in a scene through a graph structure (Armeni et al., 2019; Koch et al., 2024a;b) and has shown great potential in various downstream robotic applications (Rana et al., 2023; Werby et al., 2024; Ekpo et al., 2024).
However, existing scene graphs suffer from notable limitations. (1) Their edges typically encode only a single type of relationship, either spatial (Jatavallabhula et al., 2023; Gu et al., 2024a; Loo al., 2025; Hughes et al., 2024; Rosinol et al., 2021) or functional (Zhang et al., 2025; Dong et al., 2022202)2 (e.g. a remote controlling TV. a knob adjusting parameters). Relying solely on spatial relationships captures geometric layout but overlooks operability. while relying solely on functional relationships ignores spatial constraints, leading to incomplete and less actionable structures. (2) Most existing methods (Wu et al., 2021; Takmaz et al., 2025; Zhang et al., 2021) are limited to static scenes and struggle to adapt to dynamic environments where object positions change object states change. (3) They lack task relevance, as they fail to emphasize information directly tied to task execution, thereby reducing efficiency and effectiveness. In contrast, cognitive science research (Uithol et al., 2021; Kondyli et al., 2020; Castanheira et al., 2025) shows that human perception in new environments is both dynamic and task-oriented. Humans do not process all information equally: instead, they flexibly adjust their attention according to the current task. This process is similar browsing map on iPad: people first take broad view roughly locate the area of interest, and then zoom in to focus on the specific details needed for the task.
Motivated by these insights, emphasize that ideal scene graph should integrate both spatial and functional relationships, include fine-grained object parts as nodes, making the represen- tation compact, adaptive to dynamic changes, and highly aligned with task instructions, thus providing more concrete guidance for embodied perception and task planning.
To achieve this goal, we present MomaGraph, a novel scene representation specifically designed for embodied agents. It is the first to unify spatial and functional relationships while introducing part- level interactive nodes, providing a more fine-gran treeggraidd, compact, and task-relevant structured repre- sentation than existing approaches. To support this representation, we build MomaGraph -aaaaaarappsssss,s.... the first dataset that jointly models spatial and functional relationships with part-level annotations, encompassing multi-view observations, executed actions, and their interactive object parts, and task- aligned scene graph annotations.
Building this foundation, we propose MomaGraph-R1, 7B llsionn model (VLM) trained with the DAPO (Yu et al., 2025) reinforcement learning algorithm on SamaGraph-Scennes We design a graph-alignment reward function to guide the model toward constructing accurate, task-oriented scene graphs. MomaGraph-R1 not only predicts scene graphs but also serves zero-shot task planner within Graph-then-Plan framework: the model first generates a structured scene graph as an intermediate representation and then performs task plan- ning based on this graph, significantly improving reasoning effectiveness and interpretability.
Despite progress in task-graph planning (Agia et al., 2022), the community still lacks a unified benchmark to systematically evaluate whether and how task-oriented scene graphs improve plan- ning performance. To address this gap, we introduce ComaGraph-Beneee a comprehensive eval- uation suite that systematically assesses six key reasoning capabilities, spanning from high-level task planning to fine-graine scene understanding.
In summary, our work makes the following key contributions:
We propose MomaGraph, the first scene graph representation that jointly models spatial and functional relationships while incorporating part-level interactive nodes, providing a compact, dynamic, and task-aligned knowledge structure for embodied intelligence.
2

---

Published as a conference paper at ICLR 2026
We construct MomaGraph-Scente the first large-scale dataset of richly annotated. task-driven scene graphs in household environments, and build NomaGraph-Bencee a unified evaluation suite that systematically measures the impact of scene graph representations on task planning across six core reasoning capabilities.
We develop MomaGraph-R1, 7B vision-language model that leverages reinforcement learn- ing to optimize ppatialffunctonal reasoning, enabling zero-shot planning in a Graph-then-Plan paradigm.
MomaGraph-R1 surpasses all open-source baseline models, delivering substantial gains across public benchmarks and translating these improvements into strong generalization and effective- ness in real-world robotic experiments.
2 RELATED WORKS
Scene Graphs for 3D Indoor Scene Understanding. Scene graphs have emerged as a structured and hierarchical representation autonomous driving (Zhang et al., 2024b; Greve et al., 2024), robot manipulation (Jiang et al., 2024; Wang et al., 2025; Engelbracht et al., 2024; Jiang et al., 2025; Maggio et al., 2024), and spatial intelligence (Yin et al., 2025; Zemskova & Yudin; Liang et al., 2025a;b) community. They function not only as a means of scene representation but also criti- cal bridge between spatial understanding (Cao et al., 2024; Yang et al., 2024; Gu et al., 2024b)aam action planning. We focus on the household scenes. However, existing works often focus a sin- gle type of scene graphs. For example, ConceptGraphs (Gu et al., 2024a) primarily model spatial layouts, representing object instances and their geometrie relations in an open-vocabulary manner. While spatial graphs (Honerkamp et al., 2024b; Yan et al., 2025) provide useful geometric and se- mantic grounding, they overlook how objects can functionally interact with one another. Conversely, functional graphs (Li et al., 2021; Dong et al., 2021; Zhang et al., 2025) highlight object affordances and control relations but do not capture the overall spatial structure. Relying solely on either spatial or functional graphs leads to incomplete and less actionable representations. This motivates us to build MomaGraph, which unifies spatial and functional relationships, incorporates part-level nodes, and explicitly models state changes, providing more comprehensive foundation for task planning.
Zero-shot Embodied Task Planning with VLMs. VLMs (OpenAI, 2023; Team et al., 2025; Ahn et al., 2022) have gained significant attention in robotic task planning (Niu et al., 2024; Yue et al., 2024: Lu et al., 2023; Liang et al., 2024; Guo et al., 2024) due to their powerful capabilities in processing multimodal inputs, such as images and language instructions. However, when directly used as task planners, VLMs (Huang et al., 2023; 2024; Ahn et al., 2022; Zheng et al., 2025a; Yang et al., 2025) often suffer from sensitivity to visual noise and shallow semantic grounding; more fundamentally, their lack of structured object-relationship representations necessitates extracting or constructing more effective representations from the same visual inputs to support accurate and liable high-level planning. Prior approaches such as SayPlan (Ahn et al., 2022) assume access to a reliable 3D scene graph, which is often unrealistic in practice. To overcome this gap, we propose the Graph-then-Plan strategy, which first generates task-specific scene graphs as an intermediate structured representation before high-level planning. By explicitly modeling objects and their re- lations, this approach significantly improves the accuracy and robustness of task planning. Unlike prior graph-then-ple
methods (Dai et al., 2024; Ekpo et al., 2024) that either assume reliable scene graphs or treat graph construction and planning as separate modules, our approach enables a single VLM to jointly generate structured, task-oriented scene graphs and perform high-level planning.
3 PRELIMINARY FINDINGS AND MOTIVATION EXPERIMENTS
To ground our analysis, before the full evaluations we perform two motivating experiments on the MomaGraph- These comparisons are designed to validate our motivation and design prin- ciples, and to reveal why our proposed model is essential for embodied task planning. In this section, aim to answer the following questions.
3.1 ARE VLMS RELIABLE FOR DIRECT PLANNING WITHOUT SCENE GRAPHS?
To examine whether direct planning from visual inputs is reliable even for strong closed-souree VLMs, design controlled evaluations real-worle household tasks such "Open the window"
3

---

Published as a conference paper at ICLR 2026
Input Images
Input Images
Open
Obtain clean
window
boiled water)
1. Fill bettle with water (tap or filtecdd
handle", handlle.. 3 handlle fully": PPush opens
We Graph
ophimi'.
" Chaose between kettle stowetop: 'Fill with frosh tap water", 'Hest until milling boll", "Harall vafely when serving"
handles
2. Flug
tum on kettle".
Wio
upward"
4. 'Pnur balled watter
3. "Wart for autommatic shut-off",
X
Wzong object and
GPT-5
Momaaraph-Ri
GPT-5
d Momaaraph--R
Graph
LoCoee the window handle on the Fight zide.
L:Taaa the hundle on the window frame);
water plicher fanest",
1. "Hill fitter jug under itucet"
degrees,
2. Grip the handle timlly,
"'ootatee handle cousterdecssweee W]
2. "Graup the hardis firmly",
B.Rottte the handll unlock positoni;
W/
dlttered water kettle", TTum an waler'
3 "Insert kettle plug into outte",
2. Your filtered water into kettlle
4 "Press kuttle poseer button"
4-Tull window inward to open'
4. PPush slide the window openn"
Missing plug-in step
before switching on
GT Scene Graph
GT Action Sequence
aatioon_type"s "rotate", "function_type" "ooen_cclee" "ppn__lose_contro""" controll,
jug with faucet water filtered Plug kettle into outlet -witth kettle ON
"objects" ('handle", "window"], functional relationship": "open close")
Direct planning often fails even for strong closed-source models like GPT-5, producing wrong actions or missing key steps, while our Graph-then-Plan approach with structured scene graphs enables accurate and complete task sequences aligned with ground truth.
Figure 2:
and "Obtain clean boiled water". In these scenarios, models must reason over functional relation- ships, spatial constraints, and multi-step dependencies (c.g., plug-in before activation, filtration be- fore boiling). As shown in Fig. 2, despite their scale, closed-source VLMs like GPT-5 produce incor- rect or incomplete plans, missing prerequisite steps, or misidentifying interaction types. In contrast, our Graph-then-Plan approach, which first generates a task-specific scene graph and then performs planning, consistently produces correct and complete action sequences aligned with ground-truie logic. This demonstrates that incorporating structured scene representations significantly enhances planning accuracy and robustness beyond what direct planning can achieve.
Preliminary Findings
In contrast to directly relying on vision-language models for task planning from raw scene images, our Graph-then-Plat strateyy-whice incorporates task-oriented scene graph generation as intermediate structured representation prior to high-level planning, sub- stantially improves both the accuracy and robustmess of task planning.
3.2 ARE SINGLE-RELATIONSAE RELATIONSHIP GRAPHS ADEQUATE FOR EMBODIED AGENTS?
To ensure a fair comparison, we retrain our model using the same graph structure as in MomaGraph, but constrain the edge types to encode only single kind of relation either spatial or functional. This setup allows us to isolate the effect of relation types while keeping the graph topology consis- tent, thereby directly examining whether single-relation representations are sufficient for task plan- ning. To ensure this finding generalizes beyond one specific architecture, we evaluate this compari- son across different base models using the same dataset and experimental configurations. As demon- strated in Table 1, both MomaGraph-RI (trainned from Qwen-2.5-VL-7B and LLaVA-Onevision consistently show superior performance with unified spatial-fung funecioonn scene graphs compared to relato-relatonnship variants, supporting our hypothesis that integrated representations are essential for effective embodied task planning. Detailed training methodology is described in the Sec. 4.2.
Table 1: Comparison between MomaGraph-Riann -land LLa VA variants across task tiers.
Models
T1
T2
T3
T4
Overall
Models
T1
T2
T3
T4
Overall
2g MomaGraph-RR RI (Spatial-unly)
69.1
67.0
58.4
45.4
59.9
LLaVA LLaVA-Ouevisien (Spatia-onll))
63.4
56.7
59.7
36,3
$1.0
4 MomaGnuph-RI (Punctional-olly
71.4
65.8
59.0
649
LLaVA-Onvviionn (Functional-only)
65.1
61.7
55.8
45.4
57.0
MomaGraph-RI (Unified)
Ouevision (Unified)
56.5
4

---

Published as a conference paper at ICLR 2026
Preliminary Findings 2
Graph representations that rely solely on spatial relationships or solely on functional re- lationships are insufficient. For embodied agents, a unified representation that jointly models both spatial and functional relationships provides more complete and effective foundation for perception and action.
METHOD
4.1
MOMAGRAPH DEFINITION
Given single indoor room, the agent receives input set of mulli-view images {2: " and a nat- ural language instruction The objective is to construct an instruction-condiiinn scene graph 9T = (NT, ET, , Here, denotes the set of nodes representing objects relevant task T. ET encodes the spatial relationships among these nodes, and ET captures their functional relationships. This task-oriented scene graph provides minimal yet sufficient structured represen- tation that grounds the instruction T in the observed scene and facilitates downstream embodied task planning. Both E. and E7 are modeled directed edges, pointing from the triggering object the affected object.
task-oriented
4.2 VLMS LEARN SCENE GRAPH REPRESENTATIONS WITH REINFORCEMENT LEARNING
Existing open-source VLMs have demonstrated limited capability in generating accurate task- oriented scene graphs GT from multi-view observations {I, and natural language instructions T. VLMs do not form structured spatial-functione representations or reason effectively about task- relevant object relationships needed for embodied tasks. To go further, we want know: Can reinforcement learning teach VLMs to build more precise and task-relevant scene graph repre- sentations with MomaGraph?
Reinforcement learning offers more principled approach by encouraging the model to explore, reason, and iteratively refine its representations through outcome-driven feedback. Rather than replicating memorized patterns, RL enables models to discover effective strategies for construct- ing task-relevant scene graphs through structured thinking and reasoning. We apply the DAPO (Yu et al., 2025). The key innovation lies in our carefully designed graph-based reward function RRggp g where gored and 9 denote the predicted and ground truth task-oriented scene graphs, respectively, which evaluates how well predicted graphs embody these principles through three key components.
Action type prediction. Given the task instruction T, we ensure correct prediction of the required action type through Raction [[dpeed dgd. where apred and dg denote the predicted and ground truth action types, respectively.
Oppatal-ffunctions integration on edges. We jointly evaluate both spatial relationships Σ and functional relationships E/ within each edge, where pred and ET represent the predicted and ground truth edge sets:
Redges
max Sedge (ei, eg)
E3
EEETT
CFEET
where (edgo e3) measures semantic similarity between edges e; and e, based on their spatial and functional relationship labels.
Node completeness. We compute intersection-over-uiio similarity for task-relevant objects in Nr. where A/rred and A/ denote the predicted and ground truth sets of task-relevant nodes: Rioutes = IAgIl
PAPE
The final reward function integrates these task-oriented design principles with format validation and length control, where Riorma ensures valid JSON structure and Reength penalizes overly verbose outputs:
R(G.Tod , Wa (Raction + Redges + Rawdees) + Wy Riierna wy Roennth
5

---

Published as a conference paper at ICLR 2026
where War wy, and wy hyperparameters controlling the relative importance of each component.
This reward design directly implements our core insight: scene graphs must simultancously capture spatial layout (E²) and functional relationships (E7) while remaining tightly coupled to task quirements (T). With RL training on MomaGraph-Sceness we develop MomaGraph-R1, a 7B vision-language model built on Qwen2.5-VL-77B-
(Qwen, 2025), which learns to generate compact, task-relevant representations that provide concrete guidance for embodied planning.
We demonstrate that RL significantly enhances both the effectiveness and generalizability of open- source VLMs for scene graph generation in the following section. This aligns with broader findings that combining structured scene representations with reasoning consistently improves VLM scene understanding. Critically, MomaGraph-R1 achieves robust performance across diverse environ- ments and task configurations, enabling practical deployment in unseen embodied scenarios.
4.3 AWARE-AWARE DYNAMIC SCENE GRAPH UPDATE
In realistic environments, multiple objects of the correspondences are often initially uncertain. Take Figure 3 as an example, a kitchen stove may have several knobs, but only one controls the burner required for the current cooking task. Simply relying on visual appearance is insufficient to determine the correct functional relationship. In this work, do not focus on the agent's 's interaction policy; instead, our emphasis lies on how capture and incorporate observed state changes in the environment into the scene graph to resolve such ambiguities.
category may coexist, and their task-related
Formally, at time step i, the task-oriented scene graph represented as:
State-Aware Dynamic Upadate
G (Ni, E (T.(1) ET,(0),TT110
(3)
Initaal Gruph
Updatd
where NZ denotes the set of tele-atlevant candidate objects, ET(()) encodes their spatial layout, and ET.(t) captures hypothesized func- tional relationships, which may initially include one-to-many mappings.
Stoves
Miss Diaetiarly
Kaobs
Final
Updlaed
After the agent executes an action o, and ob- serves the environment state St+1, the scene graph refined as:
Figure MomaGraph captures state changes in the environment and dynamically updates the task-specific scene graph accordingly, enabling the graph to evolve as interactions occur and re- flecting updated spatial-functional relationships.
- (0=")=((h"" at, st,
(4)
where the update function U((-) removes incon- sistent hypotheses and strengthens confirmed correspondences based on the observed state transi- tion. As illustrated in Fig. 3, if rotating a specific knob ignites the burner while others have no effect, the functional edge [control] between that knob and the burner established, while edges from other knobs are pruned. This process enables the scene graph to evolve from ambigu- ous, many hypotheses into a compact, state-aware dynamic representation with unique and reliable object-to-objer correspondences.
DATASET AND BENCHMARK
5.1 MOMAGRAPH-SCENTE DATASET
Existing scene graph datasets for 3D indoor environments are often constrained to a single rela- tionship: some focus exclusively on spatial layouts of objects (Armeni et al., 2019; Koch et al., 2024b), while others emphasize functional interactions (Dong et al., 2021; Zhang et al., 2025). However, these scene graph representations that are restricted to a single relationship type are in- sufficient for embodied agents, task execution in household environments requires reasoning about both where objects are and how they can be used. To address these limitations, we in- troduce MomaGraph-Scenns. the first dataset designed provide more comprehensive and task-relevant scene representation. MomaGraph-Scent and functional relationships, covering 9 spatial relationship types and functional relationship
jointly encodes spatial relationships

---

Published as a conference paper at ICLR 2026
types, explicitly representing interactive elements such handles and buttons. Our dataset con- sists of approximately 1,050 task-oriented subgraphs and 6278 multi-view RGB images, collected from combination of manually collected real-world data, re-annotated existing datasets (Zhang et al., 2025; Delitzas et al., 2024), and simulated environments built with A12-THOR (Kolve et al,, 2017). These samples span more than 350 diverse household scenes and encompass 93 distinct task instructions. Compared with prior datasets, our annotations are significantly more detailed, and capturing interaction semanties at both the object and part levels. This broad coverage en- sures rich variability in scene layouts, object configurations, and interaction types, supporting robust learning and evaluation of embodied reasoning.
5.1.1 DATASET ANNOTATION
Multi- View Observation. The multi-view images provided for each graph are not constrained to always contain every relevant object within each single view. We also do not impose restrictions on the number of viewpoints or their exact configurations. This flexible setup better reflects realistic perception conditions, where embodied agents must reason across partial and diverse observations to build consistent scene graph representations.
Task Instruction. It is worth noting that the task instructions in our dataset do not explicitly mention all the objects required to accomplish the task. Instead, they are expressed in simple and natural forms (e.g., "Fill the bathtub"), where the relevant objects such as the bathtub, faucet, and button must be inferred by the model. This design encourages the model learn how to ground natural instructions into the appropriate set of objects and relationships, rather than relying on object names being explicitly stated.
Nodes. NT primarily consists of the objects necessary to accomplish the instruction. When the task execution requires interacting with specific parts, the graph may additionally include part-level interactive elements (e.g., handles, knobs, buttons). For example, for the instruction "Open the fridge," " N₂ includes both the fridge and its handle; for the instruction "Turn on the light." Nr consists of the switch and the ceiling light.
Edges. Edges in the task-oriented scene graph capture both functional and spatial relationships between nodes.
Functional Relationships. We define functional relationship as the ability of one object to change the state of another object. In indoor environments, common tasks can be broadly cate- gorized as Parameter Adjustment, Device Control, Open/Close the Cabinet or Door, Water Flow Control, Power Supply, and Assembly. Accordingly, we identify six major types:
[OPEN OR CLOSE), [ADJUST], [CONTROL], [ACTIVATE], [POWER BY), [PAIR WITH). Notably. [PAIR WITH] does not alter the internal state of objects but instead modifies their spatial configuration, which essential for assembly tasks (Qi et al., 2025). Since such tasks are criti- cal for robotic interaction and task planning, we explicitly include [PAIR WITH] as a functional relationship. Through this definition, our dataset extends beyond physical and electronic interac- tions to encompass fine- mee-grained reasoning about assembly and pairing, enhancing its utility for downstream action execution and planning.
Spatial Relationships. Capture geometric dependencies between objects and parts. The dataset primarily annotates:
Directional: [LEFT OF], [RIGHT OF], IIN FRONT OF]. [BEHIND], [HIGHER THANJ,
[LOWER THAN].
Distance-based: [CLOSEJ, [FAR], [TOUCHING).
These annotations provide the rich context necessary for reasoning about layout, reachability, and interaction feasibility.
5.2 MOMAGRAPII BENCHMARK AND EVALUATION
We introduce MomaGraph-Benche the first benchmark that jointly evaluates fine-grained scene understanding and task planning abilities across diverse levels of difficulty. Our design principle for MomaGraph-Bench is to evaluate whether advances in scene understanding provide tangible improvements in downstream task planning and reasoning. Our evaluation framework examines six essential reasoning capabilities in four tiers of difficulty levels: (1) Action Sequence Reasoning, (2)
7

---

Published as a conference paper at ICLR 2026
É. Action Reasoning
Spatial Reasoning
Affordance Reasoning
What the correct action complete the
What spatial relationship between button/ knob and bathroom sink?
What primary functional relationship radiatos?
task Open the window.?
knoh Aj Power C) Open
(1) Pull the witdoow
Rutaste
Pundo the winddow
C) Far from
A) Highar than
B) right of
DV Clooe
Adfjust
handle
Dy Lower tham
Correct Ansteen
Correct Ansuver B
Conret
Pbject Selection
Se Effect Prediction
ba Visual Correspondance
To complete Power on the dayer', which objecc should interact with firs??
ieserling plug electric outllet, what well
AfTowel
happen?
Eleettne
A) The humictifer will powerred on
c) Power button
B) The drawer will opann
Which point correspanding the point?
Dj Cord
C) The cabinet will open
D) The book will be moved
Point,
Connect Ansover
Puint Di Poist
Conrect Ausmvr:
Cornect Answve:
Dynamic Verification
ass I Long-horizon task decomposition
Between Scene and Scene 3 only one switch toggled. Which lamp does that switch control?
What is the correct sequence obtain clean
boiled wates?
luttle Switch kettle Fill
ceiling light weill
Fill filter jog with kottle
water Flog
outeet
lamps
Switch faucert CN Switch kettle CN Plug kettle into outflt
D) Netther lamp
Plug kettle into outict only
Connec Ausuee:: A
Cornect Ansaver: B
Figure 4: Examples of evaluation Multi-Choices VQA tasks in the MonaGraph--ench. We show- case example questions covering six reasoning capabilities. Beyond these core capabilities, further design tasks on Dynamic Verification and Long-horizon Task Decomposition to evaluate tem- poral reasoning and multi-steps planning.
Spatial Reasoning, (3) Object Affordance Reasoning, (4) Precondition & Effect Reasoning, (5) Goal Decompoxition, and (6) Visual Correspondence (with concrete examples shown in Fig. 4).
MonaGraph--Bnch formulated as multi-choice VQA task which comprises 294 diverse in- door scenes with 1,446 multi-view images, featuring 352 task-oriented scene graphs spanning 1 1,315 instances that range from simple step object manipulation(Tier to complex multi-step replanning (Tier 4) scenarios (detailed breakdown in Appendix A.4). MomaGraph-Bent comprehensive assessment for embodied agents' capacity generalize across tasks and scenarios. To ensure that the evaluation truly reflects generalization rather than memorization, all scenarios are drawn from entirely unseen environments.
offers the most
6 EXPERIMENTS
6.1 BENCHMARK EVALUATION FOR EMBODIED TASK PLANNING
We compare the performance of our MomaGraph-R1 with other models across all task tiers in MomaGraph-Bench to rigorously assess embodied planning, including state-of-the-art closed source models (Claude-4.5- -Snnnet GPT-5, Gemini-2.5-Pro and leading open source models (In- structBLIP, I.L.VA-V..5 DeepSeek- V1.2, V1.2.55 LLLVA-OOeVissonn Qwen2.5). We further examine whether Graph-then-Plun brings performance gains by evaluating each model under two controlled settings: (i) Direct Plan (w/o Graph): the model is directly evaluated on task planning in MomaGraph--Bnch using multi-view observations and instructions; (ii) Graph-then- thppp----e--lan (w/ Graph): the model first generates a task-oriented scene graph Gy, capturing nodes, spatial and func- tional edges, and action types, and then performs task planning conditioned on the graph.
6. RESULT ANALYSIS.
The results in Table yield several key insights:
(1) Effectiveness of Graph-then-Plan. Across all models, the w/ Graph setting consistently outper- forms the w/o Graph baseline, demonstrating that explicitly structuring task-oriented scene graphs provides a tangible benefit for downstream planning. This validates our central hypothesis that dis- entangling scene representation from action generation improves reasoning reliability.
8

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on the MomaGraph-Bench. We report accuracy (%) across four tiers (T1-T4) and the overall score, with and without traph-based reasoning.
MemnaGrap Benchmark
Type Models
Overalf
w/o Graph
Graph
wio Graph
w/ Graph.
wio Graph
w/ Gouph
wio Grophi
w/ Graph
win Graph
wi Gragli
Clande-4,5 Sonnet
77.3
83.7
67.0
70.3
69.7
72.3
66.2
69.5
69.8
73.9
Cheen Source
GPT-5
77.3
79.8
63.4
68.2
70.8
75.0
71.6
Gemini- 2.5- Pro
76.6
79.0
65.8
69.5
67.5
72.7
60.8
65.2
67.6
71.6
IestracBLLPPP
43.1
44.1
42.6
36.3
39.5
LLaVA, V1.5-B
7B
51.0
53.4
46.3
48.7
40.2
36,3
38.9
40.9
44.1
44.8
Open
DrepSeek
54.2
IntemVV1.. 2.888
6B
53.6
$1.0
51.2
53.0
55.8
59.7
33.3
40.9
48.4
$1.1
LLVVA-Onevisin--8
60.0
63.8
524
QwenZ 5-VL-BB Instruct
78
02.1
60.3
38.5
58.5
51.9
57.1
36.5
39.0
57.1
60.2
MomaGraph-RR1(Orrr
Table 3: Performance comparison the BLINK and MomaGraph-Benchh By enforcing multi- view consistency, our method significantly improves correspondence reasoning across all open- source models.
Model
LlaVA-Cnenision
Qwen M-V.2...-V--7---lma Insttrrre
DeepSeck-V1.22
4g MonaGraph-R1
BLINK
Momssmmph-Bent
BLINK
Momssingh-Bmnch
BLINK
BLINK
MemaGraph-Bench
BLINK
MomoGrapt-Bench
66.1
81.2
59.7
70.7
58.7
72.7
57.4
68.4
63.5
77.5
(2) Competitiveness of MomaGraph-RRL Our MomaGraph-R1 achieves performance on par with closed-source giants like Claude-4.5-Sonnet and GPT-5, while clearly surpassing all leading open- source VLMs. Notably, MomaGraph-Ri delivers a +11.4% relative improvement over its base model (Qwen2.5-VL-7B) under w/ Graph, highlighting the effectiveness of reinforcement learning with graph-based rewards.
(3) Scalability with Task Complexity. As task complexity increases from Tier to Tier 4, the per- formance of most open-source baselines drops sharply, reflecting their limited ability to generalize to multi-step reasoning.
contrast, MomaGraph-R1 exhibits much smaller degradation, pre- serving strong performance in Tier 3 and Tier 4. This indicates superior scalability to long-horizon planning scenarios, a crucial capability for embodied agents.
(4) General Trend Across Communities. Closed-source models still maintain the highest absolute performance, benefiting from larger-scale pretraining and proprietary data. However, the consis- tent gap reduction achieved by MomaGraph-RI shows that reinforcement learning with graph- structured intermediate representations can substantially narrow the divide, offering practical path toward competitive open-source systems.
6.2 BENCIMARK EVALUATION FOR VISUAL CORRESPONDENCE
As the model learns scene representations from multi-vier observations, it exhibits an emergent ability of cross-view consistency, which can reason about the same point across different viewpoints. This capability is most evident in visual correspondence tasks. As shown in Table 3, we compare model performance on visual correspondence tasks from public benchmark BLINK Fu et al. (2024) and our MomaGraph-Bench. Scene graph representations enhance performance universally by reducing VLM hallucinations in visual perception. By prompting models to first generate structured scene graphs (w/ Graph) and then answer questions in single-turn interactions, we force them to ex- plicitly reason about spatial and functional relationships between objects before answering. We pri- marily evaluate perception on multi-vies -iew reasoning and visual correspondence tasks from BLINK, Our MomaGraph-R1 achieves
as well as multi-view correspondence MomaGraph-Bench late-offtthe-art performance among open-source VLMs, leading by 3.8% BLINK and 4.8% on our correspondence benchmark compared the best competing open-source models. These results confirm that MomaGraph-RI enables more nuanced and detailed perception of complex indoor scenes, effectively mitigating hallucinations and enabling more reliable scene perception.
6.3 REAL ROBOT DEMONSTRATIONS
Setup. To validate the effectiveness our model in real-world settings, we deploy on the RobotEra Q5, bimanual humanoid platform with mobile base. An Intel RealSense D455 camera mounted

---

Published as a conference paper at ICLR 2026
Turn off the Light
the Microwave
Turn on the TV
Open the Cabinet
Figure 5: Real Robot experiments on the RobotEra Q5 with D455, demonstrating four household tasks that require spatial, functional, and part-level interactive elements reasoning for task execution.
enhance RGB-D perception. Importantly, all evaluation scenes unseen, ensuring that perfor- mance reflects true generalization.
Tasks. We design four representative tasks (Figure 5). consisting of two local interactions (c.g., opening a cabinet, opening microwave) and two remote interactions (e.g., turning on the TV, turning off light).
Deployment. Prior to execution, the robot performs active perception by adjusting its head pose acquire multi-view observations. MomaGraph-RI processes these observations together with the task instruction generate task-specific subgraph, which explicitly encodes the relevant ob- jects and their spatial-functional relationships, see more deployment details B.3. Following the Graph-then- Plan paradigm, MomaGraph-R1 then functions tured action sequence. These specifications are subsequently instantiated as low-level trajectories through library of parameterized primitive skills. We note that the primitive skills task task-specific and derived from teleoperation data for each scenario; the primary contribution of this work lies in the high-level planning and scene graph generating enabled by MomaGraph-R1.
task planner, producing struc-
Summary. Our real-world evaluations show that MomaGraph-RI delivers robust scene under- standing and task planning even in unseen scenarios, while remaining directly compatible with stan- dard mobile humanoid systems. This combination underscores the strength of our model and its practicality for real-world deployment.
7 CONCLUSION
This work addresses to the fundamental limitations of existing graphs for embodied agents: reliance on a single type of relationship, inability to adapt to dynamic environments, and lack of task relevance. To overcome these issues, introduce MomaGraph, novel scene representation that unifies spatial and functional scene graphs with interactive elements. To learn this represen- tation, we construct large-scale dataset MomaGraph-Scenes and propose MomaGraph-RI. a 7B VLM trained with reinforcement learning, which predicts task-oriented scene graphs and serves as a zero-shot task planner under Graph-then-Plc framework. Furthermore, we design the MomaGraph-Bench. comprehensive benchmark that rigorously evaluates both fine-grained reasoning and high-level planning. Through extensive experiments, we demonstrate that our ap- proach achieves state-of-the-art performance among open source models, remains competitive with closed source systems, and transfers effectively to public benchmarks and real robot experiments. We hope that MomaGraph will serve foundation for advancing scene representations, fostering stronger connections between the spatial VLM and robotics communities, and ultimately enabling more intelligent and adaptive embodied agents.
10

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGEMENTS
We would like to express our heartfelt thanks to Chenyangguang Zhang, Prof. Florian Shkurti, and Prof. Tom Silver for their insightful suggestions and constructive feedback. We also thank Guowei Zhang, Yuman Gao, Bike Zhang, Gechen Qu, Lihan Zha, Yuanbang Zhang, and Yu Qi for their valuable assistance in the collection of benchmark data. We thank Robot Era for providing their Q5 Mobile Manipulator for our experiments.
Liang, Lee and Huang are supported by DARPA Transfer from Imprecise and Abstract Models to Autonomous Technologies (TIAMAT) 80321, DARPA HR001 AFr Force Office of Scientific Research under award number FA9550-23-I-005 National FAI, National Science Foundation NAIRR240045, National Science Foundation TRAILS Institute (2229885). Private support was provided by Peraton and Open Philanthropy.
DOD-
Science Foundation
The work by Ju, Wang, and Sreenath was supported by The Robotics and Al Institute.
REFERENCES
Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vi- neet, Mustafa Mukadam, Liam Paull, and Florian Shkurti. Taskography: Evaluating robot task planning large 3d scene graphs. In Conference on Robot Learning, pp. 46-58 PMLR, 2022.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.010 10691,166000001000 2022.
Iro Armeni, Zhi- Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5664-5673, 2019.
Yang Cao, Yuanliang Jv, and Dan Xu. dets-det: Empower 3d gaussian splatting with boundary guidance and box-focused sampling for 3d object detection. arXiv preprint arXiv:2410.01647, 2024.
Jason da Silva Castanheira, Nicholas Shea, and Stephen M Fleming. How attention simplifies mental representations for planning, arXiv preprint rxXi:25066095500 2025.
Zhirui Dai. Arash Asgharivaskasi, Thai Duong, Shusen Lin, Eliza-EEi Tzes, George Pappas, and Nikolay Atanasov. Optimal scene graph planning with large language model guidance. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 14062-14069. IEEE, 2024.
Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Fran- cis Engelmann. SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Ang Dong, Li Feng, Dengcheng Yang, Shuang Wu, Jinshuai Zhao, Jing Wang, and Rongling Wu. Fungraph: A statistical protocol reconstruct omnigenic multilayer interactome networks for complex traits. Star Protocols, 2(4): 2021.
Daniel Ekpo, Mara Levy, Saksham Suri, Chuong Huynh, and Abhinav Shrivastava. Verigraph: Scene graphs for execution verifiable robot planning. arXiv preprint arXiv:2411.1042 2024.
Tim Engelbracht, René Zurbrügg, Mare Pollefeys, Hermann Blum, and Zuria Bauer. Spot- light: Robotic scene understanding through interaction and affordance detection. arXiv preprint arXiv:2409.11877 2024.
Xingyu Fu, Yushi Hu, Bangzheng Li. Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma. and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference Computer Vision, pp. 148-166. Springer, 2024.

---

Published as a conference paper at ICLR 2026
Elias Greve, Martin Büchner, Niclas Vödisch, Wolfram Burgard, and Abhinav Valada. Collaborative dynamic 3d scene graphs for automated driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 11118-11124 IEEE, 2024.
Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishma Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa. et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Con- ference Robotics and Automation (ICRA), pp. 5021-5028. IEEE, 2024a.
Qiao Gu, Zhaoyang Lv, Dunean Frost, Simon Green, Julian Straub, and Chris Sweeney. Egolifter: Open-world 3d segmentation for egocentric perception. In European Conference Computer Vision, pp. 382-400, 2024b.
Yanjiang Guo, Yen-Jen Wang, Lihan Zha, and Jianyu Chen. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment. In 2024 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 12124-12131, IEEE, 2024.
Daniel Honerkamp, Martin Büchner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada. Language-groune grounded dynamic scene graphs for interactive object search with mobile manipula- tion. IEEE Robotics and Automation Letters, 2024a.
Daniel Honerkamp. Martin Büchner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada. Language-groundee dynamic scene graphs for interactive object search with mobile manipula- tion. IEEE Robotics and Automation Letters, 2024b.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05593. 2023.
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024.
Nathan Hughes, Yun Chang, Siyi Hu, Rajat Talak, Rumaia Abdulhai, Jared Strader, and Luca Car- lone. Foundations of spatial perception for robotics: Hicrarchical representations and real-time systems. The International Journal of Robotics Research, 43(10);1457-1503 2024.
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd. Omama, Ganesh Iyer, Soroush Saryazdi, Tao Chen, Alaa Maalouf, Shuang Li, Nikhil Varma Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, K. Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfusion: Open-set multimodal 3d mapping. Robotics: Science and Systems, 2023.
Guangqi Jiang, Yifei Sun, Tao Huang, Huanyu Li, Yongyuan Liang, and Huazhe Xu. Robots pre- train robots: Manipulation-cenee centrrc robotic representation from large-scale robot datasets. 2025.
Hanxiao Jiang, Binghao Huang, Ruihai Wu, Zhuoran Li, Shubham Garg, Hooshang Nayyeri, Shen- long Wang, and Yunzhu Li. Roboexp: Action-condittione scene graph via interactive exploration for robotic manipulation. arXiv preprint arXiv:2402. 15487, 2024.
Yuanchen Ju, Kaizhe Hu, Guowei Zhang, Gu Zhang, Mingrun Jiang, and Huazhe Xu. Robo-abe: Affordance generalization beyond categories via semantic correspondence for robot manipulation. In European Conference on Computer Vision (ECCV), 2024.
Sebastian Koch, Pedro Hermosilla, Narunas Vaskevicius, Mirco Colosi, and Timo Ropinski. Lang3dsg: taaguage-basel contrastive pre-training for 3d scene graph prediction. In 2024 Inter- national Conference on 3D Vision (3DV), pp. 1037-1047. IEEE, 2024a.
Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open- set relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pp. 14183-14193, 2024b.
12

---

Published as a conference paper at ICLR 2026
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsami, Daniel Gordon. Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 1rX1v:1712.05474. 2017.
Vasiliki Kondyli, Mehul Bhatt, and Jakob Suchan. Towards human-centred cognitive model of visuospatial complexity in everyday driving. arXiv preprint htt::20060000550 2020.
Qi Li. Kaichun Mo, Yanchao Yang, Hang Zhao, and Leonidas Guibas. Ifr-explore: Learning inter- object functional relationships in 3d indoor scenes. arXiv preprint arXiv:2112.05298, 2021.
Yongyuan Liang, Tingqiang Xu, Kaizhe Hu, Guangqi Jiang, Furong Huang, and Huazhe Xu. Make- an-agent: A generalizable policy network generator with eechavir-prompter diffusion. 2024.
Yongyuan Liang. Wei Chow, Feng Li, Ziqiao Ma, Xiyao Wang, Jiageng Mao, Jiuhai Chen, Jiatao Gu, Yue Wang, and Furong Huang, Rover: Benchmarking reciprocal cross-modal reasoning for omnimodal generation. arXiv preprint arXiv:2511.01163, 2025a.
Yongyuan Liang, Xiyao Wang, Yuanchen Ju, Jianwei Yang. and Furong Huang. Lemon: A unified and scalable 3d multimodal model for universal spatial understanding. arXiv preprint 1rXiv: 2025b.
Joel Loo, Zhanxin Wu, and David Hsu. Open scene graphs for open-world object- goal navigation. arXiv preprint arXiv:2508.0405 2025.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. 2023.
Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, and Luca Carlone. Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 2024.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406. 11855, 2024.
OpenAl. Gpt-4 technical report. Technical report, OpenAl, 2023. URL https://api. semanticscholle wtwpolarr.rrrreoooooooee wtt.:/ouaa..ggooooouuioo.....--. 1D:
Yu Qi, Yuanchen Ju, Tianming Wei, Chi Chu, Lawson LS Wong, and Huazhe Xu. Two by two: Learning multi-task pairwise objects assembly for generalizable robot manipulation. CVPR 2025, 2025.
Ri-Zhao Qiu, Yafei Hu, Yuchen Song, Ge Yang, Yang Fu, Jianglong Yc, Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, et al. Learning generalizable feature fields for mobile ma- nipulation. arXiv preprint arXiv:2403.07563 2024.
Qwen. Qwen2.5-vl, January 2025. URL https: https://qweninn www.:gggith....oooooo 5-v1/.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task plan- ning. arXiv preprint arXiv:2307.06/33. 2023.
Antoni Rosinol, Andrew Violette, Mareus Abate, Nathan Hughes, Yun Chang. Jingnan Shi, Arjun Gupta, and Luca Carlone. Kimera: From slam to spatial perception with 3d dynamic scene graphs. The International Journal of Robotics Research, 40( 40(12-14):1110-1444 2021.
Ayca Takmaz, Alexandros Delitzas, Robert W Sumner, Francis Engelmann. Johanna Wald, and Federico Tombari. Search3d: Hierarchical vocn-vocy 3d segmentation. IEEE Robotics and Automation Letters, 2025.
13

---

Published as a conference paper at ICLR 2026
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Cean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong. Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20022 2025.
Sebo Uithol, Katherine L Bryant, Ivan Toni, and Rogier B Mars. The anticipatory and task-driven nature of visual perception. Cerebral Cortex, (112)::354-5366) 2021.
Yixuan Wang, Leonor Fermoselle, Tarik Kelestemur, Jiuguang Wang, and Yunzhu Li. Curious- bot: Interactive mobile exploration via actionable 3d relational object graph. arXiv preprint arXiv:2501.13338, 2025.
Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. Hierarchical open-vocabulary 3d scene graphs for language -ooungeed robot navigation. In First Workshop on Vision-Languaye Models for Navigation and Manipulation at ICRA 2024, 2024.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 1788):1077-1122, 2023.
Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scene- graphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7515-7525, 2021.
Zhijie Yan, Shufei Li, Zuoxu Wang. Lixiu Wu, Han Wang, Jun Zhu, Lijiang Chen, and Jihong Liu. Dynamic open-vocabulary 3d scene graphs for long-term language-guided mobile manipulation. IEEE Robotics and Automation Letters, 2025.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu,
Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: foundation model for multimodal ai agents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14203-14214, 2025.
Timing Yang, Yuanliang Ju, and Li Yi. Imov3d: Learning open vocabulary point clouds 3d object detection from only 2d images, Advances in Neural Information Processing Systems, 37:141261- 141291, 2024.
Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. Spatial mental modeling from limited views. arXiv preprint arXiv:2506.21458 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Tatiana Zemskova and Dmitry Yudin. 3dgraphllm: Combining semantic graphs and large language models for 3d referred object grounding.
Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, and Francis Engelmann. Open-vocabulary functional 3d scene graphs for real-world indoor spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 19401-19413, 2025.
Shoulong Zhang, Aimin Hao, Hong Qin, et al. Knowledge-inspired 3d scene graph prediction in point cloud. Advances Neural Information Processing Systems, 14:18600-18633 2021.
Yuanhang Zhang, Tianhai Liang, Zhenyang Chen, Yanjie Ze, and Huazhe Xu. Catch it! learning to catch in flight with mobile dexterous hands. arXiv preprint 4rXiv:2409.1031 10399, 2024a.
14

---

Published as a conference paper at ICLR 2026
Yunpeng Zhang, Deheng Qian, Ding Li, Yifeng Pan, Yong Chen, Zhenbao Liang, Zhiyao Zhang, Shurui Zhang, Hongxu Li, Maolei Fu, et al. Graphad: Interaction scene graph for end-to-end autonomous driving. arXiv preprint arXi::243. 2024b.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. 2025a.
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng. Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. https: h/tit//uiittt com/ Siiyouga/Easy/ee 2025b.
Junzhe Zhu, Yuanchen Ju, Junyi Zhang, Muhan Wang, Zhecheng Yuan, Kaizhe Hu, and Huazhe Xu. Densematcher: Learning 3d semantic correspondence for category-level manipulation from a single demo. International Conference on Learning Representations (ICLR) Spotlight, 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MOMAGRAPH-SCENT DATASET
A.1.1 REAL.-WORLA. DATASET SOURCE AND COLLECTION.
Our dataset built through a synergistic integration of newly curated data and existing public re- sources. We manually collected substantial portion of the data in real-world household envi- ronments, capturing diverse interaction scenarios under natural conditions. To further enrich the dataset, we incorporated samples from two public benchmarks, OpenFunGraph (Zhang et al., 2025) and SceneFun3D (Delitzas al., 2024), both of which contain vidcos depicting human-objer in- teractions in indoor contexts. From these videos, we carefully curated representative keyframes to derive nulti-view RGB observations, ensuring comprehensive coverage of interaction dynamics and spatial variability.
A.1.2 SIMULATION DATA COLLECTION
To complement the real-world data, we additionally generated samples within the A12-THOR sim- ulation environment Kolve et (2017). We strategically positioned the embodied agent diverse, reachable viewpoints and captured multi-view observations from varying perspectives, as illustrated in Fig. Throughout this process, we applied manual post- post-filtering to exclude non-interactable elements, thereby ensuring that the curated dataset remains focused on actionable objects and em- phasizes functional relevance critical for downstream embodied reasoning tasks.
Top-Iosen Viaws
Beachable Location
Mellview Observation
Floor 15
Hoor 22N
Floor 301
Figure 6: Simulated indoor environments in our benchmark. Each row shows three scenes (Floor 15, Floor 224 and Floor 30/) with top-down view of the layout, reachable locations for the robot, and multiview observations from different viewpoints.
A.1.3 DATASET ANNOTATION AND FORMAT.
Annotation and Format. Each task-oriented subgraph in MomaGraph-Scennt is stored in structured ISON format and linked to its corresponding scene. Annotations include a subgraph iden- tifier, the associated scene identifier, the action type, the functional category, the natural language task instruction, a set of nodes, and a set of edges. Nodes correspond to the objects part-level interactive elements required to accomplish the task, while edges capture both functional relation- ships (e.g., control, open or close) and spatial relationships (e.g., close, in fnfotoo_of. lower.than).
This example corresponds the instruction "Turn the television", where the relevant nodes are the remote control and the TV, connected by a control functional edge and spatial relations lower.than, in. front.ooof and close.
In addition, each subgraph is grounded multi-view observations. For every scene, we provide synchronized RGB images captured from multiple viewpoints. This multi-view grounding allows the annotated subgraphs to be consistently aligned with visual evidence, supporting both instruction- conditioned graph prediction from perception and multi-view reasoning tasks.
16

---

Published as a conference paper at ICLR 2026
"aubgraph_id"; "abpll1992----------------------
"scene_id"; *466828*,
postion_tppe": "press",
"Eunetien_yyee": "device_control",
stask_instruction, "rurn on the television.",
"nodes":
8
("label"; "cemote control", "ldt;
("label"; "tv", "id": 0106611--00---------0000000-0--0
10
"edgee": [
12
13
"relation_id" te3biieenof-2244455555acaa
14
"functional Lellattenaatt "control",
15
"ebject"%; ["label*; "zemote control", *id"; "I15474de 7b35 4a5e acda cc02193360b3" *object2*; [*label"; "tv", "id;;
17
"apatial_relation"" ["lower_than", "in_Cront_of", "close"),
"is_touching"; false
Figure Example JSON annotation for the task "Turn on television."
A.1.4 MULEI-ASPECT STATISTICS OF THE TRAINING DATASET
Our dataset consists of approximately 1,050 subgraphs and 6278 multi-view RGB images, collected across more than 350 diverse houschold scenes and encompassing 93 distinct task instructions. This broad coverage ensures rich variability in scene layouts, object configurations, and interaction types.
To provide comprehensive overview of our training data, we present aspect statisties cov- ering scene context, action diversity, functional relationships, and object distributions. As shown Fig. 8, the dataset spans four common household room types and captures the correspondence between action types and functional categories, reflecting the diversity and richness of real-world manipulation scenarios. Fig. 9 illustrates the distribution of action types across different room con- texts, while Fig. 10 summarizes the prevalence of various functional relationships and Fig. 11 sum- marizes the frequency of object occurrences. Together, these statistics highlight the diversity and task relevance of our dataset, ensuring broad coverage of spatial- functional interactions essential for embodied planning and reasoning.
Type
Anden Tepo Pursionn Type
20.9
Liiing
(a) Room-type distribution.
(b) Action-function correspondence.
Figure 8: Dataset statistics: (a) Distribution across four room types; (b)Heatmap showing the corre- spondence between action types and functional types.
17

---

Published as a conference paper at ICLR 2026
Action Distrbusioon
Action
Dutvoo
XCTN
300
PRM
Adden
Figure 9: Task distribution across four room types: kitchen, living room, bedroom, and bathroom.
Chart)
Relationhip Chart)
300
Figure 10: Distribution of functional relationships across tasks dataset.
A.2 TRAINING DETAILS
We train our model using 8x 80GB A100 GPUs for approximately hours based on the EasyRI (Zheng et al., 2025b) training framework. The complete training configuration for DAPO algorithm is presented in Table 4.
A.3 TRAINING CURVE
Figure 12 and 13 shows the training curves during DAPO optimization. The training and validation curves closely align across all metries, indicating good generalization without significant overfitting. The overall reward converges to ~0.93, while accuracy reward stabilizes at ~0.9. The format reward quickly reaches 1.0 within the first 25 steps, showing the model rapidly learns to produce valid ISON-structured outputs.
A.4 MOMAGRAPH BENCHMARK
A.4.1 BENCHMARK DESIGN
To rigorously evaluate opatial-functions reasoning and task planning capabilities, we design com- prehensive multi-choice VQA benchmark based on the scenes and tasks our dataset. Rather than manually crafting all questions, we leverage large vision-language models (VLMs) generate them in a scalable and diverse manner. Specifically, we provide the model with structured prompts de- scribing the scene images, state-aware question-ansee answer pairs that probe different reasoning skills, such as spatial relation understanding, affordance inference, precondition reasoning, and goal decomposition. To ensure the reliability and
graph, and task instructions, and instruct it to produce
18

---

Published as a conference paper at ICLR 2026
Top 20 Objects
hands
Object
Word
Cloud
button
/
knob
ki
sink
f
faucet
handle
/hadd
knoo
toaster
nandle
Kitshen
Sabinet
toaster Lever
edl
ceiling light
mirowaee cun
renote
knob
toilet
stove
burner
dacher cabinet
crowaaee
oven
busoon 1000
gathroom
button
eloctric
outler
sink
doo
cabinet
Light
Lightsswiter
fradge
bathtub
1gmt
stove
knob
WeseW
renooe comDo
BC0
Figure 11: Statistics of object occurrences, highlighting the most frequent objects in tasks.
Train Overall Reward
Train Accuracy Rlaward
Train Pormet Reward
1.0
00
0.0
30
25
100
125
175
125
350 175
Figure 12: Training reward curves during MomaGraph-R1 training.
correctness of the benchmark, all generated questions and answers undergo several rounds of manual verification, during which ambiguous low-quality samples are refined removed.
Moreover, since the benchmark is formulated as a multi-choice VQA task with clearly defined cor- rect answers, does require complex evaluation metries. Model performance can be directly measured by simple accuracy - i.c., the proportion of correctly answered questions - which pro- vides an intuitive and reliable indicator of spatial-fund fantiiolfu reasoning and planning capabilities. This simplicity enables straightforward comparison across models while ensuring that the evalua- tion remains rigorous and meaningful.
Data Source and Task Scope. We leverage long video sequences from SceneFun3D (Delitzas et al., 2024) that capture human-recorded layouts of entire indoor environments, from which key frames are extracted and manually annotated with task-specific graphs. To enhance diversity and coverage, we additionally collect data from real indoor scenes. Our benchmark spans four representative indoor room categories: bathroom, kitchen, living room, and bedroom. The task scope is organized into four levels of difficulty:
Single-step actions: e.g., turning on a light, pulling a drawer, opening a door. T2 Two complementary steps: e.g., filling a bathtub by first pressing the drain button and then turning on the faucet.
T3 Multi-step or preconditioned tasks: e.g., making coffee (pick up a cup - add water + start the coffee machine).
T4 Dynamic verification tasks: e.g., when the target object is missing, the system must per- form graph-based replanning and identify alternative interactive objects.
Action Sequence Reasoning: examines whether models understand the order and dependency of actions and can plan efficient sequences.
19

---

Published as a conference paper at ICLR 2026
Table 4: DAPO Training Configuration
Parameter
Value
Model Configuration
Base Model
Qwen2.5-VL. L-V--B--nnsstue
Mixed Precision
b
Training Setup
Total Epochs
25
Training Steps
175
Actor Global Batch Size
128
Critic Global Batch Size
256
Micro Batch Size (Actor)
1
Micro Batch Size (Critic)
4
Optimization
Learning Rate
le-6
Optimizer
AdamW
Weight Decay
0.01
Betal, Beta2
0.9, 0.999
Gradient Clipping
1.0
DAPO Algorithm
KI. Coefficient
0.01
KI. Penalty
low_var_k
Disable KL
True
Clip Ratio Low
0.2
Clip Ratio High
0.28
Clip Ratio Dual
3.0
Reward Function
Format Weight
0.2
Max Response Length
2048
Overlong Penalty Factor
0.5
Generation Config
Temperature
Top-p
1.0
Rollout Samples
5
Accurucy Roward
Pormat Reward
120
125
120
175
22 50
150
175
104
122
150 175
Figure 13: Validation reward curves during MomaGraph-R1 training.
Spatial Reasoning: focuses on reasoning over spatial relations such as left of or in front of. judging reachability, and selecting the most suitable object among candidates.
Object Affordance Reasoning: evaluates whether models infer the functionality of objects (e.g., knobs can be turned, cabinets can be opened), match objects to task requirements, and reason about indirect tool use.
20

---

Published as a conference paper at ICLR 2026
Precondition & Effect Reasoning: assesses whether models understand the preconditions and effects actions, such as a door needing to be closed before it can be opened, and can predict possible side effects.
Goal Decomposition: measures the ability break down complex tasks into sub- sub-goals, prioritize them, and determine parallel versus sequential execution strategies.
Visual Correspondence (extended capability): tests whether models can maintain object con- sistency across multiple views and integrate information under viewpoint changes.
B ADDITIONAL ABLATION STUDIES
B.1 COMPARISON WITH SET AND ICL BASELINES
To validate our choice of RL-based training over alternative learning paradigms, we compare our method against two additional baselines:
SFT baseline: We fine-tune WWLn2.5-VL--B on MomaGraph-Scenee using supervised learning only (without RI.), with the same graph-alignment objectives as our full method.
ICL baseline: We evaluate the base model with 3-5 in-context graph examples provided in the prompt (same setting as Owen2.55VL-7BB- main paper).
(w/ Graph) in Table 2 and of the
Method
BLINK
MomaGraph-Bench (Overall)
SFT baseline
60.4
63.9
ICI. baseline
58.7
60.2
RL w/ Graph (Ours)
63.5
71.6
Table 5: Comparison of our RL-based training with SFT and ICL baselines. Our method achieves substantially better performance on both benchmarks.
As shown in Table 5, our RL training method achieves clearly superior performance compared to both the SFT baseline (+3.1 on BLINK, +7.7 MomaGraph- Benchh and the ICL baseline (+4.8 on BLINK, +1 1.4 on MomaGraph-Benehh This demonstrates that the RL formulation crucial for learning high-quality scene graph generation that effectively improves downstream planning performance.
B.2 REWARD WEIGHT SENSITIVITY STUDY
We follow the original DAPO implementation in the EasyRI framework for default settings of wy and in Eq. 2 of the main paper. We conduct sensitivity study by varying (Wa, Wy wi) around the default configuration:
Setting ID
Wf
BLINK
MomaGraph-Bench (Overall)
A
0.5
0.5
0.5
61.3
68.2
B
0.7
0.3
0.5
63.1
70.9
C
0.8
0.2
0.7
63.7
71.2
Default
0.8
0.2
0.5
63.5
71.6
Table 6: Sensitivity analysis of reward weights (Wa, wy, wi) in our DAPO training. The model's performance remains stable across different weight configurations.
As shown in Table 6, the model's performance remains stable across these weight configurations, with variations of less than 2.4% on BLINK and 3.4% on SamaGraah-BBnch This indicates low sensitivity reward-weight choices and demonstrates the robustness of training approach.
21

---

Published as a conference paper at ICLR 2026
B.3 DETAILED WORLD DEMONSTRATIONS.
To provide a closer look into the behavior of our system, this section presents fine-grained real- world examples. We illustrate how the model processes raw images captured in realistic household environments, transforms them into task-oriented scene graphs, and generates corresponding planner outputs. These case studies highlight the system's ability capture subtle details, encode them into structured graphs, and reason over them to produce actionable plans. To validate the effectiveness
Environment Settings
Monilor
Lamp1
lamp2
Swilch
Neeiin
Figure 14: Quantitative real-robot evaluation. (a) Environment setup of the real-robot experiment. (b) Failure analysis illustrating success/failure rates across different reasoning stages.
of our approach real-world settings, we deploy the system on mobile manipulator to perform a variety of everyday tasks, as shown in Fig. 15. These tasks span multiple functional categories, such as turning off light, opening a microwave, turning on TV, and opening cabinet. In each case, the robot leverages the predicted spatial-functional scene graph to plan and execute sequence of actions without task-specific fine-tuning. The successful completion these tasks demonstrates the system's 's ability to generalize from structured graph representations real-world interaction scenarios, highlighting its potential for practical household assistance.
B.4 QUANTITATIVE REAL-ROBOT EVALUATION
To provide rigorous quantitative validation of our system's robustness, conduct comprehen- sive evaluation on complex multi-step long-horizon task. This evaluation includes success rates and failure analysis across different stages validate overall system performance under realistic, sequential conditions (see Figure 14).
Task Sctup. We evaluate the following natural language instruction that requires sequential rea- soning and manipulation: "I need better lighting. Turn the light closest to the remote I can find and turn on the monitor to watch." " To assess system robustness, we conducted 10 experimental trials, changing the camera viewpoint in each trial.
This task requires spatial reasoning (finding the switch and the remote), functional understanding (linking switches, lights, remote, monitor), and state-dependent planning (lighting affects percep- tion). Additionally, there's 's object uncertainty (multiple similar lamps or switches), complex spatial relations between objects, and sequential manipulation under partial observability.
Results. As shown in Figure 14, our system achieves an 80% success rate in graph generation, 87.5% success rate in planning (conditioned correct graphs), and an overall task success rate of 70% over 10 trials.
The main failure modes were: (1)spatial relation errors missing nodes during graph generation; and (2) action sequencing errors in the planning phase, suggesting that the system sometimes plans the right actions but in a suboptimal order.
These results demonstrate that MomaGraph remains robust across multiple reasoning and execution stages, achieving 70% overall success rate on a complex multi-step task. This validates the sys- tem's 's reliability under realistic long-horizon conditions where enrors can compound across stages.
22

---

Published as a conference paper at ICLR 2026
Open the Cabinet
Open the Microwave
Turn on the TV
Furn off the Light
Spadi Mour
Uight
RobofEra
Q5
strdoer
Soshh
Speaal striican Peak Vaw
Figure 15: Real-world robot execution of household tasks.
Open the Cabinet
Multiview Observations
Sandii
Spalts wlathee 'de'' Selee Ind's pul
MomaGraph-R1 Answer:
Scene Graph Analysis:
First, I understand the task: "Open the cabinet." Then, I identify the key objects needed: two handles and cabinet. Next, I analyze their spatial relationships: knobl in front of and close the cabinet, while knob2 is in front of and close to the cabinet, requiring "pull" action for opening/closing operations.
Answer: ("task instruction": "Open the cabinet.", "nodes": ["handle1", "handle2", "cabinet"], "edges": [['unctional relationship": "openorclose", " "object1": "handlel", "handle2": cabinet", "spatial_rehee relations": frin_front_or ponnt_of", "close"], "is dis_touching" truel, ("functional "ecirnnnnlraiiishhpppp "openorclose", "object1": "handlel", "handle2": cabinet", oopatial_relations ["in front_of", "close"], "is tsstoochhin" true)], "action_type"" "pull", "function_type". "open_close_conttettt
Action Sequence:
1. Approach the target area and identify the cabinet.
2. Locate the handlel, which is in front of and close the cabinet.
3. Grasp the handlel and pull it to the right.
4. If the cabinet does not open, locate the handle2, which is in front of and close to the kitchen cabinet.
5. Grasp the handle2 and pull it to the left.
6. Verify that the kitchen cabinet
open.
Figure 16: Real-world example of MomaGraph-R1 performing the task "Open the Cabinet." From multiview images, the system generates a scene graph capturing spatial-functional relations and outputs the corresponding action plan.
23

---

Published as a conference paper at ICLR 2026
Turn off the Light
Swishh
uutlnnippe" pour "b
Multiview Observations
MomaGraph-Ri Answer:
Scene Graph Analysis:
First, I understand the task: "Turn off the light." Then, I identify the key objects needed: a light switch and light. Next, analyze their spatial relationships: the light switch is to the right of and far from the light, requiring "press" action to turn off the light.
Answer: ('task instruction": "Tum off the light.", "nodes": ["light switch", "light"J, "edges": [["functional_relan relationship": "control", "object1"; "light switch", "object2": "light", "spatial_relations". relationsl:: ["right off,, "far"], "is_ touching" false]], action_type"" "press", "function_ _uupe: "device_control"
Action Sequence:
1. Approach the target area: Move towards the light switch, which is located to the right of and far from the light.
2. Identify the key objects mentioned in the scene graph: Locate the light switch and the light.
Press the light switch: Use your hand to press the light switch.
4. Verify the task is completed: Check if the light is off by looking at the light.
5. Confirm the task completion: Ensure that the light is off, indicating that the task "Turn off the
light" has been successfully completed.
Figure 17: Real-world example of MomaGraph-R1 performing the task "Turn off the light." From multiview images, the system generates a scene graph capturing catial-functionall relations and outputs the corresponding action plan.
24

---

Published as a conference paper at ICLR 2026
Open the Microwave
Multiview Observations
MomaGraph-Rl Answer:
Scene Graph Analysis: First, understand the task: 'Open the microwave." Then, iddetify the key objects needed: a microwave and a microwave button. Next, analyze their spatial relationships: microwave button is in front of close to the microwave, requiring prress' action for opening/closse operations.
Answer: l"task instruction': "Open the microwave.", "nodes": I'microwave button", "microwave"] "edges": [["functional relationship". "openorclose", "bjectt":""iicrro""" "microwave button", "cbjer22".'microwave"""" ['in. front of", "close"], "is touching" truel]. "action type": "press", "function type": "open close control")
"microwave" "spatial_relae relationrel
Action Sequence:
1. Approach the microwave door, which is in front of the microwave and currently closed.
2. Identify the microwave button and the microwave as the key objects mentioned in the some graph.
Press button the microwave with your hand. fully open.
Verify that the microwave
Figure 18: Real-world example of MomaGraph-R1 performing the task "Open the microwave." From multiview images, the system generates a scene graph capturing spatial-functio functional relations and outputs the corresponding action plan.
25

---

Published as a conference paper at ICLR 2026
Tern on the TV
Multiview
Observations
Toads serdee))
MomaGraph-R1 Answer:
Scene Graph Analysis:
First, understand the task: 'Tum on the IV." Then, I lidentify the key objects needed: a remote and TV. Next, analyze their spatial relationships: The remote is to the right of and in front of the TV, requiring a "press" action for activation.
Answer: ("task instruction "Tum the TV.", "nodes": ["remate", "TV"], "edges" [ relationsiip""' relations": ["in front_of", "right_of"), "s_fouching"" false)),
"control", "object1": "remote", "object2": "TV", rspatial_relations::: aecion_typel "press", "function_ type": "device_control""
Action Sequence:
1. Approach the target area: Move towards the TV and the remote, ensuring they are within reach.
2. Identify the key objects mentioned in the scme graph: Locate the remote and the TV.
Hold the remote: Use your hand to grasp remote, ensuring is in your control.
Aim
the Position effective control.
in front the TVV, making sure close enough the surface
5. Press the power button: Locate the power button the remote and press it firmly. This action should tum the TV.
6. Verify the task is completed: Check if the TV is turned on by looking for the power indicator light or by attempting to charge channels or adjust the volumne.
Figure 19: Real-world example of MomaGraph-R1 performing the task "Turn on the TV." From multiview images, the system generates a scene graph capturing spatiaa-functional relations and outputs the corresponding action plan.
26


Reviews:
[{"review_id": "Z7zcGVqD7c", "reviewer": "Reviewer_GYm6", "summary": "At its core, this paper argues that embodied agents should reason with scene graphs that encode both spatial and functional (i.e. task-specific) relations. The authors (i) curate a dataset (MomaGraph-Scenes), (ii) introduce a multi-choice evaluation suite (MomaGraph-Bench), and (iii) train an RL-tuned 7B VLM (MomaGraph-R1) with a graph-alignment reward. On the benchmark, the model reaches 71.6% accuracy (+11.4% over the best open-source baseline), with transfer to public benchmarks and qualitative real-robot demos.", "questions": "1. Data diversity claim vs. skew. The paper claims that MomaGraph-Bench is diverse. However, Figure 8 shows that a majority of scene graphs come from two action-function pairs (pull-open_close_control, and press-device_control), which, to me, seems fairly narrow. Could you elaborate on your claim and situate your benchmark in the context of similar benchmark suites (such as BLINK)?\n\n1. Currently, the only transferable task measured is BLINK. I would like to see evaluation of atleast one other task, such as EmbodiedBench, which aligns with your ultimate goal of embodied agent manipulation. \n\n[1] https://embodiedbench.github.io/", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Transfer to established benchmarks such as BLINK is solid. MomaGraph-R1 leads strong open-source baselines on BLINK, consistent with the claim that graphing first reduces VLM hallucinations. Table 2 describes the most interesting result, since SOTA VLMs achieve atmost 60-65% accuracy on BLINK. \n\n1. The benchmark has breadth across reasoning skills. The benchmark spans tiered tasks (T1–T4). Table 1 shows that unified spatial+functional graphs beat spatial-only / functional-only graphs across tiers.\n\n1. Interesting downstream real-robot evaluation. The paper evaluates four household tasks on a mobile manipulator illustrate the full Graph-then-Plan pipeline (multi-view perception -> graph -> plan -> skills).", "weaknesses": "1. Ablations are thin on training choices. The paper motivates RL but doesn’t include SFT/ICL baselines trained to produce the same JSON graphs, nor a sensitivity study on reward weights (only a config table/curves). \n\n1. Unclear novelty vs. prior “Graph-then-Plan\" work. Prior work already builds or consults scene/task graphs before planning, e.g., GRID [1]  and VeriGraph [2]. Please spell out what’s new here: unified spatial+functional + part nodes, the graph-alignment RL objective, and state-aware updates, and which components drive the gains. \n\n1. Robot results are qualitative. The real-robot section describes four tasks but omits success rates, number of trials, failure modes, and timing—making it hard to assess robustness. Per-task SR/NSR and common failure modes would strengthen the evidence. \n\nReferences\n\n[1] Dai et al. Optimal Scene Graph Planning with Large Language Model Guidance. https://arxiv.org/abs/2309.09182\n\n[2] Ekpo et al. VeriGraph: Scene Graphs for Execution Verifiable Robot Planning. https://arxiv.org/abs/2411.10446", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y1szBMJR9x", "reviewer": "Reviewer_xi2i", "summary": "This paper targets the pain point that (i) representations that are “spatial-only” or “function-only” cannot robustly support long-horizon, executable tasks, and (ii) VLMs tend to hallucinate when asked to directly plan. In response, it proposes **MomaGraph**, a **part-level, state-aware unified task scene graph** \\(G_T=(N_T, E_T^{s}, E_T^{f})\\). After each interaction step, it **dynamically updates** the functional/spatial relations via  \n\\(G_T^{(t+1)} = U(G_T^{(t)}, a_t, s_{t+1})\\),  \nso that an initial “one-to-many” hypothesis (one goal, many possible objects/affordances) progressively converges toward “one-to-one”.  \n\nThe reasoning paradigm is **Graph-then-Plan**: multi-view observations + instruction → generate the task graph → plan an action sequence from the graph → execute and write back updates. This realizes a closed loop of “perception → representation → planning → execution → updated representation.”\n\nThe authors also release **MomaGraph-Scenes** (~1,050 subgraphs, 6,278 images, 350+ scenes, 93 instructions) and a hierarchical benchmark **MomaGraph-Bench** (six capability types × four difficulty levels). They train on Qwen2.5-VL-7B using DAPO reinforcement learning with graph-alignment rewards.\n\nExperiments show that **a unified (spatial + functional) representation** and **Graph-then-Plan** both outperform “single relation only” or “direct planning,” across both open- and closed-source baselines. Their 7B variant reaches **71.6%** on the benchmark, a **+11.4%** absolute gain over the base model, degrades less on high difficulty and under cross-view correspondence (including BLINK subsets), and is demonstrated on real robots.  \n\nOverall, MomaGraph offers an integrated “representation + algorithm + data + benchmark” framework that systematically reduces hallucination while improving executability and consistency.", "questions": "On the dynamic update \\(U(\\cdot)\\) and convergence:\n\nPlease provide an explicit, executable update rule including:\n(a) How uncertainty is represented (per-node / per-edge confidence? prior vs. posterior maintenance?);\n(b) The conditions for pruning and confirmation (thresholds, evidence accumulation across multiple consistent observations, conflict detection and rollback policy);\n(c) The convergence criterion (when do you “lock in” a one-to-one mapping from an initially one-to-many affordance hypothesis? how do you prevent oscillation or premature commitment?);\n(d) Where this update module sits in the full inference loop, ideally with pseudocode or a flow diagram.\n\nConcretizing these details would move “interaction → write-back → convergence” from a conceptual claim to something reproducible, and would strengthen the argument that performance gains are truly due to Graph-then-Plan’s structured constraint.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Originality: The paper unifies spatial and functional relations in a single task scene graph, refined down to part-level nodes (e.g., handles, buttons). It then applies state-aware dynamic updates \\(G_T^{(t+1)}=U(\\cdot)\\) so that one-to-many hypotheses collapse toward one-to-one matches as the agent interacts. By pairing this with Graph-then-Plan, it decouples representation from planning and offers a holistic “representation + algorithm + data + benchmark” solution.\n\nQuality: The training data covers ~1,050 subgraphs / 6,278 multiview images / >350 scenes / 93 instructions, annotated at part level and serialized in JSON. The benchmark spans 294 scenes / 1,446 images / 352 graphs / 1,756 instances, and evaluates six capability types across four difficulty tiers. The DAPO RL setup with graph-alignment rewards (actions, spatial/functional edges, node completeness, plus format/length constraints) is clearly specified, and the ablations/error analyses convincingly explain *why* the method works.\n\nClarity: The paper formalizes \\(G_T=(N_T,E_T^s,E_T^f)\\), illustrates dynamic graph updates (e.g., knob-to-stovetop examples), and shows multiview alignment and question types (action sequencing, spatial reasoning, affordance, precondition-effect, goal decomposition, cross-view correspondence). Figures are illustrative and make reimplementation feel achievable.\n\nSignificance: On MomaGraph-Bench, Graph-then-Plan consistently outperforms direct planning across open- and closed-source models. MomaGraph-R1 (7B) reaches 71.6%, +11.4% over the base model, with significantly smaller performance drop at higher difficulty tiers. On BLINK / cross-view correspondence subsets, it achieves new SOTA-like performance (+3.8% / +4.8%), demonstrating that structured intermediate representations reduce hallucination and boost robustness in realistic household long-horizon tasks.", "weaknesses": "The paper does not yet evaluate the robustness of the dynamic update module \\(U(\\cdot)\\) under perception noise (false positives, false negatives, latency, occlusion). The core claim is that interaction plus write-back makes one-to-many → one-to-one, thereby reducing hallucination. This only generalizes if the update remains stable under imperfect observations.\n\nTwo actionable asks:\n1. Add robustness stress tests: during evaluation, inject controlled noise (missed detections / spurious detections / delayed observations / occlusions) into the perceived scene before calling \\(U(\\cdot)\\). Keep the rest of the pipeline fixed. Compare “no-graph / direct planning” vs. “Graph-then-Plan” on overall and per-capability metrics.\n2. Report simple stabilization baselines, e.g. (a) hysteresis / confirmation delay (require repeated evidence before pruning edges or committing affordances), and (b) temporal smoothing such as EMA over edge/part confidence. This would clarify whether the proposed dynamic convergence is robust in non-ideal conditions and help attribute gains specifically to the structured Graph-then-Plan constraint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1n9hoUBfPv", "reviewer": "Reviewer_uWxD", "summary": "The paper proposes MomaGraph, a task-conditioned scene-graph representation that unifies spatial and functional relations and includes part-level interactive nodes. Authors introduce (i) MomaGraph-Scenes, a new dataset of richly annotated, task-driven scene subgraphs with multi-view images; (ii) MomaGraph-Bench, a multi-choice VQA benchmark covering six reasoning capabilities from action sequencing to visual correspondence; and (iii) MomaGraph-R1, a 7B VLM trained with RL (DAPO) using custom graph-alignment reward to predict task-oriented scene graphs and then plan under a Graph-then-Plan paradigm. The method achieves 71.6% acc on MomaGraph-Bench and narrows gap to closed-source models, with +11.4% over Qwen2.5-VL-7B base. Also show it transfers to real-robot tasks (cabinet, microwave, TV, light).", "questions": "- How sensitive is performance to the reward function weights (wa, wf, wl in Eq. 2)?\n- What is distribution of failure modes? Would like to see some type of fail case analysis of this system\n- What is the computational cot of graph generation at inference time?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- well-motivated problem as there are clear limitations in existing scene graphs (single relationship types, static scenes, etc) with convincing preliminary experiments\n- I like how the work provides a complete comprehensive pipeline from representation design to dataset collection, to model training with RL, to benchmark construction, to real-world deployment\n- I think the joint modeling of spatial + functional relationships with part-level nodes is intuitive and well-executed\n- Strong empirical results (+11.4% over base model) and I like the comprehensive evaluation including in-person", "weaknesses": "- state-aware dynamic updates are presented as a major contribution but have no experimental evaluation. How often are these updates needed? How accurate are they? I would like to see ablation comparing task success with dynamic updates vs without. I also would like to see examples of success vs. failed disambiguation and in what number % of cases, this disambiguation was needed.\n- the dataset size of 1050 graphs seems modest. This may limit the generalization to diverse households or household tasks.\n- missing comparison with OpenFunGraph, a directly comparable method for functional scene graph generation with part-level interactive elements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Q28qk2Ui4", "reviewer": "Reviewer_YBG5", "summary": "The paper proposes 1) MomaGraph, a unified scene representation that integrates both spatial and functional relationships, 2) MomaGraph-Scenes, a dataset of annotated, task driven scene graphs in household environments, 3) MomaGraph-Bench, a VQA benchmark, and 4) MomaGraph-R1, a 7B vision-language model RL fine-tuned on MomaGraph-Scenes. The paper shows MomaGraph-R1 is the best performing open-source model on MomaGraph-Bench. The paper validates the system in real-world on a humanoid in unseen environments.", "questions": "- From the definition of MomaGraph (Section 4.1), the method is constrained to a single indoor room. Is there any particular for the constraint?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written, motivates the contents progressively, and thus easy-to-read. \n- The paper introduces a lot of content, from unified scene graph representation, to dataset, to benchmark, to model.", "weaknesses": "- Aside from its own proposed task, the paper only evaluated on one external benchmark, BLINK. It would be more compelling to see the scene graph and model being compared on more commonly used benchmark such as Habitat and Matterport 3D. \n- While the paper introduced the mechanism of dynamic scene graph update (Section 4.3), the defined task is static visual question answer, instead of interactive embodied question answer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)