ESCA: Contextualizing Embodied Agents via Scene-Graph Generation
Authors: Jiani Huang
Amish Sethi
Matthew Kuo
Mayank Keoliya
Neelay Velingker
JungHo Jung
Ser-Nam Lim
Ziyang Li
Mayur Naik
Venue: NeurIPS 2025 spotlight ()
Keywords: neurosymbolic
scene graph
multimodal
MLLM agents
Primary area: applications
TL;DR: Contextualizing MLLM-based agents with grounded scene graphs boosts their performance.

Abstract: Multi-modal large language models (MLLMs) are making rapid progress toward general-purpose embodied agents. However, existing MLLMs do not reliably capture fine-grained links between low-level visual features and high-level textual semantics, leading to weak grounding and inaccurate perception. To overcome this challenge, we propose ESCA, a framework that contextualizes embodied agents by grounding their perception in spatial-temporal scene graphs. At its core is SGCLIP, a novel, open-domain, promptable foundation model for generating scene graphs that is based on CLIP.  SGCLIP is trained on 87K+ open-domain videos using a neurosymbolic pipeline that aligns automatically generated captions with scene graphs produced by the model itself, eliminating the need for human-labeled annotations. We demonstrate that SGCLIP excels in both prompt-based inference and task-specific fine-tuning, achieving state-of-the-art results on scene graph generation and action localization benchmarks. ESCA with SGCLIP improves perception for embodied agents based on both open-source and commercial MLLMs, achieving state of-the-art performance across two embodied environments. Notably, ESCA significantly reduces agent perception errors and enables open-source models to surpass proprietary baselines. We release the source code for SGCLIP model training at https://github.com/video-fm/LASER and for the embodied agent at https://github.com/video-fm/ESCA.

ESCA: Contextualizing Embodied Agents via Scene-Graph Generation
Jiani Huang University of Pennsylvania j jianih@seas. upenn. edu
Amish Sethi University of Pennsylvania asethi040seas. upenn edu
Matthew Kuo
University of Pennsylvania mkuo@seas upenn. edu
Mayank Keoliya University of Pennsylvania mkeoliya@seas. upenn. edu
Neelay Velingker University of Pennsylvania neelay@seas upenn. edu
JungHo Jung University of Pennsylvania j76 unggseesss upenn. edu
Ser-Nam Lim University of Central Florida sernam@ucf.C edu
Ziyang Li
Mayur Naik
Johns Hopkins University ziyang0cs. jhu. edu
University of Pennsylvania mhnaik@seas. upenn. edu
Abstract
Multi-modal large language models (MLLMs) are making rapid progress toward general-purpose embodied agents. However, existing MLLMs do not reliably capture fine-grained links between low-level visual features and high-level textual semantics, leading to weak grounding and inaccurate perception. To overcome this challenge, propose ESCA, framework that contextualizes embodied agents by grounding their perception in spatial-temporal scene graphs. At its core is SGClip, novel, open-domain, promptable foundation model for generating scene graphs that is based on CLIP. SGClip trained on 87K+ open-domain videos using neurosymbolic pipeline that aligns automatically generated captions with scene graphs produced by the model itself, eliminating the need for human-labeled annotations. We demonstrate that SGClip excels in both prompt-based inference and task-specific fine-tuning, achieving state-of-the-art results on scene graph generation and action localization benchmarks. ESCA with SGClip improves per- ception for embodied agents based on both open-source and commercial MLLMs, achieving state of-the-art performance across embodied environments. Notably, ESCA significantly reduces agent perception errors and enables open-source mod- els surpass proprietary baselines. We release the source code for SGCLIP model training at https: at htttts::/g wwtp:icoooiiiiieeeeeeeeeeeeoeeie Fart0offmESSOO
com wwwwvideoooooooowoooooeooooooooe fm/LASER and for the embodied agent
Introduction
Recent advances in large-scale pretraining have enabled foundation models to assist with wide range of tasks, from language and vision understanding (2
mathematical problem solving [96] [8] and code generation 61 However, remains open challenge to realize embodied agents that are capable doing household chores, training alongside humans physical activities, or providing care for the aging 39 13 A critical first step toward this goal is equipping agents with fine-grained perception to ground abstract goals physical interactions.
15| g3
These authors contributed equally to this work
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Instruction
ESCA (ours)
Task Planning
Action
to prepare toasted bre- ad for breskfast, plee-
Concept
Scene Graph
SeCTEp
Executable
Fove right by 8.25 unsta.
Extraction
Gamini
25);
Transfer Protocor
entities:
Language Plan
attrabutien
Embodied Agent
relations:
[yellos]]
|ool
aborr
1. Hove forward by 0.25..
Reffect Reason
Feedback
Entity
CEDA
Environment
Identification
RAM
Visual Description
EmbodledBench
Nove succeeded
target: toasterrs objects:
155),
CB EE-Neriget
(B-Rasipsi
no collision dotected
Figure An overview of embodied agent pipeline augmented with ESCA. each cycle, the agent takes in instruction and the environmental feedback and outputs concrete executable action, through a sequence of perception, reasoning, and planning. The action is then executed and the environment will provide the next state. Notably, ESCA contextualizes the task planner with grounded visual features represented as a scene graph.
Despite progress, multi muuli--ooda large language models (MLLMs) still struggle to build spatially and temporally grounded world models. Their inability reliably ground visual features with spatial- temporal relations creates a disconnect between conceptual semanties and pixel-level observations [89 60 This lack of structured, fine-grained scene understanding severely limits their effectiveness in embodied environments. In fact, our empirical analysis shows that up to 69% of agent failures stem from perception errors, highlighting the need for frameworks that can bridge this gap.
To address this challenge, we propose integrating structured scene graphs into the perception, rea- soning, and planning pipelines of ML.LM-base embodied agents. While prior work has explored enhancing MLLMs with external visual grounding modules, these approaches typically rely on open-domain object detection models such Grounding DINO [56) and YOLO [27]. However, these models are primarily designed for object identification and often overlook semantic attributes, inter-object relationships, and temporal consistency.
In this work, we introduce ESCA (Embodied and Scene-Grape Contextualized Agent), a framework designed to contextualize MLLMs through open-domain scene graph generation (Figure D Much like the bioluminescent lure deep-sea anglerfish, which illuminates its surroundings to reveal otherwise hidden prey, ESCA provides structured visual grounding that helps MLLMs make sense of complex and ambiguous sensory environments. A key feature of ESCA is selective grounding: rather than injecting full scene graphs, which may degrade performance, the MLLM first identifies the subset of objects, attributes, and relations most pertinent to the instruction, then determines the essential entities for task completion. This mechanism is supported by transfer protocol, which performs probabilistic reasoning over object names, attributes, and spatial relations to construct prompts enriched with the most relevant scene elements. At its core is SGClip, CLIP-based model that captures semantic visual features, including entity classes, physical attributes, actions, interactions, and inter-object relations.
Through experiments on four challenging embodied environments, we demonstrate that ESCA consistently improves the performance of all evaluated MILMs, including both open-source and proprictary models. By providing structured and grounded scene graphs, ESCA significantly reduces perception errors, laying the foundation for more reliable reasoning and planning. Beyond its integration with MLLM-based agents, we show that SGClip, when evaluated independently, exhibits strong zero-shot generalization, is promptable for task-specific scene understanding, and remains fine-tunable for downstream tasks such as action recognition.
In summary, our contributions are as follows: (1) we present ESCA, general framework for contextualizing MLLM-based embodied agents through selective scene graph generation; (2) we introduce the transfer protocol for enriching prompts with probabilistically inferred scene-specific information for diverse embodied benchmarks; (3) we introduce SGClip, generalizable and fine- grained scene graph generation model, along with ESCA- Video-87K. MLLM annotated dataset;

---

L
Concept Extraction
Object Identification
Scene Graph Prediction
Summarization Validation
Visual Description
hg
Interni
Internit
trames before:
You are a robot operating in heee. You are glven the andtroo rnrromeent for performting the taxt: vipually sdentirlable features more sustle ones. Please deduce the ooject nomes with synonye For exseple, stbetstute "freshly baked beguette" "lnaves". You are suppaxed to output JSON.
You need to deseribe current visual state, sumariie interactions, any retrrneent feedback, and foiled. The
prev trame: 18, 75, =l
curr from:: 1820, 52.J Description: an gettin closer to the teaster but need also pick.
objects
48 dos:ouutoorto 0. 3. :o:ntoot top I- the bread is on the countertop coonfidence: 0.92).. The carget state: breed is toasted by touster; goel.
Figure 2: A detailed illustration of the visual description module, which involves concept extraction, object identification, scene graph prediction, and visual summarization. We also illustrate sample MLLM prompts used in a kitchen environment for the concept extraction and summarization steps.
(4) we conduct extensive evaluations demonstrating the effectiveness and versatility of ESCA and SGClip across both embodied agent and scene understanding tasks.
2
ESCA: A Framework for Embodied Agents
2.1 Background
Embodied Environments. Recent research on embodied agents has been accelerated by the avail- ability of simulated environments such as VistalhgentBench [57] and EmbodiedBench [91], which provide rich, multimodal task suites covering navigation, manipulation, and interaction across diverse scenarios. These benchmarks pose challenges that require agents to operate in both a) low-level action spaces such as continuous control signals over robot joints, and b) high-level action spaces such as programmatic instructions and skills that abstract over low-level actions.
These tasks are typically modeled as Partially Observable Markov Decision Processes (POMDPs), represented as 7-tuple (S, A, Ω, T, O, Lims R), where S is the unobservable state space, A is the task-specife action space, f is the visual perception space where I f is an image frame at time t, T 5 x A + S is the state transition dynamics, O S + relates the underlying state to the observations, Lins is the natural language instruction for the agent, and R S + {0, 1 is the reward function indicating whether the task has been completed.
Embodied Agents and MLLM-Based Embodied Agents. A typical embodied agent interacts with the environment by maintaining a history of observations and actions: h; = (Lo: ag. Li,.... at-1, 1t). where a, is the action taken by the agent at time t. The agent selects the next action by conditioning on the instruction and history he via a policy x (at Lins; ht). An WiLLM-based embodied agent realizes such a policy by leveraging multi-modal model that processes both: imagery data I,, and b) textual data, including the instruction Lins and textual representations of actions a E A.
Established by [91, recent MaLLed agent architectures often decompose the policy evaluation process into structured stages inspired by cognitive reasoning workflows (Figure D 1) Visual Description: extracting and summarizing visual inputs; 2) Reflection: integrating observations with historical context build situational awareness; 3) Reasoning: inferring rell-ael insights based on the combined context; 4) Language Plan: generating high-level plans or action sequences in natural language; 5) Executable Plan: translating plans into concrete actions executable by the agent.
Challenges of ALLM-Based Embodied Agents. Despite recent progress, MLLM-based embodied agents remain fragile in complex environments due to compounded errors across perception, reason- ing, and planning 33 [13. Perception errors include hallucinated objects, misrecognized entities or actions, and incorrect spatial relationships. Reasoning errors arise when agents fail correctly infer spatial relations or recognize task termination states. These issues propagate into planning, where agents may skip critical steps or generate invalid plans due to inaccurate state estimation.
2.2
The ESCA Framework
At the core of ESCA is a simple yet effective idea: contextuaizzng MLLLM-based agents with grounded, structured scene-graph information to improve visual description. Specifically, given
3

---

a language instruction Lins and interaction history hy, the agent generates multi mudii message sequence of images and text. While existing approaches rely on end-to-end MLLMs implicitly perform this step, ESCA decomposes the process into four modular stages below (Figure(2.
Selective Concept Extraction. This module extracts concepts with MILM guided by carefully designed prompts based on the instruction Lins and the history fę. Rather than producing only free-form natural descriptions, ESCA requires the MLLM explicitly extract structured concepts that are most relevant to the query, which can be classified as follows. a) entity classes such as (car, knife, person, cup); b) attributes including physical properties (red, small, broken) and semantic states (close-by, by, far, moving, sitting); c) relations covering spatial relations (behind, above) and interactions (cutting, cooking).
Concept extraction is guided by two signals: attributes, or relations the agent should focus on, and 2) the visual feedback I,, which provides spatial context such as environmental structures (e.g., barriers, pathways) and dynamic interactions (e.g.. objects being manipulated). Overall, this MLLM-generated structured output provides a targeted concepts set غ = {c1, (2,,02,... .2,... } for subsequent object identification and scene graph generation.
1) the instruction Lins, which highlights target entities,
Object Identification. Given the extracted concepts and the agent's visual feedback, the second module grounds these concepts to specific image segments ō = {01, .2,... } each represented by bit-mask. This step isolates visual elements in the frame that correspond to classified entities or attributes, enabling downstream semantic inference over localized visual units rather than raw pixels.
The object identification module is implemented using multi-stage pipeline. First, Grounding DINO (GD) [56), a vision-language detection model, takes the extracted concepts and the visual input to predict bounding boxes for the mentioned entities. These bounding boxes are further refined into precise segmentation masks using SAM2 [67], a segmentation model that produces pixel-accurate regions. Leveraging the GD SAM2 pipeline improves computational efficiency and offers better generalization to both attribute and relational concepts, as further detailed in the Appendix.
Scene Graph Prediction. We then construct a scene graph (SG) [8 87 34 by grounding the extracted concepts into structured visual elements. Specifically, the SG contains two types of probabilistic facts: 1) unary facts of the form :: c;())). each describing an entity o. with their classes, attributes, or states represented as c; 2) relational facts of the form p :: ci(a,, ok), each representing that a relation or an interaction e; between a pair of grounded entities o, and dg. Notably, each predicted fact associated with a confidence score denoted as p. This probabilistic formulation allows the SG to capture uncertainty and distributions over possible scene interpretations.
Implementation-wiss we build SGClip, a CLIP-based model 651 fine-tuned for open-domain scene graph prediction. The design of SGClip is guided by three aforementioned key desiderata. First, it supports open-domain concept coverage, enabling it to generalize beyond fixed taxonomies. Second, it is adaptable to extracting diverse types of information, including entity classes, attributes, and injer- relations. Third, it produces probabilistic predictions, allowing the model capture uncertainties. We provide a more detailed discussion of SGClip and its training in Section 3
Visual Summarization and Validation. This module distills these multi-modal signals into list of messages to contextualize the agent's 's reasoner and planner. Concretely, the summarizer takes prompt and is responsible for transforming the structured scene graph into natural descriptions, while also validating the consistency between the visual feedback and the underlying structured scene graph. While the summary is customizable via our transfer protocol, the of messages include 1) the current view (and potentially historic ones) augmented with visualized bounding boxes served as markers, 2) image segments corresponding to key entities, 3) the textual description of scene graph and segments, and 4) an analysis of history actions and how they caused the current scene.
2.3 Transfer Protocol
To enable ESCA to generalize across different downstream tasks, we define a general transfer protocol based on the customization of two prompt templates, positioned at the entry and exit points of the entire visual description module. The goal of this unified transfer protocol is to maximize adaptability of ESCA across tasks with diverse planning strategies, action spaces, and reasoning requirements, while maintaining consistent interface.

---

above, bread
cabinet
not
brown
behsnnd
cab behladd, bread
toaster
Sec11p
not big
b1p
SGC1ip
cUttang
oab, outting,
<norele
SeC1ip
(a) Entity classes
(b) Attributes
(c) Binary relations
Figure 3: Illustration of the inference modes of SGClip for three types of concepts: entity classes, attributes, and binary relations. While the model stays the same, the three inference modes perform different pre- and post-processing for a more accurate semantic estimation of probabilities.
Specifically, the first prompt, the Concept Extraction Prompt, specifies the required JSON output format and indicates, enumerates if possible, task-specific concepts that the agent should focus on. The second prompt, the Visual Summarization Prompt, guides the model to produce contextualized summary that integrates both grounded image segments and task-specific textual elements. such as target objects, desired states, and environmental constraints. Together, these prompts provide principled way adapt ESCA to diverse embodied AI tasks without retraining the core system. We present case study of the transfer protocol in Figure 2 and provide further details in Section
SGClip Model
To enable the generation of spatial-temporal scene graphs in open-domain setting, especially the embodied environments, we develop SGClip. a CLIP- based foundation model [65] for structured scene understanding, SGClip is designed operate in open-domain fashion, recognizing wide and extensible set of concepts. Secondly, it must adapt to different types of concepts, while be capable of generalizing unseen visual and textual domains. Finally, it must produce probabilistic predictions to capture uncertainty.
To meet these goals, SGClip builds CLIP's vision-language architecture, which naturally supports joint reasoning over images and textual phrases. However, deploying CLIP directly is insufficient, as it lacks specialization for structured scene graph prediction. To bridge this gap, we fine-tune SGClip to balance adaptability and generalizability. In this section, we describe 1) how to handle different types of concepts through inference time adaptation (Section (.11 2) how to overcome the lack of data by collecting a model-driven self-supervision dataset (Section 3.2), and 3) how to learn without relying on human annotations via self-supervised neurosymbolic learning pipeline (Section 3.3
3.1 Model Architecture and Inference Time Adaptation
At its core, SGClip (Scene Graph CLIP) is single CLIP--base model designed to score concept relevance within an image. Formally, it operates as SGClip(a, a) E [R| where a the input image and ē is the set of candidate concepts, producing logit score for each concept that reflects the model's confidence in its presence. SGClip supports three distinct inference modes to handle different types of concepts: entity classes Eelass attributes Tatrr and binary relations Tzela. Illustrated in Figure 3 each mode requires a specialized formulation of the input concepts and scoring process, effectively allowing SGClip operate flexibly across these concept types:
Entity classes. In this setting, SGClip is used to identify the most likely entity class presented in an image segment. Let Calas denote the list of candidate entity classes. Since entity is typically assumed to belong to a single class, we apply softmax normalization over the logit scores produced by SGClip for these candidates: softmax softmax(SGClip (SGClip(a, Gelass)).
Attributes. To estimate the likelihood that a specific segment possesses a particular at- tribute we construct binary contrast between the attribute and its negation by evaluating softmax (SGClipp, (c, e])), where -e denotes the negated textual phrase (e.g., "not red" for the attribute "red"). The first element of the resulting probability distribution corresponds to the model's s estimated likelihood. To improve computational efficiency, we perform batched evaluation by merging all cerrbuttecootradictton pairs into single concept set T Cattr U {-c C.nr).
5

---

ESCA-Video-87K
Scene Graph Generation
Object Tracking & Segmentation
Object Traces
LLaVA-Video-1780 117K
Visual Ingut
Meta SAM 2
SGClip
Extracted Concepts
Textual Input
Weakly- supervised Learning
Video Captioning
Concept Extraction
entitiea: lbus, car, .l: attr: [red]; state: [forra..
Weak- supervision Label
Synthetic Captions
Prog. Specifications
forward, traffic
2,,,, opoce) a (atorsecc(a) bustb) 1ightt3) [farts.e] a sontb, 1)) Ofred(1) close(k,e) ongo, 1)]
LASER
Spatial-Tempora
paosting durting Light.
Alignment Check Semantic
Semantic Parsing
Figure 4: Illustration of the construction of ESCA- Viico--V dataset and the model-driven self- supervised fine-tuning pipeline of our SGClip model. In addition videos and their natural language captions, ESCA- Video-8 -oceo--KKKK includes object traces, open-doman concepts, and programmatic speci- fications for 87K dideo-caption pairs. The dataset then used to train SGClip via LASER [34] neurosymbolic learning procedure based on spatial-tempont alignment.
Binary relations. For binary relation prediction, the goal is to determine whether a relation e holds between two segments d, and σ. To do this, we first compute bounding region ofy that tightly encloses both segments. Within this region, we apply distinct color tinting to o, and ay to indicate their directional roles (subject and object). To provide additional relational context, especially for interactions like "cutting," we augment the relation phrase by including the predicted entity classes of the subject and object, generating "(robot, cutting, cabbage)", Relation predictions are thus conditioned on the classes of both the subject and the object. Specifically, for each segment os we compute its most likely class ii by selecting the top prediction from the entity class:
Celass 14 where argmax I.l..|/a Ses....a,sSSClipio.o.. SGClip((;, Celass
We then form the augmented relation phrase as (,i, Similar attribute prediction, contrast the candidate relation with a special token <norel> denoting "no relation," and compute softmax (SGClip(a") {(P,, c, p2), <norel>])), to obtain the probability of whether the relation e holds between object and j. In practice, this process is batched over all segment pairs and relation concepts to maximize efficiency: Ca = {V,, c, 12)) i, 1... |ō], |...|ōl,e
G U {<norel>).
3.2 ESCA- VSC Dataset
We adopt the neurosymbolic weak-supervision pipeline introduced in LASER 34), which enables learning fine-grained STSGs from weak supervision signals derived from spatial-temporal program- matic specifications, eliminating the need for costly manual annotations. While the details this dataset,
learning pipeline are provided in Section 3.3] begin by introducing the the dataset we curate and use to train SGClip.
The ASCA-Video-877 dataset is constructed from the publicly available LLaVA- -77K dataset 197, and consists of 87K short video clips, each paired with natural language captions generated by GPT-4 [35]. As illustrated in Figure 4 these captions are first processed to extract relevant concepts which are then fed into GD 56 and SAM2 [67] to obtain object traces, which sequences of object segmentations that evolve across multiple video frames. In addition, these concepts are also used to assist in generating spatial-temporal programmatic specifications, expressed in linear temporal based language. To construct these specifications, we develop semantic parsing pipeline, again leveraging GPT-4, which converts high-level captions into structured tem- poral statements. These specifications formally describe how the semanties of object traces evolve, capturing temporal relations using operators such as "until", "finally", or "always".
In summary, each data point in ESCA-Video-877 Video-87K is represented as 5-tuple (I, Leap. E, 2, Φ), where I = (1,, I2.... is the video, Leap is the associated natural language caption, Σ {ō1, ō2, ...

---

is the set of object traces, č = {c1, cz.... is the set of extracted concepts, and 6 is the spatial- temporal programmatic specification. This rich, multi-level annotation enables training models like SGClip without requiring manual scene graph labeling. We defer additional details about the dataset construction process and data statistics to the Appendix.
3.3 Neurosymbolic Learning Pipeline
Given the ACCA-Video-775 our goal is to fine-tune the SGClip model using the provided object traces, concepts, and spatial-tempont programmatic specifications. This is achieved by aligning the scene graphs generated by SGClip with the expected specifications [34], where the degree of alignment serves as the learning signal (Figure 4 To perform this alignment in a differentiable manner, we leverage the Scallop programming language (50), enabling symbolic alignment checks to be integrated into end-to-end gradient-basee learning.
Specifically, the alignment loss computation mirrors the inference-time adaptation procedure de- scribed in Section 3.T where different types of concepts are processed differently but unified under the same model. The pipeline is further enhanced with semantic losses, derived from evaluating common-senss and temporal constraint satisfaction, as well as a contrastive loss that encourages the model to distinguish between matched and unmatched scene suuph-ppecification pairs. Additional details of the training process are provided in the Appendix.
4 Embodied Environments and Transfer Protocol Setup
benchmark suite designed to assess MLLM-- based embodied agents. We focus on two environments: EB-Navigation and EB-Manipulation, each of which requires different levels of perception, reasoning, and control, and may benefit from a contextualized visual description. To adapt ESCA these tasks, we apply our transfer protocol by designing two specialized prompts for each environment. Full prompt templates are provided in the Appendix; here, we summarize the core challenges and how ESCA addresses them.
We evaluate our approach on EmbodiedBench 45
EB-Navigation is built on AI2-THOR [42] and focuses on visual navigation tasks where the agent must locate target objects based on language instructions, such as "navigate to the laptop." The agent relics solely on egocentric visual input and textual feedback, navigating through a space using eight low-level movement and rotation actions. Existing models often fail by generating correct high-level plans but producing incorrect low-level actions due to poor spatial grounding from an egocentric perspective. ESCA addresses this by generating accurate scene graphs that capture spatial relations and object positions. With the scene graphs, we design prompts that incorporate numerical bounding box data and lemporal movement cues, helping the agent localize targets more precisely.
EB-Manipulation extends VLMBench [00] for evaluating low-level robotic manipulation. The agent controls 7-DoF robotic arm using discretized action spaces, with additional signals such as YOLO bounding boxes [27] and global 3D object pose estimates to assist manipulation. Existing models struggle to ground object concepts into actionable spatial representations. leading to perception failures that disrupt downstream planning and control. ESCA improves this by grounding target features into precise visual segments, enabling prompts that describe object attributes, semantic relations, and 3D spatial coordinates, giving the agent more reliable geometric context.
EB-Habitat builds upon Language Rearrangement task [78 simulated via Habitat 2.0 (77). and primarily evaluates high-level task decomposition and planning capabilities. The action space limited to atomic high-level actions, such as agent is instructed to complete tasks such as "Find a toy airplane and move it to the right counter". Common failure cases of existing models include invalid actions arising from failure to identify and remember object displacements in the scene. Our model improves this by managing scene graphs of the current and desired state of the target object, which improves awareness of task progression and limits the number of focus objects.
from which the
EB-Alfred is based on the ALFRED dataset 722 and A12-THOR [42]. It evaluates agents on high- level household tasks involving eight skill types like "pick up" or "turn off." The agent receives egocentric observations and textual feedback on action validity, performing actions on objects. While agents receive egocentric observations and action feedback, existing models tend to repeat the same mistakes because they fail to reflect on how past actions influence the current state. ESCA addresses
7

---

Io Base D GD
I| ESCA
10 Base 01
YOLO I ESCA
60
40
54.67
34.44
51.66
19.33
Success Rate
50
42
30
21.3
21.91
40
20
14.59
30
10
20
IVT.2.5
Qwen2.5
Gem2.0
GPT4o
IVL.2.5
Qwen2.5
Gem2.0
GPT4o
(a) EB-Navigation
(b) EB-Manipulation
1| Base Dn
GD In ESCA
Dn Base
10
GD In ESCA
70
60
Success Rate (%)
54.67
54.5
60
55
56.33
50
44.66
40
38.5
40
33.80
30
20
20
IVL.2.5
Qwen2.5
Gem2.0
GPT4o
IVL2.5
Qwen2.5
Gem2.0
GPT4o
(c) EB-Habitat
(d) EB-Alfred
Figure 5: The overall performance EB-Navigation and EBBManipulation environments. We show the performance of four hase models, Swwn2.5-VLL72B-insee
(IVI.2.5), Genimi-2.0-flaan (Gem2.0), (Qwen2.5), and GPT-4o (GPT4o), as well their performance when accompanied with baseline visual grounding modules such as Grounding DINO (GD) or YOLO. With ESCA and SGClip, all models consistently outperform the baselines.
this by generating scene graphs that describe both the current and target states symbolically, enabling prompts that support causal reasoning. This allows the agent recognize how previous actions led to the current situation and to deduce the necessary state changes to achieve the task goal.
Empirical Evaluation
Our experiments designed to address two key research questions: (1) How effectively does ESCA, together with SGClip, improve embodied agent performance through structured scene graph generation? and (2) How generalizable and adaptable is SGClip when evaluated independently open-domain, zero-shot, and downstream transfer tasks? We now detail our experimental setup and present empirical results addressing both questions.
Experimental Setup. We evaluate ESCA EB-Navigation, EB-Manipulation, EB-Habitat, and EB-Alfred, four environments that demand fine-graines perception to support both low-level and high-level control. To assess the general applicability of ESCA, we integrate it with four diverse MLLMs: Intern -LL-2.5-8BB-MB
[14] Owen2.5-VL-722 Ins B] Gemini-2.0- O-fllshh (62), and GPT- 4o 33 For each MILMM experiment, we use the model for both the concept extraction and visual summarization steps (Figure 2). To further benchmark ESCA's 's impact, compare performance of MILMS augmented with existing visual grounding modules, including Grounding DINO (56] and MirraltticsYOLLO)) 27)
For evaluating SGClip independently, we consider out-of-domain scene graph benchmarks, including OpenPVSG 190). Action Genome [36], and VidVRD 70), comparing SGClip against strong baselines such as CLIP 65). VL-erB [15,, BIKE and Text4Vis [84]. To assess SGClip's downstream
adaptability beyond structured scene graph prediction, further test the fine-tunability the ActivityNet action recognition dataset by applying a transfer protocol.
'The three top-level error types are Perception, Reasoning, and Planning. The second-level errors are Hallucination, Wrong Recognition, Spatial Understanding. Spatial Reasoning, Retlection Error, Inaccurate Action, and Collision. For clarity, the figure these acronyms label the different types.

---

InternVL
Reasoning Plan
Co
HG
The kettle located in the center of the Image, siightly tawards the pack, on the countertop near the stave, My plan is to move rightwerd to bypess the desk.
Pert.
Inter VL + +ESCA
(44%)
Reas.
LA
(266%)
Exploration
Decisive Progression
RF
SR
InternVL
Reasoning Plan
IA Plun. (20%)
The image shows a kitchen with refrigerator
the The kettle is not visible in the current view. I have rotated to the right once, but the kottle still not visible.
RE Reas
InternVI
(
(09%)
Behavioral Degeneration
Exploration
Figure 6: Two traces by InternVL with and without ESCA, on the task of "navigate the kettle on the stove". With ESCA, the kettle can be marked in the image and textual description, encouraging the agent to decisively move towards the goal.
Figure 7:
Error decomposi-
tior Intern with with- out ESCA, manually inspected on 60 EB-Navigation tasks.
ESCA for Embodied Agents. As shown in Figure 5 ASCCA-augmente MLLMs consistently outperform their non-contextualized baselines across both -a-iiiggttion and EB-Manipulation. Remarkably, on -B--aviga,,,n, even the open-source VLtem.VL-225. when augmented with ESCA, surpasses the base performance of the proprietary GPT-4o model. While integrating Grounding DINO or YOLO improves baseline models, ESCA provides additional, substantial gains. For example, Gemini-2.0 with ESCA achieves over 10% improvement, while GPT-4o, already boosted by YOLO, still benefits from additional 6% performance gain EB- AB-Manipulation.
Through qualitative analysis end-to-end agent behaviors, we observe that ESCA consistently improves the agent's perceptual grounding, leading to more effective task execution. As illustrated in Figure [] an agent powered by InternVL. with ESCA successfully identifies the kettle early in the episode and navigates directly toward it. In contrast, the base Intern model fails to recognize target and ultimately collapses onto the wall. This observation is further supported by the error decomposition analysis shown in Figure 7 where we find that ESCA reduces the overall perception emor rate from 69% to 30%. We provide additional detailed experimental results in the Appendix.
Generalizability and Adaptability of SGClip. Evaluating SGClip's zero-shot generalization, Fig- ure shows that SGClip trained on Action Genome, and VidVRD, demonstrating strong out-of-domain robustness. Further, SGClip shows strong adaptability, achieving notable improvements when fine-tuned on VidVRD (details provided in the Appendix). Beyond scene graph tasks, Figure [0gure[O]man highlighh SGClip's 's downstream transferability to action recognition on ActivityNet. Fine-tuned with only 1% of the training data, SGClip outperforms state-of-the-art zero-shot video recognition baselines. With 5% of the data (approximately 800 videos), SGClip achieves 92. 10% accuracy, approaching the performance of Intern Video2-6B with end-to-end finetuning on the ActivityNet dataset.
consistently outperforms CLIP on OpenPVSG,
6 Related Works
Scene Graph planning. Scene graphs [3 87 31 33 are symbolic representations that encode
the semantic structure of an image video by identifying objects as nodes and their relationships as edges [58] 43 They play a central role in variety of vision-related tasks, including visual question
answering 44 28 64], image captioning [92 102), and image generation [26] 46). More recently,
scene graphs have been increasingly adopted the domain of robotic planning, for enhanced robust-
ness 330 81, verifiable planning [7 66 16]. To enable scamless integration with multimodal
large language model (MLLM) agents, ESCA constructs scene graphs from 2D image inputs and dynamically updates them through embodied interaction with the environment.
are autonomous systems that perceive, reason, and interact within physical or simulated environments. To evaluate their capabilities, varies bench- marks 45
Embodied Agents. Embodied agents
86 have been proposed, ranging from vision-language navigation 68 and object manipulation
19 53 to interactive instruction following and long-horize planning 772 73 20).

---

OSe
Action Genome
VidVRD
SGClip CLIP
75
75
100
InterVV
VidVRD
Action Genorn
Accuracy (%)
02
50
50
90
Recall
OSenPVSG
80
BIKB
25
25
80
VidVRD OpenPVSG
Textivis
Action Genome
0
0
70
1K
10K
87K
IK
10K
87K
0% %
1%
5%
Data used for fine-tuning
(a) Entity class prediction R@ (b) Binary relation prediction R@ 10
Figure 9: The zero-shot performance of SGClip compared to CLIP (shown in dashed lines) on OpenPVSG, Action Genome, and VidVRD datasets. We showcase the Recall metrics on entity class prediction, as well as the Recall 10 metrics on binary relation prediction. To illustrate efttaaefffcnney, we include the performance of checkpoints of SGClip when trained on IK, 10K, or 87K (full) portion of ESCA- -iie---8..-
Down-stream fine- tunability on action recognition, evaluated on ActivityNet dataset. We also illustrate zero-shot base- lines (BIKE and Text4vis) as well as fully-supervised base-
Figure 10:
line (InternVI.-6B)
80 and multimodal large lan- 52 83 are driving progress toward general-purpose embodied Furthermore, recent MLLMs have been trained end-to-end directly generate low-level numerical control commands [T0 7 ESCA introduces a general framework for augmenting vision-driis -diven. MLLM-bane -base agents with structured scene graph information.
Recent advances in large language models (LLMs) S
35
79
guage models (MLLMs)
12
15
agents 5 9 32
94
59
Neurosymbolie Methods with LLM and MLLM. A growing trend for enhancing the reasoning capabilities and robustness of large language models (LLMs) is to incorporate structured represen-
tations and leverage symbolic algorithms to reason over them [40 23
93
48
47
51
76 6].
These efforts span diverse domains, including code generation 21
61
55 mathematical problem
solving [96] 18). and verifiable planning [73 54 12 (95. Recent work has extended this paradigm the low level control domain, training MLLMS with structured scene graphs in end-to-end manner 7 71 63, [01]. ESCA follows this neurosymbolic direction by introducing SGClip, which trained in Maug-augeettedd self-supervised manner enabled by neuro-symbolic methodology. using structured scene graphs as an intermediate representation to guide learning.
7 Conclusion and Limitations
We introduced ESCA, a framework for contextualizing embodied agents through scene graph generation, powered by SGClip, promptable, open-domain scene graph model. Through general transfer protocol, ESCA adapts to diverse tasks and consistently improves agent performance across multiple environments and MLLMs. Beyond embodied tasks, SGClip demonstrates strong generalization and adaptability on open-domain scene graph and action recognition benchmarks.
Limitations. Despite its strong performance, our framework has several limitations. First, the use of large language models for high-level planning introduces latency, making unsuitable for real-time low-level control. Second, the system relies on 2D visual inputs and lacks support for 3D representations like point clouds, limiting depth-aware reasoning and spatial precision. Finally, while ESCA leverages MLLMs to generate coherent plans, it lacks formal mechanisms for verifying intermediate and final states during execution.
Acknowledgements. This research was supported by the ARPA-H program on Safe and Explainable AI under award #D24AC00253-00, the NSF under award #2313010, Google Research Award, and gift from AWS AI to ASSET-Penn Engineering Center on Trustworthy AI.
10

---

References
[1] Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li. Abhi Cupta, Tommi Jaakkola, Josh Tenenbaum.
Leslie Kaelbling. Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36:22304-22325. 2023.
[2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. / 12966, 1(2):3, 2023.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Gc, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[4] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006. 131-1 2020.
[5] Suncel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hicrarchics using language. arXiv preprint 0:Xiv:2403.01823 2024.
[6] Paul Biberstein. Ziyang Li, Joseph Devietti, and Mayur Naik. Lobster: A gpu acceleraeeee framework for neurosymbolic programming. arXiv preprint arXiv:2503.21937, 2025. Accepted at ASPLOS 2026.
[7] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail. Michael Equi, Chelsea Finn, Niccolo Fusai. Lachy Groom, Karol Hausman, Brian lchter, et al. T0: A Aasion-lanuuageacttion flow model for general robot control, 2024. URL https//arrii org/abs/2410.24116
[8] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
[9] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Assion-language-ctiinn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 2023.
[10] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. Rt-1: Robotics transformer for real- world control at scale. arXiv preprint ww..xx:2222.0611111 2022.
[11] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video henchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961-970, 2015.
[12] Alessio Capitanelli and Fulvio Mastrogiovanni. A framework for neurosymbolic robot action planning using large language models. Frontiers Neurorobotics, 18: 18:1342786, 2024.
[13] Yanan Chen, Pesaranghader, Tanmana Sadhu, and Dong Hoon Can we rely on IIm agents to draft long- hong-horizon plans? let's take travelplanner as an example, 2024.
[14] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang. Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang. Tan Jiang, Bo Wang. Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu. Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou. Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance houndaries of open-source multimodal models with model, data, and test-time scaling, 2025.
[15] Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Sealing up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185-24198, 2024.
[16] Zhirui Dai. Arash Asgharivaskasi. Thai Duong, Shusen Lin, Nikolay Atanasov. Optimal scene graph planning with large language model guidance, 2024.
Tzes, George Pappas, and
11

---

[17] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, al. Robothor: An open simulation-to- real embodied platform. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3174, 2020.
[18] Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap. Danilo Jimenez Rezende, Yoshua Bengio, Michael C Mozer, and Sanjeev Arora. Metacognitive capabilities of llms: An exploration in mathematical prohlem solving. Advances in Neural Information Processing Systems, 17:19783-19812, 2024.
[19] Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kemb- havi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object manipulation. P Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 4497-4506, 2021.
[20] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar. Yuncong Yang. Haoyi Zhu, Andrew Tang. De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 2022.
[21] James Finnie-Ansley, Paul Denny, Brett Becker, Andrew Luxton-Reilly and James Prather. The robots are coming: Exploring the implications of openai codex on introductory programming. Proceedings of the 24th Australasian computing education conference, pages 10-19, 2022.
[22] Stan Franklin. Autonomous agents as embodied ai. Cybernetics & Systems, 28(6):99,-520, 1997.
[23] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[24] Raghav Goyal. Samira Ebrahimi Kahou, Vincent Michalski. Joanna Materzynska, Susanne Westphal, Heuna Kim. Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Aueller-Freitad al. The" something something" vidco database for learning and evaluating visual common sense. Proceedings of the IEEE international conference on computer vision, pages 5842-5844 5850, 2017.
[25] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 1995-19012, 2022.
[26] Roci Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph to image generation. In Computer Eision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pages 210-227. Springer, 2020.
[27] Priyanto Hidayatullah, Nurjannah Syakrani, Muhammad Rizqi Sholahuddin, Trisna Gelar, and Refdinal Tubagus. Yolov8 to yo 1: A comprehensive architecture in-depth comparative review, 2025.
[28] Marcel Hildebrandt. Hang Li, Rajat Koner, Volker Tresp, and Stephan Günnemann. Scene graph reasoning for visual question answering. arXiv preprint arXiv:2000.01002,, 2020.
[29] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd, et al. spacy: Industrial-strenge natural language processing in python. 2020.
[30] Joy Hsu, Jiayuan Mao, Josh Tenenhaum, and Jiajun Wu. What's left? concept grounding with logic- enhanced foundation models. Advances Neural Information Processing Systems, 36:38798-38814, 2023.
[31] Cassie Huang, Stuti Mohan, Ziyi Yang, Stefanie Tellex, and Li Zhang. Language model as planner and formalizer under constraints. 2025.
[32] Cassie Huang and Li Zhang. On the limit of language models as planning formalizers, 2025.
[33] Jiani Huang. Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, Le Song, and Xujie Si. Scallop: From probabilistic deductive databases to scalable diffierentiblee reasoning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 25 25145. Curran Associates, Inc., 2021.
12

---

[34] Jiani Huang, Ziyang Li, Mayur Naik, and Ser-Nam Lim. LASER: A neuro-symbolic framework for learning spatio-lemporal scene graphs with weak supervision. In The Thirteenth International Conference on Learning Representations, 2025.
[35] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AI Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXi::2410.21276. 2024.
[36] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10236-10247, 2020.
[37] Ziyuan Jiao, Yida Niu, Zeyu Zhang, Song Bann-Chun220 Zhu, Yixin Zhu, and Hangxin Liu. Sequential manipu- lation planning on scene graph. In 2022 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 8203-8210. IEEE, 2022.
[38] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bersteiin, and Li Fei- Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668-3678, 2015.
[39] Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy, Lims can't plan, but can help planning Imm-oouuo frameworks, 2024,
[40] Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. Beliefbank: Adding memory to a pre- pre-trained language model for systematic notion of belief. arXiv preprint arXiv:2109.1477 2021.
[41] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 170:::10000000 170500600,,, 2017.
[42] Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: an interactive 3d environment for visual AL. CoRR, abs/ fin//172200574, 2017.
[43] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, ct al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73 33- 2017.
[44] Soohyeong Lee, Ju- Whan Kim, Youngmin Oh, and Joo Hyuk Jeon. Visual question answering over scene graph. In 2019 First International Conference on Graph Computing (GC), pages 45-50. IEEE, 2019.
[45] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang. Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking ulss for embodied decision making. Advances in Neural Information P Processing Systems, 37: 11000022---554,,,,-1111111104044 2024.
[46] Yikang Li, Tao Ma. Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. Pastegan: A semi-parametric method to generate image from scene graph. Advances in Neural Information Processing Systems, 32, 2019.
[47] Ziyang Li, Saikat Dutta, and Mayur Naik. Iris: Llm-assisted static analysis for detecting security vulnerabilities. In The Thirteenth International Conference on Learning Representations, 2025.
[48] Ziyang Li, Jiani Huang, Jason Liu, Felix Zhu, Eric Zhao, William Dodds, Neelay Velingker, Rajeev Alur, and Mayur Naik. Relational programming with foundational models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10635-10644, 2024.
[49] Ziyang Li. Jiani Huang. Jason Liu, Felix Zhu, Eric Zhao, William Dodds, Neelay Velingker, Rajeev Alur, and Mayur Naik. Relational programming with foundational models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(9):
Mar. 2024.
[50] Ziyang Li, Jiani Huang, and Mayur Naik. Scallop: A language for neurosymbolic programming, 2023.
[51] Ziyang Li, Jiani Huang, and Mayur Naik. Scallop: A language for neurosymbolic programming. Proc. ACM Program. Lang., 7(PDDI) June 2023.
[52] Bin Lin, Yang Yc, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 10122, 2023.
[53] Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 432 448. PMLR, 2021.
13

---

[54] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+p: Empowering large language models with optimal planning proficiency, 2023.
[55] Fang Liu,, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang. Li Zhang, Zhongqi Li, and Yuchi Ma. Exploring and evaluating hallucinations in Imm-powered code generation. arXiv preprint arXiv:2404.00971, 2024.
[56] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024.
[57] Xiao Liu,, Tianjic Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song. Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang. Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng. Hao Yu, Hanchen Zhang. Wenyi Hong, Ming Ding. Lihang Pan, Xiaotao Gu, Aohan Zeng. Zhengxiao Du, Chan Hee Song. Yu Su, Yuxiao Dong, and Jie Tang. Visualagenthench: Towards large multimodal models as visual foundation agents, 2024.
[58] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Fei-Fee. Visual relationship detection with language priors. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 852-869. Springer, 2016.
[59] Yecheng Jason Ma, William I. Liang, Guanzhi Wang, De-An Huang, Oshert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. ICLR, 2024.
[60] Damiano Marsili, Rohun Agrawal, Yisong Yue, and Georgia Gkioxari. Visual agentic ai for spatial reasoning with dynamic api. 2025.
[61] Theo X Olausson, Alex Gu, Benjamin Lipkin, Cedegao E Zhang, Armando LezaraLe Joshua B Tenenbaum, and Roger Levy. Line: A neurosymbolic approach for logical reasoning by combining language models with first-order logie provers. arXiv preprint 23X1v:2310.15164 2023.
[62] Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. Introducing gemini 2.0: Our new ai model for the agentic era, Dec 2024.
[63] Jianing Qian, Yunshuang Li, Bernadette Bucher, and Dinesh Jayaraman. Task-oriented hierarchical object decomposition for visuomotor control. arXiv preprint arXiv:2411.0/22 2024.
[64] Tianwen Qian, Jingjing Chen, Shaoxiang Chen, Bo Wu, and Yu-Gang Jiang. Scene graph refinement network for visual question answering. IEEE Transactions Multimedia, 25:3950-3961, 2022.
[65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CORR, abs/2103.0000 2021.
[66] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. arXiv preprint arXiv:2307.061133 135, 2023.
[67] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam Segment anything in images and videos, 2024.
[68] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339-9347, 2019.
[69] Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user senergeneateed vidcos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pages 279-287. ACM, 2019.
[70] Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection, In Proceedings of the 25th ACM International Conference on Multimedia, pages 1300-1308, 2017.
[71] Junyao Shi, Jianing Qian, Yecheng Jason Ma, and Dinesh Jayaraman. Composing pre-traine trained object- centric representations for robotics from" what" and" where" foundation models. In 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 15424 15432. IEEE, 2024.
14

---

[72] Mohit Shridhar, Jesse Thomason, Daniel Gordon. Yonatan Bisk. Winson Han, Roozbeh Mottaghi. Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday lasks. In Proceedings of the IEEE/CVT conference on computer vision and pattern recognition, pages 10740-10749) 2020.
[73] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03766 2020.
[74] Gunnar A Sigurdsson, Gil Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Holly- wood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th Furopean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings Part I 14, pages 510-526. Springer, 2016.
[75] Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano- Lozano-Pééez, and Leslie Pack Kaelbling. Pddl planning with pretrained large language models. In NeurlPS 2022 foundation models for decision making workshop. 2022.
[76] Alaia Solko-Breslin, Seewon Choi, Ziyang Li, Neelay Velingker, Rajeev Alur, Mayur Naik, and Eric Wong. Data-efficient learning with neural programs, 2024.
[77] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Tumeer, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-266. 2021.
[78] Andrew Szot, Max Schwarzer. Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev. Large language models generalizable policies for embodied tasks. In ICLR, 2024.
[79] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[80] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[81] Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, and Yang Gao. Programmatically grounded, compositionally generalizable robotic manipulation. arXiv preprint arXiv:2304. 13826, 2023.
[82] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang. et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint 20X7022307.0642.. 2023.
[83] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang. Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint 2rXiv:2303.04667, 2023.
[84] Wenhao Wu, Zhun Sun, and Wanli Ouyang, Revisiting classifier: Transferring vision-languee models for video recognition, 2023.
[85] Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang. and Wanli Ouyang. Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6620-6630, 2023.
[86] Fei Xia, Amir Zamir, Zhiyang Hc, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9068-9079 2018
[87] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 5410-5419, 2017.
[88] Hongwei Xue, Tiankai Hang. Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang. Jianlong Fu, and Baining Guo. Advancing resh-resolution languaan representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5036-5045, 2022.
15

---

[89] Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces, 2024.
[90] Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, and Ziwei Liu. Panoptic video scene graph generation, 2023.
[91] Rui Yang. Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang. Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.05500, 2025.
[92] Xu Yang. Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition. pages 10685-10669 2019.
[93] Hanlin Zhang, Jiani Huang. Ziyang Li, Mayur Naik, and Eric Xing. Improved logical reasoning of language models via differentiable symbolic programming. arXiv preprint 230. 2023.
[94] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.000 024455 2023.
[95] Li Zhang, Peter Jansen, Tianyi Zhang, Peter Clark, Chris Callison- Burchh, and Niket Tandon. Pddlego: Iterative planning in textual environments. arXiv preprint arXiv:2405. 1979:99339999 2024.
[96] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang. Yu Qiao. et al. Mathverse: Does your multi-modal I1m truly see the diagrams in visual math problems? European Conference on Computer Vision, pages (69-186. Springer, 2024.
[97] Yuanhan Zhang. Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Vidco instruction tuning with synthetic data, 2024.
[98] Yuanhan Zhang. Jinming Wu, Wei Li. Bo Li. Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data, 2025.
[99] Zhonghan Zhao, Wenhao Chai, Xuan Wang, Boyi Li, Shengyu Hao, Shidong Cao, Tian Ye, and Gaoang Wang. See and think: Embodied agent in virtual enviroument. European Conference on Computer Vision, pages 204. Springer. 2024.
[100] Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Eric Wang. VImbench: A composi- tional benchmark for sssion-and-langeeee manipulation, 2022.
[101] Xi Zheng, Ziyang Li, Ivan Ruchkin, Ruzica Piskac. and Miroslav Pajic. Neurostrata: Harnessing neurosymbolic paradigms for improved design, testability, and verifiability of autonomous cps, 2025.
[102] Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph decomposition. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XIV 16, pages 211 1299. Springer, 2020.
[103] Luowei Zhou, Chenliang Xu, and Jason Corso. Procnets: Learning to segment procedures in untrimmed and unconstrained videos, arXiv preprint
2(6)77, 2017.
[104] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang. Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining n-modality by language-bass based semantic alignment, 2024.
16

---

8
Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made four claims in the introduction. All of them are supported by our pipeline and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
We have an explicit section dedicated to conclusion, limitation, and future works.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA].
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes].
Justification: We include the experimental details in the appendix, and our full code base in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
Justification: In the supplementary material, we provide our codebase for (1) preprocess- ing the ESCA- ACCA-Video-87K dataset and (2) implementing the ESCA pipeline and transfer protocols. We will release the dataset and open-source the code upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
aablit//uiide//ddeeabbmmsss soteesaubbbsssiooooooccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.cc/pulli//uiiiee
swtssssssiiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification:
We include these specifications in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No].
Justification: We evaluate large language models with the temperature set to 0 to reduce output variance. Moreover, querying these models multiple times with the same prompt neither environmentally sustainable nor economically feasible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes].
Justification:
We include the experimental details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /
Answer: [Yes]..
Justification: No human objectives are involved in our research. Our dataset is created on the top of fully open source dataset with Apache License 2.0.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: This work focuses on evaluating embodied agent performance simulated environments, with accuracy measured against predefined, human-authored criteria. As such, the study does not directly engage with real-world deployment or societal impact, making discussion of positive negative societal effects not applicable in this context.
20

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification:
The LLava-Viide Video-1788 dataset is under Apache 2.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, datasets.paperssiii paperswi thcode. www.com//aaaa datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: we provide the whole repository that generates our dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes].
Justification: ESCA is a new method augmenting MILMs with scene graphs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

Algorithm 1: Video Mask Propagation with New Object Discovery
Input:
Video V .To..... }r), mask generator G. propagation model P, state S Batch size B, thresholds: Tio; Tacore; Timers stride
Input:
Output: Mask dictionary __deo_segment Dict [Frame] [ObjectID] Mask
video_ Lideo_seements-ee f- 0, now aow_frame- 0;
2 while not saturated do
I + V nhow_frame]:/ // Load current image frame
M +- -()):// Generate masks for current frame
// Find object masks not yet propagated from this frame Mnew = =fitte(M,, video_segements [now_frame] Fleus Tscores Tinner;
5
// Register the masks prompts
Pnur-update(Meew)::
// Prompt the mask generator with updated maaks initialize condition P.reset_state($)
7
foreach (oid, frame_id, mask) in Pin do P.add__ees Badd__ew_matte Bedd__e__mss_pooptteee frrroopptssfrmeeeeeeeee frame_id, o, m)) end
// Propagate the mask prompts through the whole video foreach (f, {o,), (4)) E P.propagate(S) do
12
if f & video_ segments then sideo_segmentan
13
t +0:
14
end foreach do
15
16
m, binarize(4)) (); [/][o;] + mi
17
video video_segmen segments
end
end
// Find the next frame to proceas with least maak coverage Compute coverage Ps over tnowfframe +,.. I....T); saturated, now frame = check <hecc_ssturationn())
21
22 end
return video_segments
A
ESCA-Video-871
A.1 Video and Caption Source
We build upon the LLaVA-Video-A GPT-generated, detailed video descriptions.
Video-178K dataset 97 which comprises diverse video sources and
ESCA-Video-87P Video-87K contains total of 87, 045 datapoints, each consisting of a video clip ranging from 0 to 30 seconds in length, along with its corresponding caption. The videos are drawn from ten primary sources, spanning a wide range of content-inchin well as crowd-sourced videos from YouTube. Specifically, these ten sources are HD-VILA- 1000 [88]
egocentric and household activities, as
Intern Vid 10M (82). VidOR (69), VIDAL (YouTube Shorts) [104], YouCook2 [103], Charades 74
ActivityNet [T Kinetics-700 [41, Something SoottinnggSomtiingg v2 [24]. and Ego4d 25 The captions
were generated using GPT-4 with a dense frame sampling technique 997]
The two main contributions of ESCA-Video-872 Video-87K are the inclusion of object trajectories and executable programmatic specifications. We leverage SAM2 [67] generate object trajectories and design custom algorithm to handle objects that do not appear in the first frame. To obtain the specifications, we use GPT-4 and develop transcompiler that converts them into executable programs.
These two additional components enable fine- tne-grained spatio-temporal alignment between the video content and the concepts expressed in the specifications (34). Notably, the only source of supervision is the video itself; all other annotations are derived using multimodal large language models (MLLMs). We refer this approach as model-driven self-supervision.
24

---

Algorithm 2: Bounding Box Prompt Buffer Class
Output: Prompt dictionary for propagation; object-to-class mapping Initialize fields: oid, frame2bboxes, frame2masks, valid_ proidd_ppppss,
overlapped_prompen
// Add predicted bounding box and mask
Function AddPrompt (Srame_id, box, mask,) if not valid(m) then
3
return end
5
frame2bbores [frame_ide append(bor); frame2masks [frame_id] (mpennd(mask) aapmmea....ee.ameaammmass
end
8 Function PophhsstDensseProoppPPaame
// Find frame f" with largest mask area
f f- arg max, Σ frame2makss(f]]
// Assign object IDs starting from oid for each bbox in foreach box in frumeebboxes//
10
valid_prompts [[]] [oid] <- box; oid 1;
12
end
// Remove prompts from process list
13
frame2bboxes. pop((")) frame2masks.pop(
return f*, valid_promptslff) end
Function RemovaDupp1cattePrompt (video_ segeeents))mmmss foreach frame_ia id in wideo_segment do if frame_ii id not in frame2masks then
17
18
19
continue
20
end
Compute IoU between predicted and prompt masks in frame_id;
// Filter prompts already covered
Remove overlapping segments from frame2bboxes and frame2maske
23
end
end
A.2
Object Trajectory Generation
We aim to generate object trajectories from videos using Segment Anything 2 (SAM2). A key challenge is discovering new objects that do not appear the first frame, which not natively supported by SAM2. To address this, we design an itcrative algorithm that identifies the next frame most likely contain new object and propagates its mask throughout the video, as illustrated in Algorithm [ To further leverage concepts generated by GPT-4, we extend this algorithm to incorporate arbitrary frame-based bounding box generators, such as Grounding DINO (56) and YOLO (77. This extended version uses prompt scheduler that prioritizes frames containing the highest number of grounded objects and iteratively propagates them across the video, as shown in Algorithm 2 and Algorithm
25

---

Algorithm 3: Mask Generation via Frame-wise Bounding Box Grounding
Input: Grounding model 9, SAM predictor P, SAM mask generator M, video frames
{Jo....,7y label set
Input: Box and text thresholds 7b, Tr, target FPS, max propagation steps
Output: Per-frame mask segments video_ segments, object-to-class mapping oid_pred Initialize prompt_memos
+ PromptBuffer()
// Step 1:
Grounding boasdddng-based object detection
2 foreach frame I in video do bboxes = G C); video_boxes ff bboxes;
end
// Step 2: Generate masks for all bounding boxes foreach (f, bbozes) E video_ bozes do
7 masks = M bboxes);
foreach (box, mask) zip( bbozes, masks) do prompt_memory add add feadd_promppfffffe boz, mask)
end end
// Step 3: Iterative propagation from dense prompt frame popp detosssdeeee_fid_proooo
Ssatt prompt <- wwtpppttmmmmmmmmee
13
video_segrnn adeo_ssements- + 0, +
while prompt t * and faar * -1 and < k do
15
Reset predictor state 5 t So;
foreach (f. fwwffaammmppppeeeee foreach (o, b) E
E --ppppt_mmmoooee valéd_prompte do
17
do
18
Add prompt (o, b) frame f' into with state S:
end end 20
19
Propagate forward
foreach (f. {o, (4]) E P.propagate(S) do video_ saadoo_sgmmetsss1-44 binarized masks {4 0} mapped to o, end
23
// Propagate backward
foreach (J, (o:), (4:) (. PP.prooaggateS.. reverse True) do [ideo_segments[[] binarized masks {4; > 0} mapped to o; end
27
httrr_mmmrorrrrree_duuuppp www.reeeduppprrneeeeeeeeeee hitpooreeduppppoopppcccoooeeeeee seggeette
fstant prompt + prompt_memori pwp__osttdeeepppeeeeepppppppppee
29
++++l end return
A.3
Concept Generation
To construct low-level supervision from high-level captions, we leverage GPT-4 to generate spatio- temporal specifications. To mitigate potential hallucinations from the language model, we design compiler that verifies the validity of the generated programs. We few-shot prompt with three examples.
Prompt 1: Role Definition
You are super user in logic programming. You are also an expert at structured data extraction.
Prompt 2: General Task Instruction
You will describe the event length and location in both natural language and fraction of the video.
26

---

The natural language description of the locations in the video can be: early, mid, late.
The natural language description of the durations of the event can be: long, medium, short
Examples of precise video locations: [1/4, 1/2], [2/3, 1]. Examples of event durations: 1/4, 2/3, 1.
Prompt 3: Few-shot Examples
Listing 1: "One of the few-shot example for concept extraction"
Caption: A man carries child and walks to the left from behind a woman holding another child. Video_ID: "Obe30efe",
Action json:
{"video_id"; "Obe30efe", "sequential descriptions": "man carry child B, women C hold child D. man is behind women C", "man A walk", "man A at left"
"time stamps":
{ "1": { "description"; "man A carry child B", "women C hold child D". "man is behind women C" ],
programmatic": [ "carrying(A, B)", name man)",
name (B, child) "holding(C, D)". "name women)
name (D, man)", "behind C)", j. "duration": "short", "duration precise": "1/4" "video location"; "early", "video location precise": "[O, 1/4]" }, "2": { description" "man walk 1, "programmatic":
"salk (A)", 1, "duration": "medium", "duration precise": "1/2 " "video location": "mid", "video location precise": [[14 3/4]" }, "3": { "description":
"man A at left" 1. "programmatic"; [ "A) ], "duration": "short", "duration precise": "1/4", "video location": "late", video location precise"; "[3/4, 1]"}}}
Prompt 4: Concept Definition
Note all the relations are name, unary binary. name relation takes in two arguments, the first is always a variable, and the second argument could be noun ("apple"), noun phrase ("ancient_ (orccient_bbidding).. location ("ationn"dar__forett" etc. For example "name (A, "apple")" means the variable refers to an apple. Please ensure no space occur in the second argument.
unary relation takes in one variable as its argument. For example, close (A) means is close to the binary relation takes in two variables as its arguments. For example, above (A, B) means The entity in the binary and unary relation are variables in the form of capitalized letters (A, B).
above
The predicate of the unary relation can be adjectives, verbs, and name
The predicate of the binary relation can be preposition, and verb.
Please include the name relation any time if applicable. Please make the preposition and adjectives into separate two relation.
27

---

For each time stamp, please only describe the events that are happening at the same time, if any sequential events occur, put them into multiple time stamps.
For example, instead of 'person enter from left, person A walk to center, person A move to couch in one time stamp, put it into three different time stamps: "person enter from left", "person A walk to center", "person A move to couch". Please only describe one single event in sequential description per time stamp.
Please use as many relations as possible to precisely describe the action.
Please generate the action json programs for the following captions in the following format: {"actions": {caption_id: action json programs}} IMPORTANT: Please REMOVE all new line characters and extra spaces in the generated json!
28

---

50% %
Youcook2
6.40%
86.99%
1.47%
NextQA
4.05%
Youtube
Charades
Figure 11: Video sources for
A.4 Dataset Statistics
We present statistics ESCA- Video-87K through three perspectives: the composition of video sources, the complexity of extracted specifications, and the word clouds of associated concepts. We selected 0-30 second clips from the VOEVV--VIEO-IIA 178K dataset 198), with the resulting source distribution shown in Figure II Our extracted spatio-temporal specifications exhibit considerable complexity. As illustrated in Figure [2] single specification can contain multiple names, relations, and events. In addition, actions, and relations. In total, our dataset includes 220,905 unique names, 57,930 unique actions, and 35,415 unique relation keywords.
actions, visualize the vocabulary diversity using word clouds for names,
2000
2300
5000
(a) Name Count
(b) Action Count
152
1903
1000
(c) Relation Count
(d) Event Count
Figure 12: Distribution of Names, Attributes, Relations, and Events
29

---

water
scene textetttt
cor door
persor
individdu
bed
player
player
stend up
walking
view
speaking
backe Kg
unniin
S
stand
Fondd
with
upo
6.0
PD
Wit on
pack tar
g
cuir scr een table
hand
charactera
woman
phone
group
objectt
den
ameraa
lin man
F100
ges turing
savint
disppl Saplly vusture
Visible
ext t sun
Soungwale wwalk sitting
ges
a appea
spak
Lign
Finter
weal
mo
Q
sitting
wear
ng
on
r
closer transition
(a) Name
(b) Action
(c) Relation
Figure 13: Word cloud visualizations of the top-50 most frequent names, attributes, and relations.
B SGClip Training Details
We build our training pipeline following the LASER work [34], which uses a approach to align video with its caption. Note that the caption used in SGClip is also generated from the video with multi-modal large language model (GPT-4o), and this leads to all labels are generated from the video itself, thus, consider this approach as a model-driven self-supervised method. In this section, we elaborate on the details of how SGClip is trained and providing details in the training procedure.
B.1 Compute
All our experiments are carried out on a device with (1) 128 32-core Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (2) 10 NVIDIA H100 PCIe GPUs SGClip takes 10 days to finetune on the 87K datapoints for 3 epochs.
B.2 Loss
We train the SGClip with three different losses: contrastive loss, temporal loss, and semantic loss.
Contrastive Loss. To help SGClip distinguish between concepts that appear in video and those that do not, we adopt contrastive loss design. For each video V. we pair it with its corresponding specification é and randomly sample an unrelated specification d' from the dataset. The objective is to maximize the alignment score between the matching pair (V, Ø) toward 1, while minimizing the score between the mismatched pair (V, ø') toward O.
key challenge arises from the fact that specifications are exceptionally long-each containing an average of 8.24 events per dntapoint-making it computationally expensive to ground the entire specification in the video. To address this, we introduce chunked event training strategy. Each specification is divided into smaller chunks containing at most 3 events, and we align each chunk with the full video. When sampling mismatched pairs, we similarly draw from these smaller chunks to maintain alignment granularity and training efficiency.
Temporal Loss. To improve alignment accuracy between each event and its corresponding location in the video, we query GPT for a fine-grained estimation of the event's temporal span. For all satisfiable event sequences, we assign higher rewards to those aligned with the desired temporal location, and reduced rewards to those outside the target range.
Semantic Loss. We further incorporate common-senss negations to improve the model's understand- ing of what is unlikely to occur in a given scenario. For example, if a scene is outdoors, is unlikely to contain a bed; if person is "standing," they are unlikely to be "sleeping" at the same time; if person is "holding" an object, is improbable that the object is "hitting" the person simultaneously.
To introduce this notion of negation, we first identify the top 5, 000 most frequent keywords across names, actions, and relations. Using word vectors from SpaCy (29, we compute 50 most semantically distant keywords within this subset for each current scenario. During training, we sample keywords per category (name, action, relation) from these 50 distant terms to compute a semantic loss that penalizes implausible pairings.
30

---

B.3 Hyperparameters
We use a learning rate of 10-6 and a batch size The video sampled at target frame rate of FPS. For the semantic loss, we sample 5 negative keywords per instance and set the semantic loss weight to 0.1. In the provenance setting for Scallop. we use difftopkproofs with a top-k value of 3 for proof extraction. We fine tune from the CLIP model with a total of 3 epochs, and we evaluate and ensemble on the best performing models from different epochs.
31

---

Subset
Model
Strategy
Avg
Base
Common
Complex
Visual
Long
Base
55.00
60.00
51.67
40.00
30.00
47.33
IncemVL-25533B-MMPP
GD
60.00
61.67
56.67
48.33
11.66
47.67
+ ESCA
58.33
61.67
53.33
58.33
26.66
51.66
Base 56.67
46.67
48.41
36.67
15.00
40.68
Gemini-2.0-ffasi
+ 55.00
53.33
50.00
38.33
6.00
40.53
+ ESCA
56.67
41.67
43.33
46.67
21.66
42.00
Base 58.34
48,30
48,30
36,70
33.33
44.99
Qwen2.5-VL-72B
+ GD
65.00
55.00
58.00
43,33
20.00
48.27
+ ESCA
60.00
48.33
60.00
46.67
31.67
49.33
Base 55.00
60.00
58.33
60.00
23.33
51.33
GPT-4o
GD
63.33
63.33
65.00
53.33
21.66
53.33
+ ESCA
66.67
62.00
63.33
55.00
26.67
54.67
Table 1: Detailed performance EB-Navigation, decomposed by subsets of tasks.
C Additional Experimental Results
C.1 EmbodiedBench: Navigation
We demonstrate that ESCA enhances agent performance in navigating to target objects embodied benchmarks, as shown in Table [I] Across four base multimodal large language models, ESCA consistently outperforms both the base models and variants using only Froonddnng-DINO We outline the design of the transfer protocol and provide qualitative studies in the following section.
Transfer Protocol: Grounding Dino The overall pipeline of ESCA with Grounding DINO consists of three main steps: concept extraction, object identification, and visual summarization and validation. Rather than performing full scene graph prediction, we directly use Grounding DINO to extract candidate target objects based on their name and attributes. Each concept input to Grounding DINO follows the format <attribute> <object name>, such "a grey rectangular object." The ESCA pipeline then passes the augmented image and the newly constructed query back to the MLLM to predict subsequent actions. Note that, based on study, Grounding DINO tends generate false positive candidate objects, which leads to performance drop on the long-horizer horizon subset. This the only task subset where the target object is not visible in the initial scenario. The box threshold for grounding DINO is 0.2, and the text threshold 0.1.
ESCA The overall pipeline of ESCA with SGClip consists of four main steps: concept extraction, object identification, scene graph generation, and visual summarization and validation. We begin by using Grounding DINO to extract candidate target objects and their related objects based on object names. Next, SGClip predicts the attributes and relationships of the identified objects and aggregates predictions across names, attributes, and relationships. To ensure precision, we select only the top-1 object identified as the target and apply a confidence threshold of 0.3. For Grounding DINO,
Transfer Protocol:
box threshold of 0.1 and text threshold of 0.1.
We present two tasks to qualitatively analyze performance on the EB-Navigation environment. Each task is solved using three configurations: GPT-4o alone, GPT-4o augmented with Grounding DINO, and GPT-4o augmented with ESCA. Compared to GPT-4o alone, both Grounding DINO and ESCA provide additional information that aids task completion. However, ESCA results in significantly fewer false negatives than Grounding DINO, leading to improved performance on long-horizon tasks.
Qualitative Studies:
32

---

Instruction: navigate to the Pillow the room and be as close as possible to it.
fails target "pillow" the room continues rotating without making progress.
GPT-4o GD: misidentifies robot's shadow as "pillow."
GPT-4o + ECSA: identifies the "pillow" correctly.
Instruction:
Get near the cylindrical object with smooth surface and handle on the side.
GPT-4o: fails locate target "cylindrical object smooth gray surface and handle the side" in the and continues rotating without making progress.
a
a
P
PM
misidentifies picture "cylindrical object with smooth gray surface on side".
a
a
GPT-4o ECSA:
Found the "cylindrical object with a smooth gray surface and handle on the side" at the first frame.
success
Figure 14: We present two qualitative examples comparing the original MLLM (GPT-40), its variant augmented with Grounding DINO, and the full ESCA pipeline. Our observations show that ESCA improves the precision and quality of target object recognition.
33

---

Algorithm 4: Entropy-Guided Assignment of Entity Names to Objects
Input:
Objects {01..... )
Entities E = fei..... 6m (with possible repetitions);
Assignment scores S O E IR, So e) denotes the score of assigning entity e object o
Output: Assignment map A O - E, assigning one entity name to each object
1
Initialize assignment map + 0;
2 Initialize name availability counter C: E N. how many times each entity name appears in E;
3 while there exists unassigned object O do foreach unassigned object o E 0 do
4
Let D. (S(o,e) Cle) 0}
// Available assignment
6
end
Select object o" arg min, Entropy (o):
// Most confident object // Highest scoring available name Assign A t n² and decrement C(n*] + C[n']
8
Let = arg max,e E:C|])so ,(o',e)
end return A;
Figure 15: Entropy- -uiided assignment algorithm that maps entity names to objects based on assign- ment scores and availability. At each step, it selects the object with the lowest entropy valid assignments, then assigns the highest scoring available entity. This process continues until all objects assigned.
Subset
Model
Strategy
Avg
Base
Common
Complex
Visual
Spatial
Base
10.42
25.00
20.83
27.78
12.50
19.31
LntemLLL2.5-38D-MPPO
YOLO
14.58
14.58
25.00
19.40
22.92
19.30
ESCA
31.25
21.00
14.58
27.78
27.08
24.30
Base
10.42
10.42
8.33
11.11
18.75
11.81
Gemini-2.0-flash
YOLO
14.60
8.30
14.60
13.90
31.30
16.54
ESCA
18.75
21.00
20.80
22.22
27.08
21.94
Base 2.08
6.25
8.33
4.16
2.78
4.72
Qwen2.5.VL VL--2.5-VL-72BB-A
YOLO
18.80
20.80
4.20
8.30
14.60
13.34
ESCA
18.80
10.41
16.67
16.67
10.42
14.59
Base
16.67
25.00
20.83
35,41
19.44
23.47
GPT-4o
+ YOLO
39.60
29.20
29.20
19.40
25.00
28.48
ESCA
33.33
31.25
37.50
38.89
31.25
34.44
Table Detailed performance on E.B-Manipulation, decomppsee by subsets of tasks.
C.2 EmbociedBench: Manipulation
EB--anniuulaiion evaluates an agent's ability to perform fine-grained object manipulation using visual input and language instructions. Unlike typical embodied AI settings, it provides full 3D spatial information (X, Y, coordinates) for all objects, removing the need for spatial inference and shifting the focus semantic grounding and planning.
To construct structured object representations, we use the provided coordinates to generate point- based prompts for SAM 2. which produces segmentation masks that are converted into bounding boxes. Each object is annotated with a unique integer label starting from we then prompt a vision-language model (VLM) to generate textual descriptions in the form <color> <object> (e.g.. "yellow star"), using handcrafted templates to ensure consistency and match the number detected objects.
Since the VL.M outputs are unordered, we use ESCA, which treats predicted entities as categorical labels and outputs probability distributions over them for each bounding box. To resolve the corre-
34

---

spondence between entities and boxes, we apply an entropy-basee matching algorithm (Figure 3). which iteratively assigns the most confident (lowest-entropy) pairings while enforcing uniqueness constraints. The resulting assignments are used to generate a structured natural language description (e.g., "Object is yellow star"), which is appended the original prompt and passed the VLM to support precise and context-aware manipulation planning.
Qualitative Studies We present two tasks qualitatively analyze performance on the EB- Manipulation environment. Each task is solved using three configurations: GPT-4o alone, GPT-4o augmented with YOLO, and GPT-4o augmented with ESCA. Compared to GPT-4o alone, both YOLO and ESCA provide additional information that aids task completion. However, ESCA results more precise recognition than using YOLO alone.
Error Analysis As shown Figure 21 we conducted an error analysis on the EB-Manipulation task. For each subtask category, randomly sampled 10 erroneous tasks from both the GPT-4o baseline and GPT-4o augmented with ESCA. The results show that ESCA significantly reduces the perception error component.
"The three top-level eroor types are Perception, Reasoning, and Planning. The second-level errors are Hallucination, Wrong Recognition, Spatial Understanding. Spatial Reasoning, Reflection Error, Inaccurate Action, and Collision. For clarity, the figure these acronyms label the different types.
35

---

Instruction: Pick the star and place it into the red container.
GPT-4o: From left right, teal triangular silver purple container, an orange cylinder, and red container. x
YOLO: right, triangular prism, gray star, purple container, red container and orange cylinder. x
GPT-4o ECSA: left right, a grey teal triangular prism, purple container, red container, and cylinder. /
success
Instruction: Stack the lime moon and the gray in sequence.
GPT-4o: From right, can moon, blue moon a gray moon, black moon. First fetches for the related blue moon, then stack the lime grey
GPT-4o YOLO: From left right, lime gray moon, blue green *
GPT-4o ECSA: From left to right, can see green moon, gray moon, a blue moon, and lime moon.
success
Figure 16: We present two qualitative examples comparing the original MILMM (GPT-4o), its variant augmented with YOLO, and the full ESCA pipeline. Our observations show that ESCA improves the precision and quality of scene graph recognition.
36

---

SU
Perc
SU
GPT-4o ESCA
Perc (40%))
GPT-4o
LA Plan. (50%)
(62%)
( (14%)
WR
Reak (10%)
Figure 17: Error decomposition of GPT-4o with and without ESCA, based on manual inspection of 50 EB-Manipulation tasks, with 10 randomly sampled from each subtasks. As we can see, ESCA has reduced perception error by large margin.
Subset
Model
Strategy
Avg
Base
Common
Complex
Visual
Spatial
Long
Base
32.00
20.00
20.00
18.00
20.00
50.00
26.67
Intern MieemVLL-2.-38B-MM
GD
28.00
36.00
34.00
32.00
30.00
47.83
34.64
ESCA
43.00
32.00
46.00
36.00
24.00
50.00
38.50
Base
68.00
58.00
52.00
46.00
42.00
52.00
53.00
Gemini-24 f.0-fai-.0-llash
GD
68.00
58.00
52.00
46.00
42.00
56.00
53.67
ESCA
70.00
54.00
56.00
54.00
44.00
50.00
54.67
Base
50.00
42.00
42.00
36.00
34.00
34.00
39.67
Qwen2.5-VL-77B-mmm
GD
52.00
38.00
50.00
46.00
40.00
42.00
44.67
ESCA
46.00
36.00
54.00
47.00
37.00
30.00
41.67
Base
64.00
48.00
62.00
46.00
50.00
54.00
54.00
GPT-4o
GD
52.00
46.00
62.00
48.00
50.00
56.00
52.33
ESCA
62.00
56.00
54.00
53.00
52.00
50.00
54.50
Table 3: Detailed performance EB-Alfred, decomposed by subsets of tasks.
C.3
EmbodiedBench:: Alfred
EB-Alfred evaluates an agent's ability to perform household tasks requiring sequential action planning and semantic grounding realistic indoor enviromments. The agent must interpret natural language instructions, navigate through scenes, interact with objects, and complete multi-step tasks such as arranging objects, cleaning, preparing items.
Tasks are divided into several subcategories: common sense tasks require inferring unstated object properties goals, complex tasks involve longer action sequences, visual appearance tasks require identifying objects by visual attributes, spatial tasks demand precise spatial reasoning, and long horizon tasks require managing extended sequences of actions. The agent must parse instructions, identify target objects and their states, explore the environment, and execute appropriate action sequences achieve the specified goals.
Transfer Protocol: Similar EB-Navigation, the overall pipeline of ESCA with SGClip consists of four main steps: concept extraction, object identification, scene graph generation, and visual summarization and validation. We begin by using Grounding DINO extract candidate target objects and related objects based on object names. Next, SGClip predicts categorical labels and spatial relationships for the identified objects, aggregating predictions across detections. To ensure precision, we retain only the most confident bounding box for each entity in the scene, applying confidence threshold of 0.3. Finally, we construct a structured scene description incorporating the identified objects, their positions, and their spatial relationships, and pass this augmented prompt along with the annotated image back to the MILMM to predict subsequent actions. For Grounding DINO, we use box threshold of 0.15 55 and text threshold of 0.1.
37

---

Instruction: Relocste written information fumiture studying sleeping
GPT-4o: tries find newspaper (which doesn't exist), generates empty plan and prematurely.
GPT-4o GD: stuck much langer failure loop, variations of the failed approach
completes
steps
Instruction.Puting away yellow
GPT-4e: Successfully places first knife but fails locate the second knife at DiningTable and continues repetitive without making progress.
Successfully
knife DiningTable and repetitive loop without making progress.
GPT-4o ECSA:
Successfully places first knile, navigates to DiningTable to retrieve second knife, and completes the task.
Figure 18: We present qualitative examples the Alfred task, comparing the original MILM (GPT-40), its variant augmented with Grounding Dino, and the full ESCA pipeline. Our observations show that ESCA improves the over all success of the task.
38

---

Ha
WR
Pere: (19%)
Pent: (29%)
WR
Plan (42%)
GPT-4o + ESCCA
(10%)
Reas
RE
GPT-4o
(71%)
Reas. (29%)
RE
Figure 19: Error decomposition of GPT-4o with and without ESCA, based on manual inspection of 60 EB-Alfred tasks, with 10 randomly sampled from each subtask. ESCA reduces perception errors (Ha, WR) and reasoning errors (RE, SR), shifting failures to planning stages where action sequencing can be refined.
Subset
Model
Strategy
Avg
Base
Common
Complex
Visual
Spatial
Long
Base
92.00
24.00
64.00
46.00
30.00
32.00
48.00
Intern MMtmmWL-2.5-38B-MMPP
GD
84.00
36.00
66.00
52.00
32.00
40.00
$1.67
ESCA
94.00
46.00
62.00
54.00
34.00
40.00
55.00
Base
66.00
20.00
32.00
32.00
20.04
10.20
30.04
Gemini-2.0-flaan
GD
68.00
18.36
32.00
24.00
20.41
10.00
28.80
ESCA
74.00
22.00
35.14
36.00
22.00
14.00
33.86
Base
58.00
18.00
42.00
40.00
24.00
18.00
33.33
Qwen2.5-VL-72B-Ins
GD
52.00
50.00
68.00
70.00
21.00
36.00
49.50
ESCA
86.00
50.00
72.00
60.00
36.00
34.00
56.33
Base
92.00
46.00
52.00
66.00
32.65
59.18
57.97
GPT-4o
GD
86.00
58.00
62.00
76.00
32.00
34.00
58.00
ESCA
88.00
62.00
58.00
72.00
32.00
48.00
60.00
Table 4: Detailed performance on EB- -B-Haaitat, decomposed by subsets of tasks.
Qualitative Studies: We present two tasks qualitatively analyze performance the EB-Alfred environment. Each task is solved using three configurations: GPT-4o alone, GPT-4o augmented with Grounding DINO, and GPT-4o augmented with ESCA. Compared to GPT-4o alone, both Grounding DINO and ESCA provide additional information with the bounding boxcs. However, ESCA demonstrates superior spatial reasoning and plan adaptation, successfully completing both tasks where the baseline and Grounding DINO fail due to repetitive loops or premature termination.
C.4 EmbodiedBench: Habitat
EB- CB-Habitat evaluates the agent's high-level task decomposition and planning capabilities. The agent performs high-level actions to complete the given task in environment that is interactively navigatable. Most tasks involve finding an object and moving it to a certain location with slightly the subcategory of the task (eg, common sense subtask only specifies what the target object will be used for without naming the specific object itself, long horizon subtask specifies multiple target objects, requiring the agent to perform multiple sequences of actions to complete the task.) Overall, the agent is required to identify the target object from the instruction, explore the environment to find the object, and correctly move all the objects to the instructed location.
different nuances in the formulation of the instruction depending
'The three topelevel error types are Perception, Reasoning, and Planning. The second-level errors are Hallucination, Wrong Recognition, Spatial Understanding, Spatial Reasoning, Reflection Error, and Inaccurate Action. For clarity, the figure these acronyms label the different error types.
39

---

Instruction: Prepare a game by delivering something to play with to the brown table.
GPT-4o: unable identify objects repeatedly performs invalid actions location.
target [brown table) concludes completed.
GPT-4o ECSA: correctly identifies that the table in sight is not the target table.
Instruction: Relocate every ball from left counter anto the black table.
GPT-4a: unable identify objects and repestedly performs invalid actions location.
GPT-4o GD: misidentifies the target object (ball) and the target location (black table) and concludes that the task been completed.
GPT-4o correctly distinguishes target object location, and continues perform the remaining actions complete the task.
Figure 20: We present two qualitative examples on habitat task comparing the original MILMM (GPT-40), its variant augmented with Grounding Dino, and the full ESCA pipeline. Our observations show that ESCA improves the over all success of the task by providing more detailed attributes in the description.
40

---

Plan
GPT-4o ESCA
Pere. (33%)
Reas, (50%)
Reas. (60%
GPT-4o
Porc. (38%)
WR
K.
Figure 21: Error decomposition of GPT-4o with and without ESCA, based manual inspection of 50 BB-Haaitat tasks, with 10 randomly sampled from each subtasks.
Transfer Protocol:
Grounding Dino Likewise EB-Navigation, we use the three-step pipeline: concept extraction, object identification, and visual summarization and validation. The objects of interest are extracted from the instruction entities with their corresponding current and target states. As EB-Habitttt tasks are mostly comprised of moving an object from one place to another, the object states are expressed as a relation between the object and the location in most cases. The raw object name is input to Grounding DINO and hence used to augment the image with bounding boxes. The thresholds used for Grounding DINO are: threshold 0.15, text threshold 0.1.
ESCA Likewise to EB-Navigation, we use the four-step pipeline, where SGClip generates the probability distributions over each entity, keeping only the most confident bounding box in the scene. We apply confidence threshold of 0.3, and use a box threshold of 0.15 and a text threshold of 0.1 for Grounding DINO.
Transfer Protocol:
Qualitative Studies We present two qualtiative examples to demonstrate the effects of ESCA. Figure compares the inference steps of GPT-4o alone, GPT-4o augmented with Grounding DINO, and 44 augmented with ESCA. In both examples, GPT-4o is unable to identify the object in sight while GPT-4o with Grounding DINO misidentifies the target state as being complete and early terminates the episodes. On the other hand, ESCA is able to filter out incorrectly identified objects and allows the model to realize that the task is not complete and hence continue its actions complete the task.
"The three top-level emor types are Perception, Reasoning, and Planning. The second-level errors are Hallucination, Wrong Recognition, Spatial Understanding. Spatial Reasoning, Reflection Error, Inaccurate Action, and Collision. For clarity, the figure uses these acronyms label the different error types.
41

---

k=1
k=5
k= 10
Model
Prec
Rec
Pree
Rec
Pree
Rec
SGCIip-CLIP
0.469 (3))
0.085 (75)
0.321 (70)
0.250 (m)
(0246
0.353
SGClip
(.4
0.087 (00)
0.350
0.270
0.278 (45)
0.385 (55)
Table 5: Maximum test precision k and recall @ on the VidVRD dataset, evaluated every five epochs during 50 epochs of finetuning. Each cell shows the score followed by the epoch at which that score was achieved.
C.5 Downstream Task: Scene Graph Generation
We further analyze the abilities of SGClip by applying it to the task of relation tagging of scene graphs. this task, the ground truth bounding boxes objects are provided, and the objective is to correctly classify their respective object class as well as the binary predicates between them. To demonstrate SGClip on this task, we use the VidVRD dataset, comprised of 1,000 videos with 35 object categories and 132 predicate categories.
For our evaluation, finetune versions of SGClip on the VidVRD training for 75 epochs: the first is the SGClip architecture initialized to the original CLIP backbone, denoted as SGClip-CLIP; the second is SGClip. The only difference between the two is that the latter has had training from the neurosymbolic learning pipeline denoted in 3.3 We next test these models on 196 videos from the test (4 are ignored for causing an OOM error on single H100 in both models.) This test evaluation happens every five epochs; we report the maximum test precision k and recall @k over all test evaluations for each evaluation in table 5 As the table shows, SGClip outperforms SGClip-CLIP across both metrics for l = 1, 5. 10, demonstrating that the proposed neurosymbolic training pipeline provides an advantageous initialization for relational tagging finetuning on VidVRD.
42

---

Category
Model
ActivityNet Accuracy (%)
SGClip
76.34
CLIP
74.37
Zero-shot
BIKE
80.00
Tex14vis
77.40
ResT
26.30
E2E
20.00
SGClip
80.10
Few-shot (1%)
CLIP
78.79
SGClip
86.05
Few-shot (5%)
CLIP
80.02
Fully finetuned
IntemVI.-66
95.90
Table 6: ActivityNet accuracy for zero-shot, few-shot, and fully finetuned models. Zero-shot includes both external baselines and our models evaluated without training.
C.6 Down-stream Task: Action Recognition
To evaluate the generalization and finetunability of our model beyond structured scene graph under- standing, we consider the task of action recognition, where each video is annotated with a single activity label. We use a combined version of ActivityNet 1.2 and 1.3, which together span 200 unique action classes and approximately 20,000 untrimmed videos across training, validation, and test splits. The action categories range from simple atomic activities such as swimming and rock climbing to more complex, composite tasks like starting campfire performing basketball layup drill.
Similar to Embodied Bench, adopt a transfer protocol tailored to the action recognition setting. For each second of the video, we generate the top-4 predicted actions along with their confidence scores. We then apply thresholding to discard low-confidence predictions, followed by temporal smoothing to aggregate results into single continuous segment per video. The final action label is selected based on a weighted combination of its average confidence score and temporal frequency. The output is a single scored segment per video, defined by the merged temporal boundaries of the selected label.
The results, summarized in Table 3, demonstrate the strong fine-tuning capability of ESCA under varying supervision levels. ESCA consistently outperforms CLIP across 0%, 1%, and 5% training subsets, highlighting its superior ability to recognize fine-grained actions. Remarkably, ESCA achieves 92.0% accuracy using only 5% of the training data (approximately 800 videos), approaching the performance of Intern VL-6B, which is trained on the full dataset. This indicates that ESCA generalizes effectively from limited supervision, whereas Intern VL-6B's 's advantage stems in part from its exposure to the entire training corpus.
43


Reviews:
[{"review_id": "vNpkoCenep", "reviewer": "Reviewer_SqXB", "summary": "The paper integrates scene graph descriptions with agents to describe object relationships, providing structured alignment between visual contexts and textual semantics. It proposes a CLIP-based, open-domain, and promptable model to generate scene graphs, and an Embodied and Scene-Graph Contextualized Agent to support MLLM-based embodied agents. A novel ESCA-Video-87K\n dataset is proposed, including object traces, open-domain concepts, and programmatic specifications for 87K video-caption pairs.", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Introducing scene graph descriptions into agents utilises spatial and semantic contexts in visual and language information, therefore facilitating the fine-grained operations.\n2. The new ESCA-Video-87K dataset with video-caption pairs contributes to exploring open-domain scene graph descriptions. \n3. The embodied agent pipeline augmented with the proposed ESCA provides a solution for accurate agent perception.\n\nWeaknesses:\n1. The paper has limited novel content for embodied agent or scene graph generation. For embodied agent construction, the only difference is introducing scene graphs, which is straightforward and cannot capture the outstanding part. For scene graph generation, the proposed SGClip is only a fine-tuned CLIP with the proposed dataset. \n2. The paper stresses the superiority of open-domain perceptions but lacks more illustrations and experimental analysis. It only performs the open-domain comparisons with closed-set SGG datasets, missing the comparison with other open-domain SGG methods. \n3. Actually, how to integrate SGG with the agent is not well described. The relationships in SGG are not shown in a qualitative manner, especially in Fig. 1. The Authors would better clarify how spatial and temporal relationships are integrated into the agent.", "was_revised": true, "final_justification": "Thank you for the long response and illustrations. The authors have carefully addressed my concerns. After reading all the reviews and responses, I would raise my rating to borderline accept.", "rebuttal": "> **SqXB-Q1. Novelty: the only difference is introducing scene graphs,  which is straightforward and cannot capture the outstanding part …  the proposed SGClip is only a fine-tuned CLIP with the proposed dataset.**\n\nWe clarify that the introduction of scene graph for the embodied agents is non-trivial, leading to our novel contributions of (a) selective grounding, (b) design of transfer protocol, and (c) the fine-tuned model SGClip and its training dataset ESCA-Video-87K which can realize the prior goals.\n\n**Selective Grounding.** Rather than injecting full scene graphs indiscriminately, which we show in the COT-SG experiment, leading to degraded performance, we focus on presenting only the most relevant information to guide the MLLM in planning. Specifically, we use the MLLM to identify the objects, attributes, and relationships that are most pertinent to the instruction, and then determine which key objects are essential for task completion with symbolic execution. For navigation, this is the object the agent must locate; for manipulation, it is the precise (x, y, z) position mapping to the desired object feature.\n\n**Design of Transfer Protocol.** We define the transfer protocol as the process of leveraging concepts extracted by the MLLM to construct prompts enriched with scene-specific information. The protocol filters and aggregates relevant scene elements tailored to the agent’s goal. For this purpose, it performs probabilistic reasoning over object names, spatial relationships, and attributes to identify the top-k most likely target objects. This fine-grained inference is enabled by our probabilistic scene graph design, which encodes SGCLIP predicted likelihood. This mechanism, detailed in Appendix C, is a technical contribution that goes beyond simply plugging in a scene graph module.\n\n**SGClip and ESCA-Video-87K.** Regarding the novelty of SGCLIP, we emphasize that the contribution lies not in architectural innovation but in the data collection and training strategy. SGCLIP is trained on a large-scale, weakly-supervised video dataset generated entirely via multimodal LLM prompting, described in Appendix A. Our approach overcomes the scarcity of manually labeled scene graph annotations in video by enabling scalable, diverse, and compositional weak supervision at scale, requiring only raw video as input. SGCLIP thus represents a practical and generalizable solution for scene graph grounding in open-world tasks.\n\nTo the best of our knowledge, ESCA-Video-87K is the largest automatically labeled video scene graph dataset featuring rich object semantics, relations, and actions. Most other annotated SG datasets are much smaller. For comparison:\n- **Ego4D-EASG** includes 221 long videos annotated with 407 object classes, 219 verbs (action types), and 16 spatial/prepositional relation types.\n- **OpenPVSG** contains 400 videos, covering 126 object classes and 57 relation types.\n- **BDD-100K**, the largest densely labeled video dataset, focuses solely on driving scenarios. While it offers 100K videos with object trajectories, it includes only 10 object classes and lacks relation and action annotations.\n\nIn contrast, ESCA-Video-87K features automatic annotations of diverse object names, inter-object relations, and object actions across 87K videos. Human-labeled fine-grained datasets like OpenPVSG are limited in scale due to the complexity of dense annotation, highlighting the scalability advantage of our weak-supervision pipeline.\n\n> **SqXB-Q2. How does ESCA compare against open-domain SGG methods? and on open-set SGG datasets?**\n\n**Embodied Bench is an open-domain SGG benchmark**\n\nA key strength of the ESCA framework is its ability to handle open-domain natural language instructions, which often contain free-form and compositional descriptions not present in the simulated environments. For example, a banana might be described as a \"curved yellow fruit\" and a loaf of bread as a \"freshly baked baguette.\"\n\nThis open-domain capability is clearly demonstrated in our quantitative analysis of the EB-Navigation dataset. Using GPT-4o as the concept extractor, our pipeline generates an average of 5.57 free-form keywords per instruction, resulting in 84 unique target object descriptions across the dataset. This diversity significantly exceeds the 26 canonical object names defined in the ground-truth vocabulary. Beyond target objects, the process also produces 199 related objects, 296 relationships, and 117 attributes as keywords. This expanded vocabulary demonstrates how SGCLIP captures rich, unconstrained concept expressions that arise from both natural language variability in instructions (e.g., \"curved yellow fruit\" instead of \"banana\") and observations derived from the visual scene.\n\nTo further demonstrate generalizability across open-domain SGG datasets, we report performance results on EB-Alfred and EB-Habitat as follows:\n\n**EB-Habitat performance (NEW)**\n| Model                          | Raw Model | w GD   | w GD + ESCA |\n|-------------------------------|-----------|--------|--------------|\n| Intern-VL-2_5-38B-MPO         | 48.00%    | 51.67% | **55.00%**       |\n| Gemini-2.0-flash (Mar 2025)   | 30.04%    | 28.80% | **33.86%**       |\n| Qwen2.5-VL-72B-Ins            | 33.33%    | 49.50% | **56.33%**       |\n\n**EB-Alfred performance (NEW)**\n| Model                      | Raw Model | w GD   | w GD + ESCA |\n|----------------------------|-----------|--------|--------------|\n| Intern-VL-2_5-38B-MPO      | 26.67%    | 34.64% | **38.50%**       |\n| Gemini-2.0-flash (Mar 2025)| 53.00%    | 53.67% | **54.67%**       |\n| Qwen2.5-VL-72B-Ins         | 39.67%    | **44.67%** | 42.00%       |\n\n**Grounding-dino and CoT-SG are two open-domain SGG methods**\n\nTo empirically validate ESCA's open-domain capability, we compare against two baseline approaches. First, as reported in our main paper, we evaluate against a baseline that omits SGCLIP and relies solely on Grounding DINO for open-world object grounding (MLLM + GD). We further evaluate against **CoT-SG**, a Chain-of-Thought approach that prompts MLLMs to generate structured scene graph representations directly without any visual grounding. This method represents a direct generation baseline where the model produces scene graphs purely through reasoning rather than structured visual-textual alignment.\n\n**EB-Navigation Results**: ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.\n\n| Model                        | Raw Model | w GD    | w ESCA| CoT-SG (NEW) |\n|-----------------------------|-----------|---------|---------------|--------|\n| Intern-VL-2_5-38B-MPO       | 47.33%    | 47.67%  | **51.66%**        | 46.33% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 40.68%    | 40.53%  | **42.00%**        | 26.33% (NEW) |\n| Qwen2.5-VL-72B-Ins          | 44.99%    | 48.27%  | **49.33%**        | 33.99% (NEW) |\n| GPT-4o                      | 51.33%    | 53.66%  | **54.67%**        | 44.00% (NEW)   |\n\n**EB-Manipulation Results**: While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.\n| Model                    | Raw Model | w YOLO | w  ESCA| CoT-SG (NEW) |\n|--------------------------|-----------|--------|---------------|---------|\n| Intern-VL-2_5-38B-MPO    | 19.31%    | 19.30% | **24.30%**        | 20.00% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 11.81% | 16.54% | **21.94%**        | 17.60% (NEW) |\n| Qwen2.5-VL-72B-Ins         | 4.72%     | 13.34% | **19.04%**        | 17.33% (NEW) |\n| GPT-4o       | 23.47%    | 28.48% | **34.44%**        | 30.43% (NEW) |\n\n> **SqXB-Q3. how to integrate SGG with the agent and how spatial and temporal relationships are integrated into the agent.**\n\nWe illustrate how the ESCA framework integrates scene graphs through the transfer protocol for EB-navigation, as described in Appendix C. We will provide more information in a revised version. \n\nConcrete Example - EB-Navigation Task: This protocol combines the object name likelihood, attribute likelihood, and spatial relation likelihood to compute the probability of each object being the target. Consider the instruction \"navigate to the loaf of bread that is on the counter”, and three objects A, B & C. Our protocol processes this as follows:\n\n* Object Recognition:  A and B recognized as \"loaf of bread\" (0.8 likelihood) and \"counter\" (0.2). C recognized as \"counter\" (0.7) and \"bread\" (0.3).\n* Spatial Relations: SGCLIP predicts A is \"above\" C (0.9), B is \"above\" C (0.1)\n* Probabilistic Aggregation: We compute the aggregated likelihood using our three-factor formula:Target Likelihood = Object Confidence × Spatial Relation Confidence × Landmark Confidence. The target likelihood for A: 0.8 (bread confidence) × 0.9 (spatial relation) × 0.7 (counter confidence) = 0.504, likewise for B = 0.8 × 0.1 × 0.7 = 0.056. This calculation enables the agent to prioritize objects that not only match in name but also satisfy the instructed spatial context. \n\n**From Scene Graph to Action:** We pass the top-k targets downstream. Assume k = 1, Object A becomes the primary navigation target, with its bounding box coordinates and confidence score (0.504) passed to the motion planner. This probabilistic grounding enables robust decision-making under perceptual uncertainty. \nThough evaluated only on EB in static environments, our framework could generalize to dynamic environments with temporal relations. For \"watch the person cook pasta\", scene graphs would track action sequences with temporal edges, aggregating probabilities across time."}, {"review_id": "yTuJgNQrT0", "reviewer": "Reviewer_szsY", "summary": "The paper proposes the ESCA framework to provide MLLM-based agents with structured visual descriptions, scene graphs, which aids in their reasoning and planning. The core contribution is SGClip, a model trained using a self-supervised neurosymbolic pipeline that leverages CLIP to generate scene graphs without requiring human annotations. Experiments on two environments in EmbodiedBench demonstrate that the ESCA framework improves agent performance. The authors also provide code to ensure reproducibility.", "questions": "See above.", "limitations": "Yes. \n\nSection 7 (Conclusion and Limitations) provides a clear, direct discussion of primary limitations including latency, dependence on 2D visual inputs, and lack of formal verification during execution. The impact of synthetic supervision is not fully detailed and could be addressed more explicitly.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Novel and Scalable Self-Supervised Training Paradigm: The paper's key innovation is a scalable neurosymbolic pipeline that trains a scene graph model from video-caption pairs without human annotation. This elegantly solves a major data bottleneck in the field.\n2. Versatile and Generalizable SGClip: SGClip demonstrates strong performance not only within the ESCA framework but also in zero-shot and downstream tasks.\n3. Comprehensive Experimental Validation: The framework's effectiveness is rigorously tested on a diverse set of both open-source and proprietary MLLMs in multiple embodied tasks. The clear reduction in perception errors provides strong evidence for its utility.\n4. Community Contribution: The commitment to release the ESCA-Video-87K dataset and codebase is a valuable contribution that fosters reproducibility and future research.\n\nWeaknesses:\n1. Limited \"Open-Domain\" Capability in Practice: The paper claims SGClip is an open-domain model, but its application within the ESCA framework appears constrained. The candidate sets for attributes and relations seem to be pre-defined, and concept extraction relies on environment-specific textual feedback. This setup limits the method's generality and raises questions about its direct applicability to real-world scenarios that lack such structured feedback. \n2. Lack of Deeper Result Analysis and Case Studies: The paper reports a more significant performance gain on EB-Manipulation than on EB-Navigation but provides no explanation for this discrepancy. Furthermore, all qualitative examples and visualizations are from the navigation task. Including case studies for the manipulation task would be crucial for a better understanding of how ESCA functions in more complex, interaction-rich scenarios.\n3. Insufficient Ablation and A Critical Missing Baseline:\n    - Unclear Contribution of SGClip: The ESCA pipeline heavily involves MLLMs in multiple stages (concept extraction, summarization), making it difficult to isolate the precise impact of the SGClip module itself versus the overall structured prompting.\n    - Missing a Stronger Baseline: The training of SGClip can be seen as a form of knowledge distillation. A critical baseline is absent: **using a powerful MLLM to directly generate the scene graph zero-shot**, aided by grounding from visual modules like GD+SAM2. Comparing against this simpler, distillation-free approach is essential to justify the complexity and necessity of training a specialized SGClip model.\n4. Ambiguous Error Decomposition: The error analysis lacks clarity. The paper does not provide formal definitions for its error categories. Because most failures are compounded, and there are likely to be some errors in the trajectory of success.\n5. Inadequate Engagement with Relevant Literature: The paper's positioning within the field could be strengthened by a more thorough discussion of related work. It overlooks several highly relevant prior works that also address grounding and planning for LLM agents in similar AI2-THOR environments, such as LLM-Planner[1], MART[2], and P-RAG[3]. A comparative discussion with these methods would provide a clearer context for ESCA's specific contributions and advantages over existing state-of-the-art approaches.\n\n[1] C. H. Song, J. Wu, C. Washington, et al., \"LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models,\" in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 2998-3009.\n\n[2] J. Yue, X. Xu, B. F. Karlsson, and Z. Lu, \"MLLM-as-Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents,\" in The Thirteenth International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://openreview.net/forum?id=K5yeB4dTtS\n\n[3] W. Xu, M. Wang, W. Zhou, et al., \"P-RAG: Progressive Retrieval Augmented Generation for Planning on Embodied Everyday Task,\" in Proceedings of the 32nd ACM International Conference on Multimedia (ACM MM), 2024, pp. 6969-6978.", "was_revised": true, "final_justification": "After the discussion with the authors during the rebuttal, my main concerns have been solved.", "rebuttal": "> **szsY-Q1. \"Open-Domain\" Capability in Practice: The candidate sets for attributes and relations seem to be pre-defined, and concept extraction relies on environment-specific textual feedback.**\n\nWe respectfully clarify that ESCA's attributes and relations are generated dynamically on-the-fly, not pre-defined. EmbodiedBench itself is an open-domain SGG benchmark, and our evaluation demonstrates this capability through comparison with other open-domain methods (Grounding DINO and CoT-SG). For a comprehensive analysis of “open domain” capability please refer to our detailed response in SqXB-Q2.\n\n> **szsY-Q2. Deeper Result Analysis and Case Studies: Can you provide a deeper analysis to explain when and why ESCA performs better?**\n\nThe impact of ESCA varies depending on the visual demands and action structure of the task. **In general, tasks that involve more detailed spatial reasoning or complex object interactions tend to benefit more.** \n\n**Task characteristics**\n| Task             | Visual Understanding                         | Action Granularity                                     |\n|------------------|----------------------------------------------|--------------------------------------------------------|\n| EB-Navigation    | Medium: pixel-wise object localization       | Fine-grained: continuous movement (e.g., \"move forward by X\") |\n| EB-Manipulation  | Low: image-to-3D coordinate mapping          | Fine-grained: 7-tuple action specifying position       |\n| EB-Habitat       | Medium: layout reasoning + object access     | High-level: predefined discrete actions                |\n| EB-Alfred        | High: composite object-task understanding    | High-level: abstract actions (e.g., \"find X\", \"pick up Y\") |\n\n\nWe illustrate this by further conducting experiments on two high-level embodied planning benchmarks: **EB-Habitat** and **EB-Alfred**. ESCA showed the smallest performance gain on EB-Alfred, which we attribute to its abstract action space and lower reliance on fine-grained visual grounding.\n\n**EB-Habitat performance (NEW)**\n| Model                          | Raw Model | w GD   | w GD + ESCA |\n|-------------------------------|-----------|--------|--------------|\n| Intern-VL-2_5-38B-MPO         | 48.00%  | 51.67% | **55.00%**  |\n| Gemini-2.0-flash (Mar 2025)   | 30.04%  | 28.80%  | **33.86%**  |\n| Qwen2.5-VL-72B-Ins            | 33.33% | 49.50%  | **56.33%**   |\n\n\n**EB-Alfred performance (NEW)**\n| Model                      | Raw Model | w GD   | w GD + ESCA |\n|----------------------------|-----------|--------|--------------|\n| Intern-VL-2_5-38B-MPO      | 26.67%   | 34.64%  | **38.50%**   |\n| Gemini-2.0-flash (Mar 2025)| 53.00%   | 53.67% | **54.67%**  |\n| Qwen2.5-VL-72B-Ins         | 39.67%  | **44.67%** | 42.00%   |\n\n\n\n> **szsY-Q3.  Can you include qualitative examples or case studies from the manipulation task to better illustrate how ESCA operates in the manipulation task?**\n\nFor a detailed qualitative analysis, we kindly refer reviewers to the full paper in the supplementary material, Appendix C, which provides fine-grained examples of intermediate scene graphs generated using different methods, including raw MLLM, Grounding DINO, and ESCA (see Figures 14 and 16). These examples demonstrate that ESCA improves scene graph accuracy in both EB-Navigation and EB-Manipulation environments.\n\nAdditionally, in Figure 17, we highlight the error decomposition analysis of GPT-4o with and without ESCA, on a manual inspection of 50 EB tasks. ESCA significantly reduces perception errors, from 40% to 24%, highlighting its effectiveness in improving grounded visual understanding.\n\n\n> **szsY-Q4. Unclear Contribution of SGClip… A critical baseline is absent: using a powerful MLLM to directly generate the scene graph zero-shot... Comparing (ESCA) against this simpler, distillation-free approach is essential to justify the complexity and necessity.**\n\nWe appreciate the reviewer's valuable feedback and we evaluate ESCA against CoT-SG, a direct baseline where scene graphs are generated through a Chain-of-Thought MLLM pipeline. This approach prompts the MLLM to produce structured scene graph representations directly, without grounding. Further studies are detailed in **kXSe-Q1**.\n\n**EB-Navigation Results:** ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.\n\n| Model                        | Raw Model | w GD    | w ESCA| CoT-SG (NEW) |\n|-----------------------------|-----------|---------|---------------|--------|\n| Intern-VL-2_5-38B-MPO       | 47.33%    | 47.67%  | **51.66%**        | 46.33% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 40.68%    | 40.53%  | **42.00%**        | 26.33% (NEW) |\n| Qwen2.5-VL-72B-Ins          | 44.99%    | 48.27%  | **49.33%**        | 33.99% (NEW) |\n| GPT-4o                      | 51.33%    | 53.66%  | **54.67%**        | 44.00% (NEW) |\n\n\n**EB-Manipulation Results:** While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.\n\n| Model                    | Raw Model | w YOLO | w  ESCA| CoT-SG (NEW)  |\n|--------------------------|-----------|--------|---------------|---------|\n| Intern-VL-2_5-38B-MPO    | 19.31%    | 19.30% | **24.30%**        | 20.00% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 11.81% | 16.54% | **21.94%**        | 17.60% (NEW) |\n| Qwen2.5-VL-72B-Ins         | 4.72%     | 13.34% | **19.04%**        | 17.33% (NEW) |\n| GPT-4o       | 23.47%    | 28.48% | **34.44%**        | 30.43% (NEW) |\n\n\n\n\n> **szsY-Q5. Error Decomposition: Can you clarify how error types are defined and disambiguated in your analysis, especially given that failures may be compounded and minor errors may occur even in successful trajectories?**\n\nWe appreciate the reviewer’s concern and would like to clarify that formal definitions of the error categories are provided in **Footnote 1 on Page 8 of the paper**, which is also consistent with the paper:\n\n> “The three top-level error types are Perception, Reasoning, and Planning. The second-level categories include Hallucination, Wrong Recognition, Spatial Understanding, Spatial Reasoning, Reflection Error, Inaccurate Action, and Collision. For clarity, acronyms are used in the figure to label these error types.”\n\nWhile eventual failures may involve compounded errors, we classify each failure case based on the **first identifiable error** in the execution trace to avoid ambiguity. Specifically, errors  typically follow a cascade, where early perception or reasoning mistakes lead to downstream planning failures. For instance, incorrect spatial understanding may lead to collisions, and hallucinations can result in inaccurate actions. \n\nAs for errors that occur during successful trajectories, they are often corrected by subsequent planning steps. For example, the model may initially fail to find the target object due to distance, but successfully locates it after exploring further. Given that these recoverable deviations do not lead to task failure, we focus our error analysis on unrecoverable failures only.\n\n> **szsY-Q6. Inadequate Engagement with Relevant Literature: LLM-Planner, MART, and P-RAG.**\n\nWe thank the reviewer for providing these highly relevant literature and we will include the following discussion in the related works section. While these methods and ESCA both improve embodied agents, they target distinct pipeline components and are complementary: LLM-Planner focuses on hierarchical plan generation with ALFRED-specific grounding, while ESCA provides general-purpose, open-vocabulary grounding via scene graphs across diverse tasks. MART enhances planning through retrieval of past successful trajectories, P-RAG introduces progressive retrieval-augmented planning, while ESCA enhances the perception module with structured visual grounding."}, {"review_id": "n8iEymmzuE", "reviewer": "Reviewer_kXSe", "summary": "This paper addresses the challenge of fine-grained perception and physical world grounding in MLLM-based embodied agents. The authors propose the ESCA framework, which enhances an agent's environmental understanding by constructing a structured scene graph. A core contribution is the introduction of SGClip, a model specifically trained to predict relationships between objects, which forms the basis of the scene graph. The framework's effectiveness is demonstrated across multiple MLLMs and in two distinct embodied AI environments. Additionally, the authors contribute the ESCA-Video-87K dataset, generated via a neuro-symbolic self-supervised pipeline, which they plan to release to the community.", "questions": "1. Could the authors provide a direct comparative analysis of SGClip's accuracy, robustness, and inference efficiency in relationship prediction against directly prompting an advanced VLM (e.g., GPT-4o) to describe inter-object relationships, assuming the same underlying Concept Extraction and Object Identification? If a general-purpose VLM can achieve similar or even superior results with lower development and inference costs, the necessity of specially designing and training SGClip would be severely challenged.", "limitations": "Yes. The authors acknowledge limitations such as latency and reliance on 2D inputs in the conclusion section. This discussion is adequate. However, I believe they have not sufficiently addressed the core weakness I raised above, namely, \"the necessity of SGClip compared to a simpler baseline using a powerful VLM to conclude spatial relationship.\" My question aims to prompt the authors to reflect on these deeper design choices and their implications.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The paper accurately identifies a core shortcoming of current MLLM-based embodied agents: their limitations in fine-grained perception and grounding in the physical world. Enhancing the model's environmental understanding through structured scene graphs is a logically clear and practically significant research direction.\n2. The experimental evaluation is comprehensive, validating the framework's generality across several advanced MLLMs and demonstrating performance improvements in two different embodied environments.\n3. The paper is clearly written and well-structured, allowing readers to quickly grasp the core design ideas.\n\n# Weaknesses\n\n1. **Necessity of the SGClip:** The paper claims its main contribution and novelty lie in the SGClip model. However, I believe the core contribution is more in the workflow design (though this pipeline is quite common in robotics and embodied AI). I am skeptical about the necessity of SGClip. I believe that modern VLMs (e.g., GPT-4o), when properly prompted with outputs from upstream Concept Extraction and Object Identification, are sufficient to accurately and efficiently infer inter-object relationships. The paper's lack of a direct comparison with such strong baselines makes me unenthusiastic about introducing and specifically training a SGClip model.\n2. **Limitations of CLIP-based Model:** It is well-known that CLIP and its variants struggle with complex compositional relationships and precise spatial orientations (e.g., difficulty distinguishing \"A is to the left of B\" from \"B is to the left of A,\" or compositional concepts like \"person riding a horse\" versus \"horse riding a person\"). While the paper proposes some engineering tricks, it lacks a thorough discussion and analysis of the inherent limitations of the CLIP model. Therefore, I am skeptical about the accuracy and robustness of using SGClip for predicting complex spatial and interaction relationships in an embodied agent setting.", "was_revised": false, "final_justification": "", "rebuttal": "> **kXSe-Q1. Necessity of the SGClip: Could the authors provide a direct comparative analysis of SGClip's accuracy, robustness, and inference efficiency in relationship prediction against directly prompting an advanced VLM (e.g., GPT-4o) to describe inter-object relationships, assuming the same underlying Concept Extraction and Object Identification?**\n\nTo clarify SGCLIP's critical contribution within the ESCA framework, we conduct a systematic ablation study comparing SGCLIP against (a) direct scene graph generation using MLLMs and (b) alternative grounding methods such as Grounding DINO (GD). We also provide qualitative case studies & failure analysis on the above models, illustrating with examples how ESCA boosts perception and embodied planning.\n\n### **(a) SGCLIP vs. Direct MLLM Scene Graph Generation**\n\nWe evaluate against CoT-SG, a direct baseline where scene graphs are generated through a Chain-of-Thought MLLM pipeline. This approach prompts the MLLM to produce structured scene graph representations directly, without grounding.\n\n**EB-Navigation Results:** ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.\n\n| Model                        | Raw Model | w GD    | w ESCA| CoT-SG (NEW) |\n|-----------------------------|-----------|---------|---------------|--------|\n| Intern-VL-2_5-38B-MPO       | 47.33%    | 47.67%  | **51.66%**        | 46.33% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 40.68%    | 40.53%  | **42.00%**        | 26.33% (NEW) |\n| Qwen2.5-VL-72B-Ins          | 44.99%    | 48.27%  | **49.33%**        | 33.99% (NEW) |\n| GPT-4o                      | 51.33%    | 53.66%  | **54.67%**        | 44.00% (NEW) |\n\n\n**EB-Manipulation Results:** While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.\n\n| Model                    | Raw Model | w YOLO | w  ESCA| CoT-SG (NEW) |\n|--------------------------|-----------|--------|---------------|---------|\n| Intern-VL-2_5-38B-MPO    | 19.31%    | 19.30% | **24.30%**        | 20.00% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 11.81% | 16.54% | **21.94%**        | 17.60% (NEW) |\n| Qwen2.5-VL-72B-Ins         | 4.72%     | 13.34% | **19.04%**        |  17.33% (NEW) |\n| GPT-4o       | 23.47%    | 28.48% | **34.44%**        | 30.43% (NEW) |\n\n \n### **(b) SGCLIP vs. Grounding DINO**\n\nResults in Figure 5 demonstrate that the full ESCA pipeline significantly outperforms the MLLM + Grounding Dino (GD) baseline across all tasks. In that experiment, we ablate the object identification component by comparing two variants: one using Grounding DINO (GD) alone, and the other using the full ESCA pipeline, which integrates GD for object identification and SGCLIP for scene graph generation. This performance gain indicates that SGCLIP's structured, probabilistic scene graph predictions provide more effective grounding for embodied planning compared to GD’s object detection alone. \n\nAs detailed in Appendix C.1 and C.2, both configurations share the same concept extraction interface and prompting strategy; the only difference lies in the transfer protocol and grounding mechanism.\n\n### **(c) Qualitative Analysis of ESCA v/s raw MLLM v/s MLLM + GD**\n\nWe demonstrate the superior scene graph representation of ESCA via case studies and error decomposition analysis, which illustrate that ESCA achieves SOTA performance on the benchmarks by generating higher-quality scene graphs & reducing perception error, particularly in identifying hard-to-see objects and reducing hallucinations. \n\n**Case Studies on EB-Navigation & EB-Manipulation:** We kindly refer reviewers to Figures 14 and 16 in the supplementary material, Appendix C, for a detailed qualitative analysis on EB-Navigation & EB-Manipulation. For instance, Figure 14 shows scene graphs generated using raw MLLM (gpt4o), MLLM+GD, and ESCA. Figure 14’s task is  “navigating to a pillow”, where the raw MLLM doesn’t recognize any pillow, MLLM+GD misclassifies a shadow as the pillow, while ESCA correctly recognizes the pillow, and navigates to it. \n\n**ESCA Reduces Perception Error:** We highlight the error decomposition analysis of GPT-4o with and without ESCA in Figure 17, which shows that ESCA significantly reduces perception errors among all errors, from 40% to 24%, highlighting its effectiveness in improving grounded visual understanding. \n\n**Error Analysis of Scene Graphs Generated via CoT-SG.**\nIn addition to figure 17, we conducted an error analysis by sampling 10 failure cases from EB-Navigation tasks, where scene graphs were generated using GPT-4o with a Chain-of-Thought (CoT-SG) approach. The failure modes were categorized as follows:\n\n- **Failure Type 1:** Failure to detect the target object (6/10 cases)\n- **Failure Type 2:** Hallucination of non-existent objects (3/10 cases)\n- **Failure Type 3:** Incorrect object identification (e.g., misclassifying a red object as lettuce) (1/10 case)\n\nAll of these failure types stem from perception errors, whereas in our ESCA framework, only 24% of failures are attributed to perception issues, as shown in Figure 17. The discrepancy highlights a key limitation: despite leveraging a powerful MLLM, the CoT-SG method lacks sufficient grounding to reliably detect objects present in the visual scene. Further, as more entities are described without corresponding visual grounding, the planning context becomes diluted. This often leads to suboptimal action sequences and ultimately, task failure."}, {"review_id": "7H9YjhTZx3", "reviewer": "Reviewer_Q5jV", "summary": "This paper introduces ESCA to improve the perception and reasoning abilities of multimodal large language model (MLLM)-based embodied agents by integrating structured scene graph representations. At the core of ESCA is SGClip, a CLIP-based and promptable scene graph generation model, trained in a self-supervised, neurosymbolic manner on a large video dataset without requiring manual scene graph annotations. ESCA decomposes the visual perception process into modular steps and provides fine-grained, grounded visual context for agent planning and action. Experimental results across simulated environments demonstrate that ESCA consistently reduces perception errors and boosts the performance of both open-source and proprietary MLLMs, setting new benchmarks in embodied navigation and manipulation tasks.", "questions": "1. Since the video captions are generated using GPT, this indicates that GPT has the ability to perceive relationships within videos. So why not directly use GPT to generate scene graphs as part of the chain-of-thought process (similar to [1]), instead of building a new SGClip model to learn the results perceived by GPT? Isn't it inefficient to distill such knowledge from GPT into SGClip?\n2. An excessive number of relation vocabulary categories can significantly increase the number of tokens required during processing. For instance, SGClip conducts binary classification for each relation keyword, utilizing a vocabulary of 35,415 unique relation keywords. In contrast, models such as Qwen2.5-VL have a maximum input capacity of 32,768 tokens. This raises concerns about whether the inclusion of too many relation keywords and actions could exceed the model's context length, potentially impacting both the effectiveness and efficiency of the MLLM.\n\n[1] Compositional Chain-of-Thought Prompting for Large Multimodal Models. CVPR 2024.", "limitations": "See weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. ESCA introduces a modular scene graph pipeline that grounds agent perception in explicit, structured representations, improving spatial and temporal understanding in complex environments.\n2. The proposed transfer protocol could adapt ESCA to diverse embodied benchmarks.\n3. Overall writing of this paper is good and easy to understand and follow.\n\nWeaknesses:\n1. The system relies on 2D visual inputs instead of video or 3D representations, which limits depth-aware reasoning and spatial prcision.\n2. The proposed ESCA-Video-87K dataset contains a large number of semantically redundant words that have not been removed. For example, the relation keywords include 35,415 unique types, which may pose obstacles for applications.\n3. The entire system requires multiple calls to models such as MLLM and Grounding DINO, which may result in low operational efficiency.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns.", "rebuttal": "> Q5jV-Q1. Why not directly use GPT to generate scene graphs as part of the chain-of-thought process (similar to [1]), instead of building a new SGClip model to learn the results perceived by GPT?\n\nWe appreciate the reviewer's valuable feedback and evaluate against CoT-SG [1], a direct baseline where scene graphs are generated through a Chain-of-Thought MLLM pipeline. This approach prompts the MLLM to produce structured scene graph representations directly, without grounding. Further studies are detailed in **kXSe-Q1**.\n\n**EB-Navigation Results:** ESCA consistently outperforms CoT-SG across all tested models. The Chain-of-Thought strategy actually degrades performance, particularly on long-horizon tasks where target objects are not initially visible. CoT-SG often fails to detect target objects after performing a few rotation actions to scan its surroundings, and it exhibits more hallucinations than the raw MLLM.\n\n| Model                        | Raw Model | w GD    | w ESCA| CoT-SG (NEW) |\n|-----------------------------|-----------|---------|---------------|--------|\n| Intern-VL-2_5-38B-MPO       | 47.33%    | 47.67%  | **51.66%**        | 46.33% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 40.68%    | 40.53%  | **42.00%**        | 26.33% (NEW) |\n| Qwen2.5-VL-72B-Ins          | 44.99%    | 48.27%  | **49.33%**        | 33.99% (NEW) |\n| GPT-4o                      | 51.33%    | 53.66%  | **54.67%**        | 44.00% (NEW) |\n\n\n**EB-Manipulation Results:** While CoT-SG shows modest improvements over raw models, ESCA maintains superior performance across all tested MLLMs, demonstrating the effectiveness of structured grounding over direct generation approaches.\n\n\n| Model                    | Raw Model | w YOLO | w  ESCA| CoT-SG (NEW) |\n|--------------------------|-----------|--------|---------------|---------|\n| Intern-VL-2_5-38B-MPO    | 19.31%    | 19.30% | **24.30%**        | 20.00% (NEW) |\n| Gemini-2.0-flash (Mar 2025) | 11.81% | 16.54% | **21.94%**        | 17.60% (NEW) |\n| Qwen2.5-VL-72B-Ins         | 4.72%     | 13.34% | **19.04%**        |  17.33% (NEW) |\n| GPT-4o       | 23.47%    | 28.48% | **34.44%**        | 30.43% (NEW) |\n\n\n> **Q5jV-Q2. … raises concerns about whether the inclusion of too many relation keywords and actions could exceed the model's context length, potentially impacting both the effectiveness and efficiency of the MLLM.**\n\n**Training vs. Inference Distinction:** While our training dataset (ESCA-Video-87K) contains a large vocabulary (35,415 relation keywords, 57,930 actions, 220,905 names) reflecting natural language diversity, this does not translate to excessive token usage during inference. SGCLIP's core mechanism relies on cosine similarity between visual and textual embeddings rather than exhaustive binary classification over the entire vocabulary.\n\n**Efficiency Design:** ESCA's four-phase architecture inherently prevents vocabulary explosion through progressive refinement. The concept extraction phase generates only task-relevant keywords specific to the given instruction and visual scene, while both object identification and scene graph prediction work within this constrained vocabulary. \n\n**Empirical Analysis:** Our quantitative analysis on EB-Navigation shows that ESCA generates an average of 5.57 keywords per instruction, producing 84 unique target objects, 199 related objects, 296 relationships, and 117 attributes across the entire dataset. Even in the most complex scenarios, the total token count for scene graph representations remains manageable. \n\n> **Q5jV-Q3. The system relies on 2D visual inputs instead of video or 3D representations, which limits depth-aware reasoning and spatial precision.**\n\nWe believe 3D representation generation is definitely important for embodied agents application. However, this direction is beyond the current scope of our work, which focuses on visual grounding using the available 2D egocentric inputs as defined by the benchmark setup. We will clarify this design choice in the paper and include 3D integration as a promising direction in our future work section.\n\n> **Q5jV-Q4. The entire system requires multiple calls to models such as MLLM and Grounding DINO, which may result in low operational efficiency.**\n\nESCA achieves significant perceptual grounding improvements while maintaining computational efficiency. Our method introduces only 1.09 seconds of additional processing per planning step, less than 5% overhead compared to baseline systems.\nTo put this in perspective: on the EB-Navigation benchmark with GPT-4o, a complete planning step takes 22.25 seconds with ESCA versus 21.16 seconds using the baseline MLLM + Grounding DINO approach. This minimal 5% increase in runtime delivers substantial benefits through structured, interpretable scene graphs that significantly enhance visual-semantic alignment.\nWe consider this an excellent tradeoff, the marginal computational cost enables ESCA to generate rich scene representations that drive superior task performance and generalization across diverse embodied planning scenarios. The interpretability and accuracy gains far outweigh the modest time investment."}]

Author rebuttal:

Meta-review: The paper integrates scene graphs with agents for embodied problems. Overall, the paper received uniformly positive reviews with 4 x Borderline Accepts, Outlined issues focused on: (1) limited novelty [SqXB], (2) potentially limited "open domain" capabilities [szsY], (3) lack of deeper result analysis [szsY], (4) missing stronger baselines [szsY], and (5) questioned the necessity of SGClip [szsY, kXSe]. Authors have addressed these concerns in the rebuttal and reviewers agree that most significant concerns have been resolved with the scores generally going up post-rebuttal.

AC has carefully considered the reviews, rebuttal and discussion that followed as well as the content of the paper itself. AC agrees that the problem domain is interesting and results are compelling. While pure technical novelty may indeed be somewhat limited, as [SqXB] suggests, paper truly excels in many other ways; including strong results on a number of SoTA models and potential impact (with new dataset and an interesting formulation) on the broader community that cares about embodied agents. As a result the AC is recommending Acceptance.
Decision: Accept (spotlight)