Title: ESCA: Contextualizing Embodied Agents via Scene-Graph Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: neurosymbolic

scene graph

multimodal

MLLM agents
Abstract: Multi-modal large language models (MLLMs) are making rapid progress toward general-purpose embodied agents. However, existing MLLMs do not reliably capture fine-grained links between low-level visual features and high-level textual semantics, leading to weak grounding and inaccurate perception. To overcome this challenge, we propose ESCA, a framework that contextualizes embodied agents by grounding their perception in spatial-temporal scene graphs. At its core is SGCLIP, a novel, open-domain, promptable foundation model for generating scene graphs that is based on CLIP.  SGCLIP is trained on 87K+ open-domain videos using a neurosymbolic pipeline that aligns automatically generated captions with scene graphs produced by the model itself, eliminating the need for human-labeled annotations. We demonstrate that SGCLIP excels in both prompt-based inference and task-specific fine-tuning, achieving state-of-the-art results on scene graph generation and action localization benchmarks. ESCA with SGCLIP improves perception for embodied agents based on both open-source and commercial MLLMs, achieving state of-the-art performance across two embodied environments. Notably, ESCA significantly reduces agent perception errors and enables open-source models to surpass proprietary baselines. We release the source code for SGCLIP model training at https://github.com/video-fm/LASER and for the embodied agent at https://github.com/video-fm/ESCA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ESCA to improve the perception and reasoning abilities of multimodal large language model (MLLM)-based embodied agents by integrating structured scene graph representations. At the core of ESCA is SGClip, a CLIP-based and promptable scene graph generation model, trained in a self-supervised, neurosymbolic manner on a large video dataset without requiring manual scene graph annotations. ESCA decomposes the visual perception process into modular steps and provides fine-grained, grounded visual context for agent planning and action. Experimental results across simulated environments demonstrate that ESCA consistently reduces perception errors and boosts the performance of both open-source and proprietary MLLMs, setting new benchmarks in embodied navigation and manipulation tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. ESCA introduces a modular scene graph pipeline that grounds agent perception in explicit, structured representations, improving spatial and temporal understanding in complex environments.\n2. The proposed transfer protocol could adapt ESCA to diverse embodied benchmarks.\n3. Overall writing of this paper is good and easy to understand and follow.\n\nWeaknesses:\n1. The system relies on 2D visual inputs instead of video or 3D representations, which limits depth-aware reasoning and spatial prcision.\n2. The proposed ESCA-Video-87K dataset contains a large number of semantically redundant words that have not been removed. For example, the relation keywords include 35,415 unique types, which may pose obstacles for applications.\n3. The entire system requires multiple calls to models such as MLLM and Grounding DINO, which may result in low operational efficiency.\n\nQUESTIONS:\n1. Since the video captions are generated using GPT, this indicates that GPT has the ability to perceive relationships within videos. So why not directly use GPT to generate scene graphs as part of the chain-of-thought process (similar to [1]), instead of building a new SGClip model to learn the results perceived by GPT? Isn't it inefficient to distill such knowledge from GPT into SGClip?\n2. An excessive number of relation vocabulary categories can significantly increase the number of tokens required during processing. For instance, SGClip conducts binary classification for each relation keyword, utilizing a vocabulary of 35,415 unique relation keywords. In contrast, models such as Qwen2.5-VL have a maximum input capacity of 32,768 tokens. This raises concerns about whether the inclusion of too many relation keywords and actions could exceed the model's context length, potentially impacting both the effectiveness and efficiency of the MLLM.\n\n[1] Compositional Chain-of-Thought Prompting for Large Multimodal Models. CVPR 2024.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of fine-grained perception and physical world grounding in MLLM-based embodied agents. The authors propose the ESCA framework, which enhances an agent's environmental understanding by constructing a structured scene graph. A core contribution is the introduction of SGClip, a model specifically trained to predict relationships between objects, which forms the basis of the scene graph. The framework's effectiveness is demonstrated across multiple MLLMs and in two distinct embodied AI environments. Additionally, the authors contribute the ESCA-Video-87K dataset, generated via a neuro-symbolic self-supervised pipeline, which they plan to release to the community.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper accurately identifies a core shortcoming of current MLLM-based embodied agents: their limitations in fine-grained perception and grounding in the physical world. Enhancing the model's environmental understanding through structured scene graphs is a logically clear and practically significant research direction.\n2. The experimental evaluation is comprehensive, validating the framework's generality across several advanced MLLMs and demonstrating performance improvements in two different embodied environments.\n3. The paper is clearly written and well-structured, allowing readers to quickly grasp the core design ideas.\n\n# Weaknesses\n\n1. **Necessity of the SGClip:** The paper claims its main contribution and novelty lie in the SGClip model. However, I believe the core contribution is more in the workflow design (though this pipeline is quite common in robotics and embodied AI). I am skeptical about the necessity of SGClip. I believe that modern VLMs (e.g., GPT-4o), when properly prompted with outputs from upstream Concept Extraction and Object Identification, are sufficient to accurately and efficiently infer inter-object relationships. The paper's lack of a direct comparison with such strong baselines makes me unenthusiastic about introducing and specifically training a SGClip model.\n2. **Limitations of CLIP-based Model:** It is well-known that CLIP and its variants struggle with complex compositional relationships and precise spatial orientations (e.g., difficulty distinguishing \"A is to the left of B\" from \"B is to the left of A,\" or compositional concepts like \"person riding a horse\" versus \"horse riding a person\"). While the paper proposes some engineering tricks, it lacks a thorough discussion and analysis of the inherent limitations of the CLIP model. Therefore, I am skeptical about the accuracy and robustness of using SGClip for predicting complex spatial and interaction relationships in an embodied agent setting.\n\nQUESTIONS:\n1. Could the authors provide a direct comparative analysis of SGClip's accuracy, robustness, and inference efficiency in relationship prediction against directly prompting an advanced VLM (e.g., GPT-4o) to describe inter-object relationships, assuming the same underlying Concept Extraction and Object Identification? If a general-purpose VLM can achieve similar or even superior results with lower development and inference costs, the necessity of specially designing and training SGClip would be severely challenged.\n\nLIMITATIONS:\nYes. The authors acknowledge limitations such as latency and reliance on 2D inputs in the conclusion section. This discussion is adequate. However, I believe they have not sufficiently addressed the core weakness I raised above, namely, \"the necessity of SGClip compared to a simpler baseline using a powerful VLM to conclude spatial relationship.\" My question aims to prompt the authors to reflect on these deeper design choices and their implications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the ESCA framework to provide MLLM-based agents with structured visual descriptions, scene graphs, which aids in their reasoning and planning. The core contribution is SGClip, a model trained using a self-supervised neurosymbolic pipeline that leverages CLIP to generate scene graphs without requiring human annotations. Experiments on two environments in EmbodiedBench demonstrate that the ESCA framework improves agent performance. The authors also provide code to ensure reproducibility.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Novel and Scalable Self-Supervised Training Paradigm: The paper's key innovation is a scalable neurosymbolic pipeline that trains a scene graph model from video-caption pairs without human annotation. This elegantly solves a major data bottleneck in the field.\n2. Versatile and Generalizable SGClip: SGClip demonstrates strong performance not only within the ESCA framework but also in zero-shot and downstream tasks.\n3. Comprehensive Experimental Validation: The framework's effectiveness is rigorously tested on a diverse set of both open-source and proprietary MLLMs in multiple embodied tasks. The clear reduction in perception errors provides strong evidence for its utility.\n4. Community Contribution: The commitment to release the ESCA-Video-87K dataset and codebase is a valuable contribution that fosters reproducibility and future research.\n\nWeaknesses:\n1. Limited \"Open-Domain\" Capability in Practice: The paper claims SGClip is an open-domain model, but its application within the ESCA framework appears constrained. The candidate sets for attributes and relations seem to be pre-defined, and concept extraction relies on environment-specific textual feedback. This setup limits the method's generality and raises questions about its direct applicability to real-world scenarios that lack such structured feedback. \n2. Lack of Deeper Result Analysis and Case Studies: The paper reports a more significant performance gain on EB-Manipulation than on EB-Navigation but provides no explanation for this discrepancy. Furthermore, all qualitative examples and visualizations are from the navigation task. Including case studies for the manipulation task would be crucial for a better understanding of how ESCA functions in more complex, interaction-rich scenarios.\n3. Insufficient Ablation and A Critical Missing Baseline:\n    - Unclear Contribution of SGClip: The ESCA pipeline heavily involves MLLMs in multiple stages (concept extraction, summarization), making it difficult to isolate the precise impact of the SGClip module itself versus the overall structured prompting.\n    - Missing a Stronger Baseline: The training of SGClip can be seen as a form of knowledge distillation. A critical baseline is absent: **using a powerful MLLM to directly generate the scene graph zero-shot**, aided by grounding from visual modules like GD+SAM2. Comparing against this simpler, distillation-free approach is essential to justify the complexity and necessity of training a specialized SGClip model.\n4. Ambiguous Error Decomposition: The error analysis lacks clarity. The paper does not provide formal definitions for its error categories. Because most failures are compounded, and there are likely to be some errors in the trajectory of success.\n5. Inadequate Engagement with Relevant Literature: The paper's positioning within the field could be strengthened by a more thorough discussion of related work. It overlooks several highly relevant prior works that also address grounding and planning for LLM agents in similar AI2-THOR environments, such as LLM-Planner[1], MART[2], and P-RAG[3]. A comparative discussion with these methods would provide a clearer context for ESCA's specific contributions and advantages over existing state-of-the-art approaches.\n\n[1] C. H. Song, J. Wu, C. Washington, et al., \"LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models,\" in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 2998-3009.\n\n[2] J. Yue, X. Xu, B. F. Karlsson, and Z. Lu, \"MLLM-as-Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents,\" in The Thirteenth International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://openreview.net/forum?id=K5yeB4dTtS\n\n[3] W. Xu, M. Wang, W. Zhou, et al., \"P-RAG: Progressive Retrieval Augmented Generation for Planning on Embodied Everyday Task,\" in Proceedings of the 32nd ACM International Conference on Multimedia (ACM MM), 2024, pp. 6969-6978.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes. \n\nSection 7 (Conclusion and Limitations) provides a clear, direct discussion of primary limitations including latency, dependence on 2D visual inputs, and lack of formal verification during execution. The impact of synthetic supervision is not fully detailed and could be addressed more explicitly.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper integrates scene graph descriptions with agents to describe object relationships, providing structured alignment between visual contexts and textual semantics. It proposes a CLIP-based, open-domain, and promptable model to generate scene graphs, and an Embodied and Scene-Graph Contextualized Agent to support MLLM-based embodied agents. A novel ESCA-Video-87K\n dataset is proposed, including object traces, open-domain concepts, and programmatic specifications for 87K video-caption pairs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Introducing scene graph descriptions into agents utilises spatial and semantic contexts in visual and language information, therefore facilitating the fine-grained operations.\n2. The new ESCA-Video-87K dataset with video-caption pairs contributes to exploring open-domain scene graph descriptions. \n3. The embodied agent pipeline augmented with the proposed ESCA provides a solution for accurate agent perception.\n\nWeaknesses:\n1. The paper has limited novel content for embodied agent or scene graph generation. For embodied agent construction, the only difference is introducing scene graphs, which is straightforward and cannot capture the outstanding part. For scene graph generation, the proposed SGClip is only a fine-tuned CLIP with the proposed dataset. \n2. The paper stresses the superiority of open-domain perceptions but lacks more illustrations and experimental analysis. It only performs the open-domain comparisons with closed-set SGG datasets, missing the comparison with other open-domain SGG methods. \n3. Actually, how to integrate SGG with the agent is not well described. The relationships in SGG are not shown in a qualitative manner, especially in Fig. 1. The Authors would better clarify how spatial and temporal relationships are integrated into the agent.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}