RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation

Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 00:57:24

Abstract: We introduce the novel task of interactive scene exploration, wherein robots autonomously explore environments and produce an action-conditioned scene graph (ACSG) that captures the structure of the underlying environment. The ACSG accounts for both low-level information (geometry and semantics) and high-level information (action-conditioned relationships between different entities) in the scene. To this end, we present the Robotic Exploration (RoboEXP) system, which incorporates the Large Multimodal Model (LMM) and an explicit memory design to enhance our system's capabilities. The robot reasons about what and how to explore an object, accumulating new information through the interaction process and incrementally constructing the ACSG.
Leveraging the constructed ACSG, we illustrate the effectiveness and efficiency of our RoboEXP system in facilitating a wide range of real-world manipulation tasks involving rigid, articulated objects, nested objects, and deformable objects. Project Page: https://jianghanxiao.github.io/roboexp-web/

Main paper: 
 introduction: 
 Imagine a future household robot designed to prepare breakfast. This robot must efficiently perform various tasks such as conducting inventory checks in cabinets, fetching food from the fridge, gathering utensils from drawers, and spotting leftovers under food covers. Key to its success is the ability to interact with and explore the environment, especially to find items that aren't immediately visible. Equipping it with such capabilities is crucial for the robot to effectively complete its everyday tasks.Robot exploration and active perception have long been challenging areas in robotics [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] . Various techniques have been proposed, including information-theoretic approaches, curiosity-driven exploration, frontier-based methods, and imitation learning [1, [13] [14] [15] [17] [18] [19] [20] [21] [22] [23] [24] [25] . Nevertheless, previous research has primarily focused on exploring static environments by merely changing viewpoints in a navigation setting or has been limited to interactions with a small set of object categories, such as drawers, or a closed set of simple actions like pushing [26] .In this work, we investigate the interactive scene exploration task, where the goal is to efficiently identify all objects, including those that are directly observable and those that can only be discovered through interaction between the robot and the environment (see Fig. 1 ). Towards this goal, we present a novel scene representation called action-conditioned 3D scene graph (ACSG). Unlike conventional 3D scene graphs that focus on encoding static relations, ACSG encodes both spatial relationships and logical associations indicative of action effects (e.g., opening a fridge will reveal an apple inside). We then show that interactive scene exploration can be formulated as a problem of action-conditioned 3D scene graph construction and traversal.Tackling interactive scene exploration poses challenges: how can we reason about which objects need to be explored, choose the right action to interact with them, and maintain knowledge about our exploration findings? With these challenges in mind, we propose a novel, real-world robotic exploration framework, the RoboEXP system. RoboEXP can handle diverse exploration tasks in a zero-shot manner, constructing complex action-conditioned 3D scene graph in various scenarios, including those involving obstructing objects and requiring multi-step reasoning (Fig. 2 ). We evaluate our system across various settings, spanning simple, single-object scenarios to complex environments, demonstrating its adaptability and robustness. The system also effectively manages different human interventions. Moreover, we show that our reconstructed action-conditioned 3D scene graph demonstrates strong capacity in performing multiple complex downstream tasks. Action-conditioned 3D scene graph advances LLM/LMM-guided robotic manipulation and decision-making research [27, 28] , extending their operation domain from environments with known or observable objects to complicated environments with unknown or unobserved ones. To our knowledge, this is the first of its kind.Our contributions are as follows: i) we propose actionconditioned 3D scene graph and introduce the interactive scene exploration task to address the challenging interaction aspect of exploration; ii) we develop the RoboEXP system, capable of exploring complicated environments with unseen objects in a wide range of settings; iii) through extensive experiments, we demonstrate our system's ability to construct complex and complete action-conditioned 3D scene graph, demonstrating significant potential for various manipulation tasks. Our experiments involve rigid and articulated objects, nested objects like Matryoshka dolls, and deformable objects like cloth, showcasing the system's generalization ability across objects, scene configurations, and downstream tasks. 
related works: 
 Scene graphs [29, 30] represent objects and their relations [31] [32] [33] in a scene via a graph structure. Previous studies generate scene graphs from images [30, 34] or 3D scenes [35] with hierarchical and semantic information, and further with the assistance of large language models (LLMs) [36] . They leverage scene graphs for image captioning [37, 38] , image retrieval and generation [29, 39] , visual-language tasks [31, 40] , navigation [41, 42] and task planning [43] [44] [45] . While previous works model scene graphs in static 2D or 3D scenes, we generate action-conditioned scene graphs that integrate actions as core elements, depicting interactive relationships between objects and actions. This action-centric approach opens avenues for physical exploration and diverse downstream robotics tasks.Robotic exploration aims to autonomously navigate, interact with, and gather information from environments it has never encountered before. It is applicable in search and rescue [1, 2, [46] [47] [48] [49] [50] [51] [52] , planetary exploration [3, 4, 53, 54] , ob- ject goal navigation [5, 6, [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] , and mobile manipulation [7, 8, [73] [74] [75] [76] . The primary guiding principle behind robotic exploration is to reduce the uncertainty of the environment [17-19, 46, 77, 78] , making uncertainty quantification key for robotic exploration tasks. Curiosity-driven exploration has recently emerged as a promising approach, showing effective results in various contexts [15, 20, 21, 79] . Most past works have focused on exploration in the context of mobility [1, 2, 5-8, 46-50, 55-69, 73-76, 80] , with the primary goal of modeling and understanding the static environment to complete specific tasks. Recently, exploration has also been studied in the context of manipulation [16, 23, [81] [82] [83] [84] , aiming to better understand the scene by changing the state of the environment. Our work introduces a new active exploration strategy for manipulation, uniquely defining a novel scene graph-guided objective to guide the exploration process. 
problem statement: 
 We unfold this section with an introduction of actionconditioned 3D scene graph, a novel scene representation illustrating interactive object relationships (Sec. 3.1). We then formulate interactive scene exploration as an actionconditioned 3D scene graph construction and traversal problem (Sec. 3.2).An action-conditioned 3D scene graph (ACSG) is an actionable, spatial-topological representation that models objects and their interactive and spatial relations in a scene. Formally, ACSG is a directed acyclic graph G = (V, E) where each node represents either an object (e.g., a door) or an action (e.g., open), and edges E represent their interaction relations. The object node o i = (s i , p i ) ∈ V encodes the semantics and geometry of each object (e.g., the semantic embedding of a fridge s i , and its shape in the form of a point cloud p i ), whereas the action node a k = (a k , T k ) ∈ V encodes high-level action type a k and low-level primitives T k to perform the actions. Between the nodes are edges encoding their relations, which we categorize into four types: 1) between objects e o→o (e.g., the door handle belongs to the fridge), 2) from objects to actions e o→a (e.g., toy can be picked up), 3) from action to objects e a→o (e.g., a banana can be reached if we open the cabinet), or 4) from one action to another e a→a (e.g., the cabinet can be opened only if we move away the condiment). Our action-conditioned 3D scene graph greatly enhances existing 3D scene graphs, as it explicitly models the action-conditioned relations between objects. Fig. 2 depicts a complete action-conditioned 3D scene graph of a tabletop scene.One advantage of our interaction-aware scene graph lies in its simplicity for retrieving and taking actions on an object. Regardless of how complicated the scene is, given our scene graph and a target object, an agent merely needs to sequentially execute all the actions on the paths from the root to the object node in a topological order to retrieve the object. For example, in Fig. 2 , to reach the tape inside a cabinet whose door is blocked by a condiment, according to the graph, one simply needs to: 1) pick up the condiment on the table that blocks the cabinet door, and 2) open the cabinet through the door handle.This subsection describes how we can construct a complete action-conditioned scene graph of a real-world scene. This is a challenging problem due to partial observability. For instance, a banana cannot be populated without opening the cabinet. To solve this task, we formulate the scene graph construction as an active perception and exploration problem using POMDP-inspired notations. Formally, at each time t, based on our past graph estimation G t-1 , and past sensor observations O t-1 , our agent takes an action A t , which causes the environment to transition to a new state, and the agent receives a new observation O t , which is used to update its current inferred graph G t . This update might include adding new nodes to the graph or updating the state of an existing node. We will then continue with exploration and keep updating the set of remaining unexplored nodes U ⊂ V (see Algorithm 1) .The goal of the exploration is simple: discover and explore all the nodes of the scene graph in as little time as possible. Towards this, we formulate a reward function with three terms:R t = R t graph + R t explore + R t time where R t graph = |V t |-|V t-1| is the graph construction term, which promotes our agent to discover as many nodes as possible to the graph, R t explore = max(0, |U t-1 | -|U t |) gives positive reward to actions that reduce unexplored node set, which prioritize the agent to explore previously unexplored nodes, and immediate reward R t time = -λ, 0 < λ < 1 is a negative time reward that optimizes the time efficiency and allows the exploration to terminate when there is no more node to explore.Intuitively, to maximize this reward at each discrete timestamp, we should prioritize exploring the unexplored nodesAlgorithm 1 Interactive Exploration 1: input: O 0 , G 0 = (V 0 , E 0 ), U 0 ← V 0 2: while |U t-1 | ̸ = 0 do 3: if choose object oi ∈ U t-1 then % explore object 4:add spatial relations % memory5:obtain action a to explore oi % decision-making6:if action a / ∈ V t-1 then 7:V t , U t = V t-1 ∪ {a}, U t-1 ∪ {a} % add node 8: E t = E t-1 ∪ {eo i →a} % add edge 9:U t = U t \ oi % mark as explored 10:end if11:else choose action a k ∈ U t-112:if no obstruction then % decision-making if found new objects O ̸ ⊂ V t-1 then 16: V t , U t = V t ∪ {O}, U t-1 ∪ {O} % add nodes17: in the current scene graph that are likely to lead to the discovery of new nodes (e.g., opening a cabinet that has not been opened, or lifting a piece of clothing that might cover a small object). The key challenge lies in how we can perceive the objects in the scene, infer possible actions and their relations from the sensory data, and take actions with the current scene graph. In the next section, we will comprehensively describe our system implementation to achieve this goal.E t = E t ∪ {ea k →O } % add edges 18: U t = U t \ 
method: 
 To tackle the task outlined in Section Sec. 3, we present our RoboEXP system, designed to autonomously explore unknown environments by observing and interacting with them. The system comprises four key components: perception, memory, decision-making, and action modules (see Fig. 3 ). Raw RGBD images are captured through the wrist camera in different viewpoints and processed by the perception modules to extract scene semantics, including object labels, 2D bounding boxes, segmentations, and semantic features. The obtained semantic information is then transmitted to the memory module, where the 2D data is merged into the 3D representation. Such 3D information serves as a valuable guide for the decision module, aiding in the selection of appropriate actions to further interact or observe the environment and unveil hidden objects. The action module is activated to execute the planned action, generating new observations for the perception modules. This closed-loop system ensures the thoroughness of our task in interactive scene exploration. Perception Module. Given multiple RGBD observations from different viewpoints, the objective of the perception module (Fig. 3a ) is to detect and segment objects while extracting their semantic features. To enhance generality, we opt for the open-vocabulary detector GroundingDINO [85] and the Segment Anything in High Quality (SAM-HQ) [86] , an advanced version of SAM [87] . For the extraction of semantic features used in subsequent instance merging within the memory module, we employ CLIP [88] . To obtain per-instance CLIP features, we implement a strategy similar to the one proposed by Jatavallabhula et al. [89] . Specifically, we extend the local-global image feature merging approach by incorporating additional label text features to augment the semantic CLIP feature for each instance. Furthermore, we exclusively focus on instance-level features, disregarding pixel-level features, thereby accelerating the entire semantic feature extraction process.Memory Module. The memory module (Fig. 3b ) is designed to construct our ACSG of the environment by assimilating observations over time. For the low-level memory, to ensure stable instance merging from 2D to 3D, we employ a similar instance merging strategy as presented in Lu et al. [90] , consolidating observations from diverse RGBD sources across various viewpoints and time steps. In contrast to the original algorithm, which considers only 3D IoU and semantic feature similarity we additionally incorporate label similarity and instance confidence. To enhance algorithm efficiency, we represent low-level memory using a voxel-based representation, which allows for more efficient computation and memory updates. Meanwhile, given the crowded nature of objects in our tabletop setting, we have implemented voxel-based filtering designs to obtain a cleaner and more complete representation of the objects for storage in our memory.The memory module handles merging across different viewpoints and time steps. To merge across different viewpoints, we project 2D information (RGBD, semantic features, mask, bounding box) to 3D and leverage the instance merging strategy mentioned earlier to attain consistent 3D information. Addressing memory updates across time steps presents a challenge due to dynamic changes in the environment. For instance, a closed door in the previous time step may be opened by our robot in the current time step. To accurately reflect such changes, our algorithm evaluates whether elements within our memory have become outdated, primarily through depth tests based on the most recent observations. This process ensures that the memory accurately represents the environment's current state, effectively managing scenarios where objects may change positions or states across different time steps.For the high-level graph of our ACSG, the memory module analyzes the relationships between objects and the logical associations between actions and objects. Depending on Table 1 . Quantitative Results on Different Tasks. We compare the performance of both the GPT-4V baseline and our system across various tasks. We assess the outcomes using five distinct metrics to illustrate diverse facets of the interactive exploration process. Our system consistently outperforms the baseline across all tasks and metrics. changes in low-level memory and relationships, the memory module is tasked with updating the graph. This involves adding, deleting, or modifying nodes and edges within our graph.Decision-Making Module. The primary goal of the decision module (Fig. 3c ) is to identify the appropriate object and corresponding skill to enhance the effectiveness and efficiency of interactive scene exploration. In the context of our task, distinct objects may necessitate distinct exploration strategies. While humans can easily discern the most suitable skill to apply (e.g., picking up the top Matryoshka doll to inspect its contents), achieving such decisions through heuristic-based methods is challenging. The utilization of a Large Multi-Modal Model (LMM), such as GPT-4V [91, 92] , shows instrumental in addressing this difficulty, as it captures commonsense knowledge that facilitates decision-making.The LMM brings commonsense knowledge to our decision-making process and serves in two pivotal roles. Firstly, it functions as an action proposer. Given the current digital environment from the memory module, GPT-4V is tasked with selecting the appropriate skill for unexplored objects in our system. For instance, when presented with a visual prompt of an object within a green bounding box from various viewpoints, GPT-4V can discern the suitable "pick up" skill for the Matryoshka doll in the environment. For unexplored objects, our ACSG includes the attribute of whether each object node is explored or unexplored. GPT-4V, in its role as the proposer, also functions to assess whether the object holds value for further exploration. If not, the corresponding node is marked as explored, indicating that no further actions are needed.Secondly, the LMM also serves as the action verifier. For the proposer role, it analyzes the object-centric attributes and doesn't consider surrounding information when choosing the proper skill. For example, if the proposed action involves opening a door, the proposer alone may struggle with cases where obstructions exist in front of the door (e.g., a condiment bottle). To address this, we use another LMM program to verify the feasibility of the action and identify any objects in the scene that may impede the action based on information from our ACSG.In summary, the decision module, with its dual roles, effectively guides our system to choose efficient actions that minimize uncertainty in the environment and successfully Action Module. In the action module (Fig. 3d ), our primary focus is on autonomously constructing the ACSG through effective and efficient interaction with the environment. We employ heuristic-based action primitives within our action module, leveraging the geometry cues in our ACSG. These primitives encompass seven categories: "open the door", "open the drawer", "close the door", "close the drawer", "pick object to idle space", "pick back object", "move wrist camera to position". Strategic utilization of these skills plays a pivotal role in accomplishing intricate tasks seamlessly within our system (more details in the Appendix). 
experiments: 
 In this section, we assess the performance of our system across a variety of tabletop scenarios in the interactive scene exploration setting. Our primary objective is to address two key questions through experiments: 1) How does our system effectively and efficiently deal with diverse exploration scenarios and successfully construct comprehensive ACSG? 2) What is the utility of our ACSG in facilitating downstream tasks?To assess our system's efficacy across various exploration scenarios, we compared it with a strong baseline by augment-ing GPT-4V with ground truth actions. We designed five types of experiments, each with 10 different settings varying in object number, type, and layout. Our quantitative analysis reveals that our RoboEXP system consistently surpasses the baseline across various tasks. Furthermore, we validate the performance of our system in constructing ACSG through qualitative demonstrations. Check the Appendix and our supplementary video for more details. Evaluation. To thoroughly assess the efficacy of our system compared to the baseline, we have designed five key metrics (Success, Object Recovery, State Recovery, Unexplored Space, Graph Edit Distance) to measure its performance. It is crucial to note that the output of our task, represented by ACSG, aligns precisely with the format of ACSG for our system. Conversely, for the baseline, we manually construct ACSG based on its actions and the new observations it uncovers. Due to the unstructured nature of the raw scene graph from the baseline, we carefully refine it according to the observable objects, providing an upper-bound baseline for comparison during evaluation.Comparison. The quantitative findings presented in Table 1 underscore the superior performance of our system compared to the baseline method. Our approach showcases a notable enhancement across all metrics, outperforming the baseline by a considerable margin. The collective assessment of success rate, object recovery, and unexplored space metrics unequivocally validates the efficacy of our system in exploring unfamiliar scenes through interactive processes. It is essential to highlight that in the case of object recovery, the baseline method may occasionally choose to randomly open certain drawers or doors to unveil objects. This randomness contributes to a seemingly higher object recovery rate for the baseline, which may not necessarily correlate with its overall success. The unexplored space metric shows that our system is much more stable in exploring all need-to-explore spaces.Moreover, both the success rate and graph edit distance underscore the close alignment of our system with human actions, highlighting the efficiency of our approach across diverse scenarios. The state recovery metric assesses whether the final state post-exploration resembles the initial state. Our system consistently shows effective state recovery; however, the baseline may trick this metric by opting not to take any action, resulting in an artificially high score in this aspect.Fig. 4a provides additional insights, illustrating that as the number of actions increases, so does the number of objects. Specifically, we present the ground truth object number alongside the directly-observable object number that can be represented by the traditional 3D scene graph. These results underscore our system's ability to achieve robust and efficient exploration throughout the exploration process. Our system excels in efficiently discovering all concealed objects, whereas the baseline fails either due to a lack of early-stage actions or an inability to explore all need-to-explore spaces even upon completion. The analysis of errors (Fig. 4b ) in both our system and the baseline reveals the specific failure cases encountered by the baselines. In contrast, our system demonstrates enhanced robustness in both perception and decision-making.Fig. 5 further illustrates various exploration scenarios along with their corresponding ACSG. These scenarios encompass ACSG with varying width or depth, highlighting our system's adaptive capability across diverse objects such as rigid, articulated objects, nested objects, and deformable objects. In addition, the scenario in Fig. 2 shows that our system is able to deal with the scenario with obstruction.The scenarios depicted in Fig. 1 exemplify the efficacy of our generated output (ACSG) in manipulation tasks. Consider the table-rearranging scenario: without our ACSG, the robot struggles to swiftly prepare the table due to the lack of precise prior knowledge about the location of objects (e.g., the fork stored in the top-left drawer of the wooden cabinet). Beyond comprehensive layout guidance, our ACSG also addresses a crucial question regarding task feasibility for the robot. For instance, if there is no spoon in the scene, the robot recognizes its inability to perform the task and asks for human help.In addition to enhancing downstream manipulation tasks, our ACSG possesses the capability to autonomously adapt to environmental changes. In the human intervention setting, our system seamlessly explores newly added components, such as a cabinet, ensuring continuous adaptability. Check our Appendix and supplemental video for more details. 
conclusion: 
 We introduced RoboEXP, a foundation-model-driven robotic exploration framework capable of effectively identifying all objects in a complex scene, both directly observable and those revealed through interaction. Central to our system is action-conditioned 3D scene graph, an advanced 3D scene graph that goes beyond traditional models by explicitly modeling interactive relations between objects. Experiments have shown RoboEXP's superior performance in interactive scene exploration across various challenging scenarios, significantly outperforming a strong GPT4V-based baseline. Notably, the reconstructed action-conditioned 3D scene graph is crucial for guiding complex downstream manipulation tasks, like preparing breakfast in a mock-kitchen environment with fridges, cabinets, and drawer sets. Our system and its action-conditioned scene graph lay the groundwork for practical robotic deployment in complex settings, especially in environments like households and offices, facilitating their everyday use. 
 
 Appendix:  


Decision: This paper proposes an approach to generate action-conditioned scene graphs and uses it for interactive exploration for several manipulation tasks. The reviewers find the paper well written, find the results promising, and appreciate the real-world experiments. However, the reviewers have also raised several concerns including lacking baselines or ablations, unrealistic scenarios in the experiments, insufficient analysis on application of the scene graph, lacking details on the proposed method, require clarity on the research questions, clarity on the metrics used,  lacking clarity regarding the assumptions of the system, among others. Each of the reviewers has listed several questions to be addressed in the rebuttal.

Post-rebuttal: Most of the concerns of the reviewers have been sufficiently addressed in the rebuttal. The paper could be stronger if results were presented in more complex realistic scenes.

Reviews: review: title: title: review for submission 192
paper_summary: paper_summary: In this paper, the authors propose a RoboEXP system, which generates an action-conditioned scene graph via interactive exploration and the LMM is incorporated.
main_review: main_review: In this paper, the authors propose a RoboEXP system, which generates an action-conditioned scene graph via interactive exploration and the LMM is incorporated. The generated scene graph is expected to facilitate a bunch of downstream manipulation tasks. The structure of the paper is clear, and it is good to the real world experiments. Here are some concerns.

- The idea of interacting with the environment to generate the scene graph is reasonable. However, the scenario presented in this paper is actually artificial. For example, we seldom put one fruit in one drawer, and only one coke in the refrigerator in reality. It is said that the LMM could bring commonsense knowledge for decisions. I wonder how much the commonsense knowledge could be leveraged in such situations. 

- In the experiments, the proposed system and the GPT-4V methods are compared and an obvious improvement can be observed. It is suggested that a comparison of efficiency of each method is also presented or discussed. Additionally, some heuristic methods such as opening every door in the view to see what is in it could be included. 

- For the downstream manipulation tasks, does the robot uses a pre-generated scene graph or with an online scene graph generation process. The analysis about the application of the scene graph is not sufficient.
questions: questions: - The setup of the scenario and how the setup and commonsense knowledge affect the results.

- The efficiency of the comparison methods. And some heuristic method is suggested.

- The process of scene graph generation in the task.
limitations: limitations: 2
review_summary: review_summary: The generalization of the task to real-word environment and some details of the method are not quite clear.
title: title: Review
paper_summary: paper_summary: The paper presents a novel way to represent interactions in the environment using action-conditioned scene graphs and exploration for manipulation problems.
main_review: main_review: This paper presents an approach to reason about interactions through an action-conditioned graph. The work is well written and clearly motivated. 

Strengths: 

The paper does a good work at illustrating the overall system. The results seem quite promising, with clearly higher success rates in various different metrics. The qualitative results also show impressive interaction and discovery in the environment. 

Weaknessness: 

There is a bit of clarity in the exact robotic implementation that should be addressed in the main document. In particular, how the robot goes about interacting and whether the actions are needed to be designed by the user a-priori for the approach to work. In addition, some of the research questions that the results address could be more specific e.g., what efficiency is expected in exploration from the action-conditioned graph, how quickly does the approach explore all interactions, etc. Furthermore, the paper could be clearer about some metrics, specifically the similarity/differences, meanings, and implications that the metrics hold.
questions: questions: Does the user need to specify all the interactions a-priori? It seems that one can interact with an object an infinite number of ways, so how does the graph, which is finite, deal with this?

What does it mean to take an action that reduces the unexplored node set? Specifically, in a real-world application, how does an action yield a node from the environment. Is this part of the perception module?

How does the agent know when the exploration is terminated? If it has access to the unexplored node-	set, then why explore it at all and enumerate each node?

What is the ACSG “output”?

The DEG metric and unexplored space metric seem like they would produce similar meanings. Could their differences and implications ve explained more clearly?
limitations: limitations: 3
review_summary: review_summary: The work seems promising, but requires some clarifications within the text.
title: title: Action-conditioned scene graph by exploration
paper_summary: paper_summary: This paper presents a scene representation called the Action-Conditioned Scene Graph, which represents both inter-object spatial relationships and the necessary actions to reveal elements of the scene graph. The paper then describes an exploration framework which constructs the proposed perceptual representation through interaction. Experiments on real-world robotic manipulation tasks demonstrate increased success rate and improved exploration metrics compared to a baseline heuristic approach.
main_review: main_review: Overall, the idea of interactive exploration is compelling and well-presented. The methods are clear and the experiments contain multiple experiments on two real robot systems. Some points of the methodology could be more clearly stated. The work would benefit from a clearer throughline, backed up with experiments, regarding the nature of the contribution. Detailed comments are below.

**Strengths:**
* The experimental analysis is thorough, providing multiple metrics to understand different elements of the method. The error breakdown is very helpful in understanding the success rate of the method and is a valuable inclusion.
* The method shows success on two real robot platforms in exploration and various task execution. The experiments involve interacting with multiple objects including articulated and deformable ones.
* The paper is presented and explained clearly with nice visuals and data presentation.

**Points of feedback:**

It is a bit unclear whether the contribution is the scene representation or the exploration framework. The experiments focus on the exploration framework, but are lacking in baselines or ablations to demonstrate the key benefits of the RoboEXP framework. For example, does the proposed decision module outperform a brute force approach (perform every possible detected action)? Such a heuristic baseline is akin to the classic frontier exploration strategy. Another ablation could be a random approach (randomly sample viewpoints or actions). I suspect that a frontier exploration-style approach would perform very well on most metrics.

A stated goal of the work is to explore with minimum time (Section 3, line 112). While a strength of the paper is the inclusion of multiple metrics in the evaluations, it would be beneficial to show the time or number of actions taken to create the scene graph to support this claim.

Finally, some fundamental assumptions from the paper are missing in order to make a compelling case for the generalization capability of the proposed representation. First, it would be helpful to elaborate on the taxonomy of relations and actions and how they were chosen, as this impacts to which tasks and scenes this method can be applied. Second, it seems the representation proposed would be dependent on the state (e.g. in Fig 5, when the drawer is already open, the “condiment” can be picked without the “open” action. Is the graph dynamically updated in this case? Third, an action is not a property of an object alone but instead depends on both the agent and object (e.g. a robot may not be able to grasp an object that is too large or too heavy). How are these cases handled by the proposed method?

*Other points of feedback:*
* What is the definition of Graph Edit Distance? How is it computed?
* The baseline would benefit from a bit more detail. Is the robot teleoperated for the baseline?
* What is the importance of the state recovery metric? This might not always be important to the task.
* Is the given ACSG unique? There may be more than one possible action to reveal an occluded object (e.g. change viewpoint or move an object).
* Is it always necessary to construct the whole scene graph (e.g. when objects unrelated to the task at hand are present in the scene)? This may be an interesting line of future work if not already handled. The authors may consider related works in graph-based object search [1] and semantic search [2].

**References:**

[1] Zhen Zeng, Adrian Röfer, Odest Chadwicke Jenkins. SLiM: Semantic Linking Maps for Active Visual Object Search. ICRA 2020.

[2] Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, Bernadette Bucher. VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation. ICRA 2024.
questions: questions: 1. How does RoboEXP compare to other exploration heuristic baselines, like random or frontier exploration?
2. How much time (or how many actions, as is relevant) is needed to construct the scene graph?
3. What are the assumptions about the environment and type of actions? 

See additional details and points above.
limitations: limitations: 3
review_summary: review_summary: The paper is clearly presented and has been evaluated on real robots. The concept of the action-conditioned scene graph is well motivated. However, more clarity is needed regarding the assumptions of the system in order to generalize across scenarios. Furthermore, comparison against heuristic baselines would strengthen the claims of the work.