RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Authors: Hanxiao Jiang
Binghao Huang
Ruihai Wu
Zhuoran Li
Shubham Garg
Hooshang Nayyeri
Shenlong Wang
Yunzhu Li
Venue: CoRL 2024 (2024)
Keywords: Action-Conditioned Scene Graph
Foundation Models for Robotics
Scene Exploration
Robotic Manipulation

Abstract: We introduce the novel task of interactive scene exploration, wherein robots autonomously explore environments and produce an action-conditioned scene graph (ACSG) that captures the structure of the underlying environment. The ACSG accounts for both low-level information (geometry and semantics) and high-level information (action-conditioned relationships between different entities) in the scene. To this end, we present the Robotic Exploration (RoboEXP) system, which incorporates the Large Multimodal Model (LMM) and an explicit memory design to enhance our system's capabilities. The robot reasons about what and how to explore an object, accumulating new information through the interaction process and incrementally constructing the ACSG.
Leveraging the constructed ACSG, we illustrate the effectiveness and efficiency of our RoboEXP system in facilitating a wide range of real-world manipulation tasks involving rigid, articulated objects, nested objects, and deformable objects. Project Page: https://jianghanxiao.github.io/roboexp-web/

RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Hanxiao Jiang! 1.2
Binghao Huang'
Ruihai Wu² Zhuoran Li
Shubham Garg² Hooshang Nayyeri
Shenlong Wang² Yunzhu Li
'Columbia University "Peking University NNational University of Singapore
University of Illinois Urbana-Champaign
*Amazon
Abstract: We introduce the novel task of interactive scene exploration, wherein robots autonomously explore environments and produce action-conditioned scene graph (ACSG) that captures the structure of the underlying environment. The ACSG accounts for both low-level information (geometry and semantics) and high-level information (action-conditioned relationships between different entities) in the scene. To this end, we present the Robotic Exploration (RoboEXP) system, which incorporates the Large Multimodal Model (LMM) and an explicit memory design to enhance what and how to explore object, accumulating new information through the interaction process and incrementally constructing the ACSG. Leveraging the constructed ACSG, we illustrate the effectiveness and efficiency our RoboEXP system facilitating a wide range of real-world manipulation tasks involving rigid, articulated objects, nested objects, and deformable objects. Project Page: https://jianghanne gi thub. ro/roboepp-web/
system's 's capabilities. The robot reasons about
Keywords: Action-Conditioned Scene Graph, Foundation Models for Robotics, Scene Exploration, Robotic Manipulation
1 Introduction
Imagine a household robot designed to prepare breakfast. This robot must perform various tasks such as conducting inventory checks cabinets, fetching food from the fridge, gathering utensils from drawers, and spotting leftovers under food covers. Key its success the ability to interact with and explore the environment, especially find items that aren't immediately visible. Equipping it with such capabilities is crucial for the robot effectively complete its everyday tasks. Robot exploration and active perception have long been challenging areas in robotics. Various techniques have been proposed, including iooommtiion-hhoretii approaches [1-7], frontier-based methods [8-10], imitation learning [1 1, 12] and reinforcement learning [13-17]. Nevertheless, previous research has primarily focused on exploring static environments by merely changing viewpoints in navigation setting.
In this work, we investigate the interactive scene exploration task, where the goal is to efficiently identify all objects, including those that directly observable and those that only be discovered through interaction between the robot and the environment (Fig. 1). Towards this goal, we present novel scene representation called condiincondiiioneed 3D scene graph (ACSG). Unlike conventional 3D scene graphs that focus on encoding static relations, ACSG encodes both spatial relationships and logical associations indicative of action effects (e.g.. opening a fridge will reveal an apple inside). We then show that our task can be formulated as a problem of ACSG construction and traversal.
To tackle the interactive scene exploration task, propose novel, real-world robotic exploration framework, the RoboEXP system. At the core of our system is large foundational model-powered instantiation of action-conme cetiin-conditioned 3D scene graph. Specifically, our framework consists of four modules: perception, memory, decision-making and action (Fig. 3).
RoboEXP can handle diverse exploration tasks in zero-shot manner, constructing complex action- conditioned 3DD scene graph in various scenarios, including those involving obstructing objects and
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Initial State
Kiter Exploration
(a) Exploration (Baild Scene Graph)
1
Resioone
Soars Brich SS
After E Expoiiation with SG
(b) Exploitaiion (Leverage Scone Graph)
Buuger
1
Pick Gandiment
Flack Cord
Figure 1: Interactive Exploration to Construct an Action-Condee Conditioned Scene Graph (ACSG) for Robotic
Manipulation. Exploration: The rohot autonomously explores by interacting with the environment generate a comprehensive ACSG. This graph is used to catalog the locations and relationships of items. (b) Exploitation: Utilizing the constructed scene graph, the robot completes downstream tasks by efficiently organizing the necessary items according to the desired spatial and relational constraints.
requiring multi-step reasoning (Fig. 2). We evaluate our system across various settings, demonstrating its udaptability and robustness. The system also effectively manages different human interventions. Moreover, show that reconstructed action-conditioned 3D scene graph demonstrates strong capacity in performing multiple complex downstream tasks. Action Action-conditioned 3D scene graph advances LLM/LMM--UU -guided robotic manipulation and decision cerision-making research [18, 19], extend- ing their operation domain from environments with known or observable objects to complicated environments with unknown or unobserved ones. To our knowledge, this is the first of its kind.
2 Related Works
Scene Graphs [20, 21] represent objects and their relations [22-24] in scene via a graph structure. Previous studies generate scene graphs from images [21, 25, 26] or 3D scenes [27]. and further with the assistance of large language models (LLLMs) [28, 29]. While previous works model scene graphs in static 2D 3D scenes, we generate action-conditioned scene graphs that integrate actions core elements, depicting interactive relationships between objects and actions. Our work is also related to Neuro-Symbolic Representations that integrate neural networks with symbolie reasoning. Prior works explored understanding scenes and describing robotic skills in symbolic texts to interpret demonstrations [30, 31], ground abstract actions for robotic primitives [32] and generate action plans [33-38]. Our proposed framework also constructs symbolic representations of the environment, but the form of action-conditioned scene graphs for robotic manipulation.
Robotic Exploration aims to autonomously navigate [9, 11, 39 39-45], interact with [12, 46, 46 46-50), and gather information [51-53] from environments it has never encountered before. The primary guiding principle behind robotic exploration is to reduce the uncertainty of the environment [5- 7, 39,9,55 54, 55], making uncertainty quantification key for robotic exploration tasks. Curiosity-driven exploration has recently emerged as promising approach, showing effective results various contexts [16, 56-58]. Recently, exploration has also been studied in the context of manipulation [59-66], aiming better understand the scene by changing the state of the environment, Our work introduces new active exploration strategy for manipulation, uniquely defining novel scene graph- guided objective to guide the exploration process. Our work is also related to Active Perception,

---

Real
F
Belong
Drawer Handle
Open
Cabinet
Belong
Drawer Handle
O Open
Inside
Banana
Belong
Door Handle
Opn
Table
On
Cabinet
Hight
Door Handle
Belong
Inside
Open
Tape
bbject
Condiment
Action Node
Figure 2: Cetion-oonditioned 3D Scene Graph from Inferactive Scene Exploration. We depict scenario wherein robot explores tabletop
containing cabinets and condiment obstructing the left door. (a) The robot actively interacts with the scene, completing the interactive showcase corresponding low-level memory in our ACSG. small graph on bottom-left of each visualization represents a segment of the final scene graph. (c) We present the high-level memory of our ACSG. The graph reveals that picking the condiment serves as precondition for opening the door, and opening the bottom drawer allows the observation of the concealed banana.
exploration process. (b)
which actively selects actions an agent to help it perceive and understand the environment [1, 67]. Unlike passive perception, actions offer more flexibility, such as control over viewpoints [2-4, 68], sensor configurations [15, 69], or adjustments environmental configurations [70]. It can also reveal certain scene properties that cannot be perceived in passive manner, such dynamic parameters [16, 71] or articulation [62, 72, 73). Our work falls into the category of actively exploring the environment to reveal what's 's inside or underneath objects. Differing from most previous active perception efforts, which are driven by handcrafted rules [74], information gain [75, 76], or reinforcement learning [16, 77]. our approach is guided by grounding the commonsense knowledge encoded in LLM/LMM into an explicit scene graph representation.
Language Models for Robotics. Large language models (LLMs) [78-80] and large multimodality models (LMMs) [81, 82) are bringing overwhelming influence into the robotics field, for their strong capacity in common-sense knowledge and task-level planning. Previous studies have harnessed the common-sense knowledge such large models to generate action candidates [83-85] and action sequences for task planning [80, 86-88), and generate code for robotie control and manipulation [18, 89, 90]. More recently, VILA [19] utilized GPT-4V [81, 82] vision- language planning. In our RoboEXP system, leverage GPT-4V for decision-making in crucial roles select and verify actions. Morcover, instead of memorizing everything using large models in brute force way, system employs explicit memory to enhance the decision-mann maakin process.
Action-Conditioned 3D Scene Graph
An eetion-conditioned 3D scene graph (ACSG) is actionable, spotial-topological representation that models objects and their interactive and spatial relations in a scene. Formally, ACSG is directed acyclic graph G = (V, E) where each node represents either an object or an action, and edges E represent their interaction relations. The object node o, = (s,. p.) E V encodes the semantics s; and geometry P. of each object, whereas the action node a; = (ak: Tx) E encodes high-level action type dg and low-level primitives Tr perform the actions. Between the nodes are edges encoding their relations, which we categorize into four types: 1) between objects e. (e.g., the door handle belongs the fridge), 2) from objects to actions Co-aa (e.g., toy be picked up), 3) from action to objects a (c.g., banana can be reached if we open the cabinet), or 4) from one

---

Gripper
Mulli-Viaw RGBD
cocsseoe Ol 00-00-03 OZ
Environment
Perception Module
"Open the Door "Opan Chaaee"
"No Obstruction
"No Chstruction" No Cisstructon" Coutuetia
Dpwn the Srawer"
Ose oe Dever
Verifler
Highllvel Memory
Low-level memory
Action
(c) Decision Makinng Module
Figure 3: Overview of Our RoboEXP System. We present comprehensive overview RoboEXP system, comprised of four modules: (a) perception, (b) memory, (c) Decision-making and (d) action module.
action to another Ca-+a (e.g., the cabinet can be opened only we move away the condiment). Our action-conditioned 3D scene graph greatly enhances existing 3D scene graphs, as it explicitly models the condin-condiiooeedd relations between objects. Fig. depicts a complete action-conditioned 3D scene graph of tabletop scene.
One advantage of our ACSG lies in its simplicity for retrieving and taking actions on an object. Regardless of how complicated the scene is, given our scene graph and a target object, an agent merely needs to sequentially exccute all the actions on the paths from the root to the object node in a topological order to retrieve the object.
Interactive Exploration. Constructing complete ACSG of real-world scene is challenging due to partial observability. To solve this task, we formulate the scene graph construction an active perception and exploration problem using POMDP-inspired notations. Formally, at each time t, based on our past graph estimation G and past sensor observations O our agent takes an action A', which causes the environment to transition to a new state, and the agent receives new observation O which is used update its current inferred graph G This update might include adding new nodes the graph or updating the state of existing node. We will then continue with exploration and keep updating the set of remaining unexplored nodes U C V (See the Appendix for the full algorithm). The goal of the exploration is simple: discover and explore all the nodes of the scene graph in as little time as possible. We thus formulate reward function with three terms: R' = R graph + R explure + R where R graph = V' - |V- promotes our agent to discover as many nodes as possible to the graph, R Explore m0 1 (U') gives positive reward to actions that reduce the unexplored node set, which prioritizes the agent to explore previously unexplored nodes (note that an action can lead to the discovery of new unexplored nodes being added to the graph), and immediate reward / = A < A < is negative time reward that enhances efficiency and terminate the exploration when there no more node to explore.
Intuitively, to maximize this reward at each discrete timestamp, we should prioritize exploring the unexplored nodes in the current scene graph that are likely to lead to the discovery of new nodes (e.g.. opening cabinet that has not been opened, or lifting a piece of clothing that might cover small object). The key challenge lies in how we can perceive the objects in the scene, infer possible actions and their relations from the sensory data, and take actions with the current scene graph. In the next section, we will comprehensively describe our system implementation to achieve this goal.
4 Interactive Robotic Exploration (RoboEXP) System
In this section, we outline the structure of our RoboEXP system, including perception, memory, decision-making, and action modules (See the Appendix for full details our implementation).
Perception Module. Given multiple RGBD observations from different viewpoints, the objective of the perception module (Fig. 3a) is to detect and segment objects while extracting their semantic

---

features. To enhance generality, we opt for the open-vocabulary detector GroundingDINO [91] and the Segment Anything in High Quality (SAM-HQ) [92], with a predefined list of tags. For the extraction of semantic features used in subsequent instance merging, we employ per-instance CLIP [93], following a similar strategy to the one proposed by Jatavallabhula et al. [94].
Memory Module. The memory module (Fig. 3b) handles merging across different viewpoints and time steps. To merge across different viewpoints, we project 2D information to 3D and leverage the instance merging strategy similar to Lu et al. [95] to attain consistent 3D information. Addressing memory updates across time steps presents a challenge due to dynamic changes in the environment. For instance, a closed door in the previous time step may be opened by our robot in the current time step. To accurately reflect such changes, our algorithm evaluates whether elements within our memory have become outdated, primarily through depth tests based the most recent observations. This process ensures that the low-level memory accurately represents the environment's current state, effectively managing scenarios where objects may change positions states across different time steps. For the high-level graph of our ACSG, the memory module analyzes the relationships between objects and the logical associations between actions and objects. Depending on changes low-level memory and relationships, the memory module is tasked with updating the graph. This involves adding, deleting, or modifying nodes and edges within our graph.
Decision-Making Module. The primary goal of the decision module (Fig. 3c) is to identify the appropriate object and corresponding skill to enhance the effectiveness and efficiency of interactive scene exploration. In the context of our task, distinct objects may necessitate distinct exploration strategies. While humans can easily discern the most suitable skill to apply (e.g., picking up the top Matryoshka doll to inspect its contents), achieving such decisions through heuristic-based methods is challenging. The LMM brings commonsense knowledge to our gpcision-making process and serves in two pivotal roles. Firstly, functions as action proposer. Given the current digital environment from the memory module, GPT-4V is tasked with selecting the appropriate skill for unexplored objects in our system. Secondly, the LMM also serves as the action verifier. For the proposer role, it analyzes the object-centric attributes but ignore surrounding information when choosing the proper skill. To address this, we use another LMM program to verify the feasibility of the action and identify any objects in the scene that may impede the action based on information from our ACSG.
Action Module. In the action module (Fig. 3d), our primary focus is on autonomously constructing the ACSG through effective and efficient interaction. We design set of action primitives that operate on the lewelevel geometric information embedded within our ACSG. These primitives encompass seven categories: "open the door", "open the drawer", "close the door", "close the drawer", "pick object to idle space", "pick back object", "move wrist camera to position". Strategic utilization of these skills plays a pivotal role in accomplishing intricate tasks seamlessly within our system.
Experiments
In this section, we assess the performance of our system across a variety of tabletop scenarios. Our primary objective is to address two key questions through experiments: 1) How does our proposed system handle diverse exploration scenarios well and build the complete ACSG? 2) How useful is our ACSG in the downstream object retrieval and manipulation tasks?
5.1 Interactive Exploration and Scene Graph Building
Robot Setups. All our experiments conducted in real-world setting. In the tabletop scenarios, we mount one RealSense-D4y camera the wrist of the robot arm to collect RGBD observations, with the execution of actions performed by the UFACTORY xArm 7. The end effector for our robot arm is the soft gripper (see Fig. 3). In the mobile setting, we choose the Hello Robot Stretch2 with the official upgraded kits. See the Appendix for more details on the full setup.
Experiment Settings. To assess our system's 's efficacy, we designed five types of experiments, each with 10 different settings varying in object number, type, and layout. We validate the performance of our system in constructing ACSG through quantitative analysis and qualitative demonstrations.
5

---

Table Quantitative Results on Different Tasks. We compare the performance of both the GPT-4V baseline
and our sy stem across various tasks. Our system consistently outperforms the baseline across all metrics.
Task (10 varianze for cach)
Drawer Only
Doce Doce-Only Only
Drawer Door
Recunive
Occlasion
GPT-4V
GPI-4V
GPPT-4V
GPT-VV
Ours
GPI-4V
Success 9
20,:
90.00.0
30a:15.2
90110.0
1011000
702:15.3
0:0.00
70.15.33
0::.00
50.:
Object Recovery %1
83::11.0
971:33
50=166.7
100:00.0
62:10.7
91-21.
202:3.3
80.11..7
17=1144
67111.9
State Recovery 9 ?
60::16.3
10010.0
80=13.3
100::0.0
70::5.33
100-0.0
704.5.3
100:0.0
10=00.00
70::5.3
Unexplored Space
15-7.6
0:000
0+0.0
0-0:0
63+15.3
15+89
85-7.6
30+15.3
Graph Edit Dist. 4
2.8+:1004
0.2+0.20
4,4-1.42
0.140.10
5.6+:1446
0.5=0.277
8.8+2.06
2.1+11499
7.3=0.97
2.5
Objects
Otjects
102.0%
Perception
Decision
Erro
10.03%
Action
26.01%
Actions
Aviiions 0.0%%
No Enor
Static
X Fuil
GPT4V RoboEXP
(8) Action-object Graph
Figure 4: Visualization of Quantitative Results, (a) The action- object graph captures the change in the number of discovered objects relative to the number of actions taken. Our RoboEXP efficiently discovers all objects. (b) The error breakdown all our quantitative experiments includes task settings with 10 variations each. We categorize errors into perception, decision, action, and no-error cases. For the GPT-4V baseline, we mamually assist in action execution, eliminating action errors. This serves as an upper bound for baseline performance. However, even with this enhancement, our RohoEXP consistently shows superior performance.
We compare our system with a strong baseline by augmenting GPT-4V with ground truth actions. This baseline operates in a closed-loop fashion, receiving RGB observations from different viewpoints. At each turn, it generates the current scene graph, encompassing hidden objects, and suggests the next action to be taken (refer the complete prompts in the Appendix). To ensure the baseline is robust, we utilize manual actions as ground truth references for the proposed actions. In contrast, in the exploration experiments described below, all actions from our system automatically executed by our action module on the physical robot. It is also crucial to note that the output ACSG from our system faithfully aligns with the task requirements. Conversely, for the baseline, we manually construct ACSG based on its actions and the new observations it uncovers. Due to the unstructured nature of the raw scene graph from the baseline, we carefully refine according to the observable objects, providing an upper-bound performance for comparison during evaluation. (See comparisons with other heuristic-based baselines in Appendix)
We design five key metrics to measure the performance of the interactive exploration task. I) Success evaluates the success percentage across 10 variants for each task. We define success
for each experiment as when the final outputted ACSG exactly matches the GT version, and otherwise; 2) Object Recovery quantifies the percentage of hidden objects successfully identified; 3) State Recovery indicates whether the final state resembles the original state before exploration; 4) Unexplored Space evaluates the percentage of successfully explored eeeploo- space to reduce the robot's iss uncertainty about the scene; 5) Graph Edit Distance (GED) measures the disparity between the outputted graph and the GT graph.
Comparison. The quantitative findings presented Tab. 2 underscore the superior performance of our system compared to the baseline method. Our approach showcases a notable enhancement across all metries, outperforming the baseline by considerable margin. The collective assessment of success rate, object recovery, and unexplored space metrics unequivocally validates the efficacy of our system in exploring unfamiliar scenes through interactive processes. It is essential to highlight that, in the case of object recovery, the baseline method may occasionally choose to randomly open certain drawers or doors, revealing objects that have already been discovered. This randomness contributes to a seemingly high object recovery rate but may not necessarily correlate with overall success. The unexplored space metric shows that our system much more stable in exploring all need-to-explons spaces. Moreover, both the success rate and graph edit distance underscore the close alignment of our system with human actions, highlighting the efficiency of our approach across diverse scenarios. The state recovery metric assesses whether the final state post-cxploration resembles the initial state. Our

---

Table
on
om
on
on
on
Cabinet
Doll
Plate
Condiment
Hamburger
(8) Articulates Objecs
1
?
beland
belong
beking
Refrigerator
Door Handle
Door Handle
Door Handle
Handle
Open
Open
Open
Inside
Inside
Open
Fork
Condiment
Inside
(b) Nested Objecct
Under
Under
Table
Matryoshka
Pick
Matryoshka
Piik
Matryoshka
Pick
Pick
Matryoshkk
Pick
Matryoshika
(c) Deformable Object
Table
on
Fabric
Mouss
Pick
Figure 5: Qualitative Results on Different Scenarios. visualize the interactive exploration process and the corresponding constructed ACSG, (a) This scenario involves a tabletop environment with two articulated objects, accompanied by additional items either on the table or concealed in storage space. The constructed scene graph demonstrates the success of system in identifying all objects within the environment through series of physical interactions, (b) This scenario includes nested objects, five Matryoshka dolls, with only the top one being directly ohservable. Our system autonomously decides to explore the contents through recursive reasoning process, showcasing its ability to construct deep ACSG. (c) This scenario involves fabric covering mouse, showcasing exploration scenarios that involve deformable object. Our system interacts with the fabric and successfully uncovers what lies beneath it.
system consistently shows effective state recovery; however, the baseline may trick this metric by opting not to take any action, resulting in artificially high score in this aspect.
Fig. 4a provides additional insights, illustrating that as the number actions increases, so does the number of objects. Specifically, we present the ground truth object number alongside the directly- observable object number that can represented by the traditional 3D scene graph. These results underscore system's 's ability to achieve robust and efficient exploration throughout the exploration process. Our system excels in efficiently discovering all concealed objects, whereas the baseline fails either due to lack of carly-stage actions or an inability to explore all upon completion. The analysis of errors (Fig. 4b) in both our system and the baseline reveals the specific failure cases encountered by the baselines. In contrast, our system demonstrates enhanced robustness in both perception and decision-making.
spaces even
Fig. 5 further illustrates various exploration scenarios along with their corresponding ACSG. These scenarios encompass ACSG with varying width or depth, highlighting system's 's adaptive capability across diverse objects such as rigid, articulated objects, nested objects, and deformable objects. In addition, the scenario in Fig. 2 shows that our system is able to deal with the scenario with obstruction.
5.2
ACSG for Object Retrieval and Manipulation Tasks
The scenarios depicted Fig. exemplify the efficacy our generated output (ACSG) in manip- ulation tasks. Consider the table-rearranging scenario: without our ACSG, the robot struggles to swiftly prepare the table due the lack of precise prior knowledge about the location of objects (e.g., the fork stored in the tep-left drawer of the wooden cabinet). Beyond comprehensive layout guidance, our ACSG also addresses crucial question regarding task feasibility for the robot. For

---

one
on
on
(a) In Drawer?
Cabinet
Table
Sofa
belong
on.
on
Drawor Handle
Drawer Handle
Doll
Plant
Open
Coke
Inside 4
Apple
or- Chair
oni Table
Chair
(b) Under Cloth??
Plck
Under
Reader
Figure 6: Experiments on Mobile Robot Scenarios. We visualize the interactive exploration pracess and the corresponding constructed ACSG in the mobile robot settings. (a) This scenario includes table, cahinet, sofa, multiple small items, and an apple hidden in the top drawer. (h) This scenario involves chairs, one table, and reader hidden by cloth above.
instance, if the task requires spoon but there is no spoon in the scene, the robot recognizes the missing object and asks for human help. In addition enhancing downstream manipulation tasks, ACSG possesses the capability autonomously adapt environmental changes. In the human intervention setting, system seamlessly explores newly added components, such cabinet, ensuring continuous adaptability, Check our Appendix and supplemental video for more details.
5.3 Extension to Mobile Robot
We also demonstrate the effectiveness of our pipeline using mobile robot in household scenarios. To adapt Stretch2, merely modified action module to follow the new kinematic structure, while keeping other modules nearly the same. Fig. 6 shows two scenes with hidden objects in drawers and under cloth. Our system is capable of exploring the scene, utilizing manipulation and mobility, to construct the full ACSG and recover the scene. By leveraging the constructed ACSG, we can easily locate the hidden apple and e-reader.
6 Limitations and Future Work
Although our system has proven effective, there is room for improvement. The breakdown of the failure rate in Fig. 4.b suggests that failures primarily arise from detection and segmentation errors within the perception module. To address this issue, envision future directions: 1) enhancing the capabilities of visual foundation models for open world semantic understanding, and 2) utilizing temporal cues and semantic fusion techniques to improve perception robustness through continuous observations. Furthermore, our system would benefit from enhanced LMM capacities and the integration of more sophisticated and generalizable skill modules. Such improvements would enhance both the decision-making and the action execution, thereby further reducing failure cases.
7 Conclusion
We introduced RoboEXP, a foundation-modee model-driven robotic exploration framework capable of effectively identifying all objects in a complex scene, both directly observable and those revealed through interaction. Central to system is action-conditionnd 3D scene graph, advanced 3D scene graph that goes beyond traditional models by explicitly modeling interactive relations between objects. Experiments have shown RoboEXP's superior performance in interactive scene exploration across various challenging scenarios, significantly outperforming a strong GPT4V-based baseline. Notably, the reconstructed action-cue iition-conditionel 3D scene graph is crucial for guiding complex downstream manipulation tasks, like setting up the table mock environment with fridges, cabinets, and drawer sets. Our system and its action-conditioned scene graph lay the groundwork for practical robotic deployment in complex settings, especially in environments like households and offices, facilitating their integration into everyday human activities.

---

Acknowledgments
Yunzhu Li is partly supported by the Amazon AICE Award. Shenlong Wang is supported by the Amazon AICE Award, IBM IIDAI Grant, Nvidia Hardware Grants, the Insper-Illinois Innovative Grant, the NCSA Faculty Fellow, NSF Award #2331 878 and #2312102, and gifts from Intel and Meta. We thank Dorie Wang for generously sharing her toys for our research, Kaifeng Zhang and Yixuan Wang for their kind discussions. This work does not relate to the positions of Shubham Garg and Hooshang Nayyeri at Amazon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] R. Bajesy. Active perception. Proceedings of the IEEE, 1988.
[2] A. Bircher, M. Kamel, K. Alexis, H. Oleynikova, and R. Siegwart. Receding horizon "next-best- view" planner for 3d exploration. In ICRA, 2016.
[3] A. Batinovic, A. Ivanovic, T. Petrovic, and S. Bogdan. shadowcasting- based next-best-view planner for autonomous 3d exploration. RA-L, 2022.
[4] M. Naazare, F. G. Rosas, and D. Schulz. Online next-best-view planner for 3d-exploration and inspection with mobile manipulator robot. RA-L, 2022.
[5] B. Charrow, G. Kahn, S. Patil, S. Liu, K. Goldberg, P. Abbeel, N. Michael, and V. Kumar. infornation-theorrtii planning with trajectory optimization for dense 3d mapping. In RSS, 2015.
[6] G. Georgakis, B. Bucher, A. Arapin, K. Schmeckpeper, N. Matni, and K. Daniilidis. Uncertainty- driven planner for exploration and navigation. In ICRA, 2022.
[7] C. Papachristos, S. Khattak, and K. Alexis. Uncertainty-aware receding horizon exploration and mapping using aerial robots. In ICRA, 2017.
[8] B. Yamauchi. A frontier-based approach for autonomous exploration. In Proceedings 1997 IEEE International Symposium on Computational Intelligence Robotics and Automation CIRA'97 "Towards New Computational Principles for Robotics and Automation's 1997.
[9] F. Niroui, K. Zhang, Z. Kashino, and G. Nejat. Deep reinforcement learning robot for search and rescue applications: Exploration unknown cluttered environments. RA-L, 2019.
[10] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In CVPR, 2023.
[11] R. Ramrakhya, E. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object-search strategies from human demonstrations scale. In CVPR, 2022.
[12] D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi Mapping instructions to actions in 3d environments with visual goal prediction. arXiv preprint arXiv: arXiv:1809.000 1809.00786. 2018.
[13] A. Mousavian, A. Toshev, M. Fiser, Kosecka, and J. Davidson. Visual representations for semantic target driven navigation. ICRA, 2018.
[14] J. Ye, D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectgoal navigation. In ICCV, 2021.
[15] P. Chen, Ji, K. Lin, W. Hu, W. Huang, T. Li, M. Tan, and C. Gan. Learning active camera for multi-object navigation. NeurIPS, 2022.

---

[16] T. Nagarajan and K. Grauman. Learning affordance landscapes for interaction exploration in 3d environments. In NeurlPS, 2020.
[17] Xia, W. Shen, C. Li, Kasimbeg, M. E. Tchapmi, A. Toshev, R. Martín-Martin, and Savarese. Interactive gibson benchmark: benchmark for interactive navigation in cluttered environments. RA-L, 2020.
[18] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[19] Y. Hu, F. Lin, T. Zhang, L. Yi, and Y. Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv: 1784. 2023.
[20] J. Johnson, R. Krishna, M. Stark, L.J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015.
[21] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In CVPR, 2017.
[22] C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei. Visual relationship detection with language priors. In ECCV, 2016.
[23] R. Wu, K. Xu, C. Liu, N. Zhuang, and Y. Mu. Localize, assemble, and predicate: Contextual object proposal embedding for visual relation detection. In AAAI, 2020.
[24] H. Zhang, Z. Kyaw, S.-F. Chang, and T.-S. Chua. Visual translation embedding network for visual relation detection. CVPR, 2017.
[25] J. Yang. J. Lu, S. Lee, D. Batra, and D. Parikh. Graph r-cnn for scene graph generation. In ECCV, 2018.
[26] N. Hughes, Y. Chang, S. Hu, R. Talak, R. Abdulhai, Strader, and L. Carlone. Foundations of spatial perception for robotics: Hierarchical representations and real-time systems. arXiv preprint arXiv: 2305.07154, 2023.
[27] I. Armeni, Z.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In ICCV, 2019.
[28] J. Strader, N. Hughes, W. Chen, A. Speranzon, and L. Carlone. Indoor and outdoor 3d scene graph generation via language-enables spatial ontologies. arXiv preprint arX1v:2312.11713, 2023.
[29] W. Chen, S. Hu, R. Talak, and L.. Carlone. Leveraging large (visual) language models for robot 3d scene understanding, arXiv preprint arXiv:
2022.
[30] J. Mao, T. Lozano-Pérez, J. B. Tenenbaum, and L. P. Kaelbling. Learning reusable manipulation strategies. In CoRL, 2023.
[31] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu. A. Torralba, and J. Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint 19000: 2019.
[32] R. Wang, J. Mao, J. Hsu, H. Zhao, J. Wu, and Y. Gao. Programmatically grounded, composi- tionally generalizable robotic manipulation. ICLR, 2023.
[33] Z. Yang, Mao, Y. Du J. Wu, J. B. Tenenbaum, Lozano-Pérez, and L. P. Kaelbling. Compositional Diffusion-Baan Based Continuous Constraint Solvers. In CoRL, 2023.
[34] W. Liu, J. Mao, J. Hsu, T. Hermans. A. Garg, and J. Wu. Composable part-based manipulation. In CoRL, 2023.
10

---

[35] Z. Chen, K. Yi, Y. Li, M. Ding, A. Torralba, J. B. Tenenbaum, and C. Gan. Comphy: Composi- tional physical reasoning of objects and events from videos. In ICLR, 2022.
[36] J. Mao, T. Lozano-Perez, B. Tenenbaum, and I.. P. Kaelbing. PDSketch: Integrated Domain Programming, Learning, and Planning. In NeurIPS, 2022.
[37] A. Ray, C. Bradley, L.. Carlone, and N. Roy. Task and motion planning hierarchical 3d scene graphs. arXiv preprint arXiv: 2403. 0803,, 2024.
[38] W. Gu, A. Sah, and N. Gopalan. Interactive visual task learning for robots. AAAI Conference on Artificial Intelligence, 2023. d6::10.4550kkXIi.332...22
[39] F. Niroui, B. Sprenger, and G. Nejat. Robot exploration in unknown cluttered environments when dealing with uncertainty. In IRIS, 2017.
[40] S. Oßwald, M. Bennewitz, W. Burgard, and C. Stachniss. Speeding up robot exploration by exploiting background information. RA-L, 2016.
[41] M. Petrlik, P. Petracek, V. Kratky, T. Musil, Y. Stasinchuk, M. Vrba, T. Baca, D. Hert, M. Pecka, Svoboda, et al. Uavs beneath the surface: Cooperative autonomy for subterrancan search and rescue darpa subt. arXiv preprint arXiv:2206.08188 2022.
[42] M. Tranzatto, T. Miki, M. Dharmadhikari, L. Bernreiter, M. Kulkarni, F. Mascarich, O. Anders- son, S. Khattak, M. Hutter, R. Siegwart, et al. Cerberus in the darpa subterranean challenge. Science Robotics, 2022.
[43] K.-Q. Zhou, K. Zheng, C. Pryor, Y. Shen, H. Jin, L. Getoor, and X. Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. ICMI, 2023.
[44] A. J. Zhai and S. Wang. Peanut: predicting and navigating unseen targets. In ICCV, 2023.
[45] N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher. Vlfm: Vision- langualan frontier maps for zero-shot semantic navigation. arXiv preprint arXiv:23/2.03275. 2023.
[46] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
[47] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. In CVPR, 2021.
[48] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik, Mordatch, R. Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint arXiv:2011.01975 2020.
[49] F. Xia, C. Li, R. Martín-Marti O. Litany, A. Toshev, and S. Savarese. Relmogen: Lever- aging motion generation in reinforcement learning for mobile manipulation. arXiv preprins arXiv:2008. 07792, 2020.
[50] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mot- taghi. Manipulathor: A framework for visual object manipulation. In CVPR, 2021.
[51] Arm, G. Waibel, J. Preisig, T. Tuna, R. Zhou, V. Bickel, G. Ligeza, T. Miki, F. Kehl, H. Kolvenbach, et al. Scientific exploration of challenging planetary analog environments with a team of legged robots. Science robotics, 2023.
[52] M. J. Schuster, M. G. Müller, G. Brunner, H. Lehner, Lehner, R. Sakagami, Dömel, L. Meyer, B. Vodermayer, R. Giubilato, et al. The arches space-analogue demonstration mission: Towards heterogeneous teams of autonomous robots for collaborative scientific sampling in planetary exploration. RA-L, 2020.

---

[53] T. Sasaki, K. Otsu, R. Thakker, S. Haesaert, and A.-a. Agha-mohammade Where to map? iterative rover-copter path planning for mars exploration. RA-I,, 2020.
[54] C. Stachniss, G. Grisetti, and W. Burgard. Information gain-based exploration using rao- blackwellized particle filters. In RSS, 2005.
[55] Chen, Martin, Huang, Wang, and B. Englot. Autonomous exploration under uncertainty via deep reinforcement learning on graphs. IROS, 2020.
[56] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of crrion learning. ICLR, 2019.
[57] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In ICML, 2017.
[58] S. Parisi, V. Dean, D. Pathak, and Gupta. Interesting object, curious agent: Learning task-agnostic exploration. In NeurIPS, 2021.
[59] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. NeurIPS, 2016.
[60] I.. Pinto and A. Gupta. Learning to push by grasping: Using multiple tasks for effective learning. In ICRA, 2017.
[61] T. Schneider, B. Belousov, G. Chalvatzaki, D. Romeres, D. K. Jha, and Peters. Active exploration for robotic manipulation. In IROS, 2022.
[62] N. Nie, S. Y. Gadre, K. Ehsani, and S. Song. Structure from action: Learning interactions for articulated object 3d structure discovery. arXiv preprint arXiv:2207.08997 2022.
[63] C.-C. Hsu, Z. Jiang, and Y. Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. arXiv preprint arXiv:2302.01299 2023.
[64] L. Chen, Song, H. Bao, and X. Zhou. Perceiving unseen 3d objects by poking the objects. In ICRA, 2023.
[65] J. Lv, Y. Feng, C. Zhang, S. Zhao, L. Shao, and C. Lu. Sam-rl: Sensing- aware model-based reinforcement learning via differentiable physics-based simulation and rendering. RSS, 2023.
[66] Y. Zaky, G. Paruthi, B. Tripp, and J. Bergstra. Active perception and representation for robotic manipulation. arXiv preprint arXiv:2003.06734, 2020.
[67] Active perception passive perception. In Proc. of IEEE Workshop Computer Vision 1985.
[68] Q. Fang, Y. Yin, Q. Fan, Xia. S. Dong, S. Wang. J. Wang, L. Guibas, and B. Chen. Towards accurate active camera localization. In ECCV, 2022.
[69] S. Chen, Y. F. Li, W. Wang, and J. Zhang. Active sensor planning for multiview vision tasks. 2008.
[70] M. Ghasemi, E. Bulgur, and U. Topeu. Task-oriented active perception and planning in environments with partially known semantics. In ICML, 2020.
[71] Y. Wang, R. Wu, K. Mo, Ke, Q. Fan, L. Guibas, and H. Dong. AdaAfford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In ECCV, 2022.
[72] R. Martín-Martin Martín and O. Brock. Building kinematic and dynamic models of articulated objects with multi-modal interactive perception. In 20/7 AAAI Spring Symposium Series, 2017.
[73] Z. Jiang, C.-C. Hsu. and Y. Zhu. Ditto: Building digital twins of articulated objects from interaction. In CVPR, 2022.
12

---

[74] Q. V. Le. A. Saxena, and A. Y. Ng. Active perception: Interactive manipulation for improving object detection. Standford University Journal, 2008
[75] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Embodied question answering. In CVPR, 2018.
[76] S. Jauhri, S. Lueth, and Chalvatzaki. Active-perceptive motion generation for mobile manipulation. arXiv preprint arXiv:23/0.00433, 2023.
[77] S. D. Whitehead and D. H. Ballard. Active perception and reinforcement learning. In Machine Learning Proceedings 1990. 1990.
[78] J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, J. F. C. Uribe, L. Fedus, L. Metz, M. Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
[79] R. OpenAL. Gpt-4 technical report. arXiv preprint wrXiv:2303008774 2023.
[80] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint vrXiv:2303.03375 2023.
[81] OpenAL. Gpt-4v(ision) system card. http:://cch.oeenniiicco//pppsspp System Card.pdf. 2023.
[82] Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision). arXiv preprint arXiv: 2309. 17021, 2023.
[83] M. Ahn, A. Brohan, Brown, Chebotar, Cortes, B. David, Finn, C. Fu, Gopalakr- ishnan, K. Hausman, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0166), 2022.
[84] I. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Advances Neural Information Processing Systems, 2023.
[85] S. Kambhampati, K. Valmeekam, L. Guan, K. Stechly, M. Verma, S. Bhambri, L. Saldyt, and A. Murthy. Llms can't 't plan, but can help planning in Ilm-modulo frameworks. arXiv preprint arXiv: 2402.01817. 2024.
[86] W. Huang. F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models, arXiv preprini arXiv:2207.05600, 2022.
[87] J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and Chai. Lim Lim-grounder: Open-vone veeebvocabulary 3d visual grounding with large language model agent. arXiv preprint arXiv:2309. /231 2023.
[88] Y. Dai, R. Peng, Li, and J. Chai. Think, act, and ask: Open- world interactive personalized robot navigation. arXiv preprint arXiv:2310.07968 2023.
[89] J. Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In ICRA, 2023.
[90] W. Shen, G. Yang, A. Yu, J. Wong, L. P. Kaelbling, and P. Isola. Distilled feature fields enable few-shot language-guidel manipulation. In CoRL, 2023.
[91] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, L., J. Yang. H. Su, J. Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint a9X1v:2303.05499 2023.
13

---

[92] L. Ke, M. Ye, M. Danelljan, Y. Liu, Y.W. Tai, C.-K. Tang, and F. Yu. Segment anything in high quality, arXiv preprint arXiv: 2306.0/567, 2023.
[93] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021.
[94] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi,
N. Keetha, A. Tewari, al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint
a:X2v:230..07241, 2023.
[95] Lu, Chang, E. P. Jing, A. Boularias, and K. Bekris. Ovir-3d: Open-vocabulary 3d instance retrieval without training on 3d data. In CoRL, 2023.
[96] Q. Gu, A. Kuwajerwala, S. Morin, K. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, C. Gan, C. de Melo, J. Tenenbaum, A. Torralba, F. Shkurti, and L. Paull. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. arXiv preprint arxiiv 2.009166500 2023.
14

---

Our contributions are as follows: i) we propose action-coni citionccondiiooed 3D scene graph and introduce the interactive scene exploration task to address the challenging interaction aspect of exploration; ii) we develop the RoboEXP system, capable of exploring complicated environments with unseen objects wide range of settings; iii) through extensive experiments, we demonstrate our system's ability construct complex and complete action-conditioned 3D scene graph, demonstrating significant potential for various manipulation tasks. Our experiments involve rigid and articulated objects, nested objects like Matryoshka dolls, and deformable objects like cloth, showcasing the system's generalization ability across objects, scene configurations, and downstream tasks.
A Additional Details of Interactive Exploration
Due to space constraints, we did not include the pseudocode of the algorithm proposed in the main paper, but include more details here for clarity. We formulate the interactive scene exploration task into active perception and exploration problem to construct the action-conditioned 3D scene graph (ACSG).
Algorithm 1 Interactive Exploration
(Vo, E.), U V 2: while - * do
input: 0 G
if' choose object o, E U then
explore object
4: add spatial relations (Algorithm 2)
% memory % decision-making
5: obtain action a to explore O;
6: if action a & V then
7:
V2. U Vt-1 {a} U [a}
% add node % add edge
8:
E U'= U O
= E
Co.
9:
mark as explored
10:
end if
11:
else choose action a. E U
12:
if no obstruction then
% decision-making % action % perception
13:
take action ag
14:
obtain new observation O
15:
found new objects O ø Vt-I then
16:
V U V'U(O, Ut- U (0)
add nodes add edges
17:
E' E U (eaa--00
18:
U U ag
% mark explored
19:
end
20:
else
21:
add action preconditions (Algorithm 3)
% memory
end if
23:
end if'
end while
25:
output: G
% final scene graph
In the above algorithm, we have demonstrated how to construct the edges from objects to actions Co-a and from actions to objects eo- however, there is a lack of description for the other two types of edges.
Add Spatial Relations. The logic involves analyzing the spatial relationships among objects using spatial heuristics and incorporating the resulting spatial relation edges between objects eo (Algorithm 2).
Add Action Preconditions. The approach is assess the feasibility of implementing the actions. We utilize the decision-maker module verify whether there are any prerequisite actions that need to be completed beforchand, and then adjust the plan accordingly (Algorithm 3).
15

---

Algorithm 2 Add Spatial Relations
1: imput: G
(V E
2:
E'
== =
3: for V do
% check relations memory % add edge
4:
if relation from o to o, then
5: ='= {e.
6: end if
7: if relation from o, then
8: E' D'=E" E' U -o)
% add edge
9: end if
10: end for
output: G
new scene graph
Algorithm 3 Add Action Preconditions
1: input: G (V E U
2: if object o obstruct then
x decision-making
3: choose action
4: V ( {a). U Ufa}
% add node
5:
E' E
lo-aa
% add edge
6: E E (ea
% add edge
7:
end if
8: output: G
% new scene graph plan
B Additional Details of RoboEXP System
We provide additional details about our system and each module in it. We then discuss system's design for the interactive scene exploration task, focusing on its application in closed-loop exploration processes that may require multi-step or recursive reasoning and handle potential interventions. Additionally, we explain the usage of our proposed ACSG.
B.1 Details of Modules in RoboEXP System
To tackle the interactive exploration task, present our RoboEXP system, designed to autonomously explore unknown environments by observing and interacting with them. The system comprises four key components: perception, memory, system ensures the thoroughness of our task in interactive scene exploration.
and action modules. This closed-loop
B.1.1 Perception Module
Raw RGBD images are captured through the wrist camera in different viewpoints and processed by the perception modules to extract scene semantics, including object labels, 2D bounding boxes, segmentations, and semantic features. As mentioned in the main paper, to obtain per-instance CLIP features, we implement a strategy similar to the one proposed by Jatavallabhula et al. [94]. Specifically, we extend the local-gles -ealbll image feature merging approach by incorporating additional label text features augment the semantic CLIP feature for each instance. Furthermore, exclusively focus instance-leve features, disregarding pixel-level features, thereby accelerating the entire semantic feature extraction process.
B.1.2 Memory Module
The memory module designed to construct our ACSG of the environment by assimilating obser- vations over time. For the low-level memory, to ensure stable instance merging from 2D to 3D, employ similar instance merging strategy presented in Lu al. [95], consolidating observations from diverse RGBD sources across various viewpoints and time steps. In contrast to the original
16

---

algorithm, which considers only 3D IoU and semantic feature similarity we additionally incorporate label similarity and instance confidence. To enhance algorithm efficiency, we represent low-level memory using a voxel-based representation with filtering designs, which allows for more efficient computation and cleaner memory updates. Meanwhile, given the crowded nature of objects in our tabletop setting, we have implemented voxel-based filtering designs to obtain a cleaner and more complete representation of the objects for storage in memory.
System: You an aocradent taaked with akinng in the constmaction a complet soee graph for a watwomp envicemmntt The objectio is to kiwntly oljects hidden from the current ocsencation in the tabretop serting Your rale imvolves selecting aporoprate actions or opting not to take any action based on commonsense Inowtddge in response to queros with current ocservations Your responses will guide a robot oticienty explorin the enstronment. Approsch each step thoughtfuly, and andlyas the fandamenial probiem desply, considering the polantal veguannes or insoouracy in the que es. Adhare to the provided formals in your instuctions.
(3) Prompts Proposer
categery, cone dering the sumoundinn objects and observetions the tabletop scons from diferent viewpoinss. The query objectipart will be endlosed in green bounding box, though i may not always be fully accurate. Format your responses queation Choose from theee seillc 1. Open the doors droweons 2. Pick up Open the top object a. No action The primary goal is to select an action that has the potentall reveal hiddon objects. The secondary gool to act effilentty portorming ony neceesary actions to uncover hidden osjects. For esample,
User: Analyr and proside fini answer for new
falinos "Arwyyol
messoning proceese: into (Final Answee) vakit= Be comprewesoww and sooid repeeting
object contains dopes drawers and can potentillyy stor somsthing makia opt for the firat skill "'ppnn the doers or drawars" an ciject has no batom sidu and can polintially cavar somathing benwath K, chocse the second skill Pick up Open the top object"; othanwise, select the thind skil "No action to ensure offidinny.
Assistant: Got will the reasoning process stap-byy-de stop, eaplain why choose
others and follow
format.
(Suery (itjent (Shvery Images)
Assistant: ploply from GP 1-4M]
environment, Your select sullable objects could obstruct actions besed quer and curmen obaer wilons. Provide guiteros moools planning process Approwch each Froughffulyy awwyyng oy the undwrlying prislem thoroughly whiw conskdering poietial veguannss ineccaracy the quariss. Follow the provided formats your instuctions.
System: assistant lasked evaluating feasibilty actions
(b) Prompis of Verifter
Provide analysis and your anower each difirent siompoints. quary ohpecipart
now cuery onjecipart canegony, the ist surronning ng objects you consider green
comespand drg the
be hay socurate Present your reasaning process
tormat "AAmtyste]" eyour reasoring processs; (Final Answert otjectse" provided the
the and
surrounding actions. object
obiects
polontaaly fins objects. Anwlyna the wction movement and identily the blocking cnjects.
opening, Include the
Assistant: Got will output the reasoning proceso step by stop, exprain why choose object but others follow the output
User: (Query Object+ youery Images)
Assistant: (Rasty from GPT-4V]
Figure 7: Prompts of the Decision-Making module. We present the full prompts for the two pivotal roles of our decision- making module, proposer (a), verifier in (b), The prompts are used for all our experiments without modification and extra examples.
B.1.3 Decision-Making Module
As illustrated in the main paper, the decision- maainn module fulfills two crucial functions within our system. The first function serves as an action proposer (Fig. 7a), proposing the appropriate skill for the query object node. The subsequent role functions as the action verifier (Fig. 7h), tasked with confirming the feasibility of implementing the action and determining the action preconditions. The complete prompts for both roles are detailed in Fig. 7.
We adhere to the standard practice for designing prompts, other papers do using LLM/LMMM [18, 83, 89]. In order not to compromise the generalization ability of our system, we consistently use the same prompts across all scenarios and experiments. Our fundamental rule for prompt design is to minimize ambiguity and ensure alignment with our task. In our experiments, the average response time from GPT-4V is about 8 seconds for each question, which is acceptable as GPT-4V is only used in high-level task planning. For low-level motion planning, the use of action primitives allows us to meet the high-frequency requirement without having constantly querying GPT-4V.
Our ACSG utilizes GPT-4V on every object node to progressively expand the graph. Hence, regardless of how complicated the scene is, each query posed to GPT-4V resides on a local node within our ACSG, essentially addressing the question. "Should proceed with exploring this object, and if so, how?" As shown in our Matryoshka scenario, RoboEXP performs well in complex scenarios featuring five levels of hierarchical scene graphs and complicated exploration procedures. The commonsense knowledge learned by GPT-4V enables our system to efficiently explore the environment without having to manually design the exploration rules for diverse objects.
17

---

B.1.4 Action Module
The action module focuses on providing useful action primitives to aid in constructing our ACSG. We have designed seven action primitives: "open the [door]", "open the [drawer]", "close the [door]", "close the [drawer]", "pick [object] to idle space", "pick back [object]". "move wrist camera to [position]" To fully support autonomous actions, we employ a heuristic-based algorithm leveraging geometric cues. The input to each action primitive is an object node of our ACSG, from which we can extract all necessary semantic, state, and low-level geometry information of the object. The integrated information can help us determine the specific grasping pose and path planning for opening and picking actions, which generalize to different instances in various positions and poses.
For opening action primitives related to doors or drawers, engagement with handles is required. In our implementation, we exploit the handle's position and geometry to discern its motion type (prismatic or revolute) and motion parameters (motion axis and motion origin). Executing this action involves utilizing the detected handle and its geometry to adeptly open doors or drawers. Upon identifying the specific handle to be operated, our system retrieves the point cloud converted from our voxel-based representation corresponding to that handle from our memory module. Subsequently, we employ Principal Component Analysis (PCA) to determine the principal direction of the handle, aiding in aligning the gripper for optimal engagement. Additionally, understanding the opening direction is pivotal for effectively handling doors or drawers. To ascertain this, we analyze neighboring points and deduce the most common normal as the opening direction. The combined information of the handle direction and the opening direction provides sufficient guidance for our robot arm to grasp the handle and open the prismatic part. However, in the case of revolute joint, the motion becomes more intricate. Therefore, we further utilize the motion parameters inferred from the geometry simulate the evolving opening direction based on the revolute joint's opening process. This well-designed heuristic empowers our system to reliably open drawers doors in our tabletop setting.
For the pickup-related primitives, we simplify the pickup logic to exclusively consider a top-down direction. Consequently, our focus narrows down to acquiring essential information such as the object's height and xy location. We achieve this by extracting the object's point cloud from its associated voxel-based representation. Subsequently, we pinpoint the highest points within the cloud, calculating their mean to determine the optimal pickup point. This calculated point serves as a precise reference for our gripping mechanism, facilitating the successful grasping of objects in the specified direction.
Regarding viewpoint change, the primitive is parameterized with the expected pose. For example, after opening the door/drawer, to see inside, we develop the heuristic to choose the proper viewpoint from the open direction the parameter for the primitive, allowing for the implementation of the action primitive.
B.2 Other Design in Interactive Exploration
One desiderata for robot exploration is the ability to handle scenarios that necessitate multi-step or recursive reasoning. An example of this is the Matryoshka doll case, which cannot be addressed using previous one-step LLM--ased code generation approaches [18, 19]. In contrast, our modular design allows agents to dynamically plan and adapt in a closed-loop manner, enabling continuous LLM--basd exploration based on environmental feedback.
To manage multi-step reasoning, our system incorporates an action stack as a simple but effective "planning" module. Guided by decisions from the decision module, the stack structure adeptly organizes the order of actions. For instance, upon picking up the top Matryoshka doll, if the perception and memory modules identify another smaller Matryoshka doll in the environment, the decision module determines to pick it up. Our action stack dynamically adds this pickup action the top of the stack, prioritizing the new action over picking back the previous, larger Matryoshka doll. This stack structure facilitates multi-step reasoning and constructs the system's 's logic in deep and coherent structure.
18

---

Moreover, for the interactive scene exploration task, maintaining scene consisteney is crucial in practice (e.g., the agent should close the fridge after exploring it). We employ greedy strategy returning objects to their original states. This approach keeps the environment close to its pre- exploration state, making RoboEXP more practical for real-world applications.
B.3 Usage of ACSG
The ACSG constructed during the exploration stage shows beneficial for scenarios that require comprehensive understanding of scene content and structure, such household environments like kitchens and living rooms, office environments, etc. We list several examples illustrating the potential usage of the scene graph in various tasks.
Judging Object Existence. A direct application of our ACSG is to determine the presence or absence of specific objects the current environment. For instance, during the exploitation stage of the scenario (App. D) set the dining table, if the spoon is missing, the rohot further seek human assistance.
Object Retrieval. One notable advantage of our ACSG its ability to capture all actions and their preconditions. Utilizing this information, retrieving any object becomes straightforward by following the graph structure and executing actions topological order along the paths from the root to the target object node. For example, in the obstruction scenario (App. D), the ACSG can provide the sequence of actions required to fetch the tape: removing the condiment blocking the cabinet door, 2) opening the cabinet via the door handle, and 3) retrieving the tape. Such insights are crucial for tasks like cooking.
Advanced Usage. The high-level representation of the environment provided by our ACSG serves as a simplified yet effective model. Similar to the approach proposed by Gu al. [96], integrating the scene graph with Large Language Models (LLM) or Large Multi-modal Models (LMM) offers enhanced capabilities, including natural language interaction. This enables the robot to respond to human preferences expressed in natural language (e.g.. fetching a coke when the person is thirsty) or through visual cues (e.g., fetching a mug when the table is dirty).
C Additional Details of Experiments
We conduct experiments in different settings to validate the effectiveness of the model. We pro- vide additional experiments and results, includ- ing those with different lighting conditions and backgrounds, using different LMMs, interven- tion experiments, and several more room-level scenarios.
C.1 Experiment Settings
Our experimental setup encompasses diverse range of objects, illustrated in Fig. 8. To as- sess the effectiveness of our system, we devised five types of experiments for the main quan- titative results, each encompassing 10 distinet settings. These settings vary terms of object number, type, and layout, illustrated in Fig. 13. food items, tapes, tableware, and fabric.
Figure 8:
All Testing Objects. present various objects utilized in our work, encompassing different types of cabinets, fruits, dolls. condiments, beverages.
C.2 Evaluation
The details of five metrics employed for evaluation follows:
19

---

1) Success:
This metric evaluates the success percentage across 10 variants for each task. We define success for each experiment as when the final outputted ACSG exactly matches the GT version, and 0 otherwise.
2) Object Recovery: This metric quantifies the percentage of hidden objects successfully identified.
3) State Recovery:
A binary value indicates whether the final state resembles the original state before exploration. This includes considerations for partial states and object positions (e.g.. in the top drawer cabinet or on the table).
4) Unexplored Space: Evaluating the percentage of successfully explored heed-to-explos space reduce the robot's uncertainty about the scene. The identification of the need-to-explore space relies on human annotation.
5) Graph Edit Distance (GED); GED measures the disparity between the outputted graph and the GT graph. We adopt simplified version of GED with six operations- thrre for nodes (add, delete, edit) and three for edges (add, delete, edit), with each operation incurring a cost of
C.3 Baselines
GPT-4V Baseline: We employ the pure GPT-4V our baseline model along with the chain-of- thoughts (CoT) to enhance its capabilities, as outlined in a method similar to that proposed by Hu et al. [19]. The full prompt of the GPT-4V baseline is illustrated in Fig. 9.
Heuristic Baseline: We also design two heuristic baselines: i) Heuristic- Open: This method opens all handles; ii) Heuristic-Full! In addition to opening all handles, this method also picks up all movable objects reveal what's 's underneath them.
For the Random baseline, an action is chosen randomly for each object, including the option to take no action. Similar to the GPT-4V baseline, we make the heuristic and random baselines stronger by assuming that all actions are successful. We conducted the same set of experiments as our original paper (5 task scenarios with 10 different settings for each scenario) for each of the baselines. All methods use the same perception module our RoboEXP to understand the influence of different decision strategies.
Random Baseline:
System: are assiaatrn i widng consiration completa soare grech labletop endiommmnt. object klantify objects hinden
ooservatio tattetop setting Your invhees selecting appropratt actions optig to any action keoutedge response quedes
obserattons. resporses will guide polental vegueness
etheretly exploring Achore provided
Approach stop the tro.ghth.ly.
problem deaply,
nest artin glven Isteet oteercaions satrenn scens diffeent torminate. The action you can should composed (sobecopparo. <skili)) speciiic ooject or part refer to. The sidll you can choose: 11. Open door. 2. Close the Open drawer Cless the drower, 5. your
User:
Anslyre and pmovid curmett azeen grich and your final ancwer
Newppiitss Each you plex action
you potental oupuu reseoning process step-by-erpp why
speni)
you Viin (Final
yail). syour
<akii>" comprshenssiv eficientty performing
repeating my The gool select
hidden oijects. The secontary inital stone wher exploration For ourput acane graph, you
third to make the need all obyects the inchrding those found the explonetio procees..
only
ricden
Assistant:
output
User: [Cuery Images]
Assistant: [Reply from GPT-4V]
User:
[Quary Images]
Assistant: [Reply GPT-4V]
Figure 9: Prompts of the GPT-4V baseline. To ensure fairness in comparison to this baseline, we choose to similar prompts, employing the chain-of-thoughts technique enhance its performance.
C.3.1 Comparison to Heuristic & Random Baseline
We present all results in the above table in Tab. 2, which shows that our RoboEXP system is more capable and efficient in nearly all scenarios. The random baseline easily fails due to improper action selections. The Heuristic-Open baseline performs well in scenarios involving only open-action judgments but fails when pick actions required. The Heuristic- Full baseline can provide better object recovery; however, its redundant pick actions create scene graph with too many redundant
20

---

Table 2: Quantitative Results Different Tasks.
We compare the performance of baselines and system across various tasks. Our system outperforms baselines across nearly all metrics.
Method
Success %
Object Recovery %
State Recovery
Unexplored Space %
Graph Edit Dist.
Drawer-Only
Random
0:::..0
20::13.3
10::10
8..17..6
6.6:10.73
Hcuristic-Opee Open
90=:10.0
97:33.3
100=0.0
0=:0.0
0.2:+0.20
Heuristic-Full
0++0.0
97+3.3
100-0.0
0+0000
3.8+0.47
GPT-4V
20:113.3
83::11.0
16.3
15:17.6
2.8.1
Ours
90:110.0
97:13.3
100=0.0
0::0.0
0,2:0.200
Door-Only
Random
0::0.00
30
0=0.00
85.17.64
6,311.01
Heuristic-Oper
90::10.0
100::0.0
100=0.0
0::0.0
0.1:10.10
Heuristic-Full
01:0.0
100110.0
100=0.0
010.00
3.8:0.477
GPT-4V
30115.2
50116.7
80=13.3
40.
4,4111522
Ours
90=110.0
100-:0.0
100=0.00
0:0..0
0.140.110
Drawer-Door
Random
0::0.0
4#:40
0=0.0
85=6.7
12.8+00.80
Heuristic-Open
704:15.3
91:44.7
100=0.00
0:0.00
0.5::027
Hcuristic-Full
0::0.0
91::4.7
100=0.0
020.00
6.1:00771
GPT-4V
10::10.0
62:+10.7
70=15.3
25=6.6
5.6+1.46
Ours
70+15.3
91+44.7
100-0.0
0+0.00
0,5++0.27
Recursive
Random
10+10.00
33+11299
60-16; 16.3
70+122.8
8.8+11911
Heuristic- Open
0:10.0
0:10.0
100-0.0
10010.0
10:11.69
Heuristic-Fil Full
40+11.33
90+010.0
100=0.0
5+5.00
18++0.63
GPT-4V
0+:0.0
20--13.3
70=15 15.3
63-355.3
8.8+
Ours
70:115.3
80:11..7
1.0
15:18.9
2.1:11.49
Occlusion
Random
0::0.0
0.10.0
0=0.0
10010.0
9.8110.33
Hcuristic-Open
0::0.0
22:11.7
0=0.0
7548.3
8.6:10.72
HeuristicFFull
0::0.0
22.111.7
0=0.0
7548.3
9,4-10.58
GPT-4V
01:0.0
17111.4
10=10.0
8517.6
7.3.10977
Ours
50116..7
67:14.9
70=15.3
30::15.3
2.511.15
Table 3: We report the average number of actions for settings in which all methods find all objects, explore all hidden spaces, and recover the states before exploration. In the recursive and occlusion scenarios, baseline methods do not have experiments that meet these requirements. fewer actions compared to the baselines and is more aligned with the ground truth of human action choices.
RoboEXP consistently requires
Method
Drawer-Onle Only
Door-Only
Drawer-Door
Recursive
Occlusion
GT (Human)
4.0
4.0
8.0
4.6
6.0
Heuristic-Full
7.0
8.0
12.0
6.0
GPT-4V
6.0
4.4
8.0
Ours
4.0
4.0
8.0
4.6
6.3
nodes, resulting in higher Graph Edit Distance most scenarios. In complicated scenarios like object- object occlusion, the heuristic baselines completely fail, as they do not reason about object-object relationships as we do using the action verifier in our system.
There are two main reasons why heuristic-based baselines are not ideal: i) Desining all potential heuristics to cover every scenario is nearly impossible. Specifying and enumerating all semantic categories tedious, and some size-based heuristics can also fail (e.g., threshold that too small may filter out objects that should be picked up. such as clothes on a Kindle, while threshold that is too large may include non-movable objects); ii) Baselines with manually defined heuristics can create excessive redundant actions. They tend pick up everything, making exploration inefficient. In fact, RoboEXP be viewed special form of frontier-base method with LMM- -MM-gddded action selection, which demonstrates better efficiency and effectiveness in our interactive exploration task.
21

---

(D) Scenano
(d) Scenanio
100 + 0.0
100 + 0.0
100 + 0.0
100 0.0
75.1 28.9
92
Random round
100
100 0.0
100 + 0.0
100 = 0.0
Figure 10: Experiments on Extreme Illumination and Random Background. We conduct experiments in four scenarios with varying lighting conditions and random backgrounds. The reported numbers are averages over four seenarios for each condition. Our system performs well across all conditions. (OR refers Object Recovery metric).
Table 4: Quantitative Results Different LMMs. We conduct experiments with GPT-4V and LLa VA acting as core RoboEXP decision module, under the same fifteen settings as in Fig. 10.
Metric
Success % 1 Object Recovery % f
State Recovery %
Unexplored Space %
Graph Edit Dist. J
Ours (ILaVA)
25+-25.6
50+-29.6
100++000
23--11.9
2.5++0.98 0.1::0.26
(GPT4V)
95--12.9
98-14.3
100:10.0
0:0.0
C.3.2 Comparison of Efficiency
The number of actions is only comparable when all hidden objects are found, all unexplored spaces are fully explored, and all object states recovered after exploration. Given that the Random and Cheuristic-Oper baselines do not provide comparable exploration results, we primarily compare the efficiency of Heuristic-Full: GPT-4V, and our RoboEXP system. We report the average number of actions in settings where all methods find all hidden objects, fully explore the unexplored space, and restore the object states after exploration the above table in Tab. 3. The results show that our RoboEXP consistently takes fewer actions construct the full ACSG during the interactive exploration process and is more aligned with human action choices.
C.4 Extreme Illumination and Random Background
RoboEXP is robust extreme lighting conditions and complex backgrounds. To demonstrate this, we tested under four different scenarios, each with varying lighting conditions and random backgrounds. Fig. 10 shows twenty different settings and their corresponding results. various conditions and scenarios, our system able to successfully conduct interactive exploration and construct the ACSG, indicating the robustness of RoboEXP to these factors.
C.5 Performance Different LMMs
RoboEXP is compatible with different multimodal foundation models beyond GPT-4V. We conducted additional experiments using the latest LLaVA-v1.6-34B the core of our decision module and compared it against GPT-4V under the same settings. Tab. 4 shows that both models can work with
22

---

our RoboEXP system, yet the capacity of LMMs does influence the overall performance. In general, GPT-4V achieves higher success rate and more consistent behaviors across different scenarios.
Tablo
Piate
Cabliet
Callnet
(a) Add Cabinet
belonce
Lolon
balord
beloroa
Drawer Handle
Drawer Handle
Drawer Handle
Drawer Handle
Drower Handle
Add Cabinet
Open
Opan
Open
Open
Open
Apple
Enatie
Apple
Remove Orange
Table
(b) Add/Remove Object
Cabinol
beloro
beloro
Mandle
Handle
Add Coke
Orange
Qualitative Results Different Intervention Scenarios. This scenario involves adding cabinet the tabletop setting, our system can auto-detect the new cabinet and explore objects inside. (b) This scenario includes removing and adding objects from and into the cabinet. Our system can monitor hand interactions and re-explore the corresponding doors.
Figure 11:
C.6 Human Intervention
Our RoboEXP system possesses the capability autonomously adapt changes the environment. We employ two types of human interventions to demonstrate these points (refer App. D).
The first type of intervention (Fig. 1 involves adding new cabinets to the scene. In this scenario, we add a cabinet to the explored area, allowing our system to automatically explore the newly added cabinets and update the ACSG.
The second type of intervention (Fig. 11b) involves adding objects to removing existing from the cabinets in the current scene. Our system can monitor human interactions and discern which objects require re-exploration. Subsequently, it autonomously updates the ACSG based on re-exploration.
C.7 Room-Level Household Scenarios
RoboEXP can work well in room-level household environments. To demonstrate this, we conducted two experiments within an apartment (see Fig. 12), specifically the dining area and bedroom. We integrated four RGB-D observations captured by handheld RealSense D455 with ICP-based multi- way alignment. Our system successfully constructs corresponding scene graphs within the room-level decision module effectively
household environments. Once the static scene graph is constructed, identifies the correct objects for exploration. Specifically, it accurately identifies the fridge in Fig. 12 (a) and the cabinet in Fig. 12 (b) for further exploration. Tab. 5 shows the complete responses from GPT-4V in our decision module on determining the actions to take in our two household scenarios.
D
Video Timeline
Scenario A. Exploration-Explooiaa
23

---

Floor
Dining Area
Fridge
Table
Chair
ong
ono
on
Mug
Laptop
Kettle
belong
Bedroom
Cablnet
on
balong
belong
TV
Handle
Handle
belonge
belong
Handle
Handle
(a) Observations with Segmentation Masks
(b) Low-Level Geometry
(c) High-Level Scene Graph
Experiments Room-Level Houschold Scenarios. We conduct experiments in two room-level household environments. dining area includes a table, fridge, and items on the table, whereas bedroom includes a bed, cabinet, and a TV. The tigure presents (a) the observations with segmentation masks; (b) the low-level reconstructed geometry; (c) the built high-level scene graph.
Figure 12:
Exploration: 00:43 01:16
Exploitation: 01:17 01:37
Scenario B. Recursive Reasoning
Exploration: 01:49 02:26 (Two scenarios)
Scenario C. Obstruction
Exploration: 02:33 02:59
Scenario D. Intervention
Exploration: 03:05 04:09 (Two scenarios)
Extension to Mobile Robot
Exploration: 04:13 04:53
24

---

(a) Only
(b) Only
(c) Drawer
(e) Occlusion
Figure 13: Experiment Settings. Varied object numbers, types, and layouts in our experimental settings of the quantitative results.
25

---

Table 5: Decision Module Logs for Household Scenarios. We present all responses from GPT-4V for household scenarios: the dining space case and the bedroom case. The inputs to our decision module are images from various viewpoints of the query object in green bounding box.
Query Object
LMM Respouse
Dining Space
Fridge
The object in question is a fridge, which typically used to store food and drinks. The green bounding box in the images highlights the fridge, which has doors that can be opened. Since the goal is to identify all objects hidden from the current observation in the tabletop setting, and a fridge is an appliance that commonly contains items inside, it is reasonable to assume that opening the fridge could reveal hidden objects.
[naalysis]:
[Final Answer]: Open the doors or drawers.
Chair
[Analysis]: The object in question is a chair, which is typically not used to store objects in way that would require opening or removing parts to reveal them. Chairs are designed for sitting, and while it is possible for items to be placed on or under them, the chair itself does not have compartments that would be opened or closed. The images show the chair from different angles, and there is no indication that there are hidden compartments or objects on the chair that would necessitate an action to uncover them. The chair does not have a top object that would be opened or picked up, and it does not have doors or drawers.
[Final Answerj: No action.
Table
[Analysis]: The object in question is a table, which is highlighted with a green bounding box in the images provided. From the different viewpoints, it is clear that the table has a flat top surface with no visible drawers or doors that could he opened to reveal hidden objects. The table does not appear to have a covering that could hide something underneath it. The table's 's design is simple, with single central support and no storage compartments. Therefore, there is no action required that would potentially reveal hidden objects in this case.
[Final Answer]: No action.
Laptop
The object in question is laptop, which is placed on table. Laptops do not have a hidden compartment that would typically contain other objects, nor do they obstruct the view of ohjects underneath them when they are in an open position, as seen in the images. The laptop is already open, and there is no indication that there is anything on top of it or that it is covering anything beneath it. Therefore, there is no action required that would reveal any hidden objects in this case. [Final Answer]: No action.
[Analysis]:
Mug
The mug in the green bounding box is sitting tabletop and does not appear to be covering anything. nor does it have lid that could be hiding something inside. Mugs are typically open contuiners and do not have compartments that would conceal objects. Therefore, there is no need to pick it up to look underneath.
[Analysis]:
[Final Answer]: No action.
Kettle
The object in question is a kettle, which is sitting on top of a table. Kettles are typically used to store other items, especially when they are placed in setting like tabletop where they are likely used for their intended purpose of boiling water. The kettle does have drawers or doors, and it is unlikely that it is obscuring any items on the table due to its shape and size. Therefore, there is need to open any part of the kettle, and it not obstructing any items underneath it. [Final Answer): No action.
[Analysis]:
Bedroom Cabinet
The object in question is a cubinet, as indicated by the green bounding box in the provided images, Cabinets typically have doors or drawers that can be opened to reveal contents stored inside, In the images, the cabinet appears to have several drawers. Since the primary goal is to select an action that has the potential to reveal hidden objects, and the cabinet has drawers that could contain items, the appropriate action would be to open these drawers. [Final Answerl: Open the doors or drawers.
[Analysis]:
Bed
The object in question is a bed, which is typically not a storage furniture picce with doors or drawers. However, some beds do come with storage options underneath, such as drawers or a lifting mechanism for accessing a storage space. In the provided image, the bed appears to have a solid base without visible drawers or doors. There is no indication that the bed has a mechanism to lift the mattress to reveal storage underneath. Therefore, it is unlikely that there are hidden objects within the bed that can be accessed by opening doors or drawers.
[Analysis]:
[Final Answer]: No action.
Television
The object in question is television, which is placed on top of a surface, likely a media console or a dresser with drawers. Televisions are not typically used to store objects behind or within them, as they are electronic devices designed for display purposes. The images show the television from two different angles, and there is no indication that there are any hidden compartments or objects behind the screen that would be revealed by moving it. Additionally, televisions ure generally not designed to have openable parts that would conceal objects. The media console or dresser beneath the television does have drawers, but since the query is specifically about the television, these are not the focus for the action to be taken. [Final Answer]: No action.
[Analysis]:
26


Reviews:
[{"review_id": "icgeSTzAxy", "reviewer": "Reviewer_Pzsb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors propose a RoboEXP system, which generates an action-conditioned scene graph via interactive exploration and the LMM is incorporated.", "summary_of_recommendation": "The generalization of the task to real-word environment and some details of the method are not quite clear.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- The setup of the scenario and how the setup and commonsense knowledge affect the results.\n\n- The efficiency of the comparison methods. And some heuristic method is suggested.\n\n- The process of scene graph generation in the task.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JfPaUG3ou1", "reviewer": "Reviewer_hwZM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel way to represent interactions in the environment using action-conditioned scene graphs and exploration for manipulation problems.", "summary_of_recommendation": "The work seems promising, but requires some clarifications within the text.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Does the user need to specify all the interactions a-priori? It seems that one can interact with an object an infinite number of ways, so how does the graph, which is finite, deal with this?\n\nWhat does it mean to take an action that reduces the unexplored node set? Specifically, in a real-world application, how does an action yield a node from the environment. Is this part of the perception module?\n\nHow does the agent know when the exploration is terminated? If it has access to the unexplored node-\tset, then why explore it at all and enumerate each node?\n\nWhat is the ACSG “output”?\n\nThe DEG metric and unexplored space metric seem like they would produce similar meanings. Could their differences and implications ve explained more clearly?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BT8kEYHY9j", "reviewer": "Reviewer_5sxQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a scene representation called the Action-Conditioned Scene Graph, which represents both inter-object spatial relationships and the necessary actions to reveal elements of the scene graph. The paper then describes an exploration framework which constructs the proposed perceptual representation through interaction. Experiments on real-world robotic manipulation tasks demonstrate increased success rate and improved exploration metrics compared to a baseline heuristic approach.", "summary_of_recommendation": "The paper is clearly presented and has been evaluated on real robots. The concept of the action-conditioned scene graph is well motivated. However, more clarity is needed regarding the assumptions of the system in order to generalize across scenarios. Furthermore, comparison against heuristic baselines would strengthen the claims of the work.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How does RoboEXP compare to other exploration heuristic baselines, like random or frontier exploration?\n2. How much time (or how many actions, as is relevant) is needed to construct the scene graph?\n3. What are the assumptions about the environment and type of actions? \n\nSee additional details and points above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to generate action-conditioned scene graphs and uses it for interactive exploration for several manipulation tasks. The reviewers find the paper well written, find the results promising, and appreciate the real-world experiments. However, the reviewers have also raised several concerns including lacking baselines or ablations, unrealistic scenarios in the experiments, insufficient analysis on application of the scene graph, lacking details on the proposed method, require clarity on the research questions, clarity on the metrics used,  lacking clarity regarding the assumptions of the system, among others. Each of the reviewers has listed several questions to be addressed in the rebuttal.

Post-rebuttal: Most of the concerns of the reviewers have been sufficiently addressed in the rebuttal. The paper could be stronger if results were presented in more complex realistic scenes.
Decision: Accept