Embodied Semantic Scene Graph Generation
Authors: Xinghang Li
Di Guo
Huaping Liu
Fuchun Sun
Venue: CoRL2021 Poster (2021)
Keywords: Semantic Scene Graph
Embodied Exploration
Learning for Visual Navigation

Abstract: Semantic scene graph provides an effective way for intelligent agents to better understand the environment and it has been extensively used in many robotic applications. Existing work mainly focuses on generating the scene graph from the sensory information collected from a pre-defined path, while the environment should be exhaustively explored with a carefully designed path in order to obtain a comprehensive semantic scene graph efficiently. In this paper, we propose a new task of Embodied Semantic Scene Graph Generation, which exploits the embodiment of the intelligent agent to autonomously generate an appropriate path to explore the environment for scene graph generation. To this end, a learning framework with the paradigms of imitation learning and reinforcement learning is proposed to help the agent generate proper actions to explore the environment and the scene graph is incrementally constructed. The proposed method is evaluated on the AI2Thor environment using both the quantitative and qualitative performance indexes. Additionally, we implement the proposed method on a streaming video captioning task and promising experimental results are achieved.

Embodied Semantic Scene Graph Generation
Xinghang Li, Di Guo, Huaping Liu", Fuchun Sun Department of Computer Science and Technology. Tsinghua University, China Beijing National Research Center for Information Science and Technology, China "Corresponding author: Huaping Liu (hpliu @ lsinghua.edu.cn)
A Abstract: Semantic scene graph provides an effective way for intelligent agents better understand the environment and it has been extensively used in many robotic applications. Existing work mainly focuses on generating the scene graph from the sensory information collected from a pre-defined path, while the envi- ronment should be exhaustively explored with carefully designed path in order to obtain a comprehensive semantic scene graph efficiently. In this paper, we pro- pose new task of Embodied Semantic Scene Graph Generation, which exploits the embodiment of the intelligent agent to autonomously generate an appropriate path explore the environment for scene graph generation. To this end, learning framework with the paradigms of imitation learning and reinforcement learning is proposed to help the agent generate proper actions explore the environment and the scene graph is incrementally constructed. The proposed method is evaluated the A12Thor environment using both the quantitative and qualitative perfor- mance indexes. Additionally, we implement the proposed method on streaming video captioning task and promising experimental results achieved.
Keywords: Semantic Scene Graph, Embodied Exploration, Learning for Visual Navigation
Introduction
The scene graph is collection of nodes in graph structure where nodes usually represent scene entities and edges represent geometrical transformation between the nodes. It provides a flexible way to track objects and their spatial re- lations within the scene. Recently, the concept of scene graph is successfully adapted in many applications in computer vision and robotics [1][2]. For example, the Visual Genome dataset [3], which contains annotations of objects and their relationships
Tranci
Biid S4 work Saow
Teat
images, has been exten- sively utilized in various scene graph genera- tion tasks [4][5][6]. In [1], the author proposes to extract the knowledge graph from recorded videos for better video understanding. Xu et [2] provides an extensive survey on the recent progress on the generation and applications of the scene graph.
The illustration for the Embodied Se- mantic Scene Graph Generation. The agent moves around and always selects appropriate action to obtain better semantic scene graph. Further, the generated semantic scene graph can be helpful to generate the streaming video captioning.
Figure 1:
Furthermore, 3D semantic scene graph is of great interest for robotic applications such as navigation, mapping, and interaction [7]. To understand the 3D environment, Kim et al. [8] proposes a 3D scene graph construction framework using RGB-D data processing pipeline, which can be used for task planning and visual question answering. Bozcan and Kalkan [9] develops Boltzmann Machines-based generative scene model bringing objects, their spatial relations and affordances together. Armeni et al. [10] presents a 3D
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

scene graph, which unifies multi-moda semantic information in the 3D space. Recently, Rosinol al. [11] proposes a 3D dynamic scene graph which extends represent dynamic scenes with moving agents.
Though great success has been achieved for semantic scene graph, all of the above work focuses on generating scene graph from the sensory information collected from a pre-defined path. How- ever, to obtain a comprehensive and informative semantic scene graph, the environment should be exhaustively explored, which requires large time consumption. In many scenarios, the disconnec- tion between the scene graph generation and the exploration path even results in unsatisfied results. Therefore, it is necessary for the agent to have the ability to autonomously explore the environment to generate a good semantic scene graph.
In this work, we adopt the embodied exploration framework to tackle the Embodied Semantic Scene Graph Generation problem, which has never been addressed before. It combines the action and perception abilities of the agent. The embodied agent generates actions to autonomously explore the environment and then incrementally constructs high-quality semantic scene graph using the detected object instances and contextual information. The collected sensor data is continuously processed and incorporated into the scene graph model along with the exploration process. The illustration is demonstrated in I Different from the existing work which focuses on generating scene graphs from acquired sensor information, our work introduces the embodiment of the agent to effectively combine the action, vision and language together for autonomous 3D semantic scene graph generation. The main contributions are summarized as follows:
1. We propose new framework for the Embodied Semantic Scene Graph Generation prob- lem, which exploits the embodiment characteristic of the intelligent agent to find an appro- priate path for scene graph generation in embodied environment.
2. We develop a learning framework with paradigms of imitation learning and reinforcement learning to help the agent acquire the intelligence to generate high-quality scene graph.
3. We test the proposed method on the A12Thor dataset and evaluate its effectiveness using the quantitative and qualitative performance indexes.
For the rest of the paper, Section II presents the related work. Problem formulation and architecture are introduced in Section III. Sections IV and V give details about the scene graph generation and navigation modules. Section VI presents the experimental results and Section VII closes this paper.
2 Related Work
This work falls into the intersection of the embodied exploration and semantic understanding. There- fore, we present a brief review on both domains.
The embodied exploration lies in the intersection of the computer vision and robotics. The rep- resentative work includes active sensing [12][13], embodied question answering [14], embodied captioning [15], embodied amodal detection [16], multi-agent embodied exploration [17]. etc. In these works, one or multiple embodied agents equipped with visual perception modules can au- tonomously explore the environment to perform various active sensing and perception tasks. Zhang and Mei [18] proposes a constructive model for collective intelligence in which each agent would explore and communicate to cooperate with each other. In recent years, researchers have combined deep reinforcement learning with active vision sensing. Han et al. [19] builds an active vision system based on DQN to guide the agent generate appropriate actions and get better images for detection. Furthermore, Chaplot et al. [20] studies the task of embodied interactive Icarning for object detec- tion. It should be noticed that in [15], the authors propose embodied captioning task. which is relevant to our work, while our work solves a more fundamental task of generating the 3D scene graph, which provides more comprehensive contextual information of the environment. Actually, the generated scene graph can be useful for many down-streaming tasks such as image captioning, streaming video captioning, embodied Questions and Answering, etc.
On the other hand, the embodied exploration task can also benefit from the semantic scene graph. Druon et al. [21] and Zeng et al. [22] study how to enhance the active object search using the spatial context and semantic linking maps respectively. The hierarchical mechanical search also shows performance improvement by using the semantic modeling [23]. Du et al. [24] proposes to learn

---

Semantic Point Cloud Updating
Global Point Cloud Updating
Global Scene Graph Updating
coud
depth frame
RGB hrame
prevous global scene graph
updee
lool
updsted global soona graph
semantic point cloud
updateed
8GB image
scone graph
Navigation Model
t+1:
previous action
new action
Figure 2:
The architecture of the Embodied Semantic Scene Graph Generation. time instant t, after taking one action, the agent comes to new viewpoint and updates the local and global scene graph, and further exploits the local and global scene graph to generate new action. This process iterates until a stop action is obtained.
object relation graph and tentative policy for visual navigation. Besides, the scene graph also plays an important role in the visual question answering system for robot manipulation [25]. These works illustrate the significance semantic scene graph in embodied perception. However, they all use existing scene graphs and do not address the generation of the scene graph.
Problem Formulation and Architecture
The goal of this work is enable the agent automatically generate sequence of actions to explore the environment and build the corresponding semantic scene graph incrementally. Con- cretely speaking, the agent starts from any location of scene with an initial empty global se- mantic scene graph GSSGo. At time instant t, the agent could construct local scene graph LSSG, (A R, R(()) } from its observation 8t, which could be extracted from observed RGB and depth images. The local scene graph LSSG, should be merged with the previous global scene graph GSSG,-1 (M(2) R(0)
get the updated global scene graph GSSG, (A? R(")). ). Please note that we follow the conventional symbols with slight modifications [26], i.e. Agl!) and R((") represent the node and semantic relationships the local scene graph LSS9,, and N( and (9) represent the node and semantic relationships for the global scene graph GSSG,. After that, a navigation module should produce an action a, for the agent obtain the next observation 81+1- The above procedure iterates until a satisfactory global semantic scene graph is achieved and the stop action is triggered. The obtained semantic scene graph can then be used to describe the environment comprehensively and solve some downstream tasks such captioning, question answering, etc.
We call this problem Embodied Semantic Scene Graph Generation, which exploits the embodi- ment capability of the agent to collect data and construct high-quality scene graph. It is significantly different from existing scene graph generation work such as [8][10]. To solve this problem, we design an architecture as shown in 22g.2. It is mainly composed of the scene graph generation and navigation module.
the scene graph generation, 3D semantic point cloud is produced from the RGB and depth images and is used for generating the local scene graph, which is then used to incrementally update the global scene graph. For the navigation module, we utilize the RGB frame, previous action and the scene graph input and an action is generated for the next step.

---

4 Scene Graph Generation
The scene graph generation module can be divided into two parts: local scene graph prediction and global scene graph generation, which are introduced as follows.
Local scene graph prediction: we mainly follow the graph convolution network based scene graph generation method which is able to pass the information between objects. It is widely used in tasks that utilize the scene graph to bridge the gap between semantic and visual information [26][27]. The generated local semantic scene graph includes the current in-sight objects and their relations with each other. We additionally introduce object class embedding and bounding box coordinates as the input for each object node, and the training objective is to restore the bounding box coordinates and labels for each object and edge. Please see the supplementary material for more details.
Global scene graph generation: At each time step, the agent takes an action to move, and the equipped camera captures the RGB and depth frames of the scene to construct local scene graph, which is further merged into the global scene graph. During this period, the detected objects in the local scene graph and the existing objects in the previous global scene graph are aligned.
Different from the mulli-view consistency considered in [10], which has the 3D mesh and multi-view visual information known in advance, we do not have any prior information about the scene and the information of each object is incrementally updated. Considering this situation, we propose a point cloud based weighted voting mechanism which utilizes the number of the pixels in the point cloud to measure the confidence of the object category. In practice, for each detected object, we maintain a score distribution across predefined classes. The score is incrementally accumulated by the weighted value of the confidence provided by the detector and the size of the acquired point cloud.
Then, we align the object point cloud with the nodes in global scene graph by computing the fraction of point cloud in the object point cloud that is inside the node's 3D bounding box. We calculate this fraction between each object point cloud and the node in the global scene graph. If the highest fraction of object point cloud of the corresponding node is higher than the preset threshold and this fraction is higher than those with other objects, the ob- ject and the node are considered aligned. This
Lecil OOeE
bidirectional alignment procedure could ensure that each aligned object and the corresponding node are in one-to-one correspondence. Fig.3 demonstrates the alignment process. Based on the aligning result for each object and the weighted voting mechanism, we could align the nodes and edges in the obtained Local Semantic Scene Graph(LSSG) with those in the Global Semantic Scene Graph(GSSGG. The updating opera- tion could involve ADD, UPDATE and REPLACE for both nodes and edges in GSSG. Please see the supplementary material for more details.
Figure The alignments between objects and edges local scene graph and global scene graph.
5
Navigation Model
5.1 Model Structure
The goal of the proposed navigation model is to guide the agent to take actions to explore the environment and build the semantic scene graph incrementally. The action space considered in this work includes Move, where the agent can take nine basic actions corresponding to 8 directions and no move, and Rotate, where the agent can rotate clockwise or counterclockwise with a fixed angle 5, 90 no rotate. Especially, if no move and no rotate are selected, a stop action is triggered.
The design of the navigation model a challenging problem since the agent is expected to build a good semantic scene graph which contains the objects and their relationships accurately. To tackle this problem, we construct a navigation architecture as shown in Fig.4. Considering that the LSSG could represent the insight semantic observations for the agent, we use an LSTM to embed the

---

BOB
CANNCY
RGB
FASG
OSSE
Op-1
BCB
Lowe ESTML
Liver STM
VLF
Natwork
COSC AOSUuE
42
Figure 4: The architecture of the navigation model. At each time step, the of action feature, RGB feature, LSSG feature and GSSG feature are firstly extracted and concatenated and fed into double- layer LSTM to predict the next step action.
LSSG sequence to help the agent build connections visited viewpoints semantic level. Further, we introduce GSSG vector aiming at help the agent have a complete perception over the global semantic information. Since the change of GSSG could reflect the adding and updating of objects and semantic relations the scene, we utilize an LSTM to make the agent capture this dynamic process. Overall, the local scene graph and global scene graph sequence respectively, and an embedding module to represent the previous action. The three feature vectors are concatenated and fed into double-layer LSTM model. Finally, an MLP network is used to generate the next step action.
ResNet extract the visual feature, two LSTMs to encode
5.2 Learning for Navigation
Since the direct learning for such an architecture is still challenging, we adopt the imitation learning as the pre-training and the reinforcement learning for fine-tunin fine-tuning.
5.2.1 Imitation Learning
The goal of imitation learning for sequential prediction problem is to train the agent to mimic the expert's behaviors. In this work, it is important generate some demonstration paths for the agent to imitate. We adopt a two-stage method to deal with this problem. In the first stage, we try to obtain way-point set and in the second stage, perform the interpolation between way-points the whole demonstration path. To evaluate the way-points, we first count the visible object at each viewpoint, and select the closest way-point that has the most number of unseen new objects, which can be expressed as:
get
8² arg max new wwloooobjtuumttt k' = arg min veO(ve,k")
Σ (EO(
new new_object.oo (n) > 0
where O k)) represents the feasible viewpoints that are & steps away from Ue, and the way-point set can be updated W W U {u*). The above procedure is repeated until maximum distance is achieved. In the second step, we implement a beam search over the way-point sequence for interpolation [28]. The loss function is defined follows:
K
C.
log no (ax, ax,0 Sk,1 "r. 8k.1)
K
where is the number demonstration paths used for training one batch, T3 the length k-th path, Št, and /r.t are the annotated input visual state and action, and 0 denotes the parameter of the exploration policy и. The process of minimizing La equals to maximize the probability of the demonstration paths' action sequence based on the annotated inputs.
5

---

5.2.2 Reinforcement Learning
After pre-training the exploration model with imitation learning, we then try to further improve its performance using the REINFORCE algorithm.
Since our task aims at formulating a scene graph for the entire scene, we directly measure the similarity between the generated scene graph and ground truth one, which be obtained from the ground truth information about the object and the relationships. Given a global scene graph GSSG which is constructed from one path, the similarity score can then be calculated as weighted sum of the precision and recall rate of the nodes and edges:
S (im(gSS)) Anode Anode(Rnode + Anode))) + Redge + Pedge
(2)
where Prode and Pedge are the precision of the nodes and edges, respectively, while Rnode and Redge are the recall rate of the nodes and edges, respectively.
On the other hand, hope encourage the observation diversity when constructing the scene graph. The diversity can be characterized as the number of observation viewpoints the detected objects. Concretely speaking, given global scene graph GSSG which is constructed from one path, we use O to denote the set of the detected objects and calculate the diversity as:
Die(GSS9)
= Σ numm_riewpoints(o)
(3)
oEO
where num_vieed (imeuueuppiins(() is the number of the viewpoints about the detected object instance o. Therefore we may formulate the score at time instant as:
Sim (GSSg) + XeDio(GSSG) pt. where the third term is used to penally the length of the path, and A p the corresponding weighting parameters. In practice, we set Anode = 0.1, > = 0.3, and Aa = = 0.001.
(4)
According to the above definition, the immediate reward is designed to be the increment of the score r(8t, (t) = Pt Pt-1 and the cumulative reward can be computed as:
R(st, ae) r(st, 0c) + Σ of (rsy
(5)
8'=t11
where R(st, a.) represents the expected accumulated reward when agent takes action a₂ at state St, the discount parameter "y is set to 0.99, and T is the length of the action and state sequence with upper bound of steps, and we use SGD optimizer with a learning rate of 10-4
Support
Placement
Hanging
Position
Connecting
Attachment
Component
Figure 5: An illustration of the considered semantic relationships.
6 Experimental Validation
6.1 Dataset
We choose AI2THOR [29] to generate our dataset since it supports a continuous navigation while MP3D[30] does not, and has higher rendering quality of images and more accurate sensors com- pared with 13b We follow the guidelines in [26] to extract 16 semantic relationships, which can be clustered into the following seven categories (Fig.5). Please refer the supplementary ma- terial for more details.

---

Table I: Performance Comparison
Reick
Stitien
Nos
Lunds
Fuods
Laass
Fedge
Rater
NoN
Huods
Flasts
Fudas
Rutse
Fladen
Random
27.367
0.311
0.474
0.705
0.144
0.239
27.367
0.223
0.317
0.278
0.0997
0.144
PRO-Map
23.915
1.0
0.484
0.682
0.73%
0.260
D.384
23.915
0.665
0.435
0.526
0.248
0.200
0.221
Prontier-based
24,597
1.0
0.547
0.702
0.717
0.289
0.412
24,597
0.643
0.425
0.512
0.253
0.194
0.220
LSTM--Act-Pss
22.815
0.563
0.720
0.689
0.300
0.418
22.815
0.587
0.526
0.546
0.235
0.208
0.221
LSSG-GSSSC(OOn))
0.688
0.815
0.339
26.657
0.564
0.561
0.230
(rralee((Oracle))
5500
1.0
0.858
0.924
0.576
0.384
0.461
2500
0.516
0.617
0.562
0.201
0.275
0.232
Table 2: Input Ablation Study: Baseline takes RGB and last step action as input, LSSG and GSSG represent local and global semantic scene graph vector respectively.
Nos
Pandd
Randa
F Flande
Patos
R,dae
Fladae
NoS
Paods
Rmods
Flande
Padar
Ractes
F Fl
Baseline GSSG
22.815
0.563
0.720
0.689
0.300
0.418
22.815
0.567
0.526
0.546
0.235
0.208
0.221
0.309
LSSG
23.778
1.0
0.657
0.793
0.678
0.337
0.450
22.829
0.541
0.539
0.540
0.229
0.213
0.224
LSSG+OSSG
22.435
0.688
0.815
0.681
0.339
0.453
26.657
0.558
0.564
0.561
0.228
0.230
0.229
Table 3: Training Paradigm Ablation Study: represents imitation learning, RL represents rein- forcement learning, IL+RL refers to our final training paradigm.
RaICis
SGGen
Ro.d-
Flaode
Padge
Redgr
Flatye
Puste
Rucae
Plaots
P.tar
R.dyr
0.632
0.663
0.556
RL
20.423
1.0
0.573
0.729
0.694
0.301
0.420
15.020
0.619
0:475
0.538
0.231
0.197 0.230
0.222
IL-RI
22.435
1.0
0,688
0815
0.681
0.339
0.453
26.657
0.558
0.564
0.561
0.228
0.229
6.2 Method and Metrics
To verify the effectiveness of our framework and the influence of the input vectors, we introduce the following methods for comparison:
1. Random The agent randomly selects an action from action space to perform scene exploration.
PPO+Map[32] The agent utilizes Resnet-18 8 to encode RGB frame, local map and global map respectively and concatenates them as the input for the LSTM to predict the next action.
Frontier-based Exploration(33] A heuristic algorithm that guides the agent detect and sweep to the closest frontier with path planning.
15] This model is widely used in embodied tasks, it encodes RGB frame, agent position and last step action as the input LSTM to predict the next step action for the agent.
4. LSTM+Act+Posee
5. LSSG+GSSG(Ours) Our framework that takes LSSG and GSSG vectors the LSTM input.
Traverse(Oracle) Ask the agent traverse every viewpoint in the scene.
To evaluate the quality of the semantic scene graph, we adopt the precision, recall rate and Fl score of the nodes and edges compared to the ground truth scene graph, denoted as Prode, Rmodes Flnodes Pedge. Redge and Fledge respectively, and the number of steps NoS as quantitative metrics. Note that Rnode could be viewed richness of the generated GSSG.
object level coverage, and Redge could represent the semantic
In this task, we align the detecting result from Mask RCNN with ground truth to obtain the correctly detected objects at each viewpoint, and offer the ground truth segmentation of each object to the agent. Since the detection results and point cloud information are always correct, task, the agent utilizes the detecting result from Mask RCNN which is consistent with real- world environment. The agent is expected to leverage both the global scene graph and navigation policy to deal with the imperfection of Mask RCNN and the incompleteness of the collected point cloud, incrementally generating an accurate and informative scene graph.
We also propose the following two tasks for evaluation. RelCIs:
precision of node always 1.0. SGGen: In this
6.3 Results
6.3.1
Quantitative results
The experimental results are shown in Table and II, from which we have the following observations:
7

---

1. In both RelCIs and SGGen, LSSG+ outperforms other baselines on the recall rate and F1- score of both the node and edge, and achieves a performance close to Traverse, which demonstrates the effectiveness of our framework. Note that Traverse asks the agent to traverse every viewpoint in the unseen scene which always requires manual assistance in reality. While our method fully exploits the embodied capacity of the robot to explore the scene automatically and reaches comparable performance with much fewer NoS.
2. The increase in recall rate is often accompanied by decrease in precision. This is mainly caused by the imperfection of the Mask RCNN and Local Scene Graph Prediction Network(LSGGNN) since the errors at some specific viewpoints could be hard to update and would accumulate as the agent increasingly covers the scene. Our method achieves the highest Fi-score on both tasks, which illustrates that our framework handles the balance of precision and recall rate well.
3. SGGen is much more difficult than RelCIs since both the label and the mask of the detecting result could be wrong, and the detecting error could further affect the generated LSSG and GSSG. This explains the drop of all metries in SGGen compared with those in ReICIs. No matter how difficult the task is, the baseline methods only implement spatial exploration and adopt the same exploring path. While our method explores the scene at a semantic level indicating that the agent is aware of GSSG and would try to update and add the semantic information of GSSG in subsequent steps.
4. From table II we could discover that in RelCIs, both LSSG and GSSG benefit the task, and the improvement of introducing LSSG is more obvious. This may be because the LSSG sequence en- coded by LSTM contains certain amount of GSSG information. While in SGGen, adding GSSG vectors worse than Base method. We think this could be caused by the more frequently changing frequency on GSSG and its vector very different from that in the training. LSSG + GSSG performs best on both tasks, which illustrate the effectiveness and robustness of our framework.
5. Table III shows the performance of our LSSG + GSSG method under different training paradigm. We can see that IL performs better than RL in ReICIs. In SGGen, the performance has an obvious drop as the behaviour cloning based method utilized by IL has poor generalization ability. In unseen testing scene, especially the ones in SGGen that have huge differences in aspect of room layout and point cloud accuracy with training scenes, IL trained model tends to perform much worse. Meanwhile, while RI. trained model has a smaller performance drop in SGGen, it has the lowest recall rate nodes and edges in RelCIs and SGGen, which demonstrates that the scene has not been completely explored. Therefore, we design the mix training paradigm that utilizes im- itation learning first pre-train the model, and then finetunes it with reinforcement learning. Our method (IL+RL) achieves the highest score over the recall rate and fl score of nodes and edges on both two tasks, demonstrating its efficiency.
6.3.2 Qualitative Analysis and Applications
The generated semantic scene graph can be used for various downstream tasks. And we have applied the generated semantic scene graph a streaming video captioning task in this paper. The caption of the video is incrementally enriched with the generated scene graph while the agent exploring the environment. Experiment results show that the quality of the generated cap- tion is superior to that generated with Rondom cxploration policy. The detailed results and further discussion are presented in the supplementary document. Please also visit our website http://cembodiiesseenngaaah vercel.app for more information.
7 Conclusion
In this paper, a novel embodied semantic scene graph generation framework is established and the hybrid imitation reinforcement learning method is developed to address this new task. With the learned policy, the agent able to autonomously explore the environment and then incrementally generate high-quality semantic scene graph. This research exploits the important embodiment characteristic of the intelligent agent and paves a new path for the agent to semantically describe the environment. Additionally, it provides fundamental component for some challenging downstream tasks such as streaming video captioning, robotic manipulation, visual navigation, etc. We would also like to extend this work to large-scale scene graph generation in more complicated scenarios.
8

---

Acknowledgements
This work was supported in part by the National Natural Science Fund for Distinguished Young Scholars (62025304) and in part by the Seed Fund of Tsinghua University (Department of Computer Science and Technology--iiement Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things.
References
[1] L.. Mahon, E. Giunchiglia, B. Li, and T. Lukasiewicz. Knowledge graph extraction from videos. arXiv preprint arXiv:2007.10040, 2020.
[2] Xu, X. Chang, L. Guo, P.-Y. Huang. X. Chen, and A. G. Hauptmann. survey of scene graph: Generation and application. EasyChair Preprint, (3385), 2020.
[3] R. Krishna, Zhu, O. Groth, Johnson, K. Hata, Kravitz. S. Chen, Y. Kalantidis, L.-J. Li,
D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123()): 2017.
[4] Y. Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang. Scene graph generation from objects, phrases and region captions. Proceedings of the IFFE International Conference on Com- puter Vision, pages 1261-1270, 2017.
[5] B. Pan, H. Cai, D.-A. Huang, K.-H. Lee, A. Gaidon, E. Adeli, and C. Niebles. Spatio- temporal graph for video captioning with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870-10879, 2020.
[6] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1410-5419, 2017.
[7] S. Garg, N. Sünderhauf, F. Dayoub, D. Morrison, A. Cosgun, G. Carneiro, Q. Wu, T.-J. Chin, I. Reid, S. Gould, et al. Semantics for robotic mapping, perception and interaction: A survey. arXiv preprint arXiv:2 2021.
[8] U.-H. Kim, J.-M. Park, T.-J. Song, and J.-H. Kim. 3-d scene graph: A sparse and semantic rep- resentation of physical environments for intelligent agents. IEEE transactions cybernetics, 50(12):4921-4933 2019.
[9] I. Bozcan and S. Kalkan. Cosmo: Contextualized scene modeling with boltzmann machines. Robotics and Autonomous Systems, 1 11:1132-14, 2019.
[10] I. Armeni, Z.-Y. He, J. Gwak, A. Zamir, M. Fischer, J. Malik, and S. Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5664 3664-5673, 2019.
[11] A. Rosinol, A. Gupta, M. Abate. J. Shi, and L. Carlone. 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. arXiv preprint arXiv:2002.06238 2020.
[12] J. Yoon, W. R. Zame, and M. van der Schaar. Deep sensing: Active sensing using multi- directional recurrent neural networks. In International Conference on Learning Representa- tions, 2018, URL https: http::://ppeeeeeeeeeeeee wttp:/iwwwweeeeeennneeo.eeeo.aee Forum? wtt///oronnnndn....maa id=r1S1X5xCb.
net
[13] V. S. Kalogeiton, K. Ioannidis, G. C. Sirakoulis, and E. B. Kosmatopoulos, Real-time active slam and obstacle avoidance for an autonomous robot based on stereo vision. Cybernetics and Systems, 50(3):239-261) 2019.
[14] A. Das. S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-10, 2018

---

[15] S. Tan, H. Liu, D. Guo, X. Zhang, and F. Sun. Towards embodied scene description. RSS, pages 1-8, 2021.
[16] Z. Fang, A. Jain, G. Sarch, W. Harley, and K. Fragkiadaki. Move see better: Towards self-supervised amodal object detection. arXiv preprint arXhv:2012.00057, 2020.
[17] S. Tan, W. Xiang, H. Liu, D. Guo, and F, Sun. Multi-agent embodied question answering in interactive environments. ECCV, pages 1 1-8, 2020.
[18] W. Zhang and H. Mei. A constructive model for collective intelligence. Review, 7(8):1273-1277 2020. doi.https://doi.cm org/ 10.1093/ss//wwaa022
National Science
[19] X. Han, H. Liu, F. Sun, and X. Zhang. Active object detection with multistep action prediction using deep q q-network. IEEE Transactions on Industrial Informaties, 15(6):3723-377 17 2019. doi:10. 1111101119911122199000
[20] D. S. Chaplot, H. Jiang, S. Gupta, and A. Gupta. Semantic curiosity for active visual learning. In European Conference on Computer Vision, pages 309-326. Springer, 2020.
[21] R. Druon, Y. Yoshiyasu, A. Kanezaki, and A. Watt. Visual object search by learning spatial context. IEEE Robotics and Automation Letters, 5(2): 179-11286 2020.
[22] Z. Zeng, A. Röfer, and O. Jenkins. Semantic linking maps for active visual object search. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1984-1990. IEEE, 2020.
[23] A. Kurenkov, R. Martín-Martín, J. Ichnowski, K. Goldberg, and S. Savarese. Semantic and ge- ometric modeling with neural message passing in 3d scene graphs for hierarchical mechanical search. arXiv preprint arXiv:2012.04060, 2020.
[24] H. Du, X. Yu, and L. Zheng. Learning object relation graph and tentative policy for visual navigation. In European Conference on Computer Vision, pages 19-34. Springer, 2020.
[25] K. Kenfack, F. Siddiky, F.
and M. Beetz. Robotvqa scene-graphe ceone-graph-an and deep-learning-based visual question answering system for robot manipulation. In IEFF/RS/ International Conference on Intelligett Robots and Systems (IROS), Las Vegas, USA, 2020.
[26] Wald, H. Dhamo, N. Navab, and Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3961-3970, 2020.
[27] Johnson, A. Gupta, and L. Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1219-1228, 2018.
[28] S. Wiseman and A. M. Rush. Sequence-to-sequence learning beam-search optimization. arXiv preprint arXiv: 1600 02060, 2016
[29] E. Kolve, Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu,
A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv, 2017.
[30] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data indoor environments. arXiv preprint arXiv: arXiv:1709.0611 1709.0618, 2017.
[31] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, Malik, Parikh, and Batra. Habitat: A Platform for Embodied AI Research. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[32] T. Chen, S. Gupta, and A. Gupta. Learning exploration policies for navigation. arXiv preprint arXiv: 1903, 0195, 2019.
[33] B. Yamauchi. A frontier-based approach for autonomous exploration. Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA '97." "owoass New Computational Principles for Robotics and Automation', pages 146
151. IEEE, 1997.
10


Reviews:
[{"review_id": "uSboOuYAz50", "reviewer": "Reviewer_vave", "summary": "The paper studies a new visual navigation problem named “Embodied Semantic Scene Graph Generation”. The task is to have an embodied agent (e.g., a navigation robot) equipped with a first-person view camera traversing an indoor environment and generating a global semantic scene graph. A semantic scene graph consists of object categories and their semantic relationships. The goal is to generate a scene graph that closely resembles a hidden ground truth scene graph of the environment. The paper presents a method that learns to carry out this task by imitating traversal demonstrations and reinforcement learning. The paper also shows that the generated scene graphs are useful for a downstream video captioning task. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the overall problem setup of embodied semantic scene graph generation. Semantic scene graph is a useful representation for reasoning about the high-level structure of an environment, which can be useful as a guidance downstream tasks like multi-step manipulation and object searching. The problem of simultaneously navigating the environment and incrementally generating a global scene graph that matches a hidden ground truth graph is a good starting point for more future works along this line. I also appreciate the effort of carrying out a large-scale study in the AI2-THOR environment. I hope the authors release the corresponding dataset in the future.\n\nAt the same time, I’m not entirely convinced that the paper is sufficient as the first study of the problem. My main concern is that the paper doesn’t really have any discussion on why the problem of embodied scene graph generation is challenging. Intuitively, there are at least two challenges behind the problem: 1) how to *efficiently* traverse the space to generate the complete scene graph and 2) generalizing the policy to a new environment. \n\nThe introduction section briefly touches upon (1) by saying that (line 51) “the environment should be 51 exhaustively explored, which requires large time consumption”. However, the proposed method squarely bypasses the problem by providing demonstrations generated by a heuristics-based policy as training data. And the key contribution of the method, i.e., to incorporate the previously collected scene graph information in decision making, isn’t designed to address this problem. \n\nFor (2), I’m not entirely sure about the problem setup. Are there training and held-out scenes or are all scenes used for training? I don’t see it mentioned anywhere in the main text of the supplementary material. I hope the author spends more text discussing the key challenges of this problem as well as the potential solutions for them. \n\nRelatedly, I’m not entirely clear on the motivation behind the navigation module. The only explanation is at Line 167-169: “The design of the navigation model is a challenging problem since the agent is expected to build a good semantic scene graph which contains the objects and their relationships accurately. To tackle this problem, we construct a navigation architecture which is shown in Fig.4.” This isn’t really a justification of why it might help the agent build a better semantic scene graph, nor does it explain why it might outperform other baselines. I hope the author provides a more thorough discussion on the method itself and how it might address the key challenges presented in the problem setup.\n\nMinor: Wrong citation at line 125. The goal of work [23] is to generate images from scene graph, not the other way around.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Like mentioned in my comment, I like the new problem setup. But I'm not entirely sure that the paper did an adequate job addressing the key challenges of this problem setup and pointing at directions for more follow up works. In particular, the paper did not justify some of the key components of the proposed method and why it might outperform other comparable baselines. Thus I recommend rejecting before seeing more comments from the authors.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See main comments.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "owvNsnzv3AS", "reviewer": "Reviewer_nnax", "summary": "The paper proposes a new embodied AI task \"Embodied semantic scene graph generation\" in which the objective is for an agent to select actions to efficiently build a semantic scene graph. It also proposes an approach to solve this problem based in imiation and reinforcement learning. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenths:\n\nThe generation of scene graphs is an interesting and important problem in embodied navigation. \n\nWeaknesses:\n\n[W1] The related work should also cover the literature related to active sensing which is very related to the problem at hand.\n\n[W2] The approach only seems to be evaluated on relatively small scenes that are on the order of 1 room and take on the order of 20 steps to explore. It would be interesting to see how the method scales to larger environments. \n\n[W3] The work seems to be a collection of components (local scene graph generation, local scene graph fusion, action selection, imitation learning, reinforcement learning) none of which on their own strikes me as particularly novel. The core claim of the paper is that the novelty is in the action selection part, but this component of the paper is relatively under-explained with many details left out and seeming to be a relatively standard. \n\nSmaller comments:\n - The text in Figure 1 is very small and hard to read. \n - replace \"Ref. [X]\" with \"Authorname et. al.\"\n - The manuscript should be proofread for grammar and spelling mistakes of which there are many\n - We can't see from Figure 6 what actions the agent took.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While I would agree that the embodied scene graph generation task is an interesting one, I find that the paper is not very clear and lacks significant novelty. Rather, it comes across more as a collection of components that have been combined. This does not rule it out as systems work is still important, but I don't feel that the bar has been achieved that the work is of significant enough importance to warrant publication at CoRL. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": " - lack of clarity of exposition\n - lack of detail in methodology\n - lack of clear novelty", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "_L3h3Ydgyi3", "reviewer": "Reviewer_iKq5", "summary": "The paper considers the problem of embodied scene graph generation, in which, the agent sequentially constructs a scene graph by interacting with the environment. At every time step, it chooses an action that determines the next observation, generates a local scene graph from the observation, and then integrates the local scene graph with the global scene graph constructed thus far. \n\nThe action sequence is modeled by an LSTM network that takes in input features from the observation (RGB-D scan) and LSTM on the two – local and global – constructed scene graphs. The action sequence is pre-trained with imitation learning to pick a waypoint that maximizes the number of unseen objects. The model is then trained using RL with a cost function that is a combination of precision and recall.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper introduces a problem of immense significance and does solid work.\n\nWeaknesses: The paper attempts to convey too many things and finds eight pages limiting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper studies a very pertinent problem of embodied scene graph generation. The proposed solution is well investigated. The paper is nicely written. Overall, the paper is a good, solid work that I have enjoyed reading.\n\nThe only limitation would be that the paper tries to do too much. Section 6.4 seems rather too much and could be left out as a future work or in the appendix. The space could have been used to explain the model architecture and the RL reward function more clearly in the main paper. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "1. Section 6.4 seems rather much. It would help the paper if instead of presenting this application – which does not go in with the main theme of the paper – could present some more details on the architecture that have been relegated to the appendix section; especially 5.1 which describes the model structure in just one-two line. \n\n2. The conclusion seems short for such a nice paper.\n\n3. I could not find a description of how miss detections are dealt with. For instance, if the agent constructs a node in the global scene graph, by a misdetection, but realizes that the object wasn’t what it thought it was or that there was no object there in future scans. How does the agent correct it?\n\nMinor:\n1. References with missing year: [15].\n2. In 5.2.1, explain what is O(v,k)?\n3. Section 1 mentions “this work embodiment of the agent to effectively combine the action, vision, and language together for autonomous 3D semantic scene graph generation” Where is the language part in the proposed architecture? \n4. Figures 3 and 4 are too small to read\n"}}, {"review_id": "2lIIBmiH6nf", "reviewer": "Reviewer_bACY", "summary": "This work presents an approach for learning to explore in order to build a semantic scene graph. The approach leverages prior work on semantic scene graph construction (which nominally uses precollected exploration data) and expands upon this to enable exploration. The exploration is learned from initially imitation learning based on a scripted policy and then reinforcement learning to improve the policy. Further details of graph construction in this setting are described. The approach is demonstrated on an AI2Thor environment to perform well.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and clear\n- The setting of embodied exploration to build a representation is interesting\n- The approach is interesting and reasonable. The description of how the scene graph is built from local observations into global observations with voting is useful. \n- The network architecture for navigation which captures the local and global scene graph is well done.\n\nWeaknesses:\n- The comparisons are reasonable, though more qualitative downstream performance would be useful (e.g., qualitatively test on planning or captioning)\n- The figures of scene graphs are difficult to see and understand due to their size, though the figures are generally good\n- A few ablations on the training and architecture would be useful\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall the paper is interesting and well written. The approach makes sense and is valuable. There are a few issues of clarity of the figures and tables that can be easily addressed. There are also some additional ablations and experiments that would help to reinforce the results.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The performance of the approach outperforms previous approaches, though only by a small margin (particularly for SGGen), but is nonetheless is interesting. There are a few open questions I believe that are left unanswered and should be discussed:\n- From the learning side, how much does the RL contribute to exploring well\n- Ablations across inputs to the navigation architecture would be useful (e.g., removing the RGB, LSSG, and GSSG)\n- The oracle compared traverses every node, it would be useful to see how the proposed method compares to an oracle at a similar number of steps.\n- Add std deviations (potentially in the appendix to values)\n\nThe paper also lacks qualitative downstream performance of the scene graph. Though the qualitative analysis of scene graph construction is reasonable, the primarily goal of the scene graph is to perform well on downstream tasks like captioning or planning. It would be interesting to see how the performance on such tasks is qualitatively changed compared to baselines. \n\nMinor notes:\n- Text on figure 1, 3, 4 is really small\n- Typo L70 “3. We testify” -> we test\n- Add more information in the captions of table 1 and 2 on what each value means.\n"}}]

Author rebuttal:

Meta-review: This paper proposed a new task of embodied semantic scene graph generation, where the goal is to have an embodied agent intelligently select the actions to traverse an environment and build a semantic scene graph of the environment incrementally from its own experiences. At the initial reviews, this paper received mixed ratings, with two Weak Rejects, one Weak Accept, and one Strong Accept. While the authors appreciated the new problem formulation, concerns have been raised in model novelty, writing clarity, and experimental validation. The meta-reviewer strongly suggested the authors respond to the issues suggested by the reviewers. For the reviewers, please take a look at each other's reviews and the authors' responses. Let's try our best to resolve the conflicting ratings and reach a consensus.

**Update:** The authors revised their manuscript and addressed the issues brought up by the initial reviews. At the end of the discussion period, Reviewer nnax updated their rating to Weak Accept and the other three reviewers retrained their original scores. Reviewer vave, who voted Weak Reject, argued that some of their concerns had been addressed by the authors' responses, yet this paper needs substantial revision to convince the reader that: 1) the new problem is interesting and challenging, and 2) the proposed solution is a meaningful starting point to address this problem. The other three reviewers appraised the problem formulation of embodied scene graph generation. They believed that the proposed model makes sense as a first attempt to tackle this problem, even though individual components of this model did not present significant novelty. The AC understood the rationale behind the positive and negative comments of this work. Considering all the factors, the AC believed that such a paper that explored this new topic should be accepted, given the fact that three of the four reviewers have supported its acceptance and no critical flaw has been spotted during the review process. The authors should further improve the quality of this manuscript by incorporating all the reviewers' comments.
Decision: Accept (Poster)