Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a new embodied AI task "Embodied semantic scene graph generation" in which the objective is for an agent to select actions to efficiently build a semantic scene graph. It also proposes an approach to solve this problem based in imiation and reinforcement learning. 
strengths_and_weaknesses: Strenths:

The generation of scene graphs is an interesting and important problem in embodied navigation. 

Weaknesses:

[W1] The related work should also cover the literature related to active sensing which is very related to the problem at hand.

[W2] The approach only seems to be evaluated on relatively small scenes that are on the order of 1 room and take on the order of 20 steps to explore. It would be interesting to see how the method scales to larger environments. 

[W3] The work seems to be a collection of components (local scene graph generation, local scene graph fusion, action selection, imitation learning, reinforcement learning) none of which on their own strikes me as particularly novel. The core claim of the paper is that the novelty is in the action selection part, but this component of the paper is relatively under-explained with many details left out and seeming to be a relatively standard. 

Smaller comments:
 - The text in Figure 1 is very small and hard to read. 
 - replace "Ref. [X]" with "Authorname et. al."
 - The manuscript should be proofread for grammar and spelling mistakes of which there are many
 - We can't see from Figure 6 what actions the agent took.

summary_of_recommendation: While I would agree that the embodied scene graph generation task is an interesting one, I find that the paper is not very clear and lacks significant novelty. Rather, it comes across more as a collection of components that have been combined. This does not rule it out as systems work is still important, but I don't feel that the bar has been achieved that the work is of significant enough importance to warrant publication at CoRL. 
issues:  - lack of clarity of exposition
 - lack of detail in methodology
 - lack of clear novelty
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper considers the problem of embodied scene graph generation, in which, the agent sequentially constructs a scene graph by interacting with the environment. At every time step, it chooses an action that determines the next observation, generates a local scene graph from the observation, and then integrates the local scene graph with the global scene graph constructed thus far. 

The action sequence is modeled by an LSTM network that takes in input features from the observation (RGB-D scan) and LSTM on the two – local and global – constructed scene graphs. The action sequence is pre-trained with imitation learning to pick a waypoint that maximizes the number of unseen objects. The model is then trained using RL with a cost function that is a combination of precision and recall.

strengths_and_weaknesses: Strengths: The paper introduces a problem of immense significance and does solid work.

Weaknesses: The paper attempts to convey too many things and finds eight pages limiting.
summary_of_recommendation: The paper studies a very pertinent problem of embodied scene graph generation. The proposed solution is well investigated. The paper is nicely written. Overall, the paper is a good, solid work that I have enjoyed reading.

The only limitation would be that the paper tries to do too much. Section 6.4 seems rather too much and could be left out as a future work or in the appendix. The space could have been used to explain the model architecture and the RL reward function more clearly in the main paper. 
issues: 1. Section 6.4 seems rather much. It would help the paper if instead of presenting this application – which does not go in with the main theme of the paper – could present some more details on the architecture that have been relegated to the appendix section; especially 5.1 which describes the model structure in just one-two line. 

2. The conclusion seems short for such a nice paper.

3. I could not find a description of how miss detections are dealt with. For instance, if the agent constructs a node in the global scene graph, by a misdetection, but realizes that the object wasn’t what it thought it was or that there was no object there in future scans. How does the agent correct it?

Minor:
1. References with missing year: [15].
2. In 5.2.1, explain what is O(v,k)?
3. Section 1 mentions “this work embodiment of the agent to effectively combine the action, vision, and language together for autonomous 3D semantic scene graph generation” Where is the language part in the proposed architecture? 
4. Figures 3 and 4 are too small to read
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work presents an approach for learning to explore in order to build a semantic scene graph. The approach leverages prior work on semantic scene graph construction (which nominally uses precollected exploration data) and expands upon this to enable exploration. The exploration is learned from initially imitation learning based on a scripted policy and then reinforcement learning to improve the policy. Further details of graph construction in this setting are described. The approach is demonstrated on an AI2Thor environment to perform well.
strengths_and_weaknesses: Strengths:
- The paper is well written and clear
- The setting of embodied exploration to build a representation is interesting
- The approach is interesting and reasonable. The description of how the scene graph is built from local observations into global observations with voting is useful. 
- The network architecture for navigation which captures the local and global scene graph is well done.

Weaknesses:
- The comparisons are reasonable, though more qualitative downstream performance would be useful (e.g., qualitatively test on planning or captioning)
- The figures of scene graphs are difficult to see and understand due to their size, though the figures are generally good
- A few ablations on the training and architecture would be useful

summary_of_recommendation: Overall the paper is interesting and well written. The approach makes sense and is valuable. There are a few issues of clarity of the figures and tables that can be easily addressed. There are also some additional ablations and experiments that would help to reinforce the results.

issues: The performance of the approach outperforms previous approaches, though only by a small margin (particularly for SGGen), but is nonetheless is interesting. There are a few open questions I believe that are left unanswered and should be discussed:
- From the learning side, how much does the RL contribute to exploring well
- Ablations across inputs to the navigation architecture would be useful (e.g., removing the RGB, LSSG, and GSSG)
- The oracle compared traverses every node, it would be useful to see how the proposed method compares to an oracle at a similar number of steps.
- Add std deviations (potentially in the appendix to values)

The paper also lacks qualitative downstream performance of the scene graph. Though the qualitative analysis of scene graph construction is reasonable, the primarily goal of the scene graph is to perform well on downstream tasks like captioning or planning. It would be interesting to see how the performance on such tasks is qualitatively changed compared to baselines. 

Minor notes:
- Text on figure 1, 3, 4 is really small
- Typo L70 “3. We testify” -> we test
- Add more information in the captions of table 1 and 2 on what each value means.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper studies a new visual navigation problem named “Embodied Semantic Scene Graph Generation”. The task is to have an embodied agent (e.g., a navigation robot) equipped with a first-person view camera traversing an indoor environment and generating a global semantic scene graph. A semantic scene graph consists of object categories and their semantic relationships. The goal is to generate a scene graph that closely resembles a hidden ground truth scene graph of the environment. The paper presents a method that learns to carry out this task by imitating traversal demonstrations and reinforcement learning. The paper also shows that the generated scene graphs are useful for a downstream video captioning task. 
strengths_and_weaknesses: I like the overall problem setup of embodied semantic scene graph generation. Semantic scene graph is a useful representation for reasoning about the high-level structure of an environment, which can be useful as a guidance downstream tasks like multi-step manipulation and object searching. The problem of simultaneously navigating the environment and incrementally generating a global scene graph that matches a hidden ground truth graph is a good starting point for more future works along this line. I also appreciate the effort of carrying out a large-scale study in the AI2-THOR environment. I hope the authors release the corresponding dataset in the future.

At the same time, I’m not entirely convinced that the paper is sufficient as the first study of the problem. My main concern is that the paper doesn’t really have any discussion on why the problem of embodied scene graph generation is challenging. Intuitively, there are at least two challenges behind the problem: 1) how to *efficiently* traverse the space to generate the complete scene graph and 2) generalizing the policy to a new environment. 

The introduction section briefly touches upon (1) by saying that (line 51) “the environment should be 51 exhaustively explored, which requires large time consumption”. However, the proposed method squarely bypasses the problem by providing demonstrations generated by a heuristics-based policy as training data. And the key contribution of the method, i.e., to incorporate the previously collected scene graph information in decision making, isn’t designed to address this problem. 

For (2), I’m not entirely sure about the problem setup. Are there training and held-out scenes or are all scenes used for training? I don’t see it mentioned anywhere in the main text of the supplementary material. I hope the author spends more text discussing the key challenges of this problem as well as the potential solutions for them. 

Relatedly, I’m not entirely clear on the motivation behind the navigation module. The only explanation is at Line 167-169: “The design of the navigation model is a challenging problem since the agent is expected to build a good semantic scene graph which contains the objects and their relationships accurately. To tackle this problem, we construct a navigation architecture which is shown in Fig.4.” This isn’t really a justification of why it might help the agent build a better semantic scene graph, nor does it explain why it might outperform other baselines. I hope the author provides a more thorough discussion on the method itself and how it might address the key challenges presented in the problem setup.

Minor: Wrong citation at line 125. The goal of work [23] is to generate images from scene graph, not the other way around.
summary_of_recommendation: Like mentioned in my comment, I like the new problem setup. But I'm not entirely sure that the paper did an adequate job addressing the key challenges of this problem setup and pointing at directions for more follow up works. In particular, the paper did not justify some of the key components of the proposed method and why it might outperform other comparable baselines. Thus I recommend rejecting before seeing more comments from the authors.
issues: See main comments.
review_title: null

Meta-review: This paper proposed a new task of embodied semantic scene graph generation, where the goal is to have an embodied agent intelligently select the actions to traverse an environment and build a semantic scene graph of the environment incrementally from its own experiences. At the initial reviews, this paper received mixed ratings, with two Weak Rejects, one Weak Accept, and one Strong Accept. While the authors appreciated the new problem formulation, concerns have been raised in model novelty, writing clarity, and experimental validation. The meta-reviewer strongly suggested the authors respond to the issues suggested by the reviewers. For the reviewers, please take a look at each other's reviews and the authors' responses. Let's try our best to resolve the conflicting ratings and reach a consensus.

**Update:** The authors revised their manuscript and addressed the issues brought up by the initial reviews. At the end of the discussion period, Reviewer nnax updated their rating to Weak Accept and the other three reviewers retrained their original scores. Reviewer vave, who voted Weak Reject, argued that some of their concerns had been addressed by the authors' responses, yet this paper needs substantial revision to convince the reader that: 1) the new problem is interesting and challenging, and 2) the proposed solution is a meaningful starting point to address this problem. The other three reviewers appraised the problem formulation of embodied scene graph generation. They believed that the proposed model makes sense as a first attempt to tackle this problem, even though individual components of this model did not present significant novelty. The AC understood the rationale behind the positive and negative comments of this work. Considering all the factors, the AC believed that such a paper that explored this new topic should be accepted, given the fact that three of the four reviewers have supported its acceptance and no critical flaw has been spotted during the review process. The authors should further improve the quality of this manuscript by incorporating all the reviewers' comments.
Decision: Accept (Poster)