Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 3D scene graph is an unified and informative representation of the scene. Many existing works utilize 3D scene graph to solve tasks that require long horizonal planning and reasoning. While previous works directly take the complete scene graph as input, this work proposes a planner-agnostic algorithm to prunes the scene graph in linear time based on the goal of the task, and further propose a procedure for learning based planner to ensure the connectivity of the pruned subgraph.
strengths_and_weaknesses: Strengths:
1.	The idea of pruning the large 3D scene graph and utilize the subgraph to solve tasks is convincing.
2.	The designed pruning algorithm SCRUB which builds the subgraph iteratively is effective since it could finish in linear time.
3.	Successive experiments have been implemented to verify the effectiveness of the proposed algorithm.
Weaknesses:
1.	The proposed pruning algorithm SCRUB is lack of innovation. I think it is similar with BFS and just directly merge all nodes and edges between robot and goal objects into the subgraph. In complex scenes and tasks, the algorithm might fail to prune enough nodes and edges to improve the efficiency.
2.	This workload of this paper with respect to robot learning is not sufficient.  It could introduce learning methods at the pruning algorithm or design a learning-based planner that fits the pruning algorithm.
3.	In real world application, the 3D scene graph always have some incorrect nodes and edges. Since the pruning algorithm heavily depends on the accuracy of the scene graph, I wonder whether the planner could work well with the pruned graph in physical world. While this work does not verify the robustness of the algorithm on the imperfect 3D scene graph. 

summary_of_recommendation: This work makes contribution to the application of 3D scene graph. Pruning 3D scene graph to improve the performance and efficiency of the planner is a novel and worthy of in-depth study direction. Meanwhile, this work provides a large scale benchmark to support further research and evaluation. The successive experiments illustrate that the pruning algorithm could work with both non-learning and learning-based planners well. However, the proposed algorithm has its limitation in terms of generalization and robustness. 
issues: Figure 4 in the paper is ambiguous since the ellipses of different methods are overlapped with each other, it should be re-drawn.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The work introduces a benchmark for 3d scene graphs while focusing on the complexities involved in symbolic planning. The authors further introduce two methods to alleviate common problems with planning over such 3dSGs. 
strengths_and_weaknesses: - The fact that there isn't a substantial benchmark on large scale complex scene graphs is a large supporting point for this work
- I like the summarizing analyses; e.g. "Interestingly, larger scenes do not appear to directly correlate with task complexity, as the performance metrics remains largely consistent between the tiny and medium splits of the same domain (Table. 2)."
summary_of_recommendation: Overall I think the proposed work introduces an extremely useful benchmark for the community and competitive baseline augmentations to existing methods; i.e. the proposed algorithms simply take advantage of observations of deficiencies of existing works and these are well documented.
issues: line 99 "lifted actions" could the authors formally define what is a "lifted" action?

I do not have any major issues with the paper since the problem domains not included in the benchmark are adressed in the Limitations section and these are totally reasonable.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a benchmark over 3D scene-graphs (3DSGs) for large-scale robotic task planning problems, along with two methods to improve existing planning and/or learning algorithms. 
The proposed benchmark builds on top of an existing dataset (Gibson) and a corresponding 3DGS with a focus only on symbolic reasoning, discarding the motion planning and manipulation aspects.
Running existing algorithms on this benchmark provides insights on how they can be improved, for which two methods are proposed: sparsifying the graph by recursively pruning the original full-scale one (strategy called SCRUB), and exploiting the hierarchical graph structure to attend only on a subset of the objects present in the environment given a (importance) scoring function (strategy called SEEK).
SCRUB is shown to improve performance of planning algorithms on grounded domains, whereas SEEK provides a better performance increase on lifted domains.   
strengths_and_weaknesses: - As the main strength of this work, well-structured and explained benchmarks are always appreciated as they tend to initiate faster progress in a particular domain.
- Even though the proposed benchmark, as it's presented, would mainly be useful for the planning community, if it's indeed tightly coupled (or easy to integrate) with the original benchmark environment (gibson) that it builds on, a broader robotics community would also be able to make use of it.
- The detailed analysis of different planning algorithms on a multitude of domains provides insights on the relative strengths and weaknesses of those methods, along with promising improvement directions.   

weaknesses:
- the paper (including the supp. mat.) does not provide sufficient information on the tasks. I know it relies on a prior dataset, but it'd help if a subset of the tasks are explained at least in the supp. mat. to make the paper more self-contained. (The website also fails to provide further info on that point)
- Exclusion of motion and/or manipulation planning aspects diminishes relevance / usefulness of the benchmark for the robotics community. This might be alleviated if the paper explains / discusses how such low-level planning can be interfaced.
- There, usually, is a strong dependency between the motion / manipulation plans and high-level decisions, e.g., due to the executability / feasibility of a low level plan, which in turn might require re-planning, or backtracking. Also, some sequential manipulation behaviors might inform high-level decisions about their (in)feasibility. None of those are captured by a pure high-level planning-based benchmark. So, it's not clear how this benchmark might be utilized by a broader robotics research community.
summary_of_recommendation: Benchmarks play an important role in research. Traditionally, robotics field did not have many such benchmarks or datasets compared to machine learning domain. Recently, there has been a trend to introduce them for the robotics research, and this paper (and thus the introduced environments/dataset) could be a good addition. However, since motion and manipulation planning aspects are excluded, it only allows testing of high-level decision planning algorithms. Decision-making is surely a critical component of robotics research, however, interaction with the environment, thus the motion and manipulation planning and execution are the distinct aspect compared to classical AI planning. I'm not totally convinced on its usefulness in that regard, as the paper lacks proper explanations and discussion on such interfaces.
The introduced SCRUB and SEEK algorithms provide improvements over existing methods but they are rather small extensions.
issues: Please also see (Strengths & Weaknesses). 

small issues:
- line 39: '... vast majority [of] the ...'
- line 202: standard deviations (detailed results) are not included in the supp. material as reported in the main text.
- line 267: missing parenthesis 
- line 283: initialze
- line 294-296: contrary to the claim here, SCRUB does not improve BFWS on Courier(10,10) task according to Fig.4 
- line 328: 'While SEEK, unlike SCRUB[,] is not ...'
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 
This paper presents a large-scale robotic planning benchmark over 3DSGs that is designed to evaluate symbolic reasoning and planning over 3D scene graphs (3DSGs). The proposed benchmark consists of 20 task planning domains, including object rearrangement and courier, totaling 3734 tasks. This work benchmarks the performance of optimal planners (e.g. Fast Downward, Satplan, Delfi, etc.), satisficing planners (e.g. Fast Forward, DecStar-satisficing, Cerberus, etc.), and learning-based planners (e.g. relational policy learning, planning with learned object importance). Given the findings of benchmarking existing planning methods, two techniques have been proposed to (1) prunes a 3DSG to produce a sparsified 3DSG to improve the efficiency of classical planners and (2) ensures the reachability of all objects to reduce the number of replanning steps for learning-based planners. The experiments show the two proposed techniques yield improve performance. I believe this work presents a promising benchmark together with two effective techniques. I am leaning toward acceptance yet still have some concerns regarding how the benchmark can scale up to handle partially observable environments with stochasticity and incorporate low-level interactions, the access to the benchmark, the evaluation of the proposed techniques, and the discussion on the related work.
strengths_and_weaknesses: 
## Paper strengths and contributions
**Motivation**
- The motivation for modeling real-world environments as 3D scene graphs is convincing.
- Utilizing 3D scene graphs modeling real-world environments to evaluate planning methods is intuitive and realistic.
- To this end, constructing a large-scale planning benchmark over 3D scene graphs is promising.

**Technical contributions**
To alleviate the issues of classical planners and learning-based planners, two techniques are proposed in this work.
- The proposed technique SCRUB prunes a 3DSG by removing vertices and edges extraneous to the task to produce a sparsified 3DSG, greatly reducing the planning time required by classical planners (e.g.  FD and FF) and enabling optimal planners to run on lifted domains.
- The proposed technique SEEK ensures that all objects are reachable from their respective room nodes to reduce the number of required replanning steps.

**Clarity**
The overall writing is very clear. The presentation is well-organized.

**Related work**
The authors did a good job explaining how this work is related to and different from other relevant prior works. 

**Experimental results**
- The presentation of the experimental results is clear.
- The findings of the experimental results are well-explained, which leads to the intuition of the two proposed techniques (i.e. SCRUB and SEEK).

## Paper weaknesses and questions

**Fully observable and deterministic domains**
The current benchmark only supports the environments that are fully observable and deterministic, which is rather unrealistic to many robotics domains that require planning. I believe it would be easy to incorporate environments with stochasticity. I would like to hear the authors' opinions on how this benchmark can scale up to deal with partially observable environments.

**Low-level interactions**
Given that this benchmark does not support low-level interactions with environments, I would hardly call it a robotics environment. It seems to me that it is more suitable to call it a planning benchmark. At this point, I believe this paper oversells a little bit by saying this is a robotics benchmark.

**SCRUB and SEEK on other planning benchmarks**
While the effectiveness of SCRUB and SEEK is well-demonstrated on this benchmark, it would be even more convincing to also evaluate them on other commonly used planning benchmarks.

**Related work**
While the related work section sufficiently covers most relevant prior works, I believe it would make the discussion more comprehensive by including some works that construct and execute programs as symbolic plans, including
- Modular multitask reinforcement learning with policy sketches
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- Programmable agents
- Program Guided Agent 
- Program Synthesis Guided Reinforcement Learning

**Other domains**
While it states that 20 domains are included in this benchmark, only 2 domains are discussed in the paper. I did not find descriptions of other domains in the Supplementary material. Am I missing anything?

**The access to the API and data**
While I do recognize the difficulty to "submit" the benchmark, I still find it difficult to evaluate how good this is without actually playing with it. It would be better if more examples such as videos are shown to demonstrates how the benchmark can be used.
summary_of_recommendation: 
I believe this work presents a promising benchmark together with two effective techniques. I am leaning toward acceptance yet still have some concerns regarding how the benchmark can scale up to handle partially observable environments with stochasticity and incorporate low-level interactions, the access to the benchmark, the evaluation of the proposed techniques, and the discussion on the related work.
issues: 
Described in the strengths and weaknesses section.
review_title: null

Meta-review: In this paper, a large-scale robotic task planning benchmark over scene graphs is established. A task-conditioned 3DSG parsification method called SCRUB, and SEEK, a procedure enabling learning-based planners to exploit 3DSG structure is proposed. All reviewers recognized the importance of the benchmark to the community. While there are some concerns about the exclusion of low-level motions, which are also considered important in robotic planning tasks. And some more detailed introduction of other tasks is lacking. 

In the rebuttal session, the authors have responded to the reviewers' concerns and more details are provided. All reviewers recognize the value of the benchmark to the robot learning community, and have reached a consensus of accepting this paper. 

Decision: Accept (Poster)