Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed an approach to manipulate unknown object with vague instructions. The proposed method learns the topology relationship between an anchor object and the unknown object (segmented though). 
strengths_and_weaknesses: Learning topology and manipulated with vague instructions could improve human robot interactions significantly, and such technique can enable many robotics applications.
However the vague instructions could add more uncertainty and I suspect the affected range surrounding the anchor object might be an sensitive parameters and have strong effects on the performance of the proposed approach.
summary_of_recommendation: The paper is well organized and easy to follow, and the proposed approach was verified with real world experiments.
issues: The 'unknown objects' in the title could be clarified a bit. e.g. the main focus is to be able to manipulated 'novel' object but still rely on segmentation performance.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a novel approach for predicting stable placements of objects in a rearrangement task. This goal of this task is to rearrange a set of objects given high-level predicates describing their target location such as "above", "near" etc. The paper decomposes this task into multiple subcomponents: (1) Object and scene encoders which project the point clouds of a pair of objects and the scene into a latent space, (2) a relation predictor that predicts a set of fixed spatial binary predicates from the latent space representations the objects and scene, (3) a pose prior used to sample new object locations given a spatial predicate, (4) a scene discriminator used to reject invalid placement proposals, (5) a placement planner to determine a target placement location.

The paper evaluates the approach in a variety of simulation and real-world experiments.
strengths_and_weaknesses: ## Strengths

The paper addresses a challenging an important problem and proposes a complex and technically sound approach. It provides a large set of both simulated and real-world experiments, and shows the general validity of the approach. The presentation is mostly clear and easy to follow. I like the decomposition of the approach and the inclusion of a scene discriminator as a "critic".

## Weaknesses

1. Presentation and experimental evaluation: The experimental sections contain various studies, and I think it is a good idea to re-order the sections as done in the revision. I now understand what the key result is and why it shows that the proposed method works better. However, the paper would still profit from a clearer guidance for the reader as to what the experimental hypotheses are, how they matter for the final performance, and whether the experiments confirm the hypotheses. 

2. Model-free vs. model-based. As discussed in the rebuttal, the dataset generation (Section 2.3) and the model-based baseline are large equivalent with the main difference being that one operates on meshes and one of partial points clouds, and that the model-based baseline requires access to CAD models. The latter difference was not obvious to me, and should be pointed out explicitly in the paper.

3. Justification on decisions for problem formulation. In their rebuttal, the authors argued that multiple cameras are difficult in non-controlled scenarios (I agree) and that collecting multiple viewpoints is too difficult / costly (I don't fully agree). I feel the paper makes some arbitrary choices as to what it expects as given and how it frames the problem - e.g. instance segmentation capabilities are considered as a given, but a dense fusion capability for acquiring and merging multiple viewpoints is not. The paper would profit from a better justification as to why these choices were made.
summary_of_recommendation: The paper proposes a complex yet mostly sound approach to the complex rearrangement task. The paper has some shortcomings wrt to clarity and the experimental evaluation, in particular in terms of its presentation.
issues: ### Clarity & Presentation

- Introduction: I couldn't really understand from the introduction what the problem the paper wants to solve and how it wants to solve it until reading section 2.
Importantly, the introduction does not explain what the rearrangement task actually *is*, and the argument in L24-29 does not really motivate the task of rearrangement. I would recommend to first explain what the rearrangement task is, why it is challenging, and how this paper decomposes this task. Then it should be explained what alternatives exist (incl prior work) for each subcomponent, and what the choice + contribution of each component of this paper is. I feel that all of this information is somewhere hidden in the introduction, but it needs to be reordered/rewritten in order to guide the reader more gently through the complex matter.

- Figure 2 & 3: I cannot really understand the depictions of the MDN. What do the different colors mean? What do we see here? These figures should be explained in the paper.

- Latent object & scene encoders. First of all, I think it should be stated more clearly what the purpose of these is. If I understand correctly, this component enables the approach to generalize over different object geometries. Second, I'm wondering what role the hard-coded radius delta plays in the generality of the approach. 

### Experimental evaluation

Table 1 reports compares the learned and the model-based baseline in terms of specificity and sensitivity, but it is actually hard to tell which one performs better. Eyeballing the numbers, it doesn't seem to me that one can be regarded the clear winner, depending on whether we put more emphasis on specificity or sensitivity. Which one should we give more weight?
(Also, what does it mean for the model-based baseline for "centered" to have 0% sensitivity and 100% specificity?)

I would suggest to use a single metric (e.g. F-Score) to compare the baselines. Furthermore, I would suggest to use bold face for the best performing approach in each row.

### Dataset generation & model-based relationship predictor

It sounds to me that the dataset generation (Section 2.3) and the model-based baseline are large equivalent with the difference that one operates on meshes and the other on partial points clouds. Is this assessment correct? Or does object geometry play a bigger role? I would like to see in the paper a more explicit explanation of the differences.

If that assessment is correct, it seems that the only problem of the model-based baseline is the partial visibility. I would like to see a more elaborate discussion in section 3 if that is the case, or why else the model-based baseline performs (allegedly, see previous point) worse.

Finally, if the partial visibility is a problem, why not add 1-2 additional RGB-D sensors? It should be justified why this is not a solution to the problem, as it would significantly simplify the machinery proposed here to infer spatial relationships.

### Minor issues

- Real world experiments. The paper makes the assumption that object points clouds are fully segmented, but no details apart from a citation of [12] are mentioned in section 3.3. The section mentions that grasping is an issue, but can failures also be attributed to the segmentation?
- L99 "optima" is plura, the singular form is of "optimum"
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a novel integrated approach combining visual perception, logical reasoning, and motion planning of a robot arm for pick-and-place tasks. 
Learned through segmented RGB-D data from simulated scenarios, semantic spatial relations between objects are inferred. 
A novel discriminator function is also learned to predict the feasibility of placement pose of an object given a semantic goal defined as a relative spatial relation between the query and a secondary anchor object.
The output of this combined visual and semantic module is fed into an of-the-shelf sampling-based motion planner to realize the pick-and-place task by the robot. 
This architecture is also shown to be transferable to a physical robot executing such plans in the real-world.
strengths_and_weaknesses: The main strength of this work is the novel combination of the learned (spatial) relational predictor and the physical validity discriminator. The overall system architecture is also well-engineered. 

Experimental analysis is detailed and supports the novelty and usefulness claims. The analysis and justification of failure cases are also clear. The real-world transfer of learned components is valuable and encouraging.

In terms of improvement points, I have a few suggestions:
The emphasis on the applicability of the proposed approach for 'unseen' objects is not justified. Because i) it's unclear which type of objects were in the training set vs the leftout 'unseen' ones and how much they resemble/differ from each other w.r.t. their geometries, and ii) the framework relies on segmented point cloud data, 'objectness' seems to become irrelevant or less important. 

Decoupling of goal pose prediction/planning and motion control/execution is a critical (design) feature of this work. I can see its benefits, however, many robotic manipulation tasks require sequential decisions that are likely to depend on each other not only in terms of logical validity but also for manipulation planning feasibility. For example, how (with which pose) the robot picks up a tool to use affects the feasible actions for the follow-up steps. Since the authors also suggest investigating long-horizon tasks, the paper can benefit from such a discussion on how the proposed approach can be applied on those scenarios. 

It's also unclear how this approach can be extended for multi-object relational predictions and planning.
summary_of_recommendation: The paper proposes a two-step process for the problem of rearranging objects in an environment by robots.
This strategy comprises learning to predict spatial semantic relations between objects and learning to infer physical validity of placement poses using RGB-D sensory data. 
Even though individual components rely on prior works, their combination provides a novel framework for simple pick-and-place tasks given spatial logical goals.
The experimental analysis demonstrates promising initial steps for the applicability of this approach for such problems.
In general, the paper is well-written, and with some revisions and clarifications it can be improved further.
issues: Some of my suggestions are above (Strengths & Weaknesses). Here, I provide some further improvement points.
Overall, the presentation is good. However, the method section encapsulates both prior work that this paper relies on and the original contribution all together. Either a separation or clear emphasis on what is based on prior work vs new would improve the paper.

The role of the scene encoder is not well motivated/justified and thus might be clarified better.

Supp. material:

Video: Timings need improvement. There is some flickering.

Appendix: There are some typos, unfinished sentences.
review_title: null

Meta-review: This paper presents a new approach for predicting stable placements of objects in a rearrangement task, based on ambiguous instructions that specify the spatial relations between the placed objects and the surrounding "anchor" objects. The proposed method decomposes  this task into multiple subcomponents: (1) object and scene encoders, (2) relation predictor, (3) a pose prior used to sample new object locations given a spatial predicate, (4) a scene discriminator used to reject invalid placement proposals, (5) a placement planner to determine a target placement location. 

While the paper addresses a challenging problem and proposes a complex and technically sound approach, there were some issues in the paper in its first version, as reported by the reviewers. Mainly, the paper was difficult to understand. The results reported in the experimental evaluation also seemed to be mixed and did not clearly indicate which method works better. 

In their rebuttal, the authors updated the paper and resolved most of the mentioned issues, especially in terms of presentation. The authors also added new results and metrics, such as the F1 score, that clearly indicate the advantage of the proposed solution.
Decision: Accept (Poster)