Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces a network architecture based on vision-transformers to produce object-centric latent embeddings, conditioned on canonical views of the objects in question. The learned embeddings are shown to be useful to estimate logical predicates that can be used for question answering and symbolic planning. Particularly interesting is their generalization to novel objects, settings, and from simulation to a real robot setting.
strengths_and_weaknesses: The paper presents a very interesting idea that is at the heart of where current approaches to robot learning struggle: generalization to unseen objects and settings. The proposed method is novel but not overly complex (which is great). The method is tested creatively in a variety of scenarios including on a real robot. The paper is also very well organized and written. I was not able to find some details of the experiments (see below) but I am sure that this can be easily fixed.

The part that I am most worried about are the different baselines. As the paper uses novel tasks or task variations that others have not published on, it is important to make sure that the baselines are the right ones and that they have been properly tweaked and optimized. Based on the text, I am not fully convinced that this is the case here (see below).
summary_of_recommendation: This is an overall very interesting paper with novel ideas that could have a significant impact on the field. Right now, there is some information missing from the experimental section and the baselines have to be improved. If these issues are resolved, I would be happy to update my rating accordingly.

EDIT: The issues were addressed by the rebuttal.
issues: ## CLEVR experiments

L159: For the 2 million questions, are only the validation parts of condition A and B counted? How many questions are used for training?

How are the canonical object views generated here? As mentioned in the paper, providing these, simplifies the task compared to the baseline as the network does not need to infer visual appearance from text. This discrepancy and the fact that the evaluation does not use the questions from the dataset should be clarified in Table 1. Are there baselines that also take in canonical views and therefore would allow a better direct comparison?

Also, are the questions turned into predicates for the proposed method but asked as text to MDETR? Could this cause issues in the comparison?

Table 1: Why does MDETR-oracle achieve a higher ValA accuracy that MDETR? Shouldn’t training on both conditions only help in condition B? Condition A is already used in the non-oracle version, right? Or is this version also trained on validation data in addition to training data? Either way, it would be good to clarify this in Table 1. Also, would it make sense to add a SORNet-oracle variant to this comparison? Why or why not?

L167-169: Based on the results, I agree that the bottleneck is generalizability, but I do not think that the data proves that “generalizability of the visual model” is the bottleneck. It could also be generalizability between the concrete scenarios and questions, e.g. between training and validation splits.

## Predicate Classification

How do the baseline methods classify predicates for unseen objects (L184) if they are not conditioned on these objects?

Table 2: The fact that the baselines do not outperform simple majority prediction indicates that there might be an issue with training the baselines on the imbalance dataset. Did the authors attempt to address label imbalance, e.g. by adapting the sampling between positives and negatives or by weighing the loss terms appropriately?

## Relative Direction Prediction

I am not sure how the relative direction prediction task is formulated here, especially for the baselines. Are the objects fixed and the models always predict the direction between all pairs of objects?

Table 5b shows that the baselines overfit here, which is not surprising given the small dataset size used in this experiment. However, there are a number of ways to reduce overfitting, such as early stopping, regularization losses, dropout etc. Did the authors apply any of these techniques to reduce overfitting in the baselines?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper addresses the problem of classifying objects' spatial relationships from vision. To do this, it proposes a novel architecture which takes in the scene image along with close ups of each object and embeds each object view and each image patch into an embedding space. A transformer jointly processes all of these embeddings to output one embedding per object, which can then be passed to downstream tasks, such as classifying spatial relations. The results show a high level of performance on a block rearrangement task and on the clevr dataset.
strengths_and_weaknesses: Strengths:
- The proposed method enables learning object agnostic relationships by conditioning on the specific objects at inference time. 
- The real world robot results show that this model can transfer well form sim to real and is robust enough for planning through stacking tasks.
- The clevr results show the model generalizes to novel object-attribute compositions. 
- While this model will not work for all manipulation tasks, it looks like this could be used to quickly put together controllers for new tasks which could be useful for robotics practitioners.

Limitations:
- As proposed, the method does not address the case of objects appearing different from different angles, or the case of needing to take the object's pose into account when manipulating it (e.g. a trapezoid or a long prism).
- The results as shown require a high supervision burden of labeling object relationships explicitly. Many prior approaches in this area (e.g. relation networks, [25]) instead aim to learn these relationships implicitly from sparser supervision. 

Questions:
- If the objects look different at different orientations, a single canonical view could be insufficient for finding the object in the scene. I wonder if simply averaging the embeddings from multiple views would be enough to address this limitation. This would be an interesting experiment to improve the paper. 


summary_of_recommendation: I like the method and overall results. The model is novel but not groundbreakingly so, and has only been shown to work in highly supervised settings. However, the object generalization and real world results are promising for future work.

+++
The update from the authors has addressed my comments and I can strongly recommend acceptance.
issues: The paper does not compare to any existing object-attention based methods, nor does it show any results with realistic objects (vs solid colored, convex, geometrical shapes). Although the paper writes that [24] "assume a segmentation model", my understanding is that [24] uses an unsupervised learning method to learn the segmentations. Other than this, the model looks very similar to SORNet, so some further explanation of sets SORNet apart would help explain the novelty.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents Spatial Object-Centric Representation Network (SORNet), an approach that takes an RBG image and several canonical object views, and outputs object-centric embeddings which are then used in 3 spatial reasoning tasks, including: spatial relationship classification, skill precondition classification and relative direction regression. In addition to this, real-world robot manipulation experiments are shown to demonstrate the ability to use these representations for task planning.

strengths_and_weaknesses: **Strengths**

- The system offers a flexible way to accommodate a number of objects in the scene.
- The paper is clear and presents thorough results.
- Convincing real world results.

**Weaknesses**

- This list of contributions seem superfluous. To recap, the states contributions are (1) architecture, (2) dataset, (3) experiments, (4) real robot demo. It's not clear to me how the architecture is a contribution, as it is essentially just a Visual Transformer (ViT). Moreover, experiments to back up your method should not be counted as a contribution; these are required in any paper to support a presented method. I would argue that there are only 2 real contributions here: the method and the dataset.
- Related work could use a bit more attention; currently only has a small paragraph each on spatial and visual reasoning. What about considering other sequential manipulation papers that don't fall within these areas, and explain why perhaps Spatial Object-Centric Representations are better suited for the style pf problem?
- Relative Direction Prediction experiment need to be motivated more; it's not clear why this is  useful thing to learn and doesn’t seem to fit with the rest of the paper. Perhaps this can be expanded upon. However, the video does briefly motivate with a visual servoing section.

**Minor**
- Missing domain rand citations (particularly in the manipulation domain); more details on the domain randomisation? Amount of noise, etc
- Typo "… direction between in the 3D space …"
summary_of_recommendation: The paper has limited technical novelty however, the proposed system instead gives strong experimental results.
issues: **Questions** 
- After you get the plan, how do you locate the pose of the blocks in the real world? This detail seems missing in the real-world section.
review_title: null

Meta-review: Post-rebuttal, there is clear and consistent agreement from all three reviewers that this new method is useful and novel, and potentially important. This is a well-executed paper built around a cool idea that follows through to real-robot experiments.
Decision: Accept (Oral)