Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a two stage hierarchical approach for learning to manipulate objects in a way that generalizes across objects, configurations, and views. The lower level consists of a library of learned or engineered behaviors that can handle a subset of these variations. These behaviors then become discrete actions of a higher level policy, which operates on an estimated 3D scene representation, transformed into the robot frame. The scene representation is then cropped and mapped onto a latent vector representation that is jointly learned with a vector representation for each behavior. At inference time, the behavior is chosen based on the similarity of these vectors. The method is evaluated in simulated and on a real robot in grasping, pushing, and transport tasks.
strengths_and_weaknesses: The proposed method combines a variety of deep learning machinery and models in a structured way. The idea of having a two-stage approach to generating behavior is not new but the way how this is done here is very original and there are many ideas in there that make a lot of sense, e.g. jointly training the input and behavior representations, making generalization over viewpoints explicit via self-supervised objectives and via estimating the scene representation in 3D and transforming it into the robot frame.

This already good paper could be further improved by extending the experimental section with qualitative results that visualize the components of the system. How good are the learned 3D representations in sim and in real? What does the embedding space of the behaviors look like? What are the typical failure cases of the proposed method?

summary_of_recommendation: This is a nice paper that includes a lot of relevant ideas that should be interesting to the field. The paper appears correct and significant to me. It includes experiments in simulation and on a real robot. The experimental evaluation could be improved with additional analysis (see above).

issues: If possible, additional evaluations as discussed above would strengthen the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contribution of this paper is a novel pipeline for learning grasping and pushing actions that integrates both raw sensory data as well as more abstract object features. 

strengths_and_weaknesses: 
The main strength of the paper in my opinion are the results with the real robot (personally, I would focused mostly on these as opposed to the results in simulation)

The main weakness of the paper is the comparison against a weak baseline, almost designed to fail. There clearly are end-to-end systems designed for grasping and pushing that in theory should be able to achieve better results then the baselines (though I understand that training such systems is an art in itself and how hypermarkets are decided can have big impact on the results). The other main weakness of the paper is the lack of many details without which it is a bit difficult to judge the level of contribution (see issues later in the review). 
summary_of_recommendation: Overall, the paper brings in some new ideas and the evaluation with a real robot is great. There are still many issue to be addressed and if there was a "borderline" option between weak accept and weak reject, I would have picked that one. 

issues: 
In section 3, the 3D bounding box would work for relatively simple objects in which the bounding box would have little else but the object inside of it; what about more complicated objects? Also, is the bounding box aligned with the principal axis of the object if there is one?

In section 3.1 when describing the training, is the randomly chosen behavior still condition on the goal for the particular trial? Also, at this point in the paper it is not clear how do you deal with behavior parametrizations, e.g., push can be parametrized with force, direction, etc?

With regards to the 3D object detector, is this learned for each individual object or do you learn a detector that works on all objects, even ones not seen at training? Also, were the test objects in the experiments excluded from the training data for the object detector?

Section 3.2, line 193, why not also change in orientation of the end effector when it comes to the action space? Currently it is stated that it can only change its position (translation) but not orientation. Ideally, the dimensionality of the action space should be stated explicitly. 

Section 3.2, line 195, needs more detail here -- what is the difference between the 25? just the goal? also, how are the objects split into train and test and why is one behavior trained on all objects?

Similarly, for grasping, it not at all clear where these 30 grasping controllers come from and also it is not clear how the grasping point relative to the center of the object is actually selected (or who selects it)

In section 4.1, there needs to be more detail on the simulation environment, in particular, how does it model things like texture, friction, etc.? Are all the objects assumed to have the same material and density and only the shape is different?

With regards to object augmentation in line 228, does this mean that on average, from each original training object, you get 3 augmented objects and configurations?

With regards to the sentence, "An episode is successful if the object centroid is within 5cm of the target at the final timestep.", how is it decided when an episode ends, is there a fixed limit?

With regards to how objects are split into train and test, do the categories play a role e.g., do you ensure that the test set only has objects from novel categories or do you ensure that for each category, some objects are in train and some in test?

For the grasping pick and place task, it would be nice to have a comparison of a traditional approach, like Agile grasp or some other classifier used for point selection and then planning to move it to a target location -- at least, what are your thoughts on how your system would compare against a more traditional approach? The current baseline comparisons are OK but in some ways, it feels like there were selected because they were doomed to fail (there is also the whole issue of how hyperparamters are selected for your approach vs baselines and without any principled methodology for doing so, I can only assume that the graduate students spent any time tuning any parameters mostly for the proposed approach)


some minor issues:

figure 3 could use a longer caption

line 63 of the intro and at other places talk about the object being "diverse" but it is not explained, diverse with respect to what? all the objects seem mostly rigid, simple shapes so what are they diverse in terms of?

line 7 in the abstract, the phrase "in hand," is a bit confusing as the object is not always in he robot's hand

The paper often puts "object location and pose" next to each other, but pose already includes location information so it's a bit redundant.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This works re-factors the problem of manipulation of diverse objects to building a diverse set of policies and building a data-driven policy selector. They show significantly better results on unsee objects than baselines without such a re-factor. Ablations show the importance of the view-predictive and affordance-aware representations
strengths_and_weaknesses: Strengths:
* Approach seems very data efficient and seems to generalize to new objects much better than the baselines.
* This approach may include a variety of policy types: learned or rule based.

Weakness:
* There is no baseline number reported for the real robot result, it seems like a challenging set of tasks and some baseline here would be more compelling.
* This work advocates for building a library of behaviors including new controllers, it would been nice to see some discussion around the trade-off between building these libraries versus just collecting more data for the single-policy approach.
* The interaction label process seems tedious, are there any heuristics used? For example, don't use pick and place if the task requires a pour. Are the baselines also retrained with these interaction label data in some way.
* It is surprising that the baselines perform so poorly in Table1. What are the success rates on the train objects?
* Were there any cases where this approach can tell you if you are missing a behavior to tackle the task?
* How were the real robot controllers designed? Did you have knowledge of the test objects during that design?

summary_of_recommendation: In some limited settings, this approach seems very efficient for manipulation tasks, however, it is not clear how generalizable the result is. Stronger baselines on real robot results would have made this work more compelling. Additionally, some discussion of the trade-offs of having to develop and maintain libraries of policies and the additional cost of the interaction labeling versus single-policy approach with more data. 
issues: * Include a baseline for the real robot result.
* Some discussion/analysis of why the sim baselines in Table 1 are so poor
* Discussion about when no appropriate behavior is in the library, does the selector give you a signal about this?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 
Within manipulation tasks (this work focuses on grasping and adjacent skills), different behaviors are needed to generalize to different kinds of objects and their various orientations - for instance, pinching a container by its lip is specific to a certain family of objects but not others. While in principle it should be possible to train a single pixel-to-actions, goal-conditioned policy to subsume all these kinds of perceptual and control generalization, the authors opt to push the challenge of generalization to the perception stack only. Given some perceived object representation, and a fixed set of open-loop grasping “scripts” with different angles and grasp elevation angles, select the script most likely to succeed at the task. Concretely, RGB-D image goes into a GRNN, which creates a view-invariant 3D scene representation. The object-centric features are “cropped” from the 3D scene representation, which are the same feature space as the retrieval policies (64x64x64x32). The alignment between the object centric 3D representation and policy-specific key results in a fixed open loop pattern being played out. 

An HRL interpretation of this work is that it’s an options policy for a 1-step MDP that learns which of N pre-defined goal-conditioned behaviors to select. The authors show that decoupling perception and planning in this manner leads to better performance compared to an end-to-end perception and policy, and also that their choice of object +behavior representation is superior to 2D images, 3D object locations, and 3D scene feature maps. Generalization to novel objects, novel viewpoints, and object poses is demonstrated on a real robot. 

This is essentially an extension of Redmon and Angelova 2014 to 3D space and multiple viewpoints. 

strengths_and_weaknesses: 
Strengths
Policy is able to generalize to new objects and poses, from different viewpoints, on tasks that would be quite challenging for an end-to-end system to learn
Authors demonstrate that alternate object representations do not generalize as well to varying viewpoints and object configurations.

Weaknesses
Although the authors demonstrate that their choice of feature representation generalizes better than a naive 2D image representation, the downstream behaviors are very simple, scripted behaviors that are hand-tuned such that succeeding at the task means classifying one of K classes correctly. 
The “library” is not really a library of different skills, but rather one or two behaviors with a discretized set of yaw and grasp elevation parameters, which drastically limits the effective DoF of behaviors to small, discrete action spaces. As mentioned in the summary, I believe the closest relevant work to this is not multi-task or options or even skills learning, but just visual grasp point detection generalized to 3D, as the learned “classifier” is just discriminating between candidate grasp poses.   
State-of-the-art closed-loop grasping behavior from 2D images is already able to generalize to multi-view (Sadeghi et al 2017), novel objects and orientations (Kalashnikov '18). Given that the proposed work is essentially an open-loop grasping system (much, much easier and less general in regards to the skills that can be learned), I would have liked to see stronger generalization capabilities in the library.
summary_of_recommendation: The overall framework is quite interesting - pushing the problem of generalization as much as possible into perception + a standard classification problem (embedding alignment), and one that can be learned via hindsight data collection with a library of skills. I would have voted to accept this if this framework were applied to a more challenging library of skills, but instead as applied, there is *no* library skill learning on the real robot - it is merely just applying a classifier to a variety of scripted motions (with specific grasp angles and heights such that the classifier merely needs to pick the right grasp pose). The choice of the term "selector" sort of obfuscates this, but at the end of the day, it's really just a classifier over a set of memorized trajectories. I think the "selector" terminology would be more appropriate if the system were able to generalize in that "key" space, i.e. select new skills from embeddings that were not used at training time.

There is some novelty in the investigation of which 3D object representations computed from RGB-D are most ideal for predicting the "options policy", 
issues: I'd like the authors to comment on how this proposed framework is any different from simply parameterizing a K-way classifier for memorized trajectories, where the classifier is parameterized using a dot product on a learned object representation layer (but otherwise equivalent to a classifier).
review_title: null

Meta-review: Strengths:

- The view invariance of the proposed method (a result of the use of GRNNs) is a key strength and seems to generate over views better than the baselines.

- The high level structure of the approach where the perceptual module and the behavior keys are trained jointly was viewed as a strength. The fact that the low level policies could be learned or rule-based was also viewed as a strength.

Weaknesses:

- Two reviewers asked for stronger baselines. This is in reference to Table 1 where the proposed method is evaluated on a class of grasping tasks and a class of pushing tasks. The baseline methods perform terribly on these tasks and on grasping in particular. However, there are lots of grasp control/detection methods available now that can do very well with grasping, especially in the single-object scenario that we consider here. There are also a variety of push control methods that can similarly do well. 

- There was a concern that the fixed set of behaviors made the approach overly simplistic. Essentially, in the real robot exps at least, the agent looks at the scene and then selects one of a fixed set of motion scripts. Joint learning of behavior policies could help here. Another approach might be to interpolate amongst learned skills at test time.

- Multiple reviewers asked for more comprehensive real robot exps.

Post Rebuttal:

The authors added a significant amount of content and comparisons into the appendix (supplemental), including grasp comparisons which we appreciate, as well as the t-sne plot which was cool. However, if this work is viewed through the lens of "k-way" classifier over pre-scripted behaviors, then (as the authors point out) the remaining contribution is the space in which the scene is represented. Possibly this is really prior work rather than a new contribution here. Nevertheless, the application to manipulation goes beyond prior work and was viewed positively.

Decision: Accept (Poster)