Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work presents an approach for learning closed-look policies for 6D grasping arbitrary objects. Given an RGBD input, the following is done: 
(1) The object is  segmented from the environment using the RGB input.
(2) The object's segmentation mask is used to get the point cloud corresponding to the object (assuming a calibrated camera, so that the depth image can be used to infer this point cloud).
(3) The point cloud is used an input to two PointNet++ networks, which act as the actor and critic.
(4) The action from the actor (a 6D offset from the current gripper pose) from the actor is executed.
(5) Once a goal grip position is reached, an attempt at grasping the object is made.

The key novel idea in the above is to add an auxiliary objective to both the actor and critic, which is to predict the final grasp pose. This can be trained via supervised learning, making the task of learning good actions and value estimates easier, since these are informed by what a good final grasp pose is.

Learning is done in via a combination of behavioral cloning and deep RL in two steps:
(1) A dataset of expert trajectories is generated by a motion planner in simulation, with grasp planning utilized to select the end positions for the motion planner. DAGGER-based behavior cloning is further used to get a more robust set of data.
(2) The above dataset is included in the replay buffer and RL is done with additional losses for BC and the auxiliary objective. A sparse reward of successful grasping is used and TD3 is used as the RL algorithm. 

Detailed evaluation and ablation experiments are done in simulation, demonstrating high (~90%) success rates and the need for all the aspects of this approach convincingly. Additional evaluation is done in a real world setting, in which a lower success rate (~70) is seen due to the sim-to-real gap. It is also demonstrated that this approach can be used to implement human-to-robot handovers of objects.
strengths_and_weaknesses: Strengths:
* Closed loop 6D grasping of arbitrary objects given raw sensor information is an important problem in robotics that is far from solved
* The paper is well written, with no significant ambiguities
* The problem formulation for training makes no assumptions about prior knowledge about the environment, and while some knowledge of object models is used the RL training can also proceed without that
* Using point clouds as inputs is a good idea, as it allows for zero-shot transfer from simulation to real with a lesser discrepancy in sensing than would be the case with RGB inputs
* The concept of the auxiliary loss is novel and fitting for this problem
* The evaluation is very well done, with an extremely well done ablation study and impressive results. The additional demonstration of object hand-over results also further demonstrates this approach's value.
* The supplementary material contains additional valuable information, in particular more detail concerning the hindsight goal training and details about the training process and informative learning curves

Weaknesses:
* The approach is primarily composed of ideas introduced in prior works. The auxiliary loss being the main novelty and is interesting, but has limited applicability in other settings.
* The method relies on having a robust segmentation algorithm for arbitrary objects
* The method requires learning in simulation, as it needs a large number of interaction steps to converge
* The performance suffers in the real world due to the sim-to-real gap due to discrepancies in physics simulation (in particular contact modeling)
* Also the approach compatible with grasping in clutter, this is not attempted in this work.
* Fine-tuning in the real world is suggested as a possible solution to the sim-to-real gap, but is not attempted in this work

Several of the weaknesses indicate directions for future work, rather than significant issues with this paper. 
summary_of_recommendation: This paper is very well done and presents a well thought out training approach for an important and as of yet rarely addressed problem. The main weakness is having few original ideas, but the strength of how different ideas are combined and the value of the auxilliary loss make this a relatively small issue.
issues: "Expert Init" is listed in Table 3, but I can't find it explained prior to this.
More details regarding failure modes in both simulation and real would be useful, as well as more discussion of what the sim-to-real gap is in this case.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method for closed-loop 6D robotic grasping which learns a policy that maps a state corresponding to a point-cloud of the scene to an action that corresponds to 6D transformation of the gripper pose. The learned model uses an actor-critic schema. The actor is trained using Behavior Cloning approximating the trajectories generated by a motion planner expert, and the critic is trained to predict the expected return of the trajectory that correspond to the grasp probability. Both modules also predicts a goal pose as an auxiliary task. The method is evaluated on Shapenet and YCB object in simulation and in real world, demonstrating nice results in static setting (grasping from table top) as well as dynamic setting (human to robot handover). The use of different modalities from inference as well as different learning features are also ablated in the experimental section.
strengths_and_weaknesses: The method in general makes sense, and the performance seems to be quite good. The results demonstrate the value of the method mostly in dynamic mode where closed-loop is actually required, and to some extent also in static scenes there is some benefit compare to an open-loop prior solution.
I believe that a comparison to other closed-loop 6D methods such as "Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations" could improve the quality of the paper.
In addition, a few references (e.g. OMG, Graspnet) are used either as sub-modules of the proposed approach or for comparison, which has some commonality but also some difference. It could help the reader to have some short description of each reference to understand the deviations. 
summary_of_recommendation: Overall the paper is quite nice.
issues: Nothing beyond what was mentioned above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a system for learning-based 6D Grasping. The method combines imitation and reinforcement learning: use imitation learning to overcome the initial exploration challenges, and use reinforcement learning to further improve the policy performance, especially on shapes that are hard to create expert demonstration for. The method is evaluated on grasping a variety of objects in both simulated and real environments, as well as a human-robot handover task.
strengths_and_weaknesses: I like the practicality of the system: it makes intuitive sense to combine imitation learning and reinforcement learning for challenging problems like learning reactive grasping. The fact that the system operates on partial point cloud observation makes sim-to-real transfer feasible. I also appreciate the final human-robot handover experiment that shows why reactive grasping is important. 

However, I found it difficult to distill the thesis of the paper: what should the readers learn from the paper beyond knowing that the system “works”? From a technical novelty perspective, the key contributions seem to be spread across a few points: 1) combining imitation learning and reinforcement 2) learning from point cloud data for 6 DoF grasping and 3) the goal auxiliary loss. 

Like mentioned in the paper, combining imitation and reinforcement learning is not a new idea in visual-motor learning, and the idea of learning from raw point cloud for grasping has been explored by a few prior works too. The goal auxiliary losses seems new, but I’m not entirely sure what’s the motivation behind this change. What challenges are these loss functions designed to address? The only place that discusses this point is 149-151: “We introduce two auxiliary tasks for the actor-critic to predict goals, which help to learn the policy and the Q-function by adding goal supervision”, but it doesn’t really motivate *why* these losses are added. I wish the authors could shed more light on the key takeaways of the paper in the next iteration.

On the other hand, the paper does fulfill its high-level goal of developing a learning-based reactive grasping system. And the system seems to fit especially well for human-robot handover tasks judging by the experiment evaluation. However, I had a similar struggle to understand the key differences between the proposed system and the prior works. The only relevant point made in the paper is that “it performs on-par with the prior work”. What are the assumptions relaxed, if any, compared to the prior work that made this result significant? I’d like to see a more in depth discussion on the result. 

Minor: 
What’s the input space for real-world experiments? I don’t see it mentioned anywhere in 4.2 and 4.3.

summary_of_recommendation: Like mentioned in the comment, I like the practicality of the system and appreciate the evaluation. But I struggled to distill the key lessons from this paper. Hence before seeing more comments from the author, I do not recommend accepting the paper.
issues: See main comments.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method for learning closed-loop 6D robot grasping policies of singulated objects, using segmented aggregated point clouds as input. It proposes a combination of behavior cloning from expert demonstrations and reinforcement learning based on an actor-critic approach. Furthermore, it introduces an auxiliary task for supervised training of the actor and critic networks, which consist of predicting the final grasping pose of the end-effector. During the behavior cloning phase, additional demonstrations are gathered following the DAGGER scheme, where the expert provides the labels for actions and goals. During the RL phase, goal labels are generated either from a set of pre-labeled goals (in case of known objects with annotations) or hind-sight goals are used (in case of unknown objects) to add a goal label to the data tuples in the replay buffer. 
Extensive ablation studies are conducted in simulation on the algorithmic choices and on different input representations underlining their benefit. The paper also show-cases the method on two real world tasks: Table-top grasping and human-to-robot handover.

strengths_and_weaknesses: Strength:
- The paper contains extensive ablation studies on the different parts of the method, which do not only give insight into the method proposed in the paper, but can also give useful hints to other research in the field.
- The combination of imitation learning and reinforcement learning is promising for an increased sample efficiency of robotics manipulation tasks, in particular in the proposed combination with DAGGER (-like) training and hind-sight goal labelling. 
- The idea of using goal prediction as auxiliary task is interesting and seems to lead to surprising performance gains as compared to goal conditioning. 

Weaknesses:
- Many important technical points were transferred to the appendix, without reference, but are actually crucial for an understanding of the proposed method,e.g. point cloud aggregation, the training of the goal prediction network. There should at least be a to the relevant section in the appendix, but a discussion in the main paper would be better. 
- There seem to be missing some numbers in the tables, which are referenced in the text. In particular table 2 and table 3.
- The benefit of 6D grasping does not become obvious to me from the experiments on grasping. A comparison of a method capable of top-down grasps only would be useful. 
summary_of_recommendation: I liked reading the paper and found  the thorough ablation studies very insightful. In particular the comparison of different input modalities is interesting and can provide very useful insights to the community. The proposed methods for to both closed loop grasping and 6D grasping constitute relevant contribution to research in that field. 
I therefore recommend the acceptance of the paper. 
issues: - The method for aggregating point clouds is not mentioned in the methodic section of the main paper. The first place where the word aggregation appears (but not explained) is in the section 'Ablation on Other Design Choices'. Since it is however an integral part of the method proposed in the paper, it should be mentioned earlier, at least with a reference to the appendix.

- In lines 212 - 219: The paper states: 'From table 2 we can see that ... iv) Comparing to the supervised BC training, our GA-DDPG algorithm further improves the policy, ...' There seems to be no mentioning of BC in that table. Can the authors clarify, what they refer to?

- In line the authors state 214-215: ii) 'The goal-conditioned policies perform worse than the goal-auxiliary counterparts. When the predicted goal is not accurate, it affects the grasping success.' Did the authors investigate that further, or is it just a hypothesis?  

- In line 226 the paper states: 'We see that the success rate is improved from 88.2% to 93.5% after fine-tuning.'  Where can we see that? It would be nice to add this result to table 3. 

-  A discussion of the sensitivity of the proposed method wrt. hyperparameters (e.g. the weighting lambda of the actor rewards ) would be interesting
review_title: null

Meta-review: The reviewers have agreed that the paper has been addressing an important problem in robotic, i.e. closed loop 6D grasping of arbitrary objects. The combination of imitation learning and reinforcement learning is sensible, and the concept of the auxiliary loss is novel and fitting for this problem. However there is a suggestion to elaborate with motivations why these losses are added. In overall, though the idea is incremental, i.e. the combination of imitation learning and reinforcement learning, the evaluation is very well done, with a well done ablation study and impressive results. There are concerns that the key differences between the proposed system and the prior works  and under what assumptions (e.g. robust segmentation algorithm, grasp a single object) are unclear. There are also suggestions that a comparison of a method capable of top-down grasps or other closed-loop grasping methods should be included. 

During the rebuttal phase, the authors have addressed most concerns by the reviewers. Though there are still open issues, e.g. comparisons to existing 6d close-loop and open-loop table-top grasping methods, the authors have demonstrated the efficiency and practicality aspects of their approach through many interesting evaluations. 
Decision: Accept (Poster)