Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper uses several different techniques in IL and RL to improve performance on 6D robot grasping. It uses an expert planner OMG to collect initial data for BC as well as for online IL via Dagger. The uses DDPG to further train as well as fine tune on new unlabeled objects.

The topic is very relevant and of current interest as more real world applications will need more than just 2D grasping that bin-picking has addressed and related work is sufficiently discussed.

The technical contribution seems weak as the paper mostly explores known methods and well-known 'trade-tricks' (goal conditioning or loss on goal) towards a grasping centric problem which is also heavily explored as part of various RL tasks in literature. The main weakness of the work however is the lack of clear motivation for why such a complicated procedure is necessary compared to the expert planner already being used - the experiments aren't designed to address this question.

- Using a planner as an expert for IL is common practice and I don't think counts as a major contribution as presented at the end of the introduction.

- The bulk of IL experiments focus on what input representation is helpful. While it is not surprising that 3D inputs like point clouds would be better for 6D grasping these are more suitable as ablations than main experiments investigating the proposed method itself, compared to other sota approaches learning based or otherwise.

- In several places 'contact-rich' and 'different dynamics' is motivated without clear explanation early on until the experiments identified what the setup was. The former does not seem to be well explored in the experiments, '...especially in those contact-rich scenarios...'. Aren't all grasping problems contact rich (unless only reaching to a pre-grasp is being considered) or were there some new scenarios constructed to specifically study the relative effects of contact?

- Results in table 1 and figure 4 present mean statistic from 3-5 runs. This seems small, variance bands should be shown in figure 4 to see if the small number of sample are sufficient to capture the full picture.

- The problems studied could be addressed by planar grasping as well. Complex setups with clutter, etc would better motivate if the presented approach is able to scale to scenarios where 6D grasping is necessary. 

Other comments:

- Does not including the BC loss for some samples in the batch (or between training iteration) cause any discontinuities or make learning unstable, as the loss landscape discontinuously changes?

[Update]
Thank you for the responses and clarifications. I appreciate the additional experiments in the real world and comparisons with the open-loop policy. Novelty still remains a concern however; in using a planner as an IL expert, it isn't clear what was challenging to adopt this strategy for the grasping problem and qualifies as a significant contribution. Additionally, the experiments to study 'contact-rich' and 'different dynamics' problems is unclear; the experiments don't indicate what aspects of the proposed method address these challenges and are able to do so with vision/depth-only feedback (no tactile); also the evaluation in simulation alone is insufficient to study such scenarios. I have updated my score accordingly.

review_title: Weak technical contribution with no major insights from experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tackles the task of closed loop 6-DOF grasping of objects in simulation. The learned policy is a closed-loop policy, in that the gripper pose is continuously adjusted as the gripper approaches the object. The paper employs a combination of imitation learning, reinforcement learning, and auxiliary losses for training this policy. The policy operates upon information from point clouds as observed from a wrist-mounted camera.

Strengths: The paper tackles the important and relevant problem of closed-loop 6DOF grasping. The proposed solution makes sound choices: a) uses fused point-clouds to represent the state, b) use of expert behavior for imitation, c) use of auxiliary losses for training. The paper also does systematic experiments in simulation to judge the importance of the different components.

Shortcomings: While the problem is interesting and important, and the proposed approach is sound, the experiments have entirely been done in simulation. Past work on this topic has studied this problem in the real world. In particular, the paper focuses on the design of closed-loop policies. Closed-loop policies are more relevant when there is noise in the motion of the robot, or there are hard to predict dynamics arising from the interaction of the gripper with the object. These are precisely the aspects that should be studied in the real world, as they are hard to model in order to study in simulation. Thus, it is not clear to me as to what aspects of this paper will be applicable to the study of this problem in the real world.

My second concern is about novelty over past work. All aspects of the technical approach of the paper have been studied in the past. Use of imitation learning and RL together has been studied (eg: DAPG Rajeswaran et al.), use of auxiliary rewards has been studied (eg: UNREAL Jaderberg et al.), use of hindsight experience replay (eg: Andrychowicz et al.). Thus, I am not sure what is the precise technical contribution made in the paper.

Thus in summary, while the paper tackles an interesting and important problem, the problem has only been studied in simulation which makes the application less interesting. At the same time, proposed approach is largely a combination of known techniques in the literature.

Update: I thank the authors for providing clarifications and additional experiments, in particular the comparison to open-loop grasping (SOTA grasp detection method from Mousavian et al.). I still find the technical novelty of the paper limited. 
review_title: Un-compelling experimental evaluation, novelty concerns

final_score: rating: 7: Good paper, accept
review_content: __Summary__
The paper targets the problem of closed-loop 6D robotic grasping with a parallel gripper based on RGB-D in-hand-camera images. The policy takes an aggregated point cloud (computed from image history) as input (using a PointNet++) and outputs the pose transformation of the gripper. There are several contributions in the specifics of the proposed method. The policy is pretrained using behavioral cloning and DAGGER on known object models, where the expert is composed by a grasp pose sampler and the OMG grasp trajectory planner. Subsequently, the pretrained policy is improved using TD3. Actor and critic networks are each regularized via a loss for solving the auxiliary task of (independently from each other) predicting the final grasping pose. As the goal poses are only available for the expert demonstrations goals are added for the policy roll-out in hindsight, similar to hindsight experience replay, which does not require access to an object model. The approach is evaluated in simulation for grasping YCB and ShapeNet objects with a Franka Emika Panda robot. The evaluation covers ablations of several algorithmical and architectural choices.


__Strong points__
- The paper is well-written
- Thorough and interesting ablations
- Sound approach

__Weak points__
- State-representation might get invalidated when an object is moved, nullifying the biggest advantage of closed-loop grasping
- no comparisons with competing methods (e.g. open-loop 6D grasping + OMG)
- no evaluations in a cluttered environment
- no real robot experiment
- no code


__Recommendation__
I recommend accepting the paper because the system looks sound and promising and is well explained. Some parts, like regularization based on goal pose prediction, are potentially useful also for very different approaches. 

__Supporting Arguments__
Overall the approach seems sound and the success rates of around 90% seem good for closed-loop 6D grasping of unknown objects. The ablations that disentangle the contributions of the different design choices are valuable and I was delighted to also find a comparison to the more common and arguably more intuitive approach of using the goal predictions as input to the networks. As the approach and the results are presently sufficiently clear, the paper provides a valuable contribution and should thus be accepted.

Still, there are also several weak points. 
- The experiments don't seem to show the benefit of closed-loop grasping by only considering an uncluttered environment and not performing comparisons with open-loop approaches (e.g., [1] or [2]).
- Some design choices do not seem to fit well to closed-loop grasping, which is mainly important for dynamic scenes, e.g. for re-grasping slipping objects. However, in such cases the accumulated point clouds can quickly become invalid, and, furthermore, lower-level control actions might be more appropriate than pose-deltas.
- The approach is not evaluated on a real robot. For all I can tell, the simulation does not account for errors in control, calibration, forward kinematics that are encountered in practice.
- The supplementary did not contain source code. I hope that the authors consider publishing the code after acceptance. 

__Questions__
1) How could the proposed method deal with changes in the object position (accumulated point cloud becomes invalid)
2) Did you consider sharing some features between the actor and critic, e.g. parts of the PointNet
3) I assume that the gripper is always closed at the end of a finite-horizon episode, is that correct?
4) I guess the foreground mask is obtained based on the z value of the transformed point cloud. The paper should be more specific here.
5) Please elaborate: "We also empirically observe that the auxiliary task with the bootstrap errors caused by value overestimation."

__Additional Feedback__
If I understand correctly (based on Appendix A) the method uses TD3 instead of "vanilla" DDPG. If this is the case, the main part should also be more concrete.
It's not clear to me what Figure 5 is supposed to show. What does contact-rich mean in this context? Is it about contacts before closing the gripper?

Typos:
extra space in Figure 1: "known objects ."
extra "the" in Section 5.1: "indicates the the aggregated point cloud"

__References__
[1] ten Pas, Andreas, et al. "Grasp pose detection in point clouds." The International Journal of Robotics Research 36.13-14 (2017): 1455-1473.
[2] Mousavian, Arsalan, Clemens Eppner, and Dieter Fox. "6-dof graspnet: Variational grasp generation for object manipulation." Proceedings of the IEEE International Conference on Computer Vision. 2019.
review_title: Sound approach with new insights but some design choices don't seem to fit to closed-loop control

Meta-review: The paper got mixed reviews ranging from 5 to 7. The main concerns of the reviewers were the missing novelty as the paper combines different well known methods for a given problem, so there is no big algorithmic contribution. The presented pipeline for closed-loop grasping using imitation learning from a planner, Dagger and subsequent deep RL with TD3 is a straightforward, but sound and intuitive combination of algorithms to address the problem of closed loop grasping.  The presented results and ablation studies also motivate these algorithmic choices. In the rebuttal the authors addressed most concerns regarding the experiments (missing comparisons to open-loop grasping and real world experiments), but more real world experiments would be necessary to evaluate the effectiveness of the approach.  

This is a borderline paper were  I unfortunately have to recommend rejection due to the missing algorithmic contribution, a major requirement for ICLR. The paper would  however fit very well to a robotics conference and the authors are  encouraged to resubmit the paper the venues such as RSS or CoRL.  
Decision: Reject