Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Authors: Lirui Wang
Yu Xiang
Wei Yang
Arsalan Mousavian
Dieter Fox
Venue: CoRL2021 Poster (2021)
Keywords: 6D Robotic Grasping
Imitation Learning
Reinforcement Learning

Abstract: 6D robotic grasping beyond top-down bin-picking scenarios is a challenging task. Previous solutions based on 6D grasp synthesis with robot motion planning usually operate in an open-loop setting, which are sensitive to grasp synthesis errors. In this work, we propose a new method for learning closed-loop control policies for 6D grasping. Our policy takes a segmented point cloud of an object from an egocentric camera as input, and outputs continuous 6D control actions of the robot gripper for grasping the object. We combine imitation learning and reinforcement learning and introduce a goal-auxiliary actor-critic algorithm for policy learning. We demonstrate that our learned policy can be integrated into a tabletop 6D grasping system and a human-robot handover system to improve the grasping performance of unseen objects. Videos and code are available at https://sites.google.com/view/gaddpg.

Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Lirui Wang', Yu Xiang", Wei Yang', Arsalan Mousavian', Dieter Fox 'University of Washington, 2NVIDIA liruiwecs wwshington. edu, fyux, weiy, amousavian, dieterf)envidia.com
Abstract: 6D robotic grasping beyond top-down bin bin-picking scenarios is a chal- lenging task. Previous solutions based on 6D grasp synthesis with robot motion planning usually operate an open-loop setting, which are sensitive grasp syn- thesis errors. In this work, we propose a new method for learning closed-loop control policies for 6D grasping. Our policy takes segmented point cloud of an object from an egocentric camera as input, and outputs continuous 6D control ac- tions the robot gripper for grasping the object. We combine imitation learning and reinforcement learning and introduce goal-auxiliary actor-critic algorithm for policy learning. We demonstrate that our learned policy can be integrated into a tabletop 6D grasping system and a human-robot handover system to improve the grasping performance of unseen objects. Videos and code are available at https:/ /ttp:///s/eeee/eeeeeeeseeeeeeees google. com/view/ ggwee//.g......
Keywords: 6D Robotic Grasping, Imitation Learning, Reinforcement Learning
Introduction
Robotic grasping of arbitrary objects is challenging task. A robot needs to deal with objects it has never seen before, and generates a motion trajectory to grasp an object. Due to the complexity of the problem, majority works in the literature focus on bin-picking tasks, where top-down grasping is sufficient to pick up an object. Both grasp detection approaches [1, 2, 3] and reinforcement learning-based methods [4, 5] are introduced tackle the top-down grasping problem. However, is difficult for these methods to grasp objects in environments where 6D grasping is necessary, i.e., 3D translation and 3D rotation the robot gripper, such as a cereal box on a tabletop or in a cabinet.
While 6D grasp synthesis has been studied using 3D models of objects [6, 7] and partial observations [8, 9, 10], these methods only generate 6D grasp poses of the robot gripper for an object, instead of generating a trajectory to reach and grasp the object. As a result, motion planner is needed to plan the grasping motion according to the grasp poses. Usually, the planned trajectory is executed in an open-loop fashion since re-planning is expensive, and perception feedback during grasping as well dynamics and contacts of the object are often ignored, which makes the grasping sensitive to grasp synthesis errors.
In this work, to overcome the limitations of open-loop 6D grasping based grasp synthesis, introduce a new method for learning closed-loop 6D grasping polices from partially-observe point clouds of objects (Fig. 1). Our policy takes segmented point cloud of an object as input and directly outputs the control action of the robot gripper, which is the relative 6D pose transformation of the gripper. Our policy can be combined with different perception methods for object segmentation.
In order to learn the policy, we combine Imitation Learning (IL) and Reinforcement Learning (RL). Because the chance of lifting object in 6D grasping is very rare if only RL is used for exploration. We obtain demonstrations using joint motion and grasp planner [11] simulation. Consequently, we can efficiently obtain large number of 6D grasping trajectories using ShapeNet objects [12] with the planner. Then we learn the grasping policy based on the Deep Deterministic Policy Gradient (DDPG) algorithm [13], which is an actor-critic algorithm in RL that can utilize off off-policy data from demonstrations. More importantly, we introduce a goal prediction auxiliary task to improve the policy learning, where the actor and the critic in DDPG are trained to predict the final 6D grasping
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

State
Action
Segmentation
S
at
Policy
Deep Neural Network
Point cloud
Closed-Loop
Relative Translation and 3D Rotation
Image
Figure I: Illustration of our learned policy for 6D grasping. The state representation is based on the segmented point cloud of an object. The action is the relative 6D pose transformation the robot gripper. The green gripper in the figure indicates the next target location of the gripper.
pose of the robot gripper as well. The supervision goal prediction comes from the expert planner for objects with known 3D shape and pose. For unknown objects without 3D models available, we can still obtain the grasping goals from successful grasping rollouts of the policy, i.c., hindsight goals. This property enables our learned policy to be fine-tuned on unknown objects, which is critical for continual learning in the real world. We that adding the auxiliary losses stabilizes training and improves the learned policy.
Overall, our contributions are: 1) We introduce Goal-Auxiliary DDPG (GA-DDPG) algorithm for joint imitation and reinforcement learning of 6D robotic grasping. 2) We demonstrate that our learned policy can be integrated into a tabletop 6D grasping system to improve grasping perfor- mance. 3) We demonstrate that our closed-loop policy can improve
handover.
Related Work
Vision-based Robotic Grasping. Grasp synthesis can be used in a planning and control pipeline for robotic grasping [2, 14, 15). Alternatively, end-to-end policy learning methods [16, 4, 17] make use of large-scale data to learn closed-loop vision-based grasping. Although RGB images widely used the state representation, requires the policy to infer 3D information from 2D images. Recently, depth and segmentation masks [3], shape completion [9]. deep visual features [18], keypoints [19], point clouds [20], multiple cameras [21], and egocentric views [22, 23] have been considered to improve the state representation. Motivated by these methods, we utilize point clouds of objects as our state representation, which transfers well from simulation the real world.
Combining Imitation Learning and Reinforcement Learning. Model-free RL [4, 5] requires a large number of interactions even with full-state information. Therefore, many works have proposed to combine imitation learning [24] in RL. [25] augments policy gradient update with demonstration data to circumvent reward shaping and the exploration challenge. [26] uses inverse RL learn dexterous manipulation tasks with a few human demonstrations in simulation. The closest related works ours are [27, 28] that utilize demonstration data with off off-policy RL. Despite the focus on different tasks, the main difference is that our demonstrations come from an expert planner instead of human demonstrators. Thus, we can obtain large number of consistent demonstrations and query the expert planner during training to provide supervision on the on- on-policy data.
Goals and Auxiliary Tasks in Policy Learning. Goals are often used extra signals to guide policy learning [29]. For goal-conditioned policies, [30. 31, 32] make the observation that every trajectory is successful demonstration of the goal state that reaches, thereby these methods label goals in rollout trajectories for effective learning. However, goals still need to be provided in testing. In 6D grasping, estimating the grasping goals is challenging problem. On the other hand, auxiliary tasks have been used to improve RL as well [33, 34, 35, 36]. We utilize the grasping goal prediction an auxiliary task that requires the policy predict how to grasp the target object.
Learning 6D Grasping Policies
Our goal is to learn closed-loop policy for 6D robotic grasping. We first introduce related back- ground knowledge and then present our method for learning the policy.

---

3.1 Background
Markov Decision Process. A Markov Decision Process (MDP) is defined using the tuple: M {S. A. O.P. P.Pon Po: po.
S, A,A, and O represent the state, action, and observation space. R : S A - R the reward function. P: S A + S is the transition dynamics. Po the probability distribution over initial states and y = [0, 1) is discount factor. Let π S + A be policy which maps states to actions. In the partially observable case, at each time 1, the policy maps a partial observation or of the environment to an action
x (ot). Our goal is to learn policy that maximizes the expected cumulative rewards E a 12 7 'rel. where r; the reward at time t. The Q-function of the policy for a state-action pair is Q a) = R(s. a) + y E x 1
2're|ss = s'), where s' represents the next state of taking action a in state s according to the transition dynamics.
Deep Deterministic Policy Gradient. DDPG [13] is actor-critic algorithm that uses off data, deterministic policy gradient, and temporal difference learning. It has successful applications in continuous control [37, 27]. Specifically, the actor in DDPG learns the policy (o(s), while the critic approximates the Q-function Qs(s, a), where 0 and @ denote the parameters of the actor and the critic, respectively. A replay buffer of transitions D {(s, a, s')3 is maintained during training, and examples sampled from it are used to optimize the actor and critic alternately. DDPG minimizes the following Bellman error with respect to Φ:
E (s,a,,s''> (oo(s.a) a) - (r + Qalld (d))
(1)
Then the deterministic policy To is trained to maximize the learned Q-function with maxe E BD (Qo(s, (o())), which resembles a policy evaluation and improvement schedule.
3.2 6D Grasping Policy From Point Clouds
We consider the task of grasping an arbitrary object in a closed-loop manner. Our goal is to learn 6D grasping policy
8 a, that maps the state 8, at time to an action or- To simplify the setting, we still use to represent the input of the policy even though observations are used. At time the action a, is parametrized by the relative 3D translation and the 3D rotation of the robot end- effector. Therefore, our learned visuomotor policy represents an operational space controller for 6D grasping. We utilize 3D point clouds of objects to represent the states, which can be computed using depth images and foreground masks of the objects. These points are transformed from the camera frame to the robot end-effector frame represent the state. Different segmentation methods can be used to obtain a foreground mask of a target object. Our learned policy is robust to noises in the segmentation. For grasping static objects, we aggregate the point clouds from previous time steps as described in Appendix Section 6.2. For dynamic objects, no aggregation is used.
3.3 Demonstrations from a Joint Motion and Grasp Planner
To obtain demonstrations for 6D grasping, we utilize the Optimization-base Motion and Grasp Planner (OMG Planner) [11] as our expert. Given a planning scene and a set of pre-defined grasps on target object from grasp planner [6. 7]. the OMG Planner plans trajectory of the robot to grasp the object. Depending on the initial configuration of the robot, it selects different grasps as the goal. This property enables to learn policy that grasps objects in different ways according to the initial pose of the robot gripper.
Let E = (To. ....., Tr) be a trajectory of the robot gripper pose generated from the OMG planner to grasp an object, where T, E SE(3) is the gripper pose in the robot base frame at time t. Then the expert action at time can be computed as at T++ T which the relative transformation of the gripper between and I. + 1. Furthermore, Tr from the planner is a successful grasp pose of the object. We define the expert goal at time t as of = T, which is the relative transformation be- tween the current gripper pose and the final grasp. Finally, the state 8 can be obtained by computing the point cloud as in Section 3.2. In this way, we construct an offline dataset of demonstrations from the planner as Despen
......,Ti
{(st. 07. 87, s where sf is the next state.
3.4 Behavior Cloning and DAGGER
Using the demonstration dataset D Dexpent, we can already train a policy network no by applying Be- havior Cloning (BC) to predict an action from a state. To minimize the distance with expert actions,
3

---

+.12
Prediction
Coordinate (30)
10
PointNet
Actor
Translation
Retation
Notation (quatemonnn 4D))
Action
Goal sipervsiin
Cloud
PointNete+
Crttc
Critkc Prediction
GoWllo
Our network architecture uses PointNet++ [40] for feature extraction. Both the actor and the critic are regularized to predict grasping goals as auxiliary tasks.
Figure 2:
different loss functions on (3) can be used. In particular, we adopt the point matching loss func- tion defined in [38] to jointly optimize translation and rotation:
LPOSE( (ros T2) = X
[T((2) - -2()))- x
Σ
where T1, T2 E SE(3), Xy denotes a set of pre-defined 3D points on the robot gripper. and L1 norm is used. Let ag (s) be an action predicted from the policy and a = x xs(s) be an action from the expert, the behavior cloning loss is LBC aa) LPosE aq).
We apply DAGGER [39] to augment supervisions for rollouts from the learner. Given a sampled initial state so, the current policy To can roll out trajectory of states Su; 8y and actions do...., Or with end-effector poses To...., Tr. The expert then treats each state as the initial state and generates trajectory to grasp the object ((st) (Te, Te.,, Tr). The first action in the plan aj T (s:) T+ is used as supervision to correct deviations of the learner from ((x)). A corresponding expert goal g.- = Tr- can also be obtained for state 8t. These action and goal labels are used to construct dataset Daugger = {(E,, ae, ai, Ir, 8))).
3.5 Goal-Auxiliary DDPG
To handle contact-rich scenarios that are rare the expert trajectories, we further improve the policy learning by combining RL with expert demonstrations. In our RL setting, each episode ends when the agent attempts a grasp or reaches a maximum horizon T. The sparse reward r is an indicator function given at the end of the episode denoting if the target object is lifted or not.
Similar to DAGGER, we collect on-policy data for DDPG training. The only difference is that do not have expert demonstration for the action az and goal of for a state st from the learner policy. Instead, we first find the nearest goal from a goal G by 9t = arg min LPOse (g, Te), where Tr is the pose of the robot gripper at time Then the heuristic goal for & is gr Te Note that the goal set G is only available if we have the 3D shape and pose of the target object. Finally, we construct a dataset for DDPG Dares = {(t,, at, Gt. rt, s))). where 7 is the reward at state & and sf is the next state of st from the on-policy rollout. The replay buffer of DDPG training D = Dexpen U Dangeer U Dadpe. where we augment Dexpent and Daageer with the sparse reward.
Our network architecture for training the actor-critic is shown in Fig. 2.
We introduce two auxiliary tasks for the actor-critic to predict goals, which help to learn the policy and the Q-function by adding goal supervision. Given data sample (s, a, 9, s') from the replay buffer, the critic loss is
T = 2 Q (8. a) y)² + LAUX 9o):
where y + (Q To. (s') + e) is the Bellman target and Is is the predicted goal. Qu and Ty. are the target networks and e is a pre-defined clipped noise as in TD3 [41]. The auxiliary loss is TAuux (g, 9+) Lposn(g, ga) for measuring pose errors in goal prediction.
Given a data sample (s, a", g)) from the replay buffer, the loss function for the actor defined as La Xanc(a", ag)) (1 -DDraa an) Txux(g.go)),
(4) where og and go are the action and the goal predicted from the actor, respectively, and À is weighting hyper-parameter to balance the losses from the expert and from the learned value func-
tion. The BC loss LBc(a", aa) LPosE(a" aa)) as defined before, which prevents the learner

---

from moving too far away from the expert policy, The deterministic policy loss is defined as LDDPG 0o) Qa((s, ag) in order to maximize Q values. Note that the expert action a* is only available if the data sample is from Dexpent Dingger- Otherwise, we do not include the BC loss in Eq. (4). Even though our method can be used with any off-policy actor-critic algorithms, in practice, we choose TD3 for its improved performance over vanilla DDPG.
3.6 Hindsight Goals For Fine-tuning on Unknown Objects
For unknown objects without 3D models, we cannot obtain neither expert demonstrations nor grasp- ing goals from pre-defined goal set. We can use hindsight goals to fine-tune our policy on unknown objects. Based on the current policy ny. each learner episode rolls out a trajectory of state-actions with end-effector poses To.... Tr. If the grasp in the episode is successful, we know that Tr is a success grasp for the target object, which is also known the hindsight goal. Then the grasp goal for state 8, in the episode as ÿt- - T, which is used supervise the actor and the critic via the goal auxiliary tasks. By using hindsight goals, we construct a dataset using on on-policy rollouts on unknown objects Phindsight (st, az. Tr, s()), where 9r is the hindsight goal. We finetune the pretrained actor-critic network on the dataset D = Dexpen U U'inndiiiht The algorithm for training with hindsight goals can be found in Appendix Section 6.1.
compute
Experiments
Simulation Environment. We experiment with the Franka Emika Panda arm, a 7-DOF arm with parallel gripper. We use ShapeNet [12] and YCB objects [42] as our object repository. A task scene is generated by dropping sampled target object with a random pose
table in the PyBullet Simulator [43]. The maximum horizon for the policy is T = 30. An episode is terminated once a grasp is completed. The observation image size is 112 112 PyBullet using a hand camera. More environment details can be found in Appendix Section 6.2.
Training and Testing. We use approximately 1, 500 ShapeNet objects from 169 different classes in the ACRONYM dataset [44] for training, where each object has 100 pre-computed grasps from use selected YCB objects with 10 scenes per object and 138
[7] for planning. For testing, holdout ShapeNet objects within 157 scenes. We train each model three times with different random seeds. We run each YCB scene times and each ShapeNet scene times and then compute the mean grasping success rate. More details on network architecture and training can be found in Appendix Section 6.3 and Appendix Section 6.4, respectively.
4.1 Ablation Studies in Simulation
blations on State Representations. We first conduct experiments with BC to investigate the effect of using different state representations. Table presents the grasping success rates. When using images, there are three variations: "RGB", "RGBD" and "RGBDM" indicating whether depth (D) or foreground mask (M) of the object are used or not. Depth and/or mask concatenated with the RGB image, and ResNet18 [45] is used for feature extraction. "Point" in Table indicates the point cloud state representation. "Offline" means fixed-size offline dataset Dexpert is used for training which contains 50,000 data points from expert demonstrations. "Online" means the expert planner is running in parallel with training, which keeps adding data to the dataset. "DAGGER" uses on-policy rollout data from the learner policy with expert supervision Dingger. and "Goal- "Gxal--uxlliary" indicates whether we add the grasping goal prediction task or not in BC training.
From Table 1, we can see that: i) using the point clouds achieves better performance compared to using images of the current view. This indicates that the 3D features in object point clouds general- izes better for 6D grasping. ii) Adding depth or foreground mask to the image- based representation improves the performance. iii) "Online" is overall better than "Offline" by utilizing more data for training. iv) Both DAGGER and adding the auxiliary task improve success rates.
Ablations on Goal-auxiliary vs. Goal-conditioned. We evaluate our goal-auxiliary DDPG algo- rithm for 6D grasping. where the dataset
Despen U Dungger U Daupg is constructed for training. We also consider another common strategy of using goals for comparison: goal- coal-condiiioned policies such as in [30, 28, 31], where a goal is concatenated with the state as the input for the network. In this case, in order to use goals in testing, we need to train a separate network to predict goals from states. Table 2 displays the evaluation results, where we test different combinations of adding the goal- auxiliary task or using the goal-conditioned input to the actor and the critic.
5

---

Input
Method DAGGER
Test
Offline Online
Goil-Auxilary
ShapeNet
YCB
RGB
43.6
45.2
RGB
50.6
51.5
RGB
$2.1
52.3
RGB
58.1
RGBD
45.3
46.3
RGBD
58.7
65.3
RGBD
60.4
67.1
RGBD
71.7
RGBDM
48.3
50.4
RGBDM
52.2
RGBDM
67.4
66.7
RGBDM
72.3
RGBDM
75.2
72.6
Point
73.6
65.6
Point
73.3
67.3
Point
72.1
Point
72.3
72.7
Point
75.8
77.2
Point
79.6
78.5
Table 1: Success rates of different models trained by behavior cloning with expert supervision.
Policy
Nome
Auxilaary
Goal Conditioned
Both
Critic
ShapeNet
YCB
ShapeNet
YCB
ShapeNet
YCB
ShapeNet
YCB
None
80.6
71.0
17.3
12.8
68.6
60.7
76.7
76.6
Goul-Auxiliary
84.6
82.8
91.3
88.2
83.4
81.3
87.8
84.1
Goal-Conditioned
81.6
73.7
70.3
63.2
82.1
Both
86.8
82.5
87.9
83.1
84.5
81.7
86.9
82.3
Table 2:
Evaluation on success rates of two strategies of using goals RL with DDPG.
From Table 2, we can see that: i) using the goal-auxiliary loss for the critic significantly improves the performance since it regularizes the Q-learning. ii) The goal-conditioned policies perform worse than the goal-auxiliary counterparts. When the predicted goal is not accurate, it affects the grasping success. We also observe some instability in goad-conditioned policy training. iii) Adding the goal- auxiliary task to both the actor and the critic achieves the best performance. iv) Comparing to the supervised BC training in Table I, our GA-DDPG algorithm further improves the policy, especially in the contact-rich scenarios. Imitation learning alone cannot handle these contacts with the target object before closing the fingers since these are rarely seen in the expert demonstrations.
A Allations on Fine-tuning with Hindsight Goals. Our policy trained on ShapetNet objects achieves 91.3 1.3% and 88.2% success rate for grasping unseen ShapeNet and unseen YCB objects, respectively, as shown in Table 2. We can further improve the policy using RI with unseen objects. To resem- ble real-world RI., we assume that the 3D shape and pose of the YCB objects are not available. Therefore, there is expert supervision on these objects. We fine-tune the policy using the dataset D = Dexpen J Dhindsight with hindsight goals. We see that the success rate is improved from 88.2% to 93.5% after fine-tuning. We also experiment with the simulation dynamics by decreasing the lateral frictions the object, which makes object easier to slide during contacts. In this case, the success rate of the pre-trained policy reduces to 81.6%. After fine-tuning with hindsight goals, the pol- icy adapts well and achieves 88.5% success rate, which demonstrates the robustness of the training procedure to contact model errors.
Ablations on Other Design Choices. We conduct ablation studies on several design choices in GA-DDPG as shown in Table 3. We can see that: i) RI. without BC failed to learn a useful policy due to the high-dimensional state-action space and sparse reward. ii) Both online interaction for adding data and DAGGER help. iii) Our aggregated point cloud representation is better than using "RGBDM" images or single-frame point clouds. iv) Early fusion of the state-action the critic is better than late fusion, i.e., concatenating features of state and action. v) "Expert Init.": expert-aided exploration by rolling out expert plans for a few steps to initiate the start states during leamner rollouts, which is beneficial. vi) The point matching loss defined in Eq. (2) is better than direct L2 loss on translation and rotation. Overall, we observe that these design choices both stabilize training and improve the performance. Ablation experiments grasp prediction performance be found in Appendix Section 6.5.
also use
Test
No BC
Ofllnn
DAGGER
Image
Aggr.
No Expert
L.2 Loss
Final
ShapeNet
2.7
72.3
83.9
83.7
83.7
79.1
87.7
81.8
91.3
YCB
3.8
63.7
85.3
78.9
83.5
82.1
86.6
79.2
88.2
Table Ablation studies on different components in our method with grasping success rates.

---

Fixed inttill joint
Varying iniiial joint
Object
Policy
Planning
Combined
Policy
Planning
Combined
YCB
35/45
38/45
37/45
31/45
26/45
34/45
Unseen
8/10
5/10
8/10
7/10
7/10
7/10
Success Rate
78.2
78.2
81.8
69.1
60.0
74.5
Table 4: Real-world grasping results for single objects table. The numbers of successful grasps among trials and the overall success rates are presented.
mustard bottle
Toam brick
bleach cleanser
red bow!
cracker box
tomato soup can
Unseen
T-shirt
water bottle
white mug
mar
small pillow
plastic container
Figure 3: Successes (left side) and failures (right side) of real-world tabletop 6D grasping using policy trained in simulation.
4.2 Tabletop 6D Grasping in the Real World
We conducted tabletop grasping experiments
Franka robot with realsense D415 camera
the robot gripper. We evaluated two different settings: fixed initial setting and varying initial setting. In the first setting, the robot always starts from the same initial position but the object positions change to evaluate the robustness of GA-DDPG with respect to different object poses. In the second setting, we evaluate the robustness of the system against different intial positions of the robot. For each setting we consider 3 baselines: a) Policy: we use GA-DDPG during the whole trajectory and grasping the object. b) Planning: we GraspNet [10] generate grasps for the object and OMG planner [11] for planning. GraspNet trains a variational autoencoder to generate 6D grasps from an input point cloud of an object, while the OMG planner is trajectory optimization method for joint motion planning and grasp selection. c) Combined: The planner plans trajectories with length of 25 steps. We execute the first 20 steps the plan and switch to GA-DDPG for the final part. Objects are segmented using unseen object instance segmentation method [46] for all the settings. We evaluated different variations on YCB objects and 10 unseen objects with varying shapes and materials. GA-DDPG is finetuned on YCB objects. For each YCB object, we tested 5 different configurations according to the fixed/varying initial settings. Table presents the numbers successful grasps among trials. Fig. 9 in the Appendix shows all the objects in our experiments.
We can see that the learned policy performs on par with the open-loop planning with fixed initial robot poses and outperforms it in the varying initial setting. Combining planning with the policy achieves the best performance in both settings. For varying initial poses, open-loop planning suffers more due to perception errors and control errors, since the robot started further away from the target object. Using the policy for the last steps of grasping can fix some open-looy failures in the combined version. However, since the policy is trained in simulation, it suffers from the sim-to-real gap, especially in contact modeling with objects. Improving the fidelity of physics simulation and also fine-tuning with real-world data would help bridge the gap. A discussion on design limitations can be found in Appendix Section 6.8. Fig. 3 shows some successful and failed grasping examples. Most failures in the real world are due to the sim-to-real gap in contacting modeling.
4.3 Human-to-Rou Handovers in the Real World
Our closed-loop policy can be applied to human-to-robot (H2R) handovers. Based on the perception system introduced in [47], we integrate GA-DDPG to compute actions for robot control. Specifi- cally, the point cloud of an object hand was segmented using an external Azure Kinect RGB-D camera according to [47]. Then the object point cloud is fed into the policy network to compute the control action. As shown in Fig. (a), we used subset of 10 household objects proposed in [47], and measured the time to successful handovers, success rate and the number of grasp attempts. We first conducted systematic evaluation to measure the performance of our approach with respect to different objects and varied handover locations. Then we conducted a user study with 6 partici- pants recruited from the lab. We asked the participants to answer questionnaire with Likert-scale questions and open-ended questions after the experiments.
7

---

MAs. Too MRANGRORG
Left: 90%
Middle: 100%
Fight: 100%
L->R 60%
R-YL 40%
Figure 4: (a) The 10 unseen household objects we used for the human-to-robbe handover exper- iments. Pleuse zoom in for the best view. (b) The systematic evaluation with varied handover locations (left, middle, right) and moving objects during the handover (from left to right, and from right to left). We report the success rate for each setting. The overall success rate is 78%
(a)
the
Approach Time (seconds)
Success Rate
Number Attempts
30006
2.0
Yong st 1RR22
35%
1.5
65%
1.0
604
Plaee
bdy
Romele
Tooml
up
boug
Strongly disagree Disagree Neither agree or disegree
Agroz
Strongly sgree
The rabot and I worked fluently toam to transfor objecte.
prusted the robot to do the right thing ot the right time.
The robot was aware of my actions.
The robot was aware of the difference of objects my hand.
robot was to take arbitrory objects from me on efficient way.
Figure 5: Top: Quantitative results of the user study with 6 participants and 10 household objects. Bottom: Participants' agreement with each statement in the questionnaire.
Systematic Evaluation. We handover each object from three handover locations, i.e., left, middle, and right, with respect to the robot base location. Then we tried to move the object from left to right, and from right to left, after the robot starts move. We conducted experiments 10 household objects, and report the success rate in Fig. 4 (b) with an example of Medicine Box on these 5 settings. Please refer to Table 7 in Appendix for detailed results. Overall, reactive handover with moving objects is more challenging than static handover. The robot sometimes cannot adjust its grasping after the object moves. In our setting, the camera is the right side. When the object is on the left side, i.e., further from the camera, we see performance drop due to point cloud sensing noises, especially for the right to left reactive handover. Additionally, the Azure Kinect only runs at 15fps, which also causes delays in reactive handover.
all the
User Study Results. As reported in Fig. 5 Top, our method achieves comparable performance against [47] which is specifically designed and tuned for the task of H2R handovers. Interestingly, our approach has higher success rate for some difficult objects, such as mug and pen, which demon- strates that our approach can not only generalize to moving objects, but also adapts well to unseen environments (non table-top settings) and views (external camera).
Subjective Evaluation. Fig. Bottom reports participants' agreement with statements from [47]. Overall, the participants agreed with the fluency and commented "[the robot] fluently moved toward the object" and "could adjust it's position when moved". They felt safe, saying "[I] never felt would overshoot or become aggressive". They also sensed the robot aware of their actions since "fit] adapted fast when moved my arm", and was aware of the difference of objects "because it used different grasps for different objects". One was "surprised it worked participants thought the robot motion was slow when it got close to the object. This is because the system evaluates the grasping score using the grasp evaluator in GraspNet [10]. Only when the score is higher than a pre-defined threshold, the robot closes its gripper.
very thin objects". Several
5 Conclusion
We introduce the goal-auxiliary DDPG algorithm for efficiently learning of 6D grasping control polices from point clouds. Our method uses demonstrations from an expert motion and grasp planner and utilizes grasping goal prediction as an auxiliary task to improve the performance of the actor and the critic. We demonstrate that our policy trained in simulation can be integrated grasping system and a human-to-robot handover system to improve grasping performance of unseen objects. For future work, we plan to investigate the sim-to-real gap in policy learning due to contact modeling in simulation and extend the method to 6D grasping in cluttered scenes.
tabletop 6D
8

---

References
[T] J. Redmon and A. Angelova. Real-time grasp detection using convolutional neural networks. In IEEE International Conference Robotics and Automation (ICRA), pages 1316-1322, 2015.
[2] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In IEEE International Conference Robotics and Automation (ICRA), 2016.
[3] Mahler, J. Liang, Niyaz, M. Laskey, Doan, X. Liu, A. Ojea, and Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries. In Robotics: Science and Systems (RSS), 2017.
[4] D. Kalashnikov, A. Irpan. P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. QT-Opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv: /806. /0293, 2018.
[5] D. Quillen, E. Jang, O. Nachum, C. Finn, J. Iharz, and S. Levine. Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off off-policy methods. In IEEE International Conference on Robotics and Automation (ICRA), pages 6284-6291, 2018.
[6] A. Miller and P. Allen. Graspit! A versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4): 10-122, 2004.
[7] C. Eppner, A. Mousavian, and D. Fox. A billion ways to grasp: An evaluation of grasp sampling schemes on a dense, physics-based grasp data set. In International Symposium on Robotics Research (ISRR), 2019.
[8] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research (IJRR), 14)1--11::44411444: 2017.
[9] X. Yan, J. Hsu, M. Khansari, Y. Bai, Pathak, Gupta, J. Davidson, and H. Lee. Learning 6- DOF grasping interaction via deep geometry-awand 3D representations. In IEEE International Conference on Robotics and Automation (ICRA), pages 1-9, 2018.
[10] A. Mousavian, C. Eppner, and D. Fox. 6-DOF GraspNet: Variational grasp generation for object manipulation. In International Conference on Computer Vision (ICCV), pages 2901- 2910, 2019.
[11] L. Wang, Y. Xiang, and D. Fox. Manipulation trajectory optimization with online grasp syn- thesis and selection, In Robotics: Science and Systems (RSS), 2020.
[12] A. X. Chang, T. Funkhouser, I.. Guibas, P. Hanrahan, Q. Huang. 7. Li, S. Savarese, M. Savva, Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An information-rich 3d model repository. arXiv:1512.03012 2015.
[13] Lillicrap, Hunt, A. Pritzel, Heess, Frez, Y. Tassa, Silver, and D. Wierstra. Con- tinuous control with deep reinforcement learning. arXiv: 1509.0297), 2015.
[14] D. Morrison, P. Corke, and J. Leitner. Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach. In Robotics: Science and Systems (RSS), 2018.
[15] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-DOF grasping for target-driven object manipulation in clutter. JEEE International Conference on Robotics and Automation (ICRA), 2020.
[16] S. Levine, Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research (IJRR), 37(4-5):421 436, 2018
[17] S. Iqbal, J. Tremblay, A. Campbell, K. Leung, T. To, J. Cheng, E. Leitch, D. McKay, and Birchfield. Toward sim-to-real directional semantic grasping. In IEEE International Con- ference Robotics and Automation (ICRA), 2020.

---

[18] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters (RA-L). 5(2):492 2019.
[19] L. Manuelli, W. Gao, P. Florence, and Tedrake. kPAM: Keypoint affordances for category- level robotic manipulation. arXiv:/903.0668: 2019.
[20] X. Yan, M. Khansari, J. Hsu, Y. Gong, Y. Bai, S. Pirk, and H. Lee. Data-efficient learning for sim-to-real robotic grasping using deep point cloud prediction networks. arXiv: /906. 909.98 2019.
[21] I. Akinola, J. Varley, and D. Kalashnikov. Learning precise 3D manipulation from multiple uncalibrated cameras. vrXiv:2002.09107, 2020.
[22] Song, A. Zeng, Lee, and Funkhouser. Grasping in the wild: Learning 6DoF closed-loop grasping from low-cost demonstrations. Robotics and Automation Letters (RA-L), 2020.
[23] S. Young. D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. arXiv:2008. 04899, 2020.
[24] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and Peters. An algorithmic perspective on imitation learning, arXiv: Ntt::///1.00 1.06711 2018.
[25] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. 17X9v 10087, 2017.
[26] Y. Zhu, Z. Wang. J. Merel, A. Rusu, Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Had- sell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. arXiv: 000.000020000 2018.
[27] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics 2017.
problems with sparse rewards. arXiv:
(28) A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation (ICRA), pages 6292-62999 2018.
[29] L. P. Kaelbling. Learning to achieve goals. In International Joint Conferences Artificial Intelligence (IJCAI), pages 1094-1099, 1993.
[30] M. Andrychowicz, Wolski, A. Ray. J. Schneider, R. Fong, P. Welinder, McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[31] Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 15298-15309, 2019.
[32] D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devine, B. Eysenbach, and S. Levine Learning to reach goals without reinforcement learning. arXiv: /9/2.06888, 2019.
[33] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference on Robotics and Automation (ICRA), pages 512-519, 2016.
[34] M. Jaderberg, V. Mnih, W. M. Czarnecki, Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv:161 10l1::9917 2016.
[35] Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and Abbeel. Deep im- itation learning for complex manipulation tasks from virtual reality teleoperation. In IEEE International Conference on Robotics and Automation (ICRA), pages 1-8, 2018.
[36] A. Srinivas, M. Laskin, and P. Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. 2xXv:20004.04166 2020.
10

---

[37] I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y. Tassa, Erez, and M. Riedmiller. Data-efficient deep reinforcement learning for dexterous manipu- lation, arXiv: 1704.03073 2017.
[38] Y. Li. G. Wang. X. Ji, Y. Xiang, and D. Fox. DeepIM: Deep iterative matching for 6d pose estimation. In European Conference on Computer Vision (ECCV), 2018.
[39] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
[40] C. R. Qi L. Yi, H. Su, and L. J. Guibas. PointNet++: Deep hierarchical feature learning on point sets metric space. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[41] S. Fujimoto, H. Van Hoof, and D). Meger. Addressing function approximation error in actor- critic methods. Nttv:8802.94777 2018.
[42] B. Calli, A. Walsman, A.
Singh, S. Srinivasa, P. Abbeel, and A. Dollar. Benchmarking in manipulation research: The YCB object and model set and benchmarking protocols. IEEE Robotics and Automation Magazine, 22(3):36-52, 2015.
[43] E. Coumans and Y. Bai. PyBullet, a python module for physics simulation for games, robotics and machine learning. http: http:://yybbllttte org, 2016-2019.
[44] C. Eppner, A. Mousavian, and D. Fox. ACRONYM: A large-scale grasp dataset based on simulation. wrXiv:2011.09584 2020.
[45] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[46] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning RGB-D feature embeddings for unseen object instance segmentation. In Conference on Robot Learning (CoRL), 2020.
[47] W. Yang, C. Paxton, A. Mousavian, Y.-W. Chao, M. Cakmak, and D. Fox. Reactive human-to- robot handovers of arbitrary objects. arXiv:201 0.0220/1100900 1.08961, 2020.
[48] J. Tobin, L. Biewald, R. Duan, M. Andrychowicz, A. Handa, V. Kumar, B. McGrew, A. Ray, Schneider, P. Welinder, W. Zaremba, and Abbeel. Domain randomization and generative models for robotic grasping. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3482-3489. IEEE, 2018
[49] D. Kingma and Ba. Adam: method for stochastic optimization. ar.iv::142.00000 2014.


Reviews:
[{"review_id": "o4Vhlw76O6U", "reviewer": "Reviewer_nhX8", "summary": "The paper presents a system for learning-based 6D Grasping. The method combines imitation and reinforcement learning: use imitation learning to overcome the initial exploration challenges, and use reinforcement learning to further improve the policy performance, especially on shapes that are hard to create expert demonstration for. The method is evaluated on grasping a variety of objects in both simulated and real environments, as well as a human-robot handover task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the practicality of the system: it makes intuitive sense to combine imitation learning and reinforcement learning for challenging problems like learning reactive grasping. The fact that the system operates on partial point cloud observation makes sim-to-real transfer feasible. I also appreciate the final human-robot handover experiment that shows why reactive grasping is important. \n\nHowever, I found it difficult to distill the thesis of the paper: what should the readers learn from the paper beyond knowing that the system “works”? From a technical novelty perspective, the key contributions seem to be spread across a few points: 1) combining imitation learning and reinforcement 2) learning from point cloud data for 6 DoF grasping and 3) the goal auxiliary loss. \n\nLike mentioned in the paper, combining imitation and reinforcement learning is not a new idea in visual-motor learning, and the idea of learning from raw point cloud for grasping has been explored by a few prior works too. The goal auxiliary losses seems new, but I’m not entirely sure what’s the motivation behind this change. What challenges are these loss functions designed to address? The only place that discusses this point is 149-151: “We introduce two auxiliary tasks for the actor-critic to predict goals, which help to learn the policy and the Q-function by adding goal supervision”, but it doesn’t really motivate *why* these losses are added. I wish the authors could shed more light on the key takeaways of the paper in the next iteration.\n\nOn the other hand, the paper does fulfill its high-level goal of developing a learning-based reactive grasping system. And the system seems to fit especially well for human-robot handover tasks judging by the experiment evaluation. However, I had a similar struggle to understand the key differences between the proposed system and the prior works. The only relevant point made in the paper is that “it performs on-par with the prior work”. What are the assumptions relaxed, if any, compared to the prior work that made this result significant? I’d like to see a more in depth discussion on the result. \n\nMinor: \nWhat’s the input space for real-world experiments? I don’t see it mentioned anywhere in 4.2 and 4.3.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Like mentioned in the comment, I like the practicality of the system and appreciate the evaluation. But I struggled to distill the key lessons from this paper. Hence before seeing more comments from the author, I do not recommend accepting the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See main comments.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "hDYwzL-HKNC", "reviewer": "Reviewer_GYJh", "summary": "The paper proposes a method for learning closed-loop 6D robot grasping policies of singulated objects, using segmented aggregated point clouds as input. It proposes a combination of behavior cloning from expert demonstrations and reinforcement learning based on an actor-critic approach. Furthermore, it introduces an auxiliary task for supervised training of the actor and critic networks, which consist of predicting the final grasping pose of the end-effector. During the behavior cloning phase, additional demonstrations are gathered following the DAGGER scheme, where the expert provides the labels for actions and goals. During the RL phase, goal labels are generated either from a set of pre-labeled goals (in case of known objects with annotations) or hind-sight goals are used (in case of unknown objects) to add a goal label to the data tuples in the replay buffer. \nExtensive ablation studies are conducted in simulation on the algorithmic choices and on different input representations underlining their benefit. The paper also show-cases the method on two real world tasks: Table-top grasping and human-to-robot handover.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper contains extensive ablation studies on the different parts of the method, which do not only give insight into the method proposed in the paper, but can also give useful hints to other research in the field.\n- The combination of imitation learning and reinforcement learning is promising for an increased sample efficiency of robotics manipulation tasks, in particular in the proposed combination with DAGGER (-like) training and hind-sight goal labelling. \n- The idea of using goal prediction as auxiliary task is interesting and seems to lead to surprising performance gains as compared to goal conditioning. \n\nWeaknesses:\n- Many important technical points were transferred to the appendix, without reference, but are actually crucial for an understanding of the proposed method,e.g. point cloud aggregation, the training of the goal prediction network. There should at least be a to the relevant section in the appendix, but a discussion in the main paper would be better. \n- There seem to be missing some numbers in the tables, which are referenced in the text. In particular table 2 and table 3.\n- The benefit of 6D grasping does not become obvious to me from the experiments on grasping. A comparison of a method capable of top-down grasps only would be useful. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I liked reading the paper and found  the thorough ablation studies very insightful. In particular the comparison of different input modalities is interesting and can provide very useful insights to the community. The proposed methods for to both closed loop grasping and 6D grasping constitute relevant contribution to research in that field. \nI therefore recommend the acceptance of the paper. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- The method for aggregating point clouds is not mentioned in the methodic section of the main paper. The first place where the word aggregation appears (but not explained) is in the section 'Ablation on Other Design Choices'. Since it is however an integral part of the method proposed in the paper, it should be mentioned earlier, at least with a reference to the appendix.\n\n- In lines 212 - 219: The paper states: 'From table 2 we can see that ... iv) Comparing to the supervised BC training, our GA-DDPG algorithm further improves the policy, ...' There seems to be no mentioning of BC in that table. Can the authors clarify, what they refer to?\n\n- In line the authors state 214-215: ii) 'The goal-conditioned policies perform worse than the goal-auxiliary counterparts. When the predicted goal is not accurate, it affects the grasping success.' Did the authors investigate that further, or is it just a hypothesis?  \n\n- In line 226 the paper states: 'We see that the success rate is improved from 88.2% to 93.5% after fine-tuning.'  Where can we see that? It would be nice to add this result to table 3. \n\n-  A discussion of the sensitivity of the proposed method wrt. hyperparameters (e.g. the weighting lambda of the actor rewards ) would be interesting "}}, {"review_id": "T8UVEqmFuD9", "reviewer": "Reviewer_7sEr", "summary": "This work presents an approach for learning closed-look policies for 6D grasping arbitrary objects. Given an RGBD input, the following is done: \n(1) The object is  segmented from the environment using the RGB input.\n(2) The object's segmentation mask is used to get the point cloud corresponding to the object (assuming a calibrated camera, so that the depth image can be used to infer this point cloud).\n(3) The point cloud is used an input to two PointNet++ networks, which act as the actor and critic.\n(4) The action from the actor (a 6D offset from the current gripper pose) from the actor is executed.\n(5) Once a goal grip position is reached, an attempt at grasping the object is made.\n\nThe key novel idea in the above is to add an auxiliary objective to both the actor and critic, which is to predict the final grasp pose. This can be trained via supervised learning, making the task of learning good actions and value estimates easier, since these are informed by what a good final grasp pose is.\n\nLearning is done in via a combination of behavioral cloning and deep RL in two steps:\n(1) A dataset of expert trajectories is generated by a motion planner in simulation, with grasp planning utilized to select the end positions for the motion planner. DAGGER-based behavior cloning is further used to get a more robust set of data.\n(2) The above dataset is included in the replay buffer and RL is done with additional losses for BC and the auxiliary objective. A sparse reward of successful grasping is used and TD3 is used as the RL algorithm. \n\nDetailed evaluation and ablation experiments are done in simulation, demonstrating high (~90%) success rates and the need for all the aspects of this approach convincingly. Additional evaluation is done in a real world setting, in which a lower success rate (~70) is seen due to the sim-to-real gap. It is also demonstrated that this approach can be used to implement human-to-robot handovers of objects.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Closed loop 6D grasping of arbitrary objects given raw sensor information is an important problem in robotics that is far from solved\n* The paper is well written, with no significant ambiguities\n* The problem formulation for training makes no assumptions about prior knowledge about the environment, and while some knowledge of object models is used the RL training can also proceed without that\n* Using point clouds as inputs is a good idea, as it allows for zero-shot transfer from simulation to real with a lesser discrepancy in sensing than would be the case with RGB inputs\n* The concept of the auxiliary loss is novel and fitting for this problem\n* The evaluation is very well done, with an extremely well done ablation study and impressive results. The additional demonstration of object hand-over results also further demonstrates this approach's value.\n* The supplementary material contains additional valuable information, in particular more detail concerning the hindsight goal training and details about the training process and informative learning curves\n\nWeaknesses:\n* The approach is primarily composed of ideas introduced in prior works. The auxiliary loss being the main novelty and is interesting, but has limited applicability in other settings.\n* The method relies on having a robust segmentation algorithm for arbitrary objects\n* The method requires learning in simulation, as it needs a large number of interaction steps to converge\n* The performance suffers in the real world due to the sim-to-real gap due to discrepancies in physics simulation (in particular contact modeling)\n* Also the approach compatible with grasping in clutter, this is not attempted in this work.\n* Fine-tuning in the real world is suggested as a possible solution to the sim-to-real gap, but is not attempted in this work\n\nSeveral of the weaknesses indicate directions for future work, rather than significant issues with this paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is very well done and presents a well thought out training approach for an important and as of yet rarely addressed problem. The main weakness is having few original ideas, but the strength of how different ideas are combined and the value of the auxilliary loss make this a relatively small issue.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "\"Expert Init\" is listed in Table 3, but I can't find it explained prior to this.\nMore details regarding failure modes in both simulation and real would be useful, as well as more discussion of what the sim-to-real gap is in this case."}}, {"review_id": "3mc6egtA8RB", "reviewer": "Reviewer_i4Zo", "summary": "The paper proposes a method for closed-loop 6D robotic grasping which learns a policy that maps a state corresponding to a point-cloud of the scene to an action that corresponds to 6D transformation of the gripper pose. The learned model uses an actor-critic schema. The actor is trained using Behavior Cloning approximating the trajectories generated by a motion planner expert, and the critic is trained to predict the expected return of the trajectory that correspond to the grasp probability. Both modules also predicts a goal pose as an auxiliary task. The method is evaluated on Shapenet and YCB object in simulation and in real world, demonstrating nice results in static setting (grasping from table top) as well as dynamic setting (human to robot handover). The use of different modalities from inference as well as different learning features are also ablated in the experimental section.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method in general makes sense, and the performance seems to be quite good. The results demonstrate the value of the method mostly in dynamic mode where closed-loop is actually required, and to some extent also in static scenes there is some benefit compare to an open-loop prior solution.\nI believe that a comparison to other closed-loop 6D methods such as \"Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations\" could improve the quality of the paper.\nIn addition, a few references (e.g. OMG, Graspnet) are used either as sub-modules of the proposed approach or for comparison, which has some commonality but also some difference. It could help the reader to have some short description of each reference to understand the deviations. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall the paper is quite nice.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Nothing beyond what was mentioned above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have agreed that the paper has been addressing an important problem in robotic, i.e. closed loop 6D grasping of arbitrary objects. The combination of imitation learning and reinforcement learning is sensible, and the concept of the auxiliary loss is novel and fitting for this problem. However there is a suggestion to elaborate with motivations why these losses are added. In overall, though the idea is incremental, i.e. the combination of imitation learning and reinforcement learning, the evaluation is very well done, with a well done ablation study and impressive results. There are concerns that the key differences between the proposed system and the prior works  and under what assumptions (e.g. robust segmentation algorithm, grasp a single object) are unclear. There are also suggestions that a comparison of a method capable of top-down grasps or other closed-loop grasping methods should be included. 

During the rebuttal phase, the authors have addressed most concerns by the reviewers. Though there are still open issues, e.g. comparisons to existing 6d close-loop and open-loop table-top grasping methods, the authors have demonstrated the efficiency and practicality aspects of their approach through many interesting evaluations. 
Decision: Accept (Poster)