Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Authors: Lirui Wang
Yu Xiang
Dieter Fox
Venue:  (2021)
Keywords: Robotics
Reinforcement Learning
Learning from Demonstration

Abstract: 6D robotic grasping beyond top-down bin-picking scenarios is a challenging task. Previous solutions based on 6D grasp synthesis with robot motion planning usually operate in an open-loop setting without considering perception feedback and dynamics and contacts of objects, which makes them sensitive to grasp synthesis errors. In this work, we propose a novel method for learning closed-loop control policies for 6D robotic grasping using point clouds from an egocentric camera. We combine imitation learning and reinforcement learning in order to grasp unseen objects and handle the continuous 6D action space, where expert demonstrations are obtained from a joint motion and grasp planner. We introduce a goal-auxiliary actor-critic algorithm, which uses grasping goal prediction as an auxiliary task to facilitate policy learning. The supervision on grasping goals can be obtained from the expert planner for known objects or from hindsight goals for unknown objects. Overall, our learned closed-loop policy achieves over $90\%$ success rates on grasping various ShapeNet objects and YCB objects in simulation. The policy also transfers well to the real world with only one failure among grasping of ten different unseen objects in the presence of perception noises.

Under review as a conference paper at ICLR 2021
ACTOR-CRITIC FOR 6D ROBOTIC GRASPING WITH POINT CLOUDS
Anonymous authors Paper under double-blind review
ABSTRACT
6D robotic grasping beyond top-down bin-picking scenarios is challenging task. Previous solutions based on 6D grasp synthesis with robot motion planning usu- ally operate in an open-loop setting without considering perception feedback and dynamics and contacts of objects, which makes them sensitive to grasp synthesis errors. In this work, we propose novel method for learning closed-loop control policies for 6D robotic grasping using point clouds from an egocentric camera. We combine imitation learning and reinforcement learning in order grasp unseen objects and handle the continuous 6D action space, where expert demonstrations are obtained from a joint motion and grasp planner. We introduce goal-auxiliary actor-critic algorithm, which uses grasping goal prediction as an auxiliary task to facilitate policy learning. The supervision on grasping goals can be obtained from the expert planner for known objects or from hindsight goals for unknown objects. Overall, our learned closed-loop policy achieves over 90% success rates grasping various ShapeNet objects and YCB objects in simulation. The policy also transfers well to the real world with only one failure among grasping of ten different unseen objects in the presence of perception noises
INTRODUCTION
Robotic grasping of arbitrary objects challenging task. A robot needs to deal with objects has never seen before, and generates motion trajectory to grasp object. Due to the complexity of the problem, majority works in the literature focus on bin-picking tasks, where top-down grasping is sufficient pick up object. Both grasp detection approaches (Redmon & Angelova 2015 Pinto & Gupta 2016 Mahler et al. 2017) and reinforcement learning-bace based methods (Kalashnikov et al.] 2018 Quillen al. 2018) 8) are introduced tackle the top-down grasping problem. However, is difficult for these methods to grasp objects in environments where 6D grasping is necessary, i.e., 3D translation and 3D rotation of the robot gripper, such as cereal box on a tabletop or in a cabinet.
While 6D grasp synthesis has been studied using 3D models of objects Miller & Allen 2004 Eppner al. (219)) and partial observations (ten Pas et al. 2017 Yan al. 2018 Mousavian al. 2019). these methods only generate 6D grasp poses of the robot grippe for an object. instead of generating trajectory of the gripper pose to reach and grasp the object. As a result, motion planner is needed to plan the grasping motion according to the grasp poses. Usually, the planned trajectory executed in an open-loop fashion since re-planning is expensive, and perception feedback during grasping well as dynamics and contacts of the object are often ignored, which makes the grasping sensitive to grasp synthesis errors.
In this work, to overcome the limitations in the paradigm of 6D grasp synthesis followed by robot motion planning, we introduce a novel method for learning closed-loop 6D grasping polices from partially- oobserely-oberreed point clouds of objects. Our policy directly outputs the control action of the robot gripper, which is the relative 6D pose transformation of the gripper. For the state representa- tion, we adopt an egocentric view with wrist camera mounted on the robot gripper, which avoids self-occlusion of the robot arm during grasping compared to using an external static camera. Addi- tionally, aggregate point clouds of the object from previous time steps to avoid ambiguities in the current view and encode the history observations. Our point cloud representation provides richer 3D information for 6D grasping and generalizes better to different objects compared RGB images.
Videos and code available at wtthtts::/sstes..eee

---

Under review as a conference paper at ICLR 2021
Train on Known Objects
Finetune on Unknown Objects
Real World Execution
Unseon
Our method learns the 6D grasping policy with goal-auxiliary task using an egocentric camera (goals denoted as green forks). We combine imitation learning with a planner (green tra- jectory) and reinforcement learning for known objects. When finetuning the policy with unknown objects, we use hindsight goals from successful episodes (red trajectory) as supervision. The policy learned in simulation can be successfully applied to the real world for grasping unseen objects.
Figure 1:
We propose to combine Imitation Learning (II.) and Reinforcement Learning (RL) in learning the 6D grasping policy. Since RL requires exploration of the state space, the chance of lifting an object is very rare. Moreover, the target object can easily fall down with bad contact and the ego-view camera may lose the object during grasping. Different from previous works (Song al. 2020 Young et al. (220)) that specifically design a robot gripper and collect human demonstrations using it, obtain demonstrations using a joint motion and grasp planner (Wang al. (020) in simulation. Consequently, we can efficiently obtain a large number of 6D grasping trajectories using ShapeNet objects (Chang et al. 2015) with the planner. Then we learn the grasping policy based the Deep Deterministic Policy Gradient (DDPG) algorithm (Lillicrap et al.] 2015), which is an actor- critic algorithm in RL that can utilize off-policy data from demonstrations. More importantly, we introduce a goal prediction auxiliary task to improve the policy learning, where the actor and the critic in DDPG are trained to predict the final 6D grasping pose of the robot gripper as well. The supervision on goal prediction comes from the expert planner for objects with known 3D shape and pose. For unknown objects without 3D models available, we can still obtain the grasping goals from successful grasping rollouts of the policy, i.e., hindsight goals. This property enables our learned policy to be finetuned on unknown objects, which is critical for continual learning in the real world. Figure I Iillustrates our setting for learning the 6D grasp polices.
We conduct thorough analyses and evaluations of our method for 6D grasping. We show that our learned policy successfully grasp unseen ShapeNet objects and unseen YCB objects (Calli et al. 2013) in simulation, and finetuning the policy with hindsight goals for unknown YCB objects im- proves the grasping success rate. In the real world, we utilized recent unseen object instance segmentation method (Xiang et al., (0200) to segment the point cloud of an target object, and then applied GA-DDPG for grasping. The learned policy is able to successfully grasp YCB objects in the real world. It only failed one among ten grasping of different unseen objects with perception noises.
Overall, our contributions are three-folds: 1) We propose to use point cloud as our state representa- tion and use demonstrations from a joint motion and grasp planner to learn closed-loop 6D grasping policies. 2) We introduce the Goal-Auxiliary DDPG (GA-DDPG) algorithm for joint imitation and reinforcement learning using goals. 3) We propose to use hindsight goals for finetuning a pre-trained policy on unknown objects without goal supervision.
2 RELATED WORK
Combining Imitation Learning and Reinforcement Learning, For nigh-dimensional continuous state-action space with sparse rewards and complex dynamics as in most real-world robotic set- tings, model-free RL provides a data-driven approach to solve the task (Kalashnikov et al. 2018 Quillen et al.) 2018). but it requires a large number of interactions even with full-state information. Therefore, many works have proposed to combine imitation learning (Osa et al. 2018 in RL. For example, Rajeswaran et al.) (0017) augment policy gradient update with demonstration data to cir- cumvent reward shaping and the exploration challenge. Zhu et al. (2018) use inverse RL to learn dexterous manipulation tasks with few human demonstrations in simulation. The closest related works to ours are Vecerik et al. (2017); Nair et al. (2018a) that utilize demonstration data with off- policy RI.. Despite the focus different tasks, the main difference is that our demonstrations
2

---

Under review as a conference paper at ICLR 2021
from an expert planner instead of human demonstrators. Therefore, we can obtain large number of demonstrations and query the expert planner during training to provide supervision on the on on-policy data. Moreover, compared with human demonstrators, expert planners are more likely to generate trajectories that are suitable for robots to execute and have consistency across the state space.
Goals and Auxiliary Tasks in Policy Learning. Goals are often used extra signals to guide policy learning (Kaelbling] 1993). For goal-conditioned policies, (Andycchowizz et al. (2017): Ding et al. 2019); Ghosh al. (019)) make the observation that every trajectory is a successful demon- stration of the goal state that it reaches, thereby these methods re-label goals in rollout trajectories for effective learning. However, goals need to be provided in testing, which can be an optimistic assumption for complex tasks. Although many works have been proposed to generate goals or sub- goals with deep neural networks (Nair et al. 2018b] Florensa et al. 2018 Sharma et al. 2019), in 6D grasping, estimating the grapsing goals for unseen objects is challenging task. On the other hand, auxiliary tasks have been employed to improve RI. as well. For instance, state reconstruction tasks (Finn et al. 2016), auxiliary control and reward tasks (Jaderberg al. 2016), pose estimation loss (Zhang et al. 2018) and contrastive loss (Srinivas et al. (2020) have been used to improve rep- resentation learning. In work, we utilize the grasping goal prediction natural auxiliary task that requires the policy to predict how to grasp the target object.
Vision-based Robotic Grasping. Grasp synthesis can be used in planning and control pipeline for robotic grasping (Pinto & Gupta 2016 Mahler et al. (017; Morrison et al. 2018 Murali et al. 2020). Alternatively, end-to-end policy learning methods (Levine al.) [2018] Kalashnikov al.,
2018 Iqbal et al., (2020) make use of large-scale data to learn closed-loop vision-based grasping. Although RGB images are widely used as the state representation, it requires the policy infer 3D information from 2D images. Recently, depth and segmentation masks (Mahler et al.) (017), shape completion (Yan al., 2018), deep visual features (Florence al. 2019), keypoints (Manuelli al, 2019). point point clouds (an et al., (019)) multiple cameras ( (Akinola et al. (020), and egocentric views (Song et al., 2020 Young et al. 2020) have been considered to improve the state representation. Motivated by these methods, we utilize the aggregated point clouds of an object from an egocentric camera as our state representation. Moreover, we explicitly learn a policy for the large action space in 6D grasping instead of using value-based representations as in most top-down grasping methods.
3 BACKGROUND
Markov Decision Process. A Markov Decision Process (MDP) is defined using the tuple: M {S, P. po: 7). S, A. and O represent the state, action, and observation space. R SxA4 - R is the reward function. P Axx.A -> S is the transition dynamics. Po is the probability distribution over initial states and γ = [0, 1) is a discount factor. Let π. S + A be policy which maps states actions. In the partially observable case, at each time t, the policy maps a partial observation o, of the environment to an action ag 7 (o:). Our goal is to learn a policy that maximizes the expected cumulative rewards E Er 7 rel, where r' the reward at time t. The Q-function of the policy for a state-action pair is (s,,a) R(s,, a) E s², ? //rr/80 = s'), where s' represents the next state of taking action a at state s according to the transition dynamics.
Deep Deterministic Policy Gradient. Deep Deterministic Policy Gradient (DDPG) (Lillicrap al., 2015) is an actor-critic algorithm that uses off-policy data and temporal difference learning, and has successful applications in continuous control (Popov al. 2017 Vecerik et al. 2017). Specifically, the actor in DDPG learns the policy (s), while the critic approximates the Q-function Qo(s, a), where 0 and é denote the parameters of the actor and the critic, respectively.
replay buffer of transitions D = {(s, a, r, }²)} is maintained during training, and examples sampled from it are used to optimize the actor-critic. DDPG minimizes the following Bellman error with respect to Φ;
[(d)= E (s.a.)))>) [ (Qo(s. a) (r + 7000s's (0((()=mmss))))))))))))))))))))) =o(s')))
Then the deterministic policy To is trained to maximize the learned Q-function with objective max; BewD(Qs (s)))) which resembles a policy evaluation and improvement schedule.
4 LEARNING 6D GRASPING POLICIES
We consider the task of grasping an arbitrary object in a closed-loop manner, where partial observa- tions come from wrist camera mounted on the end-effector of a robot. Our goal is to learn 61)
3

---

Under review as a conference paper at ICLR 2021
Current
Observation
Aggregated Observation
Figure 2: Illustration of our point aggregation procedure. We show the RGB, depth, foreground mask and point cloud at time and the aggregated point cloud up to time
grasping policy т St - az that maps the state s; at time to an action To simplify the setting, still use s to represent the imput of the policy even though observations are used. At time the action is parametrized by the relative 3D translation and the 3D rotation of the robot end-effector. Therefore, our learned visuomotor policy represents an operational space controller for 6D grasping.
4.1 6D GRASPING FROM POINT CLOUDS
We utilize 3D point clouds of objects to represent the states, which can be computed using depth images and foreground masks of the objects. better utilize the 3D information and resolve ambi- guities of the current view due to local geometry, we propose to aggregate point clouds as the gripper moves in time for our state representation. Figure illustrates our point aggregation procedure.
In order to aggregate 3D points from different time steps, we need to transform them into com- mon coordinate system. We choose the coordinate of the robot base to aggregate the 3D points. Specifically, at time let X²" IR denote the 3D point cloud of the target object from the wrist camera, where n, represents the number of points at time t. Since the camera is attached to the robot hand, first transform X to the end-effector frame using the camera extrinsics denoted Using forward kinematics, we can compute the transformation of the end-effector in the robot base as Te E SE(3). Then state representation at time t is 8, Te( U T (X:)), i.e., aggregating the point clouds in the robot base up to time and then transforming the points to the end-effector frame at time t. Using 3D points suffers less from the sim-to-real gap.
X,-
transform the point cloud to the robot base by T, "(x,). Finally, our
4.2 DEMONSTRATIONS FROM A JOINT MOTION AND GRASP PLANNER
To obtain demonstrations for 6D grasping, we utilize the recent Optimization- based Motion and Grasp Planner (OMG Planner) (Wang ct al. 2020) as our expert. Given a planning scene and a set of pre-defined grasps 2019). the OMG Planner plans trajectory of the robot to grasp the object. Depending on the initial configuration of the robot, the OMG Planner selects different grasps the goal. This property enables us to learn a policy that grasps objects in different ways according to the initial pose of the robot gripper, which helps alleviate the multi-modality problem in 6D) grasping.
target object from grasp planner (Miller & Allen 2004 Eppner al.
Let { = (To TI....,TTy Tr) be a trajectory of the robot gripper pose generated from the OMG planner grasp an object, where T, SE(3) is the gripper pose in the robot base frame time t. Then the expert action at time can be computed aj Tr+ 1 T, which is the relative transformation of the gripper between / and ++ 1. Furthermore, Tr from the OMG Planner is successful grasp pose of the object. We define the expert goal at time as 9f T - which is the relative transformation between the current gripper pose and the final grasp pose. Finally, the state can be obtained by computing the aggregated point cloud as in Section 4.1 In this way, we construct an offline dataset of demonstrations from the OMG Planner as Dolline
(St. ai. {()). where s is the next state.
4.3 BEHAVIOR CLONING AND DAGGER
Using the demonstration dataset D Dottines we can already train a policy network Tp by applying Be- havior Cloning (BC) to predict an action from state. Different loss functions on STE(3 can be used.
4

---

Under review as a conference paper at ICLR 2021
goal prediction
512
1024
PointNet++
actor
point
1024
PointNet++
critic
translation
goal prediction
O Value: 0.81
Figure 3: Our network architecture with the PointNet++ network eQi al. 2017 for feature extrac- tion. Both the actor and the critic are regularized to predict grasping goals as auxiliary tasks. Note that we use quaternions in grasping goal prediction and Euler angles for the action.
In partieular, adopt the point matching loss function to jointly optimize translation and rotation:
Loose (Ti, T2)
Σ x
[TT(2) - T2(x)l11,
X
where T1, T2 E SE((), Xg denotes a set of pre-defined 3D points on the robot gripper, and L1 norm is used. Let a (T(s) ) be an action predicted from the policy network and a* = x (s) be an action from the expert, the behavior cloning loss is Lac(a", a²) Lrosu (a" aa))
We apply DAGGER (Ross et al., 2011) to augment supervisions for rollouts from the learner. Given a sampled initial state so, the current policy wy can roll out a trajectory states Sp, ..., ST and actions 40 or with end-effector poses .o., Tr. The expert then treats each state as the initial state and generates a trajectory to grasp the object E(+t) = (Tr. Te+1, Tr). We use the first action from Te+1 T, to correct deviation the learner from {(*t). A corresponding expert goal 9," T, can also be obtained for state st. These action and goal
the expert as supervision az = x (*{))
labels are used construct a dataset D Ddugger (St, at ag gf s()}
4.4 GOAL-AUXILIARY DDPG
To handle contact-rich scenarios that are rare in the expert trajectories, further improve the policy learning by combining RL with expert demonstrations. In our RL setting, each episode ends when the agent attempts grasp reaches maximum horizon T. The sparse reward is indicator function given at the end the episode denoting if the target object is lifted or not.
Similar to DAGGER, do not have expert demonstration for the action ot and goal g," for a state 8, from the learner policy. Instead, first find the nearest goal from goal set & by gt arg wigginnee. |gg Tell where T; is the pose of the robot gripper at time t. Then the heuristics goal for S; is = ge T Note that the goal set G is only available if we have the 3D shape and pose of the target object. Finally, can construct an on-policy dataset for DDPG Datpy = [(8e, Gt. Gt. V2, {))}. where r is the reward at state 8; and sy the next state of 8; from the on-policy rollout. The replay buffer of DDPG training is D = Dottine U Daggee U Dapp. where we augment Dottine and Dangger with the sparse reward.
collect on-policy data for DDPG training. The only difference is that
We introduce auxil- iary tasks for the actor-critic to predict goals, which help to learn the policy and the Q-function by adding goal supervision. Given data sample (s, a, 9, r, s') from the replay buffer, the critic loss
Our network architecture for training the actor-critic is shown in Figure 3
Lo (s, a, r, g. g = 2/2 (s(,s)) (r aaa (s', Te (s') + } + LAux(g. g)
(3)
where 9 is the predicted goal, Q and Tp. are the target networks and is pre-defined clipped noise as in TD3 Fujimoto et al. (2018). The auxiliary loss is LAUX gd) Lpose(g, g.)
Given data sample (s, a", .)) from the replay buffer, the loss function for the actor defined as Tine (8, a" a 9999 ALno(a" ,a²) + (1 -) A)DPP a))) LAux(9, g²)
(4)
5

---

Under review as a conference paper at ICLR 2021
where a" and of are the action and the goal predicted from the actor, respectively. and À is a weight- ing hyper-parameeer to balance the losses from the expert and from the learned value function. The BC loss L (a", ad) LPoSE(a* a²) defined before, which prevents the learner to move too far away from the expert policy. The deterministic policy loss is defined as T-DDPG (8, a²) = - Q0((, a a²), and the auxiliary loss is LAUx(g- 9¹) = LgosE(g: g)). Note that the expert action a is only available if the data sample is from Dottine or Dangger. Otherwise, we do not include the BC loss in Eq. ( Even though our method can be used with any actor-critic algorithms, in practice, we choose TD3 for its improved performance over vanilla DDPG.
4.5 HINDSIGHT GOALS FOR FINETUNING ON UNKNOWN OBJECTS
If would like deploy the trained policy the real world, the agent needs to potentially deal with unknown objects and different dynamics. In this case, we cannot obtain neither expert demon- strations nor grasping goals from pre-defined goal set. We introduce using hindsight goals to finetune our policy on unknown objects for self-supervised continual learning.
Based on the current policy Tyg, each learner episode rolls a trajectory of state-actions with end- effector poses To, ..., Tr. If the grasp in the episode is successful, we know that Tr is a success grasping pose for the target object, which is also known as the hindsight goal. Then we can compute the grasp goal for state in the episode as Jt- ,, which is used to supervise the actor and the critic via the goal auxiliary tasks. By using hindsight goals, we construct a dataset using on on-policy rollouts on unknown objects Phindsight {se, ar 9t. re, s() where Jr is the hindsight goal. The new loss functions for the critic and the actor can be rewritten as
PT) = (o((s 1 (8, (5) + =)]] + LAUx(0,9)) rT (n, a" a² 9999 FT) ALBO(a", a²) - - A)LDDPG a²) + LAUX (0) IT. (5)
where rT E f0. 1} is the binary reward for the last state in the episode. Therefore, we only have the goal auxiliary losses for successful episodes during finetuming unknown objects. We finetune the pretrained actor-critic network on the dataset D = Domine Dhindiight. Note that we cannot collect extra data for D Dangger and Dudpy with unknown objects, but we can potentially use the ones that already collected. Our algorithm of using hindsight goals is described in Appendix Section A
5 EXPERIMENTS
We conduct experiments and present our findings to the following questions. (1) For BC with super- vised learning, can using point clouds achieve better generalization for grasping unseen objects than using images? (2) With RL, how can different strategies of using grasping goals affect the perfor- mance? (3) Can our method further improve on unknown objects using hindsight goals, even if the dynamics has changed? We also conduct ablation studies different components in our method, comparison with an open-loop baseline and real-world experiments for 6D grasping.
Task Environment. We experiment with the Franka Emika Panda arm, 7-DOF arm with parallel gripper. We use ShapeNet (Chang et al.] (015) and YCB objects (Calli et al.) 2015) as our object repository. A task scene is generated by dropping sampled target object with random pose table in the PyBullet Simulator (Coumans & Bail 2016-2019). The maximum horizon for the policy is T 30. An episode is terminated once grasp completed. The observation image size is 112 x 112 in PyBullet. More environment details can be found in Appendix Section B.
Training and Testing. We use approximately 1,500 ShapeNet objects from 169 different classes for training, where each object has 100 pre-computed grasps from [ppnnrr et al. (2019) for planning. For testing, we use selected YCB objects with 10 scenes per object and 138 holdout ShapeNet objects within 157 scenes. We train each model three times with different random seeds. We run each YCB scene times and each ShapeNet scene times and then compute the mean grasping success rate. More details on network and train-test can be found in Appendix Section B.2 and B.3
5.1 BEHAVIOR CLONING WITH EXPERT SUPERVISION
We first conduct experiments with BC to investigate the effect of using different state representa- tions. Table I presents the grasping success rates. When using images, there three variations:

---

Under review as a conference paper at ICLR 2021
Methed DAGGER
Test
Inpul
Ofline
Auxliaux
ShapeNet
YCB
KGB
0.436
0.452
RGB
0,521
0.523
RGBD
0.453
0.463
RGBD
0.604
0.671
0,600
RGBDM
0,483
0.504
RGBDM
0.674
0.667
RGBDM
/
0.752
0.726
Point
/
0.733
0.673
Point
0.723
0.727
0.772
Point
0.796
0.785
Table 1: Success rates of different models trained by behavior cloning with expert supervision.
0.8
CA-DEPO
0.4
BRAN
BC JOWTLAI OON.IE
NO. DAGIBI
PONT LONLNE
COAL
ROROM AUR DADDEN OLUME
RONO AIX DAO ONLIN
J0,80 DAGRED DNLINE
BE non DAGGER ONLINE
00100
100008 120000
100303 120000
Stop
Trainiing Stop
Figure 4: (a) Learning curves of different BC models. (b) Learning curves of different RL. models. The mean and the 1 standard deviation range are plotted for each model.
"RGB", "RGBD" and "RGBDM" indicating whether depth (D) or foreground mask (M) of the ob- ject are used or not. Depth and/or mask are concatenated with the RGB image, and ResNet (He) al. (016) is used for feature extraction. "Point" Table T iiniiaates the aggregated point cloud representation we propose. "Offline" means fixed-size offline dataset Domine is used for training which contains 50,000 data points from expert demonstrations. "Online" means the expert planner is running in parallel with training, which keeps adding new data to the dataset. "DAGGER" uses on-policy rollout data from the learner policy with expert supervision Dauggers and "Gxil-Auxiliary"" indicates whether add the grasping goal prediction task or not in BC training.
From Table [ we can see that: i) using the aggregated point clouds achieves better performance compared to using images of the current view. This indicates that the 3D features in object point clouds generalizes better for 6D grasping. ii) Adding depth or foreground mask to the image-based representation improves the performance. iii) "Online" is overall better than "Offline" by utilizing more data for training. iv) Both DAGGER and adding the auxiliary task improve grasping success rate. Figure aa) shows the learning curves of several BC models.
5.2 GOAL-AUGMENTED REINFORCEMENT LEARNING
We evaluate our goal-auxiliary DDPG algorithm for 6D grasping, where the dataset D Domine U Daagger U Dadgg is constructed for training. We also consider another common strategy of using goals for comparison: condicondi policies such as in An/Aycchoowiczz et al. (2017): Nair et al.
2018a): Ding et al. (019)) where a goal is concatenated with the state as the input for the network.
Policy
None
Goal-Auxiliary
Gioal-Cnddfioned
Both
Critic
ShapeNet
YCB
ShapeNet
YCB
ShapeNet
YCB
ShapeNet
YCB
Nune
0,806
0.710
0.773
0.728
0.686
0.607
0.767
0.766
Goal-Auxiliary
0.846
0.828
0.913
0.882
0.834
0.813
0.878
0.841
Coal-Conditioned
0.816
0.737
0.793
0.761
0.703
0.632
0.821
0.779
Both
0.868
0.825
0.879
0.831
0.845
0.817
0.869
0.823
Table 2: Evaluation on success rates of two strategies of using goals in RL with DDPG.

---

Under review as a conference paper at ICLR 2021
Test
No BC
Ottine
No DAGGER
Image
No Aggr.
Late Fusion
No Expert Init.
Loss
Final
ShapeNet
0.027
0.723
0,839
0.837
0.837
0.791
0.877
0.818
0.913
YCB
0.038
0.637
0.853
0.789
0.835
0.821
0.866
0.792
0.882
Table 3: Ablation studies on different components in our method with grasping success rates.
In this case, in order to use goals in testing, we need train a separate network to predict goals from states. T displays the evaluation results, where we test different combinations of adding the goal-auxiliary task or using the condiiioo input to the actor and the critic.
From Table 2 we can see that: i) using the goal-auxiliary loss for the critic significantly improves the performance since it regularizes the Q-learning. ii) The goal-conditioned policies perform worse than the -axillary counterparts. Mainly because predicting the 6D grasping goals itself is a challenging problem and treating grasping as reaching the target goal is simplistic. When the pre- dicted goal is not accurate, it affects the grasping success. We also observe some instability in goal-conditioned policy training. iii) Adding the goal-auxiliary task to both the actor and the critic achieves the best performance. iv) Comparing to the supervised BC training, our GA-DDPG al- gorithm further improves the policy, especially in those contact-rich scenarios. Imitation learning alone cannot handle these contacts with the target object before closing the fingers since these rarely seen in the expert demonstrations.
5.3 FINETUNING WITH HINDSIGHT GOALS
Our policy trained on ShapetNet objects achieves 91.3% and 88.2% success rate for grasping unseen ShapeNet and unseen YCB objects, respectively, as shown in Table 2 We can further improve the policy using RL with unseen objects. To resemble real-world RL, we assume that the 3D shape and pose of the YCB objects are not available. Therefore, there is no expert supervision on these objects. We finetune the policy using the dataset D = Dotfine U DDhindight with hindsight goals. We see that the success rate is improved from 88.2% to 93.5% after finetuning. We also experiment with changing the dynamics in simulation by decreasing the lateral frictions of the object, which makes object easier to slide during contacts. In this case, the success rate of the pre-trained policy reduces to 81.6%. After finetuning with hindsight goals, the policy achieves 88.5% success rate, which demonstrates the effectiveness of using hindsight goals for finetuning.
5.4 ABLATION STUDIES
We conduct ablation studies on several design choices in GA-DDPG as shown in Table 3 We can see that: RL without BC failed to learn useful policy due to the nigh-dimensional state-action space and sparse reward. ii) Both online interaction for adding data and DAGGER help. iii) Our aggregated point cloud representation is better than using "RGBDM" images or single-frame point clouds. iv) Early fusion of the state-action in the critic is better than late fusion, i.c., concatenating features of state and action. v) We also use expert-aided exploration by rolling out expert plans for few steps to initiate the start states during learner rollouts and denote it as "Expert Init.", which is beneficial. vi) The point matching loss defined in Eq. (2) is better than a L.2 loss on translation and rotation. Overall, we observe that these design choices both stabilize training and improve the performance. Figure shows the learning curves of several RL models. Ablation experiments on grasp prediction performance can be found in Appendix Section C
5.5 COMPARISON WITH AN OPEN-LOOP BASELINE
We compare our learned closed-loop policy with traditional manipulation pipeline consisting of grasp detection from partial point clouds, motion planning to generate trajectories to reach grasps, and open-loop execution. Specifically, we use a state-of-the-art grasp detection method, 6-DOF GraspNet (Mousavian et al. 2019), to generate a set of grasps. We then use the top 100 feasible grasps according to their grasp quality scores as the grasp set for the OMG planner. The planner selects grasp and generates a trajectory to reach the grasp. Finally, the generated trajectory is exe- cuted in PyBullet for grasping. This open-loop method achieves 75.7% success rate in the simulator on grasping YCB objects. Our learned closed-loop policy from GA-DDPG achieves 91.3% success rate without finetuning on YCB objects, which is significantly better than this open-loop baseline. Our finetuned policy matches the performance of the open-loop expert with perfect grasps around 94%. This experiment demonstrates that closed-loop 6D grasping is critical as open-loop methods are heavily affected by the grasp detection performance.
8

---

Under review as a conference paper at ICLR 2021
Object
Cracker
Sugar
Tomato Soup
Mustard
Meat Can
Bleach
Bowl
Mug
Foam Brick
Unseen
Mean
Fix
515
575
3/5
475
3/5
475
4/5
475
575
9/10
0.34
Varying
5/5
2/5
4/5
4/5
5/5
4/5
0.78
Table 4: Real-world grasping results. The numbers of successful grasps among trials are presented.
YCB
mustard bottle
foam brick
bleach cleanser
red bowl
cracker box
Unseen
spray
wood block
white mug
tape
small pillow
Unseen
T-shirt
marker
water bottle
plastic container
pump bottle
Failures
tomato can
sugar box
meat
plastic container tape dispenser
Figure 5: Success and failure of real-world 6D grasping using our policy trained in simulation.
5.6 REAL-WORLD EXPERIMENTS
Finally, we conduct real-world grasping experiments with GA-DDPG. We directly applied our pol- icy learned in PyBullet to the real world. To obtain the segmented point cloud of an object, utilized an unseen object instance segmentation method (Xiang et al. 2020). The policy outputs the relative 6D pose transformation of the robot end-effector each time step, which is used control the robot for grasping. For the YCB objects, the robot grasped each object 5 times in a setting with fixed robot initial pose and setting with varying robot initial poses. In addition, 10 unseen ob- jects with various shapes and materials are tested. Table 4 presents the numbers of successful grasps among trials. The learned policy achieves around 80% success rate. It only failed for one object among the ten unseen objects in both settings. Figure shows some successful grasping examples with different objects. The last row in Figure shows some examples of failure cases. We see two main categories of failures. The first one is due to inaccurate grasp poses from the policy. The sec- ond one due to slippery and mismatch physics between simulation and the real world, where the same hand pose can succeed in simulation but fail in the real world. Making the simulation better matching the real world or finetuning the policy in the real world would improve the policy.
CONCLUSION
We have proposed a novel method for efficiently learning 6D grasping control polices from point clouds. Our method combines demonstrations from an expert motion and grasp planner with RL to explore on-policy data. It learns a closed-loop control policy which takes into account dynamics and contacts with objects in grasping. We introduce the goal-auxiliary DDPG algorithm for policy learning, which uses the goal prediction as an auxiliary task to improve the performance of actor and critic. We demonstrate that our method trained on ShapeNet objects can successfully applied to grasping unseen objects. Furthermore, we show that our policy can be finetuned unknown objects using hindsight goals to achieve continual learning. We also demonstrated that our policy trained in simulation can be applied the real world. It achieves robustness and generalization across diverse objects and initial robot poses in the real world for grasping unseen objects. As for future work, we plan to extend method to 6D grasping in cluttered scenes.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Iretiayo Akinola, Jacoh Varley, and Dmitry Kalashnikov. Learning precise 3D manipulation from multiple uncalibrated cameras. arXiv preprint arXiv:2002.09107, 2020.
Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems (NeurIPS), pp- 5048-5058, 2017.
Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron Dollar. Benchmarking in manipulation research: The YCB object and model set and benchmarking pro-
tocols. IEEE Roboties and Automation Magazine, 22(3):36-52) 2015.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-riee 3d model repository. arXiv preprint arXiv: 2015.
Erwin Coumans and Yunfei Bai. PyBullet, python module for physics simulation for games, robotics and machine learning. http::///ppttt htttp:/pybbullt..co org 2016-2019.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems (NeurlPS), pp. 15298-15309, 2019.
Clemens Eppner, Arsalan Mousavian, and Dieter Fox. billion ways to grasp: An evaluation of grasp sampling schemes on dense, physics-based grasp data set. In International Symposium Robotics Research (ISRR), 2019.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519, 2016.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492 4994999999999 2019.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning (ICML), pp- 1515-1528, 2018
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 0802.8007277777 2018
Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devine, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. arXiv preprint arXiv:1912.06088 060888 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Shariq Iqbal, Jonathan Tremblay, Andy Campbell, Kirby Leung, Thang To, Jia Cheng, Erik Leitch, Duncan McKay, and Stan Birchfield. Toward sim-to-real directional semantic grasping. In IEEE International Conference Robotics and Automation (ICRA), pp. 7247-7253, 2020.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czamecki,, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.088 005397, 2016.
Leslie Pack Kaclbling. Learning to achieve goals. In International Joint Conferences Artificial Intelligence (ICAI), pp. 1094-1099, 1993.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806. 10293, 2018.
10

---

Under review as a conference paper at ICLR 2021
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal Robotics Research (IJRR), 37(4-5):421-435 2018.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint VrXi::/509.02997 2015.
Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Apari- cio Ojea, and Ken Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. Robotics: Science and Systems (RSS), 2017.
Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kPAM: Keypoint affordances for category-level robotic manipulation. arXiv preprint arXiv: 06684, 2019.
Andrew Miller and Peter Allen. Graspit! A versatile simulator for robotic grasping. IEFE Robotics & Automation Magazine, 1 11(4): 10-122, 2004.
Douglas Morrison, Peter Corke, and Jürgen Leitner. Closing the loop for robotic grasping: A real- time, generative grasp synthesis approach. In Robotics: Science and Systems (RSS), 2018.
Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-DOF GraspNet: Variational grasp gen- eration for object manipulation. In International Conference on Computer Vision (ICCV), pp- 2901-2910, 2019.
Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner, Chris Paxton, and Dieter Fox. 6- DOF grasping for target-driven object manipulation in clutter. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In IEEE International Con- ference on Robotics and Automation (ICRA), pp. 6292-6299, 2018a.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Sys- tems (NeurIPS), pp. 9191-9200, 20188 8b.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. urXiv preprint arXiv: 1,06711, 2018.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In IEEE International Conference on Robotics and Automation (ICRA). pp- 3406-3413, 2016.
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Ve- cerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep rein- forcement learning for dexterous manipulation. arXiv preprint arXiv:1704.03000 2017.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep hierarchical
feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems (NeurIPS). pp. 5099-5108, 2017.
Deirdre Quillen, Eric Jang, Ofir Nachum, Chelsea Finn, Julian Ibarz, and Sergey Levine. Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off-po methods. In IEEE International Conference on Roboties and Automation (ICRA), pp- 6284-6291, 2018.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 1eXXiv::00011000
2017.
Joseph Redmon and Anelia Angelova. Real-time grasp detection using convolutional neural net- works. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1316-1322, 2015.
11

---

Under review as a conference paper at ICLR 2021
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 627-635, 2011.
Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-person visual imitation learning via decoupled hierarchical controller. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Shuran Song. Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learning 6DoF closed-loop grasping from low-cost demonstrations. Robotics and Automation Letters, 2020.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.04136, 2020.
Andreas ten Pas, Mareus Gualtieri, Kate Saenko, and Robert Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research (IJRR), (6(13-14)114555-4773 2017.
Josh Tobin, Lukas Biewald, Rocky Duan, Marcin Andrychowicz, Ankur Handa, Vikash Kumar, Bob McGrew, Alex Ray, Jonas Schneider, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization and generative models for robotic grasping. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3482-3489, IEEE, 2018.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 1XXiv:707700817. 2017.
Lirui Wang, Yu Xiang, and Dieter Fox. Manipulation trajectory optimization with online grasp synthesis and selection. In Robotics: Science and Systems (RSS), 2020.
Yu Xiang, Christopher Xie, Arsalan Mousavian, and Dieter Fox. Learning RGB-D feature embed- dings for unseen object instance segmentation. In Conference on Robot Learning (CoRL), 2020.
Xinchen Yan, Jasmined Hsu, Mohammad Khansari, Yunfei Bai, Arkanath Pathak, Abhinav Gupta, James Davidson, and Honglak Lee. Learning 6-DOF grasping interaction via deep geometry- aware 3D representations. In JEEE International Conference on Robotics and Automation (ICRA), pp. 1-9, 2018.
Xinchen Yan, Mohi Khansari, Jasmine Hsu, Yuanzheng Gong, Yunfei Bai, Sören Pirk, and Honglak Lee. Data-efficient learning for sim-to-real robotic grasping using deep point cloud prediction networks. arXiv preprint 1UXiv:1906.08988 2019.
Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy. arXiv preprint arXiv:2008. 04499,999, 2020.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Picter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8, 2018.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, al. Reinforcement and imitation learning for diverse visuomotor skills. arxiv preprint 1rXiv:1802.095660 2018.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A HINDSIGHT GOAL AUXILIARY TRAINING ALGORITHM
Algorithm 1: Hindsight Goal Auxiliary Training (policy TT., dataset hiidsightt critic Q+)
for
i ...., N do Execute ωe in environment for an episode Log trajectory
(50, ao, To..., ST, (T, (T,
Compute Oe from and augment goals to the dataset D mindsight
Optimize the Qs. To with Eq. (5)
end
Compared with the common relabeling strategy for goal-conditioned policies (Andrychowicz al. 2017 Ding et al. 2019 Ghosh et al. 2019). we do not modify the reward of an episode. While using every hindsight goal might allow the robot to reach diverse goals, it hurts grasping performance use unsuccessful grasp poses goals. Moreover,
do not treat grasping as a goal-reaching task since solving the grasp detection problem also a challenge. will also ignore contacts during grasping and require perfect grasp pose input at inference time.
B IMPLEMENTATION DETAILS
B. TASK ENVIRONMENT DETAILS
The expert planner is tuned to have around 95% success rates on both YCB and ShapeNet objects. For point aggregation at time step I, sample m. points among the ni, observed points uniformly to limit the number of points. Note that a closer view of the object leads to a larger foreground mask. Therefore, n, increases as goes from 0 to T. In order to balance points sampled from different time steps, we aggregate observed point clouds in an exponential decaying schedule. Specifically, my [a ], where a 0.95. The network input contains 1024 sampled points from state st.
Compared to the YCB dataset which composes of objects that are manipulated in daily life, some objects in the ShapeNet repository such as Shelves, Desks, and Beds do not resemble real-world tabletop objects. Therefore we resize them to be graspable by the Panda gripper and use the same object physics for all the objects (Eppner al. 2019 Tobin et al. 2018). We randomize initial states Pn from an upper-sphere pose distribution facing toward the target object in distance from 0.2m to 0.5m. Compared to the restricted top-down grasping, the diversity of the initial arm configurations and object shapes and poses forces the policy to learn grasping in the 6D space. When RGB images are used, we apply domain randomization during training by randomly changing the textures.
For the task setup, we have also tried using RGB images from overhead static camera input and joint positions/velocities as the actions. For 6D grasping, we believe that using the ego motion from wrist camera generalizes better unseen objects in different backgrounds and suffers less from occlusions during grasping. Moreover, the aggregated point cloud suitable for sim-to-real transfer since it has no reliance color information and camera parameters, and often shows robustness to depth noises from sensors. It also allows motion based on history observations even when the current timestep has limited sensory feedback.
B.2 TRAINING AND TESTING DETAILS
The dataset Dofflne contains 50,000 data points in successful episodes from expert demonstrations and extra 120,000 failure data points in the case of offline RL in our ablation studies. These data contain few rollouts for each object to resemble the real-world setting where the demonstration data has a fixed size. Our online setting has around 3 million interaction steps. The replay buffer contains 2- 10 transitions for points and 2- 10 transitions for images due the CPU memory limit.
During each iteration of the DDPG training, we sample 14 parallel rollouts of the actor to collect experiences. We then perform 20 optimization steps mini-batch of size 250 which has 70% of
13

---

Under review as a conference paper at ICLR 2021
Figure 6: Complex contacts between the robot gripper with different objects during grasping.
the data from the on- on-policy buffer and the rest from expert buffer. We add decaying uniform noises from 3cm, degrees to 0.5em, degree for the exploration on translation and rotation, respectively. During expert rollouts, we randomly apply motion perturbations to increase the diverse state space explorations. The balancing ratio of the BC loss and the DDPG loss = 0.2. We use a discount factor of y 0.95 for the MDP. We adopt the TD3 algorithm (Fujimoto al. 2018), and the policy network is updated once every two Q-function updates. The target network of the actor and the first target network of the critic are updated every iteration with Polyak averaging of 0.999. The second target network used in Q-learning uses 3, 000 update step delay (Kalashnikov et al. 2018). In the pose loss LPOSE, the X, contains keypoints the finger tips, finger bases, and the palm base of the robot gripper.
B.3 NETWORK ARCHITECTURE DETAILS
The feature extractor backbone for point clouds is PointNet++ (Qi et al. 2017) with model size around 10M, while the backbone network for images is ResNet18 (He et al. 2016) which has much more parameters with model size of 45M. The light-weighted structure of PointNet++ supports efficient computation distributed RI., large batch size, and less overfitting with faster convergence. We note that other 3D networks should also be applicable. For ResNet18 with depth and foreground masks, we concatenate the depth image and the mask to the RGB image. The weights of the first convolution layer corresponding to the additional channels are initialized with all zeros.
The remaining timestep 7 - L. is concatenated in the features to provide information about the remaining horizon. The actor network and the critic network are both implemented by multi-layer perceptron with three fully-connected layers using the ReLU activation function. We use Adam as the optimizer for training and decrease the learning rate piece-wise step schedule. We use separate feature extractors for the actor and the critic. For finetuning with hindsight goals, we fix the weights the feature extractor backbones, and only finetune the actor and critic networks.
B.4 DESIGN LIMITATIONS
The underlying assumption for using point aggregation and hindsight goals in our method is that the object pose does not change dramatically due robot interaction, which we find quite applicable for precision grasping. Moreover, we use the depth region between the two fingertips to determine the robot should close the fingers and retract the arm complete a grasp. In the real world where depth and mask perception are limited as the end-effector is close to the target, aggregated points inside the cage region the gripper as well as the time step / to determine ter- mination. Although this heuristic would not allow re- re-grasp other learned grasp closure behavior, we find it generalized and robust enough to simplify the task. the simulator, our value function in the actor-critic can improve the policy, especially in those contact-rich scenarios (Figure 6 even physics parameters are varied. However, when deploying the learned policy to the real world, we still observe the sim-to-real domain gap in contact modeling. Finally, while action representation of end-effector delta pose makes the policy learning easier by removing some of the complexity of the configuration space and dynamics, is less suitable for high-frequency control domains.
use the
C GRASP PREDICTION PERFORMANCE
Since method involves grasping goal prediction. evaluate the grasp detection quantitatively and qualitatively. In this experiment, we train a policy network using BC on the offline dataset and point clouds our input.
14

---

Under review as a conference paper at ICLR 2021
Figure 7: Grasp prediction examples on several ShapeNet objects. The red fork denotes the gripper pose predicted from our policy network.
Grasp- Trans lation (m)
Grasp-Rotatin (*)
Grasp- Doitt-Disttnce (m)
Method
SeapeNet
YCB
ShapeNet
YCB
ShapeNet
YCB
Mean
Min
Mean
Gual-Auxilaay
0.015
0.045
0.020
0,047
6.621
17,989
7.610
19,465
0.023
0.065
0,028
0.069
Gral-Conditioned
0.016
0.039
0.025
0.045
6.987
17,638
7,821
19.864
0.025
0.056
0.033
0.071
Table Evaluation on grasp prediction performance
Table shows the grasp prediction results. We use the target goals from the expert planner as the ground truth, and compute translation errors in meters, rotation errors in degrees and the point matching distances in meters as used in the loss function in Eq. (2). We compute both the mean error and the minimum eтror along the trajectories from the expert.
In Table 5 "Goal-Auxiliary indicates our policy network with auxiliary goal prediction using the aggregated point cloud representation. "Goal-Conditioned" indicates a separate network trained for grasp prediction only, which is used for goal-conditioned models. We can see that the performance of the two networks are quite similar, which demonstrates that our training is able to optimize the auxiliary task well. Overall, jointly training the policy and grasp prediction does not affect the grasp prediction performance. We also observe that the grasp prediction error decreases as the camera gets closer to the object in closed-loop setting. Note that the regression errors affected by the discrete nature the pre-defined grasp set and the multi modaiitto,llty,, especially for certain symmetric objects such bowls. Figure 7 illustrates some grasp prediction examples ShapeNet objects.
15


Reviews:
[{"review_id": "z5AUHpoTcgL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses several different techniques in IL and RL to improve performance on 6D robot grasping. It uses an expert planner OMG to collect initial data for BC as well as for online IL via Dagger. The uses DDPG to further train as well as fine tune on new unlabeled objects.\n\nThe topic is very relevant and of current interest as more real world applications will need more than just 2D grasping that bin-picking has addressed and related work is sufficiently discussed.\n\nThe technical contribution seems weak as the paper mostly explores known methods and well-known 'trade-tricks' (goal conditioning or loss on goal) towards a grasping centric problem which is also heavily explored as part of various RL tasks in literature. The main weakness of the work however is the lack of clear motivation for why such a complicated procedure is necessary compared to the expert planner already being used - the experiments aren't designed to address this question.\n\n- Using a planner as an expert for IL is common practice and I don't think counts as a major contribution as presented at the end of the introduction.\n\n- The bulk of IL experiments focus on what input representation is helpful. While it is not surprising that 3D inputs like point clouds would be better for 6D grasping these are more suitable as ablations than main experiments investigating the proposed method itself, compared to other sota approaches learning based or otherwise.\n\n- In several places 'contact-rich' and 'different dynamics' is motivated without clear explanation early on until the experiments identified what the setup was. The former does not seem to be well explored in the experiments, '...especially in those contact-rich scenarios...'. Aren't all grasping problems contact rich (unless only reaching to a pre-grasp is being considered) or were there some new scenarios constructed to specifically study the relative effects of contact?\n\n- Results in table 1 and figure 4 present mean statistic from 3-5 runs. This seems small, variance bands should be shown in figure 4 to see if the small number of sample are sufficient to capture the full picture.\n\n- The problems studied could be addressed by planar grasping as well. Complex setups with clutter, etc would better motivate if the presented approach is able to scale to scenarios where 6D grasping is necessary. \n\nOther comments:\n\n- Does not including the BC loss for some samples in the batch (or between training iteration) cause any discontinuities or make learning unstable, as the loss landscape discontinuously changes?\n\n[Update]\nThank you for the responses and clarifications. I appreciate the additional experiments in the real world and comparisons with the open-loop policy. Novelty still remains a concern however; in using a planner as an IL expert, it isn't clear what was challenging to adopt this strategy for the grasping problem and qualifies as a significant contribution. Additionally, the experiments to study 'contact-rich' and 'different dynamics' problems is unclear; the experiments don't indicate what aspects of the proposed method address these challenges and are able to do so with vision/depth-only feedback (no tactile); also the evaluation in simulation alone is insufficient to study such scenarios. I have updated my score accordingly.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HhZAWB_UvEr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "__Summary__\nThe paper targets the problem of closed-loop 6D robotic grasping with a parallel gripper based on RGB-D in-hand-camera images. The policy takes an aggregated point cloud (computed from image history) as input (using a PointNet++) and outputs the pose transformation of the gripper. There are several contributions in the specifics of the proposed method. The policy is pretrained using behavioral cloning and DAGGER on known object models, where the expert is composed by a grasp pose sampler and the OMG grasp trajectory planner. Subsequently, the pretrained policy is improved using TD3. Actor and critic networks are each regularized via a loss for solving the auxiliary task of (independently from each other) predicting the final grasping pose. As the goal poses are only available for the expert demonstrations goals are added for the policy roll-out in hindsight, similar to hindsight experience replay, which does not require access to an object model. The approach is evaluated in simulation for grasping YCB and ShapeNet objects with a Franka Emika Panda robot. The evaluation covers ablations of several algorithmical and architectural choices.\n\n\n__Strong points__\n- The paper is well-written\n- Thorough and interesting ablations\n- Sound approach\n\n__Weak points__\n- State-representation might get invalidated when an object is moved, nullifying the biggest advantage of closed-loop grasping\n- no comparisons with competing methods (e.g. open-loop 6D grasping + OMG)\n- no evaluations in a cluttered environment\n- no real robot experiment\n- no code\n\n\n__Recommendation__\nI recommend accepting the paper because the system looks sound and promising and is well explained. Some parts, like regularization based on goal pose prediction, are potentially useful also for very different approaches. \n\n__Supporting Arguments__\nOverall the approach seems sound and the success rates of around 90% seem good for closed-loop 6D grasping of unknown objects. The ablations that disentangle the contributions of the different design choices are valuable and I was delighted to also find a comparison to the more common and arguably more intuitive approach of using the goal predictions as input to the networks. As the approach and the results are presently sufficiently clear, the paper provides a valuable contribution and should thus be accepted.\n\nStill, there are also several weak points. \n- The experiments don't seem to show the benefit of closed-loop grasping by only considering an uncluttered environment and not performing comparisons with open-loop approaches (e.g., [1] or [2]).\n- Some design choices do not seem to fit well to closed-loop grasping, which is mainly important for dynamic scenes, e.g. for re-grasping slipping objects. However, in such cases the accumulated point clouds can quickly become invalid, and, furthermore, lower-level control actions might be more appropriate than pose-deltas.\n- The approach is not evaluated on a real robot. For all I can tell, the simulation does not account for errors in control, calibration, forward kinematics that are encountered in practice.\n- The supplementary did not contain source code. I hope that the authors consider publishing the code after acceptance. \n\n__Questions__\n1) How could the proposed method deal with changes in the object position (accumulated point cloud becomes invalid)\n2) Did you consider sharing some features between the actor and critic, e.g. parts of the PointNet\n3) I assume that the gripper is always closed at the end of a finite-horizon episode, is that correct?\n4) I guess the foreground mask is obtained based on the z value of the transformed point cloud. The paper should be more specific here.\n5) Please elaborate: \"We also empirically observe that the auxiliary task with the bootstrap errors caused by value overestimation.\"\n\n__Additional Feedback__\nIf I understand correctly (based on Appendix A) the method uses TD3 instead of \"vanilla\" DDPG. If this is the case, the main part should also be more concrete.\nIt's not clear to me what Figure 5 is supposed to show. What does contact-rich mean in this context? Is it about contacts before closing the gripper?\n\nTypos:\nextra space in Figure 1: \"known objects .\"\nextra \"the\" in Section 5.1: \"indicates the the aggregated point cloud\"\n\n__References__\n[1] ten Pas, Andreas, et al. \"Grasp pose detection in point clouds.\" The International Journal of Robotics Research 36.13-14 (2017): 1455-1473.\n[2] Mousavian, Arsalan, Clemens Eppner, and Dieter Fox. \"6-dof graspnet: Variational grasp generation for object manipulation.\" Proceedings of the IEEE International Conference on Computer Vision. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-4HuPNLTv69", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the task of closed loop 6-DOF grasping of objects in simulation. The learned policy is a closed-loop policy, in that the gripper pose is continuously adjusted as the gripper approaches the object. The paper employs a combination of imitation learning, reinforcement learning, and auxiliary losses for training this policy. The policy operates upon information from point clouds as observed from a wrist-mounted camera.\n\nStrengths: The paper tackles the important and relevant problem of closed-loop 6DOF grasping. The proposed solution makes sound choices: a) uses fused point-clouds to represent the state, b) use of expert behavior for imitation, c) use of auxiliary losses for training. The paper also does systematic experiments in simulation to judge the importance of the different components.\n\nShortcomings: While the problem is interesting and important, and the proposed approach is sound, the experiments have entirely been done in simulation. Past work on this topic has studied this problem in the real world. In particular, the paper focuses on the design of closed-loop policies. Closed-loop policies are more relevant when there is noise in the motion of the robot, or there are hard to predict dynamics arising from the interaction of the gripper with the object. These are precisely the aspects that should be studied in the real world, as they are hard to model in order to study in simulation. Thus, it is not clear to me as to what aspects of this paper will be applicable to the study of this problem in the real world.\n\nMy second concern is about novelty over past work. All aspects of the technical approach of the paper have been studied in the past. Use of imitation learning and RL together has been studied (eg: DAPG Rajeswaran et al.), use of auxiliary rewards has been studied (eg: UNREAL Jaderberg et al.), use of hindsight experience replay (eg: Andrychowicz et al.). Thus, I am not sure what is the precise technical contribution made in the paper.\n\nThus in summary, while the paper tackles an interesting and important problem, the problem has only been studied in simulation which makes the application less interesting. At the same time, proposed approach is largely a combination of known techniques in the literature.\n\nUpdate: I thank the authors for providing clarifications and additional experiments, in particular the comparison to open-loop grasping (SOTA grasp detection method from Mousavian et al.). I still find the technical novelty of the paper limited. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper got mixed reviews ranging from 5 to 7. The main concerns of the reviewers were the missing novelty as the paper combines different well known methods for a given problem, so there is no big algorithmic contribution. The presented pipeline for closed-loop grasping using imitation learning from a planner, Dagger and subsequent deep RL with TD3 is a straightforward, but sound and intuitive combination of algorithms to address the problem of closed loop grasping.  The presented results and ablation studies also motivate these algorithmic choices. In the rebuttal the authors addressed most concerns regarding the experiments (missing comparisons to open-loop grasping and real world experiments), but more real world experiments would be necessary to evaluate the effectiveness of the approach.  

This is a borderline paper were  I unfortunately have to recommend rejection due to the missing algorithmic contribution, a major requirement for ICLR. The paper would  however fit very well to a robotics conference and the authors are  encouraged to resubmit the paper the venues such as RSS or CoRL.  
Decision: Reject