Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds

1 INTRODUCTION . Robotic grasping of arbitrary objects is a challenging task . A robot needs to deal with objects it has never seen before , and generates a motion trajectory to grasp an object . Due to the complexity of the problem , majority works in the literature focus on bin-picking tasks , where top-down grasping is sufficient to pick up an object . Both grasp detection approaches ( Redmon & Angelova , 2015 ; Pinto & Gupta , 2016 ; Mahler et al. , 2017 ) and reinforcement learning-based methods ( Kalashnikov et al. , 2018 ; Quillen et al. , 2018 ) are introduced to tackle the top-down grasping problem . However , it is difficult for these methods to grasp objects in environments where 6D grasping is necessary , i.e. , 3D translation and 3D rotation of the robot gripper , such as a cereal box on a tabletop or in a cabinet . While 6D grasp synthesis has been studied using 3D models of objects ( Miller & Allen , 2004 ; Eppner et al. , 2019 ) and partial observations ( ten Pas et al. , 2017 ; Yan et al. , 2018 ; Mousavian et al. , 2019 ) , these methods only generate 6D grasp poses of the robot gripper for an object , instead of generating a trajectory of the gripper pose to reach and grasp the object . As a result , a motion planner is needed to plan the grasping motion according to the grasp poses . Usually , the planned trajectory is executed in an open-loop fashion since re-planning is expensive , and perception feedback during grasping as well as dynamics and contacts of the object are often ignored , which makes the grasping sensitive to grasp synthesis errors . In this work , to overcome the limitations in the paradigm of 6D grasp synthesis followed by robot motion planning , we introduce a novel method for learning closed-loop 6D grasping polices from partially-observed point clouds of objects . Our policy directly outputs the control action of the robot gripper , which is the relative 6D pose transformation of the gripper . For the state representation , we adopt an egocentric view with a wrist camera mounted on the robot gripper , which avoids self-occlusion of the robot arm during grasping compared to using an external static camera . Additionally , we aggregate point clouds of the object from previous time steps to avoid ambiguities in the current view and encode the history observations . Our point cloud representation provides richer 3D information for 6D grasping and generalizes better to different objects compared to RGB images . 1Videos and code are available at https : //sites.google.com/view/gaddpg We propose to combine Imitation Learning ( IL ) and Reinforcement Learning ( RL ) in learning the 6D grasping policy . Since RL requires exploration of the state space , the chance of lifting an object is very rare . Moreover , the target object can easily fall down with a bad contact and the ego-view camera may lose the object during grasping . Different from previous works ( Song et al. , 2020 ; Young et al. , 2020 ) that specifically design a robot gripper and collect human demonstrations using it , we obtain demonstrations using a joint motion and grasp planner ( Wang et al. , 2020 ) in simulation . Consequently , we can efficiently obtain a large number of 6D grasping trajectories using ShapeNet objects ( Chang et al. , 2015 ) with the planner . Then we learn the grasping policy based on the Deep Deterministic Policy Gradient ( DDPG ) algorithm ( Lillicrap et al. , 2015 ) , which is an actorcritic algorithm in RL that can utilize off-policy data from demonstrations . More importantly , we introduce a goal prediction auxiliary task to improve the policy learning , where the actor and the critic in DDPG are trained to predict the final 6D grasping pose of the robot gripper as well . The supervision on goal prediction comes from the expert planner for objects with known 3D shape and pose . For unknown objects without 3D models available , we can still obtain the grasping goals from successful grasping rollouts of the policy , i.e. , hindsight goals . This property enables our learned policy to be finetuned on unknown objects , which is critical for continual learning in the real world . Figure 1 illustrates our setting for learning the 6D grasp polices . We conduct thorough analyses and evaluations of our method for 6D grasping . We show that our learned policy can successfully grasp unseen ShapeNet objects and unseen YCB objects ( Calli et al. , 2015 ) in simulation , and finetuning the policy with hindsight goals for unknown YCB objects improves the grasping success rate . In the real world , we utilized a recent unseen object instance segmentation method ( Xiang et al. , 2020 ) to segment the point cloud of an target object , and then applied GA-DDPG for grasping . The learned policy is able to successfully grasp YCB objects in the real world . It only failed one among ten grasping of different unseen objects with perception noises . Overall , our contributions are three-folds : 1 ) We propose to use point cloud as our state representation and use demonstrations from a joint motion and grasp planner to learn closed-loop 6D grasping policies . 2 ) We introduce the Goal-Auxiliary DDPG ( GA-DDPG ) algorithm for joint imitation and reinforcement learning using goals . 3 ) We propose to use hindsight goals for finetuning a pre-trained policy on unknown objects without goal supervision . 2 RELATED WORK . Combining Imitation Learning and Reinforcement Learning . For high-dimensional continuous state-action space with sparse rewards and complex dynamics as in most real-world robotic settings , model-free RL provides a data-driven approach to solve the task ( Kalashnikov et al. , 2018 ; Quillen et al. , 2018 ) , but it requires a large number of interactions even with full-state information . Therefore , many works have proposed to combine imitation learning ( Osa et al. , 2018 ) in RL . For example , Rajeswaran et al . ( 2017 ) augment policy gradient update with demonstration data to circumvent reward shaping and the exploration challenge . Zhu et al . ( 2018 ) use inverse RL to learn dexterous manipulation tasks with a few human demonstrations in simulation . The closest related works to ours are Vecerik et al . ( 2017 ) ; Nair et al . ( 2018a ) that utilize demonstration data with offpolicy RL . Despite the focus on different tasks , the main difference is that our demonstrations are from an expert planner instead of human demonstrators . Therefore , we can obtain a large number of demonstrations and query the expert planner during training to provide supervision on the on-policy data . Moreover , compared with human demonstrators , expert planners are more likely to generate trajectories that are suitable for robots to execute and have consistency across the state space . Goals and Auxiliary Tasks in Policy Learning . Goals are often used as extra signals to guide policy learning ( Kaelbling , 1993 ) . For goal-conditioned policies , Andrychowicz et al . ( 2017 ) ; Ding et al . ( 2019 ) ; Ghosh et al . ( 2019 ) make the observation that every trajectory is a successful demonstration of the goal state that it reaches , thereby these methods re-label goals in rollout trajectories for effective learning . However , goals need to be provided in testing , which can be an optimistic assumption for complex tasks . Although many works have been proposed to generate goals or subgoals with deep neural networks ( Nair et al. , 2018b ; Florensa et al. , 2018 ; Sharma et al. , 2019 ) , in 6D grasping , estimating the grapsing goals for unseen objects is a challenging task . On the other hand , auxiliary tasks have been employed to improve RL as well . For instance , state reconstruction tasks ( Finn et al. , 2016 ) , auxiliary control and reward tasks ( Jaderberg et al. , 2016 ) , pose estimation loss ( Zhang et al. , 2018 ) and contrastive loss ( Srinivas et al. , 2020 ) have been used to improve representation learning . In our work , we utilize the grasping goal prediction as a natural auxiliary task that requires the policy to predict how to grasp the target object . Vision-based Robotic Grasping . Grasp synthesis can be used in a planning and control pipeline for robotic grasping ( Pinto & Gupta , 2016 ; Mahler et al. , 2017 ; Morrison et al. , 2018 ; Murali et al. , 2020 ) . Alternatively , end-to-end policy learning methods ( Levine et al. , 2018 ; Kalashnikov et al. , 2018 ; Iqbal et al. , 2020 ) make use of large-scale data to learn closed-loop vision-based grasping . Although RGB images are widely used as the state representation , it requires the policy to infer 3D information from 2D images . Recently , depth and segmentation masks ( Mahler et al. , 2017 ) , shape completion ( Yan et al. , 2018 ) , deep visual features ( Florence et al. , 2019 ) , keypoints ( Manuelli et al. , 2019 ) , point clouds ( Yan et al. , 2019 ) , multiple cameras ( Akinola et al. , 2020 ) , and egocentric views ( Song et al. , 2020 ; Young et al. , 2020 ) have been considered to improve the state representation . Motivated by these methods , we utilize the aggregated point clouds of an object from an egocentric camera as our state representation . Moreover , we explicitly learn a policy for the large action space in 6D grasping instead of using value-based representations as in most top-down grasping methods . 3 BACKGROUND . Markov Decision Process . A Markov Decision Process ( MDP ) is defined using the tuple : M = { S , R , A , O , P , ρ0 , γ } . S , A , andO represent the state , action , and observation space . R : S×A → R is the reward function . P : S×A → S is the transition dynamics . ρ0 is the probability distribution over initial states and γ = [ 0 , 1 ) is a discount factor . Let π : S → A be a policy which maps states to actions . In the partially observable case , at each time t , the policy maps a partial observation ot of the environment to an action at = π ( ot ) . Our goal is to learn a policy that maximizes the expected cumulative rewards E π [ ∑∞ t=0 γ trt ] , where rt is the reward at time t. The Q-function of the policy for a state-action pair is Q ( s , a ) = R ( s , a ) + γ E s′ , π [ ∑∞ t=0 γ trt|s0 = s′ ] , where s′ represents the next state of taking action a at state s according to the transition dynamics . Deep Deterministic Policy Gradient . Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) is an actor-critic algorithm that uses off-policy data and temporal difference learning , and has successful applications in continuous control ( Popov et al. , 2017 ; Vecerik et al. , 2017 ) . Specifically , the actor in DDPG learns the policy πθ ( s ) , while the critic approximates the Q-function Qφ ( s , a ) , where θ and φ denote the parameters of the actor and the critic , respectively . A replay buffer of transitions D = { ( s , a , r , s′ ) } is maintained during training , and examples sampled from it are used to optimize the actor-critic . DDPG minimizes the following Bellman error with respect to φ : L ( φ ) = E ( s , a , r , s′ ) ∼D [ 1 2 ( Qφ ( s , a ) − ( r + γQφ ( s ′ , πθ ( s ′ ) ) ) 2 ] . ( 1 ) Then the deterministic policy πθ is trained to maximize the learned Q-function with objective maxθ E s∼D ( Qφ ( s , πθ ( s ) ) ) , which resembles a policy evaluation and improvement schedule .

Summary: The paper targets the problem of closed-loop 6D robotic grasping with a parallel gripper based on RGB-D in-hand-camera images. The policy takes an aggregated point cloud (computed from image history) as input (using a PointNet++) and outputs the pose transformation of the gripper. There are several contributions in the specifics of the proposed method. The policy is pretrained using behavioral cloning and DAGGER on known object models, where the expert is composed by a grasp pose sampler and the OMG grasp trajectory planner. Subsequently, the pretrained policy is improved using TD3. Actor and critic networks are each regularized via a loss for solving the auxiliary task of (independently from each other) predicting the final grasping pose. As the goal poses are only available for the expert demonstrations goals are added for the policy roll-out in hindsight, similar to hindsight experience replay, which does not require access to an object model. The approach is evaluated in simulation for grasping YCB and ShapeNet objects with a Franka Emika Panda robot. The evaluation covers ablations of several algorithmical and architectural choices.