Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds

1 INTRODUCTION . Robotic grasping of arbitrary objects is a challenging task . A robot needs to deal with objects it has never seen before , and generates a motion trajectory to grasp an object . Due to the complexity of the problem , majority works in the literature focus on bin-picking tasks , where top-down grasping is sufficient to pick up an object . Both grasp detection approaches ( Redmon & Angelova , 2015 ; Pinto & Gupta , 2016 ; Mahler et al. , 2017 ) and reinforcement learning-based methods ( Kalashnikov et al. , 2018 ; Quillen et al. , 2018 ) are introduced to tackle the top-down grasping problem . However , it is difficult for these methods to grasp objects in environments where 6D grasping is necessary , i.e. , 3D translation and 3D rotation of the robot gripper , such as a cereal box on a tabletop or in a cabinet . While 6D grasp synthesis has been studied using 3D models of objects ( Miller & Allen , 2004 ; Eppner et al. , 2019 ) and partial observations ( ten Pas et al. , 2017 ; Yan et al. , 2018 ; Mousavian et al. , 2019 ) , these methods only generate 6D grasp poses of the robot gripper for an object , instead of generating a trajectory of the gripper pose to reach and grasp the object . As a result , a motion planner is needed to plan the grasping motion according to the grasp poses . Usually , the planned trajectory is executed in an open-loop fashion since re-planning is expensive , and perception feedback during grasping as well as dynamics and contacts of the object are often ignored , which makes the grasping sensitive to grasp synthesis errors . In this work , to overcome the limitations in the paradigm of 6D grasp synthesis followed by robot motion planning , we introduce a novel method for learning closed-loop 6D grasping polices from partially-observed point clouds of objects . Our policy directly outputs the control action of the robot gripper , which is the relative 6D pose transformation of the gripper . For the state representation , we adopt an egocentric view with a wrist camera mounted on the robot gripper , which avoids self-occlusion of the robot arm during grasping compared to using an external static camera . Additionally , we aggregate point clouds of the object from previous time steps to avoid ambiguities in the current view and encode the history observations . Our point cloud representation provides richer 3D information for 6D grasping and generalizes better to different objects compared to RGB images . 1Videos and code are available at https : //sites.google.com/view/gaddpg We propose to combine Imitation Learning ( IL ) and Reinforcement Learning ( RL ) in learning the 6D grasping policy . Since RL requires exploration of the state space , the chance of lifting an object is very rare . Moreover , the target object can easily fall down with a bad contact and the ego-view camera may lose the object during grasping . Different from previous works ( Song et al. , 2020 ; Young et al. , 2020 ) that specifically design a robot gripper and collect human demonstrations using it , we obtain demonstrations using a joint motion and grasp planner ( Wang et al. , 2020 ) in simulation . Consequently , we can efficiently obtain a large number of 6D grasping trajectories using ShapeNet objects ( Chang et al. , 2015 ) with the planner . Then we learn the grasping policy based on the Deep Deterministic Policy Gradient ( DDPG ) algorithm ( Lillicrap et al. , 2015 ) , which is an actorcritic algorithm in RL that can utilize off-policy data from demonstrations . More importantly , we introduce a goal prediction auxiliary task to improve the policy learning , where the actor and the critic in DDPG are trained to predict the final 6D grasping pose of the robot gripper as well . The supervision on goal prediction comes from the expert planner for objects with known 3D shape and pose . For unknown objects without 3D models available , we can still obtain the grasping goals from successful grasping rollouts of the policy , i.e. , hindsight goals . This property enables our learned policy to be finetuned on unknown objects , which is critical for continual learning in the real world . Figure 1 illustrates our setting for learning the 6D grasp polices . We conduct thorough analyses and evaluations of our method for 6D grasping . We show that our learned policy can successfully grasp unseen ShapeNet objects and unseen YCB objects ( Calli et al. , 2015 ) in simulation , and finetuning the policy with hindsight goals for unknown YCB objects improves the grasping success rate . In the real world , we utilized a recent unseen object instance segmentation method ( Xiang et al. , 2020 ) to segment the point cloud of an target object , and then applied GA-DDPG for grasping . The learned policy is able to successfully grasp YCB objects in the real world . It only failed one among ten grasping of different unseen objects with perception noises . Overall , our contributions are three-folds : 1 ) We propose to use point cloud as our state representation and use demonstrations from a joint motion and grasp planner to learn closed-loop 6D grasping policies . 2 ) We introduce the Goal-Auxiliary DDPG ( GA-DDPG ) algorithm for joint imitation and reinforcement learning using goals . 3 ) We propose to use hindsight goals for finetuning a pre-trained policy on unknown objects without goal supervision . 2 RELATED WORK . Combining Imitation Learning and Reinforcement Learning . For high-dimensional continuous state-action space with sparse rewards and complex dynamics as in most real-world robotic settings , model-free RL provides a data-driven approach to solve the task ( Kalashnikov et al. , 2018 ; Quillen et al. , 2018 ) , but it requires a large number of interactions even with full-state information . Therefore , many works have proposed to combine imitation learning ( Osa et al. , 2018 ) in RL . For example , Rajeswaran et al . ( 2017 ) augment policy gradient update with demonstration data to circumvent reward shaping and the exploration challenge . Zhu et al . ( 2018 ) use inverse RL to learn dexterous manipulation tasks with a few human demonstrations in simulation . The closest related works to ours are Vecerik et al . ( 2017 ) ; Nair et al . ( 2018a ) that utilize demonstration data with offpolicy RL . Despite the focus on different tasks , the main difference is that our demonstrations are from an expert planner instead of human demonstrators . Therefore , we can obtain a large number of demonstrations and query the expert planner during training to provide supervision on the on-policy data . Moreover , compared with human demonstrators , expert planners are more likely to generate trajectories that are suitable for robots to execute and have consistency across the state space . Goals and Auxiliary Tasks in Policy Learning . Goals are often used as extra signals to guide policy learning ( Kaelbling , 1993 ) . For goal-conditioned policies , Andrychowicz et al . ( 2017 ) ; Ding et al . ( 2019 ) ; Ghosh et al . ( 2019 ) make the observation that every trajectory is a successful demonstration of the goal state that it reaches , thereby these methods re-label goals in rollout trajectories for effective learning . However , goals need to be provided in testing , which can be an optimistic assumption for complex tasks . Although many works have been proposed to generate goals or subgoals with deep neural networks ( Nair et al. , 2018b ; Florensa et al. , 2018 ; Sharma et al. , 2019 ) , in 6D grasping , estimating the grapsing goals for unseen objects is a challenging task . On the other hand , auxiliary tasks have been employed to improve RL as well . For instance , state reconstruction tasks ( Finn et al. , 2016 ) , auxiliary control and reward tasks ( Jaderberg et al. , 2016 ) , pose estimation loss ( Zhang et al. , 2018 ) and contrastive loss ( Srinivas et al. , 2020 ) have been used to improve representation learning . In our work , we utilize the grasping goal prediction as a natural auxiliary task that requires the policy to predict how to grasp the target object . Vision-based Robotic Grasping . Grasp synthesis can be used in a planning and control pipeline for robotic grasping ( Pinto & Gupta , 2016 ; Mahler et al. , 2017 ; Morrison et al. , 2018 ; Murali et al. , 2020 ) . Alternatively , end-to-end policy learning methods ( Levine et al. , 2018 ; Kalashnikov et al. , 2018 ; Iqbal et al. , 2020 ) make use of large-scale data to learn closed-loop vision-based grasping . Although RGB images are widely used as the state representation , it requires the policy to infer 3D information from 2D images . Recently , depth and segmentation masks ( Mahler et al. , 2017 ) , shape completion ( Yan et al. , 2018 ) , deep visual features ( Florence et al. , 2019 ) , keypoints ( Manuelli et al. , 2019 ) , point clouds ( Yan et al. , 2019 ) , multiple cameras ( Akinola et al. , 2020 ) , and egocentric views ( Song et al. , 2020 ; Young et al. , 2020 ) have been considered to improve the state representation . Motivated by these methods , we utilize the aggregated point clouds of an object from an egocentric camera as our state representation . Moreover , we explicitly learn a policy for the large action space in 6D grasping instead of using value-based representations as in most top-down grasping methods . 3 BACKGROUND . Markov Decision Process . A Markov Decision Process ( MDP ) is defined using the tuple : M = { S , R , A , O , P , ρ0 , γ } . S , A , andO represent the state , action , and observation space . R : S×A → R is the reward function . P : S×A → S is the transition dynamics . ρ0 is the probability distribution over initial states and γ = [ 0 , 1 ) is a discount factor . Let π : S → A be a policy which maps states to actions . In the partially observable case , at each time t , the policy maps a partial observation ot of the environment to an action at = π ( ot ) . Our goal is to learn a policy that maximizes the expected cumulative rewards E π [ ∑∞ t=0 γ trt ] , where rt is the reward at time t. The Q-function of the policy for a state-action pair is Q ( s , a ) = R ( s , a ) + γ E s′ , π [ ∑∞ t=0 γ trt|s0 = s′ ] , where s′ represents the next state of taking action a at state s according to the transition dynamics . Deep Deterministic Policy Gradient . Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) is an actor-critic algorithm that uses off-policy data and temporal difference learning , and has successful applications in continuous control ( Popov et al. , 2017 ; Vecerik et al. , 2017 ) . Specifically , the actor in DDPG learns the policy πθ ( s ) , while the critic approximates the Q-function Qφ ( s , a ) , where θ and φ denote the parameters of the actor and the critic , respectively . A replay buffer of transitions D = { ( s , a , r , s′ ) } is maintained during training , and examples sampled from it are used to optimize the actor-critic . DDPG minimizes the following Bellman error with respect to φ : L ( φ ) = E ( s , a , r , s′ ) ∼D [ 1 2 ( Qφ ( s , a ) − ( r + γQφ ( s ′ , πθ ( s ′ ) ) ) 2 ] . ( 1 ) Then the deterministic policy πθ is trained to maximize the learned Q-function with objective maxθ E s∼D ( Qφ ( s , πθ ( s ) ) ) , which resembles a policy evaluation and improvement schedule .

Summary: This paper tackles the task of closed loop 6-DOF grasping of objects in simulation. The learned policy is a closed-loop policy, in that the gripper pose is continuously adjusted as the gripper approaches the object. The paper employs a combination of imitation learning, reinforcement learning, and auxiliary losses for training this policy. The policy operates upon information from point clouds as observed from a wrist-mounted camera.