Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives

1 Introduction . Meaningful exploration remains a challenge for robotic reinforcement learning systems . For example , in the manipulation tasks shown in Figure 1 , useful exploration might correspond to picking up and placing objects in different configurations . However , random motions in the robot ’ s joint space will rarely , if ever , result in the robot touching the objects , let alone pick them up . Recent work , on the other hand , has demonstrated remarkable success in training RL agents to solve manipulation tasks [ 4 , 24 , 26 ] by sidestepping the exploration problem with careful engineering . Levine et al . [ 26 ] use densely shaped rewards , while Kalashnikov et al . [ 24 ] leverage a large scale robot infrastructure and Andrychowicz et al . [ 4 ] require training in simulation with engineered reward functions in order to transfer to the real world . In general , RL methods can be prohibitively data inefficient , require careful reward development to learn , and struggle to scale to more complex tasks without the aid of human demonstrations or carefully designed simulation setups . An alternative view on why RL is difficult for robotics is that it requires the agent to learn both what to do in order to achieve the task and how to control the robot to execute the desired motions . For example , in the kitchen environment featured at the bottom of Figure 1 , the agent would have to learn how to accurately manipulate the arm to reach different locations as well as how to grasp different objects , while also ascertaining what object it has to grasp and where to move it . Considered independently , the problems of controlling a robot arm to execute particular motions and figuring out the desired task from scalar reward feedback , then achieving it , are non-trivial . Jointly learning to solve both problems makes the task significantly more difficult . †Equal advising 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . s * robot Sample Primitive Select Argument Parameters Mean Variance Sample Arguments Feedback Control Loop srobot s′ robot ek ( s , s * ) Ck ( e , s ) Primitives are re-used across robots , environments , and tasks π Grasp : Lift : Push : Twist : Image Input Sparse Rewards 0 Hk.66Hk.33Hk In contrast to training RL agents on raw actions such as torques or delta positions , a common strategy is to decompose the agent action space into higher ( i.e. , what ) and lower ( i.e. , how ) level structures . A number of existing methods have focused on designing or learning this structure , from manually architecting and fine-tuning action hierarchies [ 14 , 27 , 32 , 47 ] , to organizing agent trajectories into distinct skills [ 3 , 20 , 41 , 50 ] to more recent work on leveraging large offline datasets in order to learn skill libraries [ 29 , 40 ] . While these methods have shown success in certain settings , many of them are either too sample inefficient , do not scale well to more complex domains , or lack generality due to dependence on task relevant data . In this work , we investigate the following question : instead of learning low-level primitives , what if we were to design primitives with minimal human effort , enable their expressiveness by parameterizing them with arguments and learn to control them with a high-level policy ? Such primitives have been studied extensively in task and motion planning ( TAMP ) literature [ 22 ] and implemented as parameterized actions [ 19 ] in RL . We apply primitive robot motions to redefine the policy-robot interface in the context of robotic reinforcement learning . These primitives include manually defined behaviors such as lift , push , top-grasp , and many others . The behavior of these primitives is parameterized by arguments that are the learned outputs of a policy network . For instance , top-grasp is parameterized by four scalar values : grasp position ( x , y ) , how much to move down ( z ) and the degree to which the gripper should close . We call this application of parameterized behaviors , Robot Action Primitives for RL ( RAPS ) . A crucial point to note is that these parameterized actions are easy to design , need only be defined once and can be re-used without modification across tasks . The main contribution of this work is to support the effectiveness of RAPS via a thorough empirical evaluation across several dimensions : • How do parameterized primitives compare to other forms of action parameterization ? • How does RAPS compare to prior methods that learn skills from offline expert data ? • Is RAPS agnostic to the underlying RL algorithm ? • Can we stitch the primitives to perform multiple complex manipulation tasks in sequence ? • Does RAPS accelerate exploration even in the absence of extrinsic rewards ? We investigate these questions across complex manipulation environments including Kitchen Suite , Metaworld and Robosuite domains . We find that a simple parameterized action based approach outperforms prior state-of-the-art by a significant margin across most of these settings2 . 2Please view our website for performance videos and links to our code : https : //mihdalal.github.io/raps/ 2 Related Work . Higher Level Action and Policy Spaces in Robotics In robotics literature , decision making over primitive actions that execute well-defined behaviors has been explored in the context of task and motion planning [ 9 , 22 , 23 , 43 ] . However , such methods are dependent on accurate state estimation pipelines to enable planning over the argument space of primitives . One advantage of using reinforcement learning methods instead is that the agent can learn to adjust its implicit state estimates through trial and error experience . Dynamic Movement Primitive and ensuing policy search approaches [ 11 , 21 , 25 , 36 , 37 ] leverage dynamical systems to learn flexible , parameterized skills , but are sensitive to hyper-parameter tuning and often limited to the behavior cloning regime . Neural Dynamic Policies [ 6 ] incorporate dynamical structure into neural network policies for RL , but evaluate in the state based regime with dense rewards , while we show that parameterized actions can enable RL agents to efficiently explore in sparse reward settings from image input . Hierarchical RL and Skill Learning Enabling RL agents to act effectively over temporally extended horizons is a longstanding research goal in the field of hierarchical RL . Prior work introduced the options framework [ 45 ] , which outlines how to leverage lower level policies as actions for a higher level policy . In this framework , parameterized action primitives can be viewed as a particular type of fixed option with an initiation set that corresponds to the arguments of the primitive . Prior work on options has focused on discovering [ 1 , 12 , 41 ] or fine-tuning options [ 5 , 14 , 27 ] in addition to learning higher level policies . Many of these methods have not been extended beyond carefully engineered state based settings . More recently , research has focused on extracting useful skills from large offline datasets of interaction data ranging from unstructured interaction data [ 49 ] , play [ 28 , 29 ] to demonstration data [ 2 , 35 , 39 , 40 , 44 , 46 , 53 ] . While these methods have been shown to be successful on certain tasks , the learned skills are only relevant for the environment they are trained on . New demonstration data must be collected to use learned skills for a new robot , a new task , or even a new camera viewpoint . Since RAPS uses manually specified primitives dependent only on the robot state , RAPS can re-use the same implementation details across robots , tasks and domains . Parameterized Actions in RL The parameterized action Markov decision process ( PAMDP ) formalism was first introduced in Masson et al . [ 31 ] , though there is a large body of earlier work in the area of hybrid discrete-continuous control , surveyed in [ 7 , 8 ] . Most recent research on PAMDPs has focused on better aligning policy architectures and RL updates with the nature of parameterized actions and has largely been limited to state based domains [ 13 , 51 ] . A number of papers in this area have focused on solving a simulated robot soccer domain modeled as either a single-agent [ 19 , 31 , 48 ] or multi-agent [ 15 ] problem . In this paper , we consider more realistic robotics tasks that involve interaction with and manipulation of common household objects . While prior work [ 42 ] has trained RL policies to select hand-designed behaviors for simultaneous execution , we instead train RL policies to leverage more expressive , parameterized behaviors to solve a wide variety of tasks . Closely related to this work is Chitnis et al . [ 10 ] , which develops a specific architecture for training policies over parameterized actions from state input and sparse rewards in the context of bi-manual robotic manipulation . Our work is orthogonal in that we demonstrate that a higher level policy architecture is sufficient to solve a large suite of manipulation tasks from image input . We additionally note that there is concurrent work [ 34 ] that also applies engineered primitives in the context of RL , however , we consider learning from image input and sparse terminal rewards . 3 Robot Action Primitives in RL . To address the challenge of exploration and behavior learning in continuous action spaces , we decompose a desired task into the what ( high level task ) and the how ( control motion ) . The what is handled by the environment-centric RL policy while the how is handled by a fixed , manually defined set of agent-centric primitives parameterized by continuous arguments . This enables the high level policy to reason about the task at a high level by choosing primitives and their arguments while leaving the low-level control to the parameterized actions themselves . 3.1 Background . Let the Markov decision process ( MDP ) be defined as ( S , A , R ( s , a , s′ ) , T ( s′|s , a ) , p ( s0 ) , γ , ) in which S is the set of true states , A is the set of possible actions , R ( s , a , s′ ) is the reward function , T ( s′|s , a ) is the transition probability distribution , p ( s0 ) defines the initial state distribution , and γ is the discount factor . The agent executes actions in the environment using a policy π ( a|s ) with a corresponding trajectory distribution p ( τ = ( s0 , a0 , ... at−1 , sT ) ) = p ( s0 ) Πtπ ( at|st ) T ( st+1|st , at ) . The goal of the RL agent is to maximize the expected sum of rewards with respect to the policy : Es0 , a0 , ... at−1 , sT , ∼p ( τ ) [ ∑ t γ tR ( st , at ) ] . In the case of vision-based RL , the setup is now a partially observed Markov decision process ( POMDP ) ; we have access to the true state via image observations . In this case , we include an observation space O which corresponds to the set of visual observations that the environment may emit , an observation model p ( o|s ) which defines the probability of emission and policy π ( a|o ) which operates over observations . In this work , we consider various modifications to the action space A while keeping all other components of the MDP or POMDP the same .

Summary: Low-level actions, such as torques or joint-velocities, provide a hard-exploration challenge for applying reinforcement learning (RL) to robotic systems. For example, it might be impossible to reach and grasp an object solely due to random torques or joint velocities. To overcome this difficulty, one can define high-level actions to enable a meaningful exploration. In this paper, the authors introduce parametrized action primitives, which encode a robotic movement and use those actions for learning with RL. The parametrized actions are just manually encoded primitives, such as moving the arm up, reaching a position $x$ and grasp, and so on. Therefore, the system proposed by the authors requires expert knowledge of the system, the task, and robotics. This simplification of the task results, according to the paper, in a more efficient RL.