Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
Authors: Murtaza Dalal
Deepak Pathak
Ruslan Salakhutdinov
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
robotic manipulation
motion primitives
hierarchical RL

Abstract: Despite the potential of reinforcement learning (RL) for building general-purpose robotic systems, training  RL agents to solve robotics tasks still remains challenging due to the difficulty of exploration in purely continuous action spaces. Addressing this problem is an active area of research with the majority of focus on improving RL methods via better optimization or more efficient exploration. An alternate but important component to consider improving is the interface of the RL algorithm with the robot. In this work, we manually specify a library of robot action primitives (RAPS), parameterized with arguments that are learned by an RL policy. These parameterized primitives are expressive, simple to implement, enable efficient exploration and can be transferred across robots, tasks and environments. We perform a thorough empirical study across challenging tasks in three distinct domains with image input and a sparse terminal reward. We find that our simple change to the action interface substantially improves both the learning efficiency and task performance irrespective of the underlying RL algorithm, significantly outperforming prior methods which learn skills from offline expert data.

Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
Murtaza Dalal
Deepak Pathak
Ruslan Salakhutdinoy
Carnegie Mellon University
(mdalal, dpathak, raaataaa 0 cs cmu. edu
Abstract
Despite the potential of reinforcement learning (RL) for building general-purpose robotic systems, training RL agents to solve robotics tasks still remains challenging due the difficulty of exploration in purely continuous action spaces. Addressing this problem is an active area of research with the majority of focus on improving RI. methods via better optimization or more efficient exploration. An alternate but important component to consider improving is the interface of the RI algorithm with the robot. this work, we manually specify a library of robot action primitives (RAPS), parameterized with arguments that are learned by an RL policy. These parameterized primitives are expressive, simple to implement, enable efficient exploration and can be transferred across robots, tasks and environments. We perform thorough empirical study across challenging tasks in three distinct domains with image input and sparse terminal reward. We find that our simple change to the action interface substantially improves both the learning efficiency and task performance irrespective of the underlying RL algorithm. significantly outperforming prior methods which learn skills from offline expert data. Code and videos at https://mihdallll http://mmiialal..guu.....b io/tap/
Introduction
Meaningful exploration remains a challenge for robotic reinforcement learning systems. For example, in the manipulation tasks shown in Figure useful exploration might correspond to picking up and placing objects in different configurations. However, random motions in the robot's joint space will rarely, if ever, result in the robot touching the objects, let alone pick them up. Recent work, the other hand, has demonstrated remarkable success in training RI. agents to solve manipulation tasks [4, 24, 26) by sidestepping the exploration problem with careful engineering. Levine et al. [26] use densely shaped rewards, while Kalashnikov et al. [24] leverage large scale robot infrastructure and Andrychowicz al. [4] require training in simulation with engincered reward functions in order to transfer to the real world. In general, RL methods can be prohibitively data inefficient, require careful reward development learn, and struggle to scale to more complex tasks without the aid of human demonstrations or carefully designed simulation setups.
An alternative view on why RL is difficult for robotics is that it requires the agent to leam both what to do in order to achieve the task and how to control the robot to execute the desired motions. For example, in the kitchen environment featured the bottom of Figure the agent would have to learn how to accurately manipulate the arm to reach different locations as well as how to grasp different objects, while also ascertaining what object has to grasp and where to move it. Considered independently, the problems of controlling a robot arm to execute particular motions and figuring out the desired task from scalar reward feedback, then achieving it, are non-trivial. Jointly learning to solve both problems makes the task significantly more difficult.
Equual advising
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Sample Primitive
Feedback Control Loop
Cite.s)
rober
c,t3,5
Select Argument
Sample Arguments
Grasp: Life
Parameters
Image Input
Sparse Rewards
Mean Variance
Twist:
3IM
RAH
H
Primitives are re-used across robots, environments, tasks
Kitchen
Metaworld
Robosuite
Figure I: Visual depiction of RAPS, outlining the process how a primitive executed on a robot. Given an input image, the policy outputs distribution over primitives and a distribution over all the arguments of all primitives, samples a primitive and selects its corresponding argument distribution parameters, indexed by which primitive was chosen, samples an argument from that distribution and executes controller in feedhack loop the robot for fixed number of timesteps (H&) to reach new
We show an example sequence of executing the lift primitive after having grasped the kettle in Kitchen environment. The agent observes the initial (0) and final states (H&) and receives a reward equal to the reward accumulated when executing the primitive. Below visualize representative tasks from the three environment suites that we evaluate on.
In contrast to training RL agents on raw actions such as torques or delta positions, common strategy decompose the agent action space into higher (i.e., what) and lower (i.e., how) level structures. A number of existing methods have focused on designing or learning this structure, from manually architecting and fine-tuning action hierarchies [14, 27, 32, 47], to organizing agent trajectories into distinct skills [3, 20, 41, 50] to more recent work on leveraging large offline datasets in order to learn skill libraries [29, 40]. While these methods have shown success in certain settings, many of them are either too sample inefficient, do not scale well to more complex domains, or lack generality due to dependence on task relevant data.
In this work, we investigate the following question: instead of learning low-level primitives, what if we were design primitives with minimal human effort, enable their expressiveness by parameterizing them with arguments and learn to control them with a high-level policy? Such primitives have been studied extensively in task and motion planning (TAMP) literature [22] and implemented as parameterized actions [19] in RI. We apply primitive robot motions to redefine the policy-robot interface the context of robotic reinforcement learning- These primitives include manually defined behaviors such as lift, push, gopagr and many others. The behavior of these primitives is parameterized by arguments that are the learned outputs of policy network. For instance, top- grasp is parameterized by four scalar values: grasp position (x.y), how much to move down (z) and the degree to which the gripper should close. We call this application of parameterized behaviors, Robot Action Primitives for RL (RAPS). A crucial point note is that these parameterized actions are easy to design, need only be defined once and can be re-used without modification across tasks.
The main contribution of this work is to support the effectiveness of RAPS via thorough empirical evaluation across several dimensions:
How do parameterized primitives compare to other forms of action parameterization?
How does RAPS compare to prior methods that learn skills from offline expert data?
Is RAPS agnostic to the underlying RL algorithm?
Can we stitch the primitives perform multiple complex manipulation tasks in sequence?
Does RAPS accelerate exploration even in the absence of extrinsic rewards?
We investigate these questions across complex manipulation environments including Kitchen Suite, Metaworld and Robosuite domains. We find that a simple parameterized action based approach outperforms prior oftte--f--h-arrr by significant margin across most of these settings?.
Please view website for performance videos and links to code: https://mihdalall wtt.alllllgggiiiiiiooooooooo iithub.io/raps/

---

2 Related Work
Higher Level Action and Policy Spaces in Robotics In robotics literature, decision making over primitive actions that execute well-defined behaviors has been explored in the context of task and motion planning [9, 22, 23, 43]. However, such methods are dependent on accurate state estimation pipelines to enable planning over the argument space of primitives. One advantage of using reinforcement learning methods instead is that the agent can learn to adjust its implicit state estimates through trial and error experience. Dynamic Movement Primitive and ensuing policy search approaches [11, 21, 25, 36, 37] leverage dynamical systems to learn flexible, parameterized skills, but are sensitive to syper-parameter tuning and often limited to the behavior cloning regime. Neural Dynamic Policies [6] incorporate dynamical structure into neural network policies for RI., but evaluate in the state based regime with dense rewards, while we show that parameterized actions can enable RL agents to efficiently explore in sparse reward settings from image input.
Hierarchical RL and Skill Learning Enabling RL agents to act effectively over temporally extended horizons is longstanding research goal in the field of hierarchical RL. Prior work introduced the options framework [45], which outlines how to leverage lower level policies higher level policy. In this framework, parameterized action primitives can be viewed as a particular type of fixed option with an initiation set that corresponds to the arguments of the primitive. Prior work on options has focused on discovering [1, 12, 41] or fine-tuning options [5, 14, 27] in addition to learning higher level policies. Many of these methods have not been extended beyond carefully engineered state based settings. More recently, research has focused on extracting useful skills from large offline datasets of interaction data ranging from unstructured interaction data [49], play [28. 29] to demonstration data [2. 35, 39, 40, 44, 46, 53J. While these methods have been shown to be successful on certain tasks, the learned skills are only relevant for the environment they are trained on. New demonstration data must be collected to use learned skills for a new robot, a new task, or even a new camera viewpoint. Since RAPS uses manually specified primitives dependent only on the robot state, RAPS can re-use the same implementation details across robots, tasks and domains.
actions for
Parameterized Actions in RL The parameterized action Markov decision process (PAMDP) formalism was first introduced in Masson et al. [31], though there is a large body of earlier work in the area of hybrid has focused on better aligning policy architectures and RL updates with the nature of parameterized actions and has largely been limited state based domains [13, 51]. A number of papers in this area have focused on solving a simulated robot soccer domain modeled as either a single-agent [19, 31, 48] multi-agent [15] problem. In this paper, we consider more realistic robotics tasks that involve interaction with and manipulation of common household objects. While prior work [42] has trained RL policies to select
control, surveyed in [7, 8]. Most recent research on PAMDPs
designed behaviors for simultaneous execution, we instead train RL policies to leverage more expressive, parameterized behaviors to solve a wide variety of tasks. Closely related to this work is Chitnis et al. [10], which develops a specific architecture for training policies over parameterized actions from state input and sparse rewards in the context of bi-manual robotic manipulation. Our work is orthogonal in that we demonstrate that higher level policy architecture is sufficient to solve a large suite of manipulation tasks from image input. We additionally note that there is concurrent work [34] that also applies engineered primitives in the context of RL, however, we consider learning from image input and sparse terminal rewards.
3 Robot Action Primitives in RL
To address the challenge of exploration and behavior learning in continuous action spaces, we decompose desired task into the what (high level task) and the how (control motion). The what is handled by the enviromment-cemriic RL policy while the how handled by a fixed, manually defined set of agent-centric primitives parameterized by continuous arguments. This enables the high level policy to reason about the task at high level by choosing primitives and their arguments while leaving the low-level control to the parameterized actions themselves.
3.1 Background
Let the Markov decision process (MDP) be defined as A, A,,,, T p()o))) 7:) in which S is the set of true states, A is the set of possible actions, R(x, a, 8') is the reward function, (s'1s,, a) is the transition probability distribution, p(sp) defines the initial state distribution, and y is the discount factor. The agent executes actions in the environment using policy xt (als)) with
3

---

corresponding trajectory distribution p(T = (su, ao -1,) = p(ss)) The goal of the RL agent is to maximize the expected sum of rewards with respect the policy: Beg 1t..:............pp... 1..7.pp((r) 2 a+)]. In the case of vision-based RI., the setup is now partially observed Markov decision process (POMDP); we have access to the true state via image observations. In this case, we include an observation space O which corresponds to the set of visual observations that the environment may emit, an observation model p(o)s) which defines the probability of emission and policy <aalo) which operates over observations. In this work, we consider various modifications to the action space A while keeping all other components of the MDP or POMDP the same.
az). (st, (
3.2 Parameterized Action Primitives
We now describe the specific nature of our parameterized primitives and how they can be integrated into RL algorithms (see Figure for an end-to-end visualization of the method). In a library of K primitives. the k-th primitive is a function fu args) that executes a controller C2 on a robot for a fixed horizon Hg, 8 is the robot state and args is the value of the arguments passed to fk- args is used to compute target robot state s* and then CA is used to drive to s". A primitive dependent error metric e (s, determines the trajectory C& takes to reach s*. C a general purpose state reaching controller, e.g. an end-effector or joint position controller; we assume access to such a controller for each robot and it is straightforward to define and tune if not provided. In this case, the same primitive implementation can be re-used across any robot. In this setup, the choice of controller, error metric and method to compute s* define the behavior of the primitive motion, how it uniquely forms movement in space. We refer to Procedure 1 for general outline of a parameterized primitive. To summarize, each skill is a feedback control loop with end-effector low level actions. The input arguments are used to define target state to achieve and the primitive executes loop to drive the error between the robot state and the target robot state to zero
As an example, consider the "lifting" primitive, which simply involves lifting the robot arm upward. For this action, args is the amount lift the robot arm, e-g. by 20cm., the robot state for this primitive is the robot end-effector position, k is the index of the lifting primitive in the library, Cs. is an end-effector controller, es (8, 8*) = 8* 8, and H₂ is the end-effector controller horizon, which in our setting ranges from 100-300. The target position 8* is computed as + [0, O, args]. I moves the robot arm for Hş steps, driving s towards. s*. The other primitives are defined in a similar manner; see the appendix for a precise description of each primitive we define.
Robot action primitives are a function of the robot state, not the world state. The primitives function by reaching set points of the robot state as directed by the policy, hence they are agent- centric. This design makes primitives agnostic to camera view, visual distractors and even the underlying environment itself. The RL policy,
Procedure Parameterized Action Primitive
Input: primitive dependent argument vector args, prim- itive index k. robot state
I: compute s (args, s)
2:
for i = L..., He low-level steps do
3:
e = es (s),
compute state error compute torques
4:
a, (e
on the other hand, is environment centric: it chooses the primitive and appropriate arguments based environment observations in order to best achieve the task. A key advantage of this decomposition is that the policy no longer has to learn how to move the robot and can focus directly on what it needs to do. Meanwhile, the low-level control need not be perfect because the policy can account for most discrepancies using the arguments.
5:
execute a, on robot
6:
end for
One issue with using fixed library of primitives is that it cannot define all possible robot motions. As result, we include a dummy primitive that corresponds to the raw action space. The dummy primitive directly takes in a delta position and then tries to achieve it by taking a fixed number of steps. This does not entirely resolve the issue as the dummy primitive operates on the high level horizon for Hş steps when called. Since the primitive is given a fixed goal for H2 steps, it is less expressive than a feedback policy that could provide a changing argument at every low-level step. For example, if the task is to move in circle, the dummy primitive with a fixed argument could not provide a target state that would directly result in the desired motion without resorting to a significant number of higher level actions, while a feedback policy could iteratively update the target state to produce a smooth motion in a circle. Therefore, it cannot execute every trajectory that a lower level policy could; however, the primitive library as whole performs well in practice.
In order to integrate these parameterized actions into the RI setting, we modify the action space of a standard RL environment to involve two operations at each time step: (a) choose a primitive

---

Primitim: LiM
Poruent grapp Argannts: 23 a.
FROM cRettal Argamats:
hust
PriaitSo udde (5g1 Cantodence 5Ph
Argaiee:: 1855
$53.0
Arganett:
Figure 2: We visualize exccution of an RL agent trained to solve a cabinet opening task from sparse rewards using robot action primitives. each time-step, display the primitive chosen, the policy's confidence in the action choice and the corresponding argument passed to the primitive in the bottom left corner.
out of a fixed library (b) output its arguments. As in Chitnis al. [10], the policy network outputs distribution over one-hot vectors defining which primitive to use well as distribution over all of the arguments for all of the primitives, design choice which enables the policy network to have a fixed output dimension. After the policy samples an action, the chosen parameterized action and its corresponding arguments are indexed from the action and passed to the environment. The environment selects the appropriate primitive function f and executes the primitive the robot with the appropriate arguments. After the primitive completes executing, the final observation and sum of intermediate rewards during the execution of the primitive returned by the environment. We do so to ensure if the task is achieved mid primitive execution, the action is still labelled successful.
We describe concrete example to ground the description of our framework. If we have 10 primitives with 3 arguments each, the higher level policy network outputs 30 dimensional mean and standard deviation vectors from which we sample 30 dimensional argument vector. It also outputs 10 dimensional logit vector from which we sample a 10 dimensional one-hot vector. Therefore in total, our action space would be 40 dimensional. The environment takes in the 40 dimensional vector and selects the appropriate argument (3-dimensional vector) from the argument vector based on the one-hot vector over primitives and executes the corresponding primitive in the environment. Using this policy architecture and primitive execution format, we train standard RL agents to solve manipulation tasks from sparse rewards. See Figure for visualization of full trajectory of policy solving a hinge cabinet opening task in the Kitchen Suite with RAPS.
Experimental Setup
In order to perform a robust evaluation of robot action primitives and prior work, we select a set of challenging robotic control tasks, define our environmental setup, propose appropriate metrics for evaluating different action spaces, and summarize our baselines for comparison.
We evaluate RAPS on three simulated domains: Metaworld [17], Kitchen [52] and Robosuite [54], containing 16 tasks with varying levels of difficulty, realism and task diversity (see the bottom half of Fig. 1). We use the Kitchen environment because it contains seven different subtasks within single setting, contains human demonstration data useful for training learned skills and contains tasks that require chaining together to four subtasks to solve. In particular, learning such temporally-extendee we evaluate on the Metaworld benchmark suite due to its wide range of manipulation tasks and established presence in the RL community. We select subset of tasks from Metaworld (see appendix) with different solution behaviors to robustly evaluate the impact of primitives on RL. Finally, one limitation of the two previous domains that the underlying end-effector control is implemented via a simulation constraint as opposed to true position control by applying torques to the robot. In order to evaluate if primitives would scale to more realistic learning setups, we test on Robosuite, benchmark of robotic manipulation tasks which emphasizes realistic simulation and control. We select the block lifting and door opening environments which have been demonstrated to be solvable in prior work [54]. We refer the reader to the appendix for detailed description of each environment.
Tasks and Environments:
behavior challenging [2, 17, 35]. Next,
We modify each task to use the environment success metric as sparse reward which returns 1 when the task is achieved, and 0 otherwise. We do so order to establish a more realistic and difficult exploration setting than dense rewards which require significant engineering effort and true state information to compute. Additionally, we plot all results against the mean task success rate since a directly interpretable measure of the agent's performance. We run each method using visual input as we wish to bring our evaluation setting closer
Sparse Reward and Image Observations
5

---

to real world setups. The higher level policy, primitives and baseline methods are not provided access to the world state, only camera observations and robot state depending on the action.
Evaluation Metrics One challenge when evaluating hierarchical action spaces such as RAPS alongside variety of different learned skills and action parameterizations, is that of defining fair and meaningful definition of sample efficiency. We could define one sample to be a forward pass through the RL policy. For low-level actions this is exactly the sample efficiency, for higher level actions this only measures how often the policy network makes decisions, which favors actions with a large number of low-level actions without regard for controller run-ting un-time cost, which can be significant. Alternatively, we could define one sample to be a single low-level action output by low-level controller. This metric would accurately determine how often the robot itself acts in the world, but it can make high level actions appear deceptively inefficient. Higher level actions execute far fewer forward passes of the policy in each episode which can result in faster execution on a robot when operating over visual observations, key point low-level sample efficiency fails to account for. We experimentally verify this point by running RAPS and raw actions on a real xArm 6 robot with visual RL and finding that RAPS executes each trajectory 32x times faster than raw actions. We additionally verify that RAPS is efficient with respect to low level steps in Figure 4.
To ensure fair comparison across methods, we instead propose to perform evaluations with respect to two metrics, namely, (a) Wall-clock Time: the amount of total time it takes to train the agent to solve the task, both interaction time and time spent updating the agent, and (b) Training Steps: the number of gradient steps taken with a fixed batch size. Wall clock time is not inherently tied to the action space and provides an interpretable number for how long it takes for the agent to learn the task. To ensure consistency, we evaluate all methods on a single RTX 2080 GPU with 10 CPUs and 50GB of memory. However, this metric is not sufficient since there are several possible factors that can influence wall clock time which can be difficult to disambiguate, such as the effect of external processes, low-level controller execution speed, and implementation dependent details. As result, we additionally compare methods based on the number of training steps, a proxy for data efficiency. The number of network updates is only a function of the data; it is independent of the action space, machine and simulator, making it a non-transient metric for evaluation. The combination of the two metries provides a holistic method of comparing the performance of different action spaces and skills operating on varying frequencies and horizons.
Baselines The simplest baseline we consider is the default action space of the environment, which we denote as Raw Actions. One way to improve upon the raw action space is to train a policy to output the parameters of the underlying controller alongside the actual input commands. This baseline, VICES [30], enables the agent to tune the controller automatically depending on the task. Alternatively, one can use unsupervised skill extraction generate higher level actions which can be leveraged by downstream RL. We evaluate one such method, Dyn-E [49], which trains an observation and action representation from random policy data such that the subsequent state is predictable from the embeddings of the previous observation and action. A more data-driven approach to learning skills involves organizing demonstration data into latent skill space. Since the dataset is guarantced to contain meaningful behaviors. it is more likely that the extracted skills will be useful for downstream tasks. We compare against SPIRL [35], a method that ingests a demonstration dataset to train a fixed length skill VAE 2 = c(01-H) ), 41-# = d(z) and prior over skills p(2[s), which is used to guide downstream RL. Additionally, we compare against PARROT [44], which trains an observation conditioned flow model on offline dataset to map from the raw action space to latent action space. In the next section, we demonstrate the performance of our RAPS against these methods across a diverse set of sparse reward manipulation tasks.
5 Experimental Evaluation of RAPS
We evaluate the efficacy of RAPS on three different settings: single task reinforcement learning across Kitchen, Metaworld and Robosuite, as well as hierarchical control and unsupervised exploration in the Kitchen environment. We observe across all evaluated settings, RAPS is robust, efficient and performant, in direct contrast to a wide variety of learned skills and action parameterizations.
In all of our results, each plot shows 95% confidence interval of the mean performance across three seeds.

---

Kitchen Kettle
Kitchen Microwave
Kitchen Light Switch
Kitchen Top Left Burner
Success Rate
lo n2- Stecesss Rate
Success Rate
Success Rate
68
bo
02-
60
29
t0
20
Wall Clock Time (hrs)
Wall Clock Time (hrs)
Wall Clock Time (hus)
Wall Clock Time (hrs)
Kitchen Hinge Cabinet
Metaworld Drawer Close
Metawoeld Peg Unplug Side
Metaworld Sweep Into
1.0
Success Rate
68
Succoss Rate
Success Rate
Succoss Rate
66
Wall Clock Time (hrs)
Wall Clock Time (hrs)
Wall Clock Time (hus)
Wall Clock Time (hrs)
Metaworld Disassemble
Metaworld Assembly
Robosuite Door
Robosuite Lift
Success Rate
Success Rate
Success Rate
Success
Wall Clock (hrs)
Wall Clock (hrs)
Wall Clock Time (hrs)
Clock Time (hrs)
RAPS (Ours)
Raw Actions
Raw Actions (Dems)
VICES
VICES (Dense)
Figure 3: Comparison of various action parameterizations and RAPS across all three environment suites using
Dreamer as the underlying RI. algorithm. RAPS (green), with sparse rewards, is able to significantly outperform
all haselines, particularly on the more challenging tasks, even when they are augmented with dense reward. See the appendix for remaining plots on al alide-cabines soccer-v2 tasks.
Kitchen Slide Cabinet
Kitchen Kettle
Kitchen Microaave
Kitchen Hinge Cabinet
Success Rate
Suecess Rate
Success Rate
Success
6K
1908 Interactions
6k
105ax Low Level Interactions
SXE
6k
5O0E
106n
5005
1000x
1930x
Low Level
Low Level
Interactions
Low Level Interactions
(Ours)
VICES
Figure 4: In the Kitchen environment suite, we run comparisons logging the number of low level interactions of RAPS, Raw actions and VICES. While the methods appear closer efficiency with respect low-level actions, RAPS still maintains the best performance across every task. We note that on real robot, RAPS runs significantly faster than the raw action space terms of wall-clock time.
5.1 Accelerating Single Task RL using RAPS
In this section, we evaluate the performance RAPS against fixed and variable transformations the lower-level action space well as state of the art unsupervised skill extraction from demonstrations. Due to space constraints, we show performance against the number of training steps in the appendix.
We compare RAPS against Raw Actions and VICES using Dreamer [18] as the underlying algorithm across all three environment suites in Figure 3. Since the default action space of Kitchen, joint velocity control, instead modify the suite to use 6DOF end-effictor control for both raw actions and VICES. We find Raw Actions and VICES are able to make progress number of tasks across all three domains, but struggle to execute the fine- mme-graieed manipulation required to solve more difficult environments cabinet, such as hinge-cabinet, assembly-v2 and di sassembly-v2 The latter two environments not solved by Raw Actions or VICES even when they are provided dense rewards. In contrast, RAPS able to quickly solve every task from sparse rewards.
Action Parameterizations
observe weak performance
the kitchen environment, from sparse rewards, no prior method makes progress on the hardest manipulation task: grasping the hinge cabinet and pulling it open to 90 degrees, while RAPS is able
7

---

Kitchen Kettle
Kitchen Slide Cabinet
Kitchen Top Left Burner
Kitchen Hinge Cabinet
Success Rate
1.00
Success Rate
Success Rate
Success Rate
0073-
875
8.5
0.23
0.23
0.2
10
20
Wall Clock Time (hrs)
Wall Clock Time (hrs)
Wall Clock Time (hrs)
Wall Clock Time (hrs)
RAPS (Ours)
SPIRI
Dyn-H
PARROT
Dreamer PARROT
Figure 5: Comparison of RAPS and skill lcarning methods the Kitchen domain using SAC as the underlying
RL algorithm. While SPIRL and PARROT are competitive or even improve upon RAPS's performance on casier
tasks, only RAPS (green) is able to solve top-left- seff-bbutt burner and hinge-cabinet.
Kettle
Slide Cabinet
Light Switch
Microwave
Top Burner
Hinge Cabinet
RL Algorithm
Raw
RAPS
Raw
RAPS
Raw
RAPS
Raw
RAPS
Raw
RAPS
Raw
RAPS
Dreamer
0.8
.93
1.0
1.0
1.0
1.0
.53
0.8
.93
1.0
0.0
1.0
SAC
.33
0.8
.67
1.0
.86
.67
.33
1.0
.33
1.0
0.0
1.0
PPO
.33
1.0
.66
1.0
.27
1.0
0.0
.66
.27
1.0
0.0
1.0
Table 1: Evaluation of RAPS across RL algorithms (Dreamer, PPO, SAC) Kitchen. We report the final
success rate each method on five evaluation trials trained over three seeds from sparse rewards. While raw action performance (left entry) varies significantly across RL algorithms, RAPS (right entry) is ablc to achieve high success rates on every task with every RL algorithm.
to quickly learn solve the task. In the Metaworld domain, peg-umplug-sides side-v2, assembly-v2 and di disassembly yy v2 are difficult environments which present challenge to even dense reward state based RL [52]. However, RAPS is able to solve all three tasks with sparse rewards directly from image input. We additionally include a comparison of RAPS against Raw Actions on all 50 Metaworld tasks with final performance shown in Figure as well as the full learning performance in the Appendix. RAPS is able to learn to solve or make progress on 43 out of 50 tasks purely from sparse rewards. Finally, in the Robosuite domain, by leveraging robot action primitives, we are able to learn to solve the tasks more rapidly than raw actions or VICES, with respect to wall-clock time and number of training steps, demonstrating that RAPS scales to more realistic robotic controllers.
Offine Learned Skills An alternative point of comparison is to leverage offline data to learn skills and run downstream RL. We train SPIRL and PARROT from images using the kitchen demonstration datasets in D4RL [16]. and Dyn-E with random interaction data. We run all agents with SAC as the underlying RL algorithm and extract learned skills using joint velocity control, the type of action present in the demonstrations. See Figure 5 for the comparison of RAPS against learned skills. Dyn-E unable to make progress across any of the domains due to the difficulty of extracting useful skills from highly unstructured interaction data. In contrast, SPIRI and PARROT manage to leverage demonstration data to extract useful skills; they are competitive or even improve upon RAPS on the casier tasks such as microwave and kettle, but struggle to make progress on the more difficult tasks in the suite. PARROT, in particular, exhibits a great deal of variance across tasks, especially with SAC, so we include results using Dreamer well. We note that both SPIRL and PARROT are limited by the tasks which are present in the demonstration dataset and unable to generalize their extracted skills to other tasks in the same environment other domains. In contrast, parameterized primitives are able to solve all the kitchen tasks and are re-used across domains as shown in Figure 3.
Generalization to different RL algorithms A generic set of skills should maintain performance regardless of the underlying RI. algorithm. In this section, we evaluate the performance of RAPS against Raw Actions on three types of RI. algorithms: model based (Dreamer), off-policy model free (SAC) and on-policy model free (PPO) on the Kitchen tasks. We use the end-effector version of raw actions as our point of comparison on these tasks. As seen in Table 1, unlike raw actions, RAPS is largely agnostic to the underlying RL algorithm.
5.2 Enabling Hierarchical Control via RAPS
We next apply RAPS to more complex setting: sequential RL., in which the agent must learn to solve multiple subtasks within single episode, as opposed to one task. We evaluate on the Kitchen Multi-Tate Task environments and plot performance across SAC, Dreamer, and PPO in Figure 7.
8

---

Durion thoss
Door Close
Dox Lock
Dower Clos
Dowe Qpen
Tummer
Puile
Pulls
Honde hil
Roch
Reach Wall
Windon Clos
Windew Open
Batlon Preve T%:
Codfee hi
Dinocablc
Dial
Rutton
Hodde
Dnton
R.
Shot Hoo
Swez Inaa
Door Open
Door Uniock
Foace Cloc Cloc
Paoot Open
Pog Tephing
Shide
Place Stoles
Plate Stide
Paak Back
Pub Wul
Box Close
Nouenbly
Bakchall
Skak Dul
Swoop
Stell Mace
Bio Picking
Pick
Pirk
RAPS
Raw Actioms
Figure 6: Final performance results for single task RL on the Metaworld domain after 3 days of training using the Dreamer base algorithm. RAPS is able to successfully learn most tasks, solving 43 out of 50 tasks while Raw Actions only able to solve tasks.
Kitchen Sequential Multi-Task
Kitchen Sequemial Multi-Task
Sequemial Multi-Task
Kitchen Sequential
Sultala Salved
Sahhaaa Solved
Sohtokk Solvod
Sabaakk Solved
SAC
vis
vik
Time (hrs)
Wall Clock Time (hrs)
Updates
Updates
Sequential
Sequential Multi-Task
Sequentia
Soquential Mahii-Task
Dreamer d Suhuols Salved
Solved
Subtodss Selved
Subtodks Solved
300
Dkkk
246
Clock Time (hrs)
Wall Clock Time (hrs)
Number Updates
Number Updatess
Soquential Multi-Tick
Sequential Multi-Task
Kitchen Sequential MultiTaskk
Kitchen Sequeniaa Multi-Task
Sabcalls Salved
Salved
Sabtales Salved
PPO
Lse
22
40
123
riol
Wall Clock Time (hrs)
Wall
Clock Time (hrs)
Nomber of Updates
Number Updates
RAPS (Ours)
Raw Actions EE
SPIRI
Dyn-E
PARROT
Figure 7: Learning performance of RAPS on sequential multi-task RL. Each row plots different base RL algorithm (SAC, Dreamer. PPO) while first two columns plot the two multi-task environment results against wall-clock time and the next two columns plot against number of updates, ie. training steps. RAPS consistently solves at least three out of four subtasks while prior methods generally to make progress beyond one or two.
Raw Actions prove to be a strong baseline, eventually solving close to three subtasks on average, while requiring significantly more wall-clock time and training steps. SPIRL initially shows strong performance but after solving one to two subtasks then plateaus and fails improve. PARROT is less efficient than SPIRL but also able to make progress up to two subtasks, though it exhibits great deal of sensitivity to the underlying RL algorithm. For both the offline skill learning methods, they struggle to solve any of the subtasks outside of kettle, mi microwave, and clide-cabing which are encompassed in the demonstration dataset. Meanwhile, with RAPS, across all three base RL algorithms, we observe that the agents are able to leverage the primitive library rapidly solve three out of four subtasks and continue to improve. This result demonstrates that RAPS can elicit significant gains in hicrarchical RL performance through its improved exploratory behavior.

---

Kitchen Top Left Burner
Kitchen Hinge Cabinet
Kitchen Top Burner
Kitchen Hinge Cabinet
Success Rate
Success Rate
Success Rate
Success Rate
10 15
ok
sic
42k
six
>ik
six
100k
Wall Clock Time (hrs)
Wall Clock Time (hrs)
Number of Updates
Number Updates
RAPS (Ouns)
Raw Actions
Figure 8: RAPS significantly outperforms raw actions in terms of total wall clock time and number of updates when fine-tuning initialized from reward free exploration.
5.3 Leveraging RAPS enable efficient unsupervised exploration
In many settings, sparse rewards themselves can be hard to come by. Ideally, we would be able to train robot without train time task rewards for large periods of time and fine-tune to solve new tasks with only few supervised labels. We use the kitchen environment to test the efficacy of primitives on the task of unsupervised exploration. We run an unsupervised exploration algorithm, Plan2explore [38], for a fixed number of steps learn a world model, and then fine-tune the model and train a policy using Dreamer to solve specific tasks. We plot the results in Figure 8 on the aop-lef-buurn and hinge-cabinet tasks. RAPS enables the agent to learn an effective world model that results in rapid learning of both tasks, requiring only hour of fine-tuning to solve the hinge-cabinet task. Meanwhile, the world model Iearned by exploring with raw actions is unable to quickly finetune as quickly. We draw two conclusions from these results, RAPS enables more efficient exploration than raw actions, b) RAPS facilitates efficient model fitting, resulting in rapid fine-tuning.
Discussion
Limitations and Future Work While demonstrate that RAPS is effective at solving a diverse array of manipulation tasks from visual input, there are several limitations that future work would need address. One issue consider that of dynamic, fluid motions. Currently, once a primitive begins executing, it will not stop until its horizon is complete, which prevents dynamic behavior that a feedback policy on the raw action space could achieve. In the context of RAPS, integrating the parameterization and environment agnostic properties of robot action primitives with standard feedback policies could be one way to scale RAPS more dynamic tasks. Another potential concern is that of expressivity: the set of primitives we consider in this work cannot express all possible motions that robot might need to execute. As discussed in Section 3, we do combine the base actions with primitives via a dummy primitive so that the policy can fall back to default action space if necessary. Future work could improve upon our simple solution. Finally, more complicated robot morphologies may require significant domain knowledge in order to design primitive behaviors. In this setting, we believe that learned skills with the agent-centric structure of robot action primitives could be an effective way to balance between the difficulty of learning policies to control complex robot morphologies [4, 33] and the time needed to manually define primitives.
Conclusion In this work we present an extensive evaluation of RAPS, which leverages parameter- ized actions to learn high level policies that can quickly solve roboties tasks across three different environment suites. We show that standard methods of re-parameterizing the action space and learning skills from demonstrations are environment and domain dependent. In many cases, prior methods are unable to match the performance of robot action primitives. While primitives are not general solution to every task, their success across wide range of environments illustrates the utility of incorporating an agent-centric structure into the robot action space. Given the effectiveness of simple parameterized action primitives, promising direction to further investigate would be how to best incorporate agent-centric structure into both learned and manually defined skills and attempt to get the best of both worlds in order to improve the interface of RL algorithms with robots.
Acknowledgments We thank Shikhar Bahl, Ben Eyesenbach, Aravind Sivakumar, Rishi Veerapa- neni, Russell Mendonca and Paul Liang for feedback on early drafts of this paper. This work was supported in part by NSF IIS1763562 NSF IIS IIS-2024594, and ONR Grant 000014181286 and the US Army. Additionally, MD is supported by the NSF Graduate Fellowship. We would also like to acknowledge NVIDIA's GPU support.
10

---

References
[1] J. Achiam, H. Edwards, D. Amodei, and Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 10X1:1807.10299, 2018. 3
[2] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning, 2021. 3,5
[3] A. Allshire, R. Martín-Martín, C. Lin, S. Manuel, S. Savarese, and A. Garg. Laser: Learning a latent action space for efficient reinforcement learning. arXiv preprini arXiv:2103. 157933 2021.
2
[4] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, al. Learning dexterous in-hand manipulation. The Interna- lional Journal of Robotics Research, 39(1):3-20, 2020. 1, 10
[5] P.-.. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[6] S. Bahl, M. Mukadam, A. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensorimotor learning, 2020. 3
[7] A. Bemporad and M. Morari. Control of systems integrating logic, dynamics, and constraints. Automatica, 35(3):407-427, 1999.
[8] M. S. Branicky, V. S. Borkar, and S. K. Mitter. A unified framework for hybrid control: Model and optimal control theory. IEEE transactions on automatic control, 43(1):31 1-54, 1998. 3
[9] Cambon, R. Alami, and F. Gravot. A hybrid approach to intricate motion, manipulation and task planning. The International Journal of Robotics Research, 28(1):104-11) 126, 2009.
[10] R. Chitnis, S. Tulsiani, S. Gupta, and A. Gupta. Efficient bimanual manipulation using learned task schemas. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1149-1155. IEEE, 2020. 3,5
[11] C. Daniel, Neumann, O. Kroemer, J. Peters, al. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 17:1-50, 2016. 3
[12] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint 1802.06670 2018. 3
[13] Z. Fan, R. Su, W. Zhang, and Y. Yu. Hybrid actor-critic reinforcement learning in parameterized action space. arXiv preprint 1931v.190.003344, 2019. 3
[14] K. Frans, J. Ho, X. Chen, P. Abbeel, and J. Schulman. Meta learning shared hierarchies. arXiv 2017. 2,
preprint arXiv:1710.0090
[15] H. Fu, H. Tang, J. Hao, Z. Lei, Y. Chen, and C. Fan. Deep multi-agent reinforcement learning with discrete-continuord hybrid action spaces. arXiv preprint UrXiv:1903.04959, 2019.
[16] Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven
reinforcement learning, 2021. 8
[17] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning, 2019. 5
[18] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination, 2020.
[19] M. Hausknecht and P. Stone. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv: 2015. 2,
[20] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embedding space for transferable robot skills, In International Conference on Learning Representations, 2018.

---

[21] A. J. Ijspeert, J. Nakanishi, and S. Schaal. Learning attractor landscapes for learning motor primitives. Technical report, 2002. 3
[22] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In 201/ IEEE International Conference Robotics and Automation, pages 1470-1477. IEEE, 2011 1. 2,
3
[23] L. P. Kaelbling and T. Lozano-Pérez. Learning composable models of parameterized skills. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 886-893. IEEE, 2017. 3
[24] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-baat -bssed robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[25] Kober and J. Peters. Learning motor primitives for robotics. In 2009 IEEE International Conference on Robotics and Automation, pages 1818.2118. IEEE, 2009. 3
[26] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334- 2016.
[27] A. C. Li, C. Florensa, I. Clavera, and P. Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. arXiv preprint arXiv:/906. 05862, 2019. 2,3
[28] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020.
3
[29] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and Sermanet. Learning latent plans from play. In Conference on Robot Learning, pages 1 13-1132. PMLR, 2020. 2, 3
[30] R. Martín-Martín, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks, 2019. 6
[31] W. Masson, P. Ranchod, and G. Konidaris. Reinforcement learning with parameterized actions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[32] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. arXiv preprint
2018. 2
[33] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning
dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020. 10
[34] S. Nasiriany, H. Liu, and Y. Zhu, Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks, 2021. 3
[35] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors, 2020. 3, 6,
[36] J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, 2010.
[37] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid roboties. In Adaptive motion of animals and machines, pages 261-280. Springer, 2006. 3
[38] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning explore via self-supervised world models, 2020. 10
[39] T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pages 8624-8633. PMLR, 2020. 3
[40] Shankar, S. Tulsiani, L.. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference Learning Representations, 2019. 2,
12

---

[41] A. Sharma, S. Gu, S. Levine, V. Kumar. and K. Hausman. Dynamics-awar aware unsupervised discovery of skills. arXiv preprint arXiv:1907.00 07670007,, 2019. 2.
[42] M. Sharma, J. Liang, J. Zhao, LaGrassa, and O. Kroemer. Learning to compose hierarchical object-centric controllers for robotic manipulation. arXiv preprinl arXiv:2011 200::20011.4477, 2020, 3
[43] A. Simeonov, Y. Du, B. Kim, F. R. Hogan, J. Tenenbaum, P. Agrawal, and A. Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. arXiv preprins arXiv:2011.08177 2020. 3
[44] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinchart, and Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10024 2020. 3, 6
[45] R. S. Sutton, D. Precup, and S. Singh. Between mdps and cemi-mdps: A framework for temporal
abstraction reinforcement learning. Artificial intelligence, 1 112(1-2):18 81-211, 1999. 3
[46] D. Tanneberg, K. Ploeger, E. Rueckert, and J. Peters. Skid raw: Skill discovery from raw trajectories. IEEE Robotics and Automation Letters, 63):4666-4700) 2021. 3
[47] S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549, PMLR, 2017.
[48] E. Wei, D. Wicke, and S. Luke. Hierarchical approaches for reinforcement learning in parame- terized action space. arXiv preprint arXiv:1810.099 2018. 3
[49] W. Whitney, R. Agarwal, K. Cho, and A. Gupta. Dynamics-aware embeddings, 2020. 3, 6
[50] K. Xie, H. Bharadhwaj, D. Hafner, A. Garg, and F. Shkurti. Latent skill planning for exploration and transfer. In International Conference on Leurning Representations, 2020.
[51] J. Xiong, Q. Wang. Z. Yang, P. Sun, L. Han, Y. Zheng, H. Fu, T. Zhang, J. Liu, and H. Liu. Parametrized deep nettworkss learning: Reinforcement learning with discrete-continuuus hybrid action space. arXiv preprint arXiv: 1810.06394, 2018.
[52] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094- 1 1100. PMLR, 2020. 5, 8
[53] W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint w:Xiv:20/1.072.0 2020.
[54] Y. Zhu, J. Wong, A. Mandlekar, and R. Martín-Martn robosuite: A modular simulation framework and benchmark for robot learning. 2020. 5
13


Reviews:
[{"review_id": "QHKND8mq4R", "reviewer": "Reviewer_mpsj", "summary": "This paper proposes a novel action space for efficient learning of robotic manipulation tasks. In contrast to typical joint space and task space control, the proposed action space consists of a set of hand-designed agent-centric primitive skills (e.g. lift arm for $d$ cm, close grippers to $d$ cm, and rotate wrist for $\\theta$) with continuous parameters (e.g. $d$ and $\\theta$). Since the primitive skills are agent-centric (i.e. conditioned only on a robot state), the primitive skills can be used across diverse environments and tasks. With this appropriately designed action space, a high-level policy learns to choose which primitive skill and corresponding parameters to execute from visual inputs and sparse rewards.\n\nThe exhaustive experiments on three robotic manipulation benchmarks and three different RL algorithms demonstrate improved sample efficiency of the proposed action space. The proposed method outperforms baselines with joint space control, task space control, and learned skills from offline data or random interactions.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: This paper proposes a novel action space that consists of multiple manually designed primitive skills, each of which is parameterized with continuous variables. This idea is somewhat similar to the idea of motion primitives but could be designed more general to different environments and tasks. \n\n**Clarity**: The paper is clearly written and easy to follow. It would be great if more details about the primitive skills and how to derive this set of skills are discussed in the main paper.\n\n**Quality**: The experiments are thorough and the proposed method successfully tackles many challenging manipulation tasks, which is impressive. \n\n**Significance**: The primitive skills are tailored to the robots and tasks, which requires domain knowledge and human efforts for designing them. The paper claims that once the primitive skills are designed, they can be transferred to other tasks and environments, but it might be still limited to similar tasks and environments, which may require new set of primitive skills for new tasks and environments. However, the proposed method seems timely as this is a simpler alternative of widely used task space and joint space control, and provides significant improvements in exploration and sample efficiency.\n\nThe paper claims that the proposed action space is suited to learn from visual inputs and sparse rewards. Intuitively, this is true as the set of well-designed primitive skills can always provide meaningful transitions, so reduce the exploration burden. However, this is generally very challenging to learn a policy from high-dimensional visual inputs and sparse reward. Especially given that the action space is not smaller than the original action space (either joint torque control or end-effector control), training the high-level policy may require significant engineering efforts. Could you elaborate whether this was not the case or there were some important implementation or training details.\n\nThe proposed action space is tailored to the robots and tasks. The reviewer wonders how diverse set of tasks can be covered with the proposed action space, which set of tasks cannot be handled with the proposed action space (e.g. velocity, force sensitive tasks like catching a ball, pouring, etc), and how this action space is scalable, as more number of primitive skills and parameters would be required for complex tasks but it makes learning of the high-level policy difficult.\n\nIt is great to compare sample efficiency in terms of both \"wall clock time\" and \"training steps\". However, the comparisons on \"wall clock time\" could be highly dependent on implementation of each method and environment. Moreover, the \"training steps\" can be prone to the proposed method as the proposed method uses longer-horizon low-level skills compared to baselines. If we want to simulate real-world scenarios, the environmental steps (the number of low-level interactions) can be the best evaluation metric as real-world interaction is the most slow, dangerous, and costly component in real-world RL. It would be great to see the comparisons over the number of interactions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FPu3XzRqjuz", "reviewer": "Reviewer_cseK", "summary": "Low-level actions, such as torques or joint-velocities, provide a hard-exploration challenge for applying reinforcement learning (RL) to robotic systems. For example, it might be impossible to reach and grasp an object solely due to random torques or joint velocities. To overcome this difficulty, one can define high-level actions to enable a meaningful exploration. In this paper, the authors introduce parametrized action primitives, which encode a robotic movement and use those actions for learning with RL.\nThe parametrized actions are just manually encoded primitives, such as moving the arm up, reaching a position $x$ and grasp, and so on.\nTherefore, the system proposed by the authors requires expert knowledge of the system, the task, and robotics.\nThis simplification of the task results, according to the paper, in a more efficient RL. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Albeit I strongly agree with the authors that RL cannot be performed on \"low-level\" action (especially if one wants to learn directly on a real system), I struggle to find any novelty or contribution.\n\nThe proposed parameterized actions are - in my understanding - just mere trajectory planners. I do not find any novelty in their introduction nor their use in RL. It is not surprising that the proposed framework is more efficient than baselines since the framework requires much more human effort. The authors argue that these action primitives are easy to implement (perhaps easy for a person who can program and understand a robotic system) and transferable between robots. \nNaturally, the authors claim that this framework works with different RL techniques, such as model-based, model-free. Again, this is not surprising since what the authors are doing is to frame a different MDP where instead of low-level actions, we have high-level ones.\nI wonder why the authors did not choose perhaps a smarter parametrization of robotic movements. The literature about movement primitives is abundant. Dynamic movement primitives (DMPs) [1] or probabilistic movement primitives (ProMPs) [2] are good examples of parametric movements that can be used along with RL. Traditionally, movement primitives can be automatically extracted by existing data using simple regression and segmentation techniques when needed.\n\nIn short, the authors propose just a higher-level action MDP, where any RL technique will benefit the abstraction. The method is not novel, does not give any new insights about RL or robotics. In my opinion, the proposed solution completely lacks novelty. Movement primitives have already been used in the context of reinforcement learning [3, 4]. \nI would recognize some novelty if the authors had proposed a novel technique for extracting movement primitives, or a new reinforcement learning technique specifically designed to use movement primitives.\nIn my perspective, none of the two has been done. \n\n-------------- UPDATE ---------------\nAfter the discussion with the authors, I am willing to give more credit to their work. \n\n\n[1] Ijspeert, Auke Jan, et al. \"Dynamical movement primitives: learning attractor models for motor behaviors.\" Neural computation 25.2 (2013): 328-373.\n[2] Paraschos, Alexandros, et al. \"Probabilistic movement primitives.\" Advances in neural information processing systems (2013).\n[3] Stulp, Freek, and Stefan Schaal. \"Hierarchical reinforcement learning with movement primitives.\" 2011 11th IEEE-RAS International Conference on Humanoid Robots. IEEE, 2011.\n[4] Li, Zhijun, et al. \"Reinforcement learning of manipulation and grasping using dynamical movement primitives for a humanoid-like mobile manipulator.\" IEEE/ASME Transactions on Mechatronics 23.1 (2017): 121-131.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations of their work.\nThe societal impact was not addressed, but, in all fairness, I believe was not necessary. ", "review_text": "", "extra_scores": {"ethical_concerns": "I do not recognize any ethical concern with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "92cY0LzfnIn", "reviewer": "Reviewer_WUWx", "summary": "The authors used a hierarchical setup to learn complex manipulation tasks. Instead of using raw joint controls, the the action space is composed of a selection of primitive actions including lifting, pushing, and dummy primitive action ,etc. Then at high level, a learned policy decides to choose which motion primitives and outputs the corresponding parameters. The authors demonstrated that using their method, RAPS, the robot arm can solve many benchmark manipulation tasks within shorter wall clock time, significantly outperform a few STOA baselines. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of this paper is that, using the RAPS architectures, many challenging tasks can be solved effectively in sim after a few hours of training. By constraining the action outputs to the motion primitive space, the framework is also insensitive to the specific learning algorithms being used. The paper is generally well written, though a few details warrant more discussion.  The following are may comments to address:\n\n(1) The dummy primitive. The authors said \"As the dummy primitive operates on the high level horizon for Hk steps when called. Therefore, it cannot execute every trajectory that a lower level policy could. \" This is a bit confusing. If the dummy primitive is already the raw 6D end-effector pose, together with the gripper state they should represent all possible states of the manipulator. If the primitive cannot be finished within \"Hk\" steps, one can simply resume the same inputs again. I think more clarifications are needed here.\n\n(2) The parameterization of the primitives. The authors mentioned that the arguments of the primitives can be sampled from a learned distribution. However, the dimension of each primitive's arguments is still different, even if one uses a multi-dimensional Gaussian. More discussions are needed to clarify how arguments are encoded in this scenario. \n\n(3) Choices of primitives. I think a detailed discussion on the importance of each primitive is needed. For example, in order to solve a single task, how many primitives are effectively used? What is the usage percentage/activation time for each of them? Also, a bar plot showing how each primitive is selected over time would be interesting. \n\n(4) On top of (3), one may perform an ablation study on the minimum set of primitives needed to solve a particular task effectively.\n\n(5) Cross robot transfer. These primitives used in appendix seem to be agnostic to robot geometry (unlike joint pose spec). Can we demonstrate how well a policy trained with one robot can be transferred to another robot that is morphologically different?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers had some concerns that were all addressed by the authors. Especially, the authors did new experiments and provided a lot of new results. All these new results are extremely useful to support the claims of the authors and *should be included* in the final version of the paper. For this reason, the paper will have to be significantly revised but, on the basis of the discussion, the reviewers are confident that this final version will describe a strong enough contribution. 
Decision: Accept (Poster)