Deep Black-Box Reinforcement Learning with Movement Primitives
Authors: Fabian Otto
Onur Celik
Hongyi Zhou
Hanna Ziesche
Vien Anh Ngo
Gerhard Neumann
Venue: CoRL 2022 Poster (2022)
Keywords: Movement Primitives
Episode-Based
Black-Box
Deep Reinforcement Learning
Trust Regions

Abstract: Episode-based reinforcement learning (ERL) algorithms treat reinforcement learning (RL) as a black-box optimization problem where we learn to select a parameter vector of a controller, often represented as a movement primitive, for a given task descriptor called a context. ERL offers several distinct benefits in comparison to step-based RL. It generates smooth control trajectories, can handle non-Markovian reward definitions, and the resulting exploration in parameter space is well suited for solving sparse reward settings. Yet, the high dimensionality of the movement primitive parameters has so far hampered the effective use of deep RL methods. In this paper, we present a new algorithm for deep ERL. It is based on differentiable trust region layers, a successful on-policy deep RL algorithm. These layers allow us to specify trust regions for the policy update that are solved exactly for each state using convex optimization, which enables policies learning with the high precision required for the ERL. We compare our ERL algorithm to state-of-the-art step-based algorithms in many complex simulated robotic control tasks. In doing so, we investigate different reward formulations - dense, sparse, and non-Markovian. While step-based algorithms perform well only on dense rewards, ERL performs favorably on sparse and non-Markovian rewards. Moreover, our results show that the sparse and the non-Markovian rewards are also often better suited to define the desired behavior, allowing us to obtain considerably higher quality policies compared to step-based RL.

Deep Black-Box Reinforcement Learning with Movement Primitives
Fabian Otto Onur Celik Hongyi Zhou', Hanna Ziesche', Ngo Anh Vien', and Gerhard Neumann"
1 Bosch Center for Artificial Intelligence, Germany
University of Tübingen, Germany Autonomous Learning Robots, Karlsruhe Institute of Technology, Germany fabian. www.ottscc.ccooooooooooo com
Abstract: Episode-based reinforcement learning (ERL) algorithms treat rein- forcement learning (RL) as a black-box optimization problem where we learn to select a parameter vector of controller, often represented as movement primi- tive, for given task descriptor called a context. ERL offers several distinct bene- fits in comparison step-based RL. It generates smooth control trajectories, can handle non-Markovian reward definitions, and the resulting exploration in param- eter space is well suited for solving sparse reward settings. Yet, the high dimen- sionality of the movement primitive parameters has so far hampered the effective use of deep RL methods. In this paper, we present new algorithm for deep ERL. It is based on differentiable trust region layers, a successful on-policy deep RL algorithm. These layers allow to specify trust regions for the policy update that are solved exactly for each state using convex optimization, which enables poli- cies learning with the high precision required for the ERI. We compare our ERL. algorithm to state-of-the-art step-based algorithms in many complex simulated robotic control tasks. In doing so, we investigate different reward formulations dense, sparse, and non-Markovian. While step-based algorithms perform well only on dense rewards, ERL performs favorably on sparse and non-Markovian re- wards. Moreover, our results show that the sparse and the mon-Markovian rewards are also often better suited to define the desired behavior, allowing us to obtain considerably higher quality policies compared step-based RL.
Keywords: Movement Primitives, Deep Episode- Based RL, Trust Region Layers
Introduction
Reinforcement learning (RL) problems be viewed from a step-based [1, 2, 3] and an episode- based perspective [4, 5, 6]. the former, most commonly found in deep RL, a policy selects an action for each state of the trajectory. Consequently, step-based RL requires Markovian reward definitions. Moreover, the exploration in action space often induces a random walk that inadequately explores the entire behavior space of the agent. As result, most step-based RL algorithms only work well with dense rewards, where the agent receives a reward signal at each time step, rather than only the last step. We refer to the latter sparse reward setting.
In the episode-based based reinforcement learning (ERL) perspective, we choose the complete behav- ior during the episode with respect to the controller parameters in the beginning of the episode. Typically, simple controllers are used that are valid only for executing single trajectory, such as movement primitives (MPs). At the beginning of an episode, the MP parameters are adapted to context vector that serves as task descriptor and may contain e, g. different initial joint configura- tions, the goal to reach, or start positions as well as object and obstacles locations. ERL allows for efficient exploration of the behavior space because exploration is implemented in the parameter space of the MP, allowing these algorithms to learn from sparse or even non-Marke Non-Markovian rewards. In addition, ERL inherently generates smooth control trajectories due the use of MPs, which often leads to more energy efficient behavior. However, ERL has so far not gained much popularity as
6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand.

---

in 1,2,..,,
Context c
Policy (o(lee)
Desired Trajectory V
Controller (8),
Or
(of. of.
Environment
8e+1
Figure 1: Overview of the proposed Black-Box Reinforcement Learning (BBRL) framework. The normal distributed policy predicts, given the context c, the parameters a movement primitive that translates to desired trajectory r A trajectory tracking controller f generates low-level actions a, given the current 8, and the desired state of from the trajectory 7".
it does not exploit the time-series structure of the RL problem and is therefore considered less data efficient, at least in the dense reward setting. So far, there is no deep learning method to adapt the MP parameters to the context, which limits ERL to linear adaptation strategies [7, 8].
This paper offers two main contributions: We propose new algorithm for deep ERL. The high- dimensional action space imposed by the MP parameters requires highly accurate action selection, hence robust and precise policy updates are needed. We adapt a recent step-based RL algorithm based on differentiable trust region projection layers (TRPLs) [9] to the ERL setting, Unlike meth- ods such as proximal policy optimization (PPO) [2], that are based on approximated trust regions, TRPL offers exact trust regions for the policy updates which is essential for learning precise policies in high-dimensional action spaces. As second contribution, (ii) we show that ERL can be superior to step-based RL for tasks that are more easily defined by sparse or non-Markovian rewards than by dense reward. To this end, we have investigated different reward formulations for variety of complex simulated robot control tasks. Our experiments show that step-based algorithms have difficulties learning high-quality policies in sparse non-Markovian reward settings, while our pro- posed algorithm scales well to these reward settings. We further show that dense Markovian rewards can be very restrictive in terms of specifying desired behavior in domains. For example, if want to achieve energy efficient, time-optimal reaching motions, sparse rewards can be used that only penalize the distance the goal in the last time step. Non-Markovian rewards useful for tasks with inherent mor-Moroovinn optimality descriptors, e. g.g., the maximum height during jump or the minimum distance of bat to ball. As consequence, step-based based RL performs poorly even after intensive engineering of the used dense rewards.
algorithm master tasks where
In summary, our findings show that our algorithm has the following benefits: (a) it produces in- herently smooth control policies, (b) offers efficient exploration in parameter space, which in turn enables learning with (c) sparse and non-Markovian rewards. These rewards also (d) allow for more direct specification of the desired behavior, which leads to (e) higher quality of the learned behavior. These benefits come at the cost of an increased data complexity (a factor of 2 to 3).
2 Background and Related Work
Black-Box Reinforcement Learning. Contextual episode-based policy search [4, 8] treats RI. as black-box optimization problem where a contextual search distribution ((wcc) over the controller parameters ur is optimized to maximize the expected return R(w, c), i.e.,
arg max [Ex((re)][ (R(w, c)]] =(wcc)
where p(c) denotes the context distribution given by the task. Due the black-box nature of the problem, no structural assumptions are made for the return function R(w, c), i.e., the return can be any non-Markovian function of the resulting trajectory. Most ERL algorithms only consider the non-contextual setting, where different optimization techniques have been used, such as pol- icy gradients [10], natural gradients [11], stochastic search strategies [12, 13, 14], or trust-region optimization techniques [5, 6, 8]. The only methods that incorporate context adaptation [8, 14] con- sider linear mapping from context to parameter space. We, in contrast, consider highly nonlinear context-parameter relationships using deep learning.
As alternative gradient- and step-based methods recent works [15, 16, 17] also proposed using full gradient-fres black-box approach for finding the optimal neural network parameters. They typi- cally consider learning the neural network policy parameters as the black-box optimization problem

---

and leverage the episode performance for evaluating network configurationss While these methods can be competitive for some tasks, they are still difficult to use for most contextual setups as they require rollouts with different neural network parameters to be comparable. Yet, in the contextual case the performance of the rollout does not only depend on the neural network parameters but also on the context (c. g. the goal) which makes the evaluation much more noisy. For example, a good neural network parametrization could still yield a poor performance because has been evaluated in a hard context. The above approaches are completely ignorant to the context and, therefore, cannot attribute the performance differences to the context.
In contrast, our setting does not perform black-box optimization on the level of a global neural network control policy with several thousand parameters but on the level of local control parameters (in the range of 20-50 dimensions). While we, unlike the above linear approaches, still use a neural network policy with high number of parameters to adapt the local control parameters to the context, parameters are not treated as black-box. Instead, the DNN policy is updated using policy gradient, which utilizes the context information as well as the derivatives of the policy.
Step-Based Reinforcement Learning. Unlike episode- based methods, step-based approaches max- imize the expected return by optimizing an oction-selection policy that chooses a new action in each time step of the trajectory. While the goal remains learn optimal trajectories, they interact directly with the environment based on raw actions, such as positions, velocities, or torques generated from the current state information. One such method is the trust region policy optimization algorithm [18], which first introduced trust region methods to deep RL, but is rather complex and difficult to scale. PPO [2] simplifies the update by introducing a clipping heuristic the policy gradient ob- jective. An alternative to the above policy gradient methods are policy iteration based approaches [19].
Differentiable Trust Region Layer. Differentiable trust region projection layers (TRPLs)) [9] present a scalable and mathematically sound approach for enforcing trust regions in step-based deep RL. While PPO [2] is also motivated by trust-region updates of the policy, cannot enforce the trust region exactly. TRPL provides an efficient way to enforce a trust region as well as more stability and control during training, while reducing the dependency on code level choices [20].
The layer takes the output of a standard Gaussian policy as input in terms of mean fu and variance Σ and projects it into the trust region if the given mean and variance violate their respective bounds. This projection is done for each input state individually. Subsequently, the projected Gaussian pol- icy distribution with parameters μ. Σ is used for any further steps, e. g. for sampling and/or loss computation. Formally, the layer solves the following two optimization problems for each state
s.t. dream (jhss Mulal S Cpe. and arg min dinean (ps, (())),
arg min dewn (S., E(s)) s.t. dear Enulss) S cx, E,
where u. and Σ, are the optimization variables for input state s and Ep and cy are the trust region bounds for mean and covariance, respectively. Finally, dinean and deov are the similarity metrics for the mean and covariance of a decomposable distance or divergence measure. Otto et al. [9] proposed three such measures, we will, however, only leverage the decomposed KL-divergence in this work. The trust region for the KL-divergence measure can be made fully differentiable as shown in [9] and also explained in more detail in Appendix A.
Reinforcement Learning with Movement Primitives. While most work on RL with movement primitives [5, 21, 22, 23] concentrates on learning single MP parameter vector for single task configuration, a few methods allow linear adaptation [6. 24, 25] of the MP's parameter vector to the context. In terms of step-based deep RL, Bahl et al. [26] propose neural dynamic processes (NDP). Their goal is to embed the structure of dynamical movement primitives (DMPs) into deep policies by reparametrizing action spaces via second-order differential equations. This can be seen as intersec- tion between step-based and trajectory methods by learning subtrajectories via DMPs spanning mul- tiple timesteps. Their approach allows for effective replanning, but unlike our approach, their main exploration still takes place at the action level rather than at the trajectory level. Hence, equivalent to standard step-based approaches, they have difficulties with sparse and non- Mon-Markovian rewards. Lastly, using DMPs requires several numerical integration steps that also need to be differentiated, which is computationally expensive.
3

---

Deep Black-Box Reinforcement Learning
Formally, we learn a policy To (w]cc that models a distribution over the MP parameters wo based on the context information c. For most traditional RL tasks, the context space is often subset of the observations space. As we learn policy in the parameter space w, exploration is also taking place at this level. Instead of sampling actions in each time step, sample parameter vector only in the beginning of each episode. This leads to exploration at the level of desired trajectories and thus to much smoother and temporally correlated exploration. An overview is shown in Figure 1.
3.1 Episode-Base Basee Reinforcement Learning Objective
Similar to step- based policy gradient [27. 18], we can also optimize the advantage function using the likelihood ratio gradient and an importance sampling estimator in the ERL setting, Our objective has thus the following form
no T c) T (m(c)) c))
[TT Toar) = (e(c,w))p(e).nee
Tona (c. un)
which is maximized w.r.t O. The term A (c, w) = [R|c, wn; m] E [R]c; 1] denotes the advantage function and n is the old policy used for sampling. We can further make use of learned context- value function (c( = E RR|e; for the advantage estimator, which is approximated by optimizing
arg min E (cww)) [V& - R(w R[
Unlike step-based methods, our approach leverages samples of the form (c, w. R) that are generated per trajectory, where R is the trajectory performance.
3.2 Choice the Deep Reinforcement Learning Algorithm
Training this type of policy can theoretically be done with most existing deep RL methods. Yet, learning in the parameter space requires to learn policies with a higher level of precision than in the step-based case. In the step-based case, smaller errors during action selection at later time step. In ERL, we only select one parameter vector w per episode, hence no error correction of these parameters is possible. As a consequence, we chose TRPL since this method has shown to be more stable and precise than other RL methods as the trust regions for the policy update are implemented exactly. Furthermore, the trust regions enforced per state while most other deep RL methods [18, 2, 28] only offer approximate trust region updates for which the trust region is enforced for the mean policy change over all states. We will now present the objective based on the adaptation of the TRPL algorithm to the ERL setup.
still be corrected
3.3 Movement Primitives as Parametrized Controllers
So far we only discussed how to learn the distribution in the parameter space, but not how to create actions. For this purpose, we use a trajectory generator ((w) that is typically translating the pa- rameters w to a desired trajectory T" = (s s ......... sT)) Common choices of trajectory generators are DMPs [29, 30], probabilistic movement primitives (ProMPs) [31], or Viapoint MPs [32]. The step-based control signal is then retrieved by employing trajectory tracking controller f(st. sf), such simple PD-controller that outputs torques given the current and desired position as well as velocity. Here, no noise is applied to the raw actions for exploration. Even more complex controllers such Cartesian impedance controllers can be readily used.
As a result, trajectory generation and execution is independent the policy training, enabling the of any trajectory generator and tracking controller. For this work, will use ProMPs [31] for trajectory generation and torque-based PD-controllers for action execution. In the case a ProMP, optimize its weight vector that linearly influences the generated trajectories. Further, can include additional parameters, such as the trajectory starting time as well as the execution speed of the motion (represented process. As our experiments show, this offers a powerful parametrization for tasks where exact timing is crucial, such as robot table tennis and throwing.
velocity of the ProMP's phase variable) to our ERL optimization

---

3.4 Advantages of Episode-Based Versus Step-Based Reinforcement Learning
Step-based RL is currently by far the more prominent perspective on RL. This seems a natural choice as step-based RL exploits the temporal structure of RL problems and is therefore expected to outper- form ERL in terms of sample efficiency. Yet. step-based RL also comes with a few disadvantages. First, exploration in action space often results fully explore the trajectory space of the agent. In addition, the stochastic action selection also re- sults in noisy returns, which translates into high variance of the policy gradient estimate. Second, the step-based exploration process also complicates the use of sparse or non-Markovian rewards in the step-based setting. While there do exist specialized solutions for some of these problems, they often require much more complex algorithms.
very jerky random walk behavior that does not
In contrast, ERL offers a simple framework for learning complex behavior with sparse and non- Markovian rewards, without special treatment of these cases. ERL only uses a fraction of data points to update the policy, since each trajectory is abstracted as only a single data point. While the small number of available training samples may initially appear to be a disadvantage, it offers three distinct benefits: (i) As the used controllers are deterministic, the returns used for the policy updates contain less variance than in the step-based case, which simplifies finding good policy updates. (ii) The exploration in parameter space allows for learning with sparse and Mon-Markovian rewards which can be used to define the desired behavior parametrization allows the inclusion of temporal parameters such as shifting or scaling of the desired trajectories. We believe that all these features constitute to superior performance of learned ERL policies in comparison to the step-based variant in most of our experiments. In conclusion, we consider ERL as a promising alternative to step-based RL in particularly for tasks where the desired behavior is hard to define with dense rewards.
more direct way. Finally, (iii) the controller
4 Experimental Results
For our evaluation, we first demonstrate the henefits of deep ERL in terms of sparse rewards, preci- sion, and energy efficiency. Afterwards, we conduct large scale study on all 50 Meta-World tasks [33] to show that we achieve competitive performance on a variety of robot manipulation tasks with highly shaped dense rewards. Lastly, we investigate multiple challenging control problems that are hard to solve in the step-based setting. We compare our method, which we call BBRL- TRPL, to the step-based methods PPO [2]. TRPL [9], SAC [1], and NDP [26] as well as to deep evolution strategies (ES) approach [16], the linear adaption method CMORE with ProMPs [8] and BBRL-PPO, which is equivalent to BBRL-TRPL but trained with PPO instead of TRPL.
Note that for NDP, the authors report the performance in terms of the used samples and not in terms of environment interactions (the original work only uses every fifth interaction). We report the to- tal number of environment interactions because we think this leads to a fairer comparison and also explains the rather poor performance of NDP in our experiments. For all tasks the context infor- mation e is represented as subset of the full observation of the first time step including only the stochastic elements, i. e. the parts that are randomly initialized, such as goal or object positions. The trajectory performance R( wr. c) is, if not specified otherwise, the full undiscounted trajectory return. We always evaluate on 20 different seeds and compute 10 evaluation runs after each iteration. Fol- lowing Agarwal et al. [34], we report the interquartile mean (IQM) with a 95% stratified bootstrap confidence interval and performance profiles where feasible (for hyperparameters see Appendix D).
4.1 Sparse Rewards Induce Energy Eflicient Behavior
extension of the reacher from OpenAl gym [35]. Instead of two actuated joints, we use five, but limit the context space, e. the location of the goal, to y 2 0. This results in increased control complexity with a slightly decreased task complexity. We investigate two types of rewards, a dense reward equivalent to the original reacher and a sparse reward that only provides the distance to the goal in the last episode time step (see Appendix B. While BBRL- TRPL and BBRL-PPO can solve the task for both rewards, NDP and ES do not succeed at either (Figure 2 left and center left). PPO and TRPL achieve a slightly better asymptotic performance than BBRI-TRPI. in the dense setting, but are not able to consistently reach the goal for the sparse reward signal. SAC achieves a comparable performance to BBRI-TRPL in the dense setting, however,
As introductory task we
5

---

Reader Spand
Pailnng- Domse
Bun Phuiking Spance
Nandon Swsiem Eatoadi
Nandoe Ravemant
TRPL
NDP
CMORE
SAC
BERL.PPO
BERL-TRPL
The learning curve for the 5D reacher with dense (left) and sparse reward (center left) signal. The box pushing task is trained with three different rewards. The dense reward (center right) has the best performance for step-based PPO, while performs poorly with sparsity in time (right, solid line) and sparsity in time and space (right, dashed line). BBRL-TRPL. solves the task in the first two settings and is even partially able do so for the more difficult sparse reward.
Figure 2:
cannot leverage the sparse reward (see Appendix C) While, CMORE performs reasonably well, is only able to cover part of context space the due to its linear adaption.
To demonstrate these learning capabilities for sparse rewards in more complex setting, also evaluate the algorithms on box pushing task. The goal is to move a box to a given goal location and orientation using simulated Franka Emika Panda (see Appendix B.2). We consider three rewards, a dense reward based on the goal and rotation distance for each time step and two sparse rewards. The first induces time-dependent sparsity by returning the reward signal only in the final time step. The second extends this to include space-dependent sparsity, and additionally requires the box to move near the goal in order receive the reward. PPO and TRPL can only solve this task in the dense setting (Figure 2 center right and right). SAC and ES even struggle with the dense reward. For SAC, penalties due to constraint violations in particular lead to this unstable behavior. The same applies for NDP, thus we did not evaluate it for the sparse rewards. BBRL-TRPL. on the other hand, learns the task in the dense and the first sparse setting without any problems. Even for the complex time and space sparsity, BBRL-TRPL. is able to partially solve the task. BBRL-PPO receives merely moderate results for the dense and first sparse setting.
Yet, why would we sparse rewards when dense rewards work well with step-based RI? A dense reward typically induces a rather fast motion with poor energy efficiency. As shown in our experiments, dense rewards do not work well if we want generate accurate motions that reach the goal after a certain time and are energy efficient as well. Here, we can use sparse rewards instead, e.e., the goal distance is penalized only in the final time step, while the energy cost is accounted for in each time step.
To illustrate the trade-off between precision and energy efficiency, we analyzed the final behaviors in both reward setups with different action penalty factors in the reward function. For each of these factors, we computed the average precision and energy consumption (Figure 3 left and center left). For all methods, decreasing the action penalty factor leads to a higher task precision. However, for the dense reward, high task precision can only be achieved with high energy costs, while the sparse reward generates behavior of similar precision with one (box pushing) or even two (reacher) factors of magnitude less energy consumption. When analyzing the behaviors, it is visible that the dense reward behavior quickly moves to the target and stays there while the sparse reward behavior reaches the target only slightly before the specified end of the episode, resulting in much slower, smoother, and more energy efficient motion.
4.2 Large Scale Robot Manipulation
On the Meta- Meta-World benchmark suite [33], we showcase our ability to learn high quality policies. We train individual policies for each environment, but use the same hyperparameters. PPO and TRPL achieve the best sample complexity (Figure center right). Nevertheless, BBRL-TRPL. has competitive asymptotic performance and even converges slightly higher than PPO. While in the ag- gregated view the gap is rather small, the corresponding performance profiles (Figure 3 right) show that BBRL-TRPL performs better above the 80% threshold. This means that BBRL-TRPL finds

---

S0 B.ar
Bex Packnng- tximg
Nin-WWold
Kina World Pofice hell
8:55
0.23
3:0
0.5
0.11
0.5
Coasd TRPL
Clar
Ntmbbrr www.eeinnieeeai. ERATIEECR..0
Stcceve Banc
PPO
NDP
ES
BBRL-PP PPO
BERL-TRPL
BERL-TRPL spurse
Figure 3: Energy efficiency (sum of squared control action costs) and task performance (distance to the target in the last time step or success rate) trade-off for reacher (left) and box pushing (center left). We average over 100 evaluation runs and all seeds and choose action penalty factors in the intervals (0, 100) for the reacher and (0, 1] for box pushing. For both tasks, BBRI-TRPL. with sparse rewards can achieve a much higher energy efficiency (1 to 2 factors of magnitude) while achieving a similar target precision as PPO in the dense reward setting. The success rate (center right) for all 50 Meta- World tasks and the corresponding performance profile (right), i. the fraction of runs that perform above the threshold given by the x axis. While the sample efficiency is lower for BBRL-TRPL, the final quality of the policy is higher than PPO.
more consistent solutions than PPO with higher precision and solves these tasks without failures. NDP, ES, and BBRL-PPO are not achieving competitive performance.
As an additional ablation study, we trained BBRL-TRPL using a sparse reward (only the final step reward of each episode is used), denoted as BBRL-TRPL. sparse. While the IQM score is lower, it still completes 50% of the tasks with 100% success rate, which is higher than PPO using the dense reward. Moreover, the slope of the performance profile is rather small, i. e. almost all tasks that can be solved are solved to perfection. After further investigation, we found that poorly performing tasks all require behavior that involves sub-goals, c. bin picking, pick and place, etc. This could be ad- dressed e. g. by sequencing several MPs and also giving the corresponding sub-goal as intermediate rewards, which is an interesting direction for future research.
4.3 Dealing with non-Markovian Rewards
As final evaluation, we demonstrate the effectiveness of our method in the presence of non- Markovian rewards. These rewards are particularly useful for complex robot learning tasks, where the whole trajectory history is needed to provide feedback to the agent. We first use modification of the hopper from Open Al gym [35], where we aim to jump as high as possible and land at a target location (see Appendix B.3). The non-Markovian reward returns the maximum height and the min- imum distance the target during the episode. We compare our results to CMORE and ES, which also use the non-M Mor-Maaoov reward. PPO, SAC, and TRPL are trained with a Markovian version that provides height and goal distance in each time step. Please note that we conducted extensive reward shaping to gain the highest performance (i. e. maximum height and minimum goal distance). Overall, BBRL-TRPL achieves a higher jump height than most other methods (Figure 4 left) ac- companied with smaller target distance (Figure 4 center left). BBRL-PPO and CMORE can match the target distance. SAC can even exceed it, but none can reliably learn a good jump height. While BBRL-TRPL charges energy and then jumps just once, the step-based methods try to maximize the height each time step leading multiple jumps in one episode (see Appendix C).
Beer pong [25] is another example where non-Markovian rewards are beneficial. The goal is to throw a ball into cup at various locations on table. The return is computed from the entire trajectory of the ball, e. g. by leveraging table contacts or the minimum distance to the cup (see Appendix B.4). Since we cannot directly train PPO on such reward and designing Markovian version is particularly difficult here, we fix the ball release time and consider the time between release of the ball and the end of the ball trajectory the reward in a similar manner as for the non-Markovian setting. We evaluate BBRL-TRPL and CMORE on the non- Mar-aanooiann reward with a learned ball release time as additional controller parameter as well as PPO for the setting described above (Figure 3 right). BBRL-TRPI. and BBRL- PPO both manage to throw the ball into the cup, while PPO struggles. CMORE can reliably throw
final time step. This allows to compute
7

---

Happer luup Min Hage
Hupper happ
Dur 7
Takll
1.5
1.7
3.5
24
0.2
3.4
New
bar
PPO
TRPL
ES
CMORE
SAC
BBRI-PPO
BBRL-TRPL
Figure 4: The maximum jumping height of the hopper's center of mass (left) and the target distance (center left). The Man-Marovvaan reward allows to jump approximately 20cm higher with increased goal precision. The beer pong task (center right) demonstrates a similar behavior. PPO has difficul- ties throwing the ball into the cup even with the fixed optimal release point, whereas BBRL-TRPP can consistently succeed in the tasks with dynamic release points. The success rate of the table tennis task (right) for hitting the ball (dashed line) and successfully returning the ball near the target position (solid line) shows BBRL-TRPL consistently hits the ball and returns in most of cases.
the ball, however only for a subset of the context space. Similar to the jumping task, BBRL-PPO exhibits a larger confidence interval it is not able consistently solve the task.
Lastly, we train an agent for simulated table tennis [25] where the context is four dimensional and given by features of the initial ball trajectory and the desired location for returning the ball (see Appendix B.5). Intuitively, the agent should be rewarded if it hits the ball and returns the ball near the designated goal position. Similar to the throwing task, for the basebased methods we consider the time after hitting the ball as one time step. For the approachs,,, we also learn the trajectory starting time as well as the speed of the desired trajectory (which is parameter of the ProMP). Both parameters help to learn the precise timing required to play table tennis. BBRL-TRPL always manages to hit the ball and successfully returns it within the vicinity of the goal in more than 60% of the cases. While BBRL-PPO and TRPL can hit the ball, they cannot return with enough precision. We also tested BBRL-TRPL with fixed timing parameters (start time and velocity) and observed a similar behavior the algorithm requires these timing parameters learn how put the ball the desired location. PPO fails to even hit the ball consistently.
Summarizing, BBRL-TRPL is able to successfully deal with non-Markovian reward structures, which can be more intuitive, easier to define, and achieve better learned behaviors than engincered dense rewards.
5 Conclusion and Limitations
In this work, we proposed a new algorithm for ERL. Our method leverages the recently introduced TRPLs [9] to guarantee precise and robust policy updates in high-dimensional parameter spaces. In thorough empirical evaluations, we have shown that we can indeed learn policies with high precision even in the presence of sparse and non-Markovian rewards, where step-based approaches typically fail. Due to the use of sparse rewards, our learned behaviors are more energy efficient and less sensitive to high action costs, which leads to almost universally well-performing policies.
The main limitation of our method and ERL in general is that they typically require more interaction time than step-based RL in the dense reward setting. While we agree this can be critical in some scenarios, still think the benefits outweigh this drawback in many scenarios. To address the sample complexity issue, we plan to investigate off-policy approaches of our method as well as re- planning of the motion primitive to obtain more training data for the policy update in future work. Another limitation of our method is that the desired trajectory is planned in advance in the beginning of the episode, and hence, cannot be altered during the execution. This might be problematic for unforeseen events or perturbations, e.e. highly complex or reactive behavior that cannot directly be modeled with the current motion representation. We believe that this limitation can also be tackled by incorporating re- re-planning the MP trajectory in the RL algorithm. For future work, we will also investigate sequencing multiple MPs to solve more complex tasks which are composed of sub- -oals.
8

---

References
[1] Haarmoja, Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[2] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. In arXiv preprint, 2017. URL htttp::/aaxiv..org//aaaaa 17077063777777 .63477.
[3] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[4] M. P. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foun- dations and trends Robotics, 2(1-2))388 103388443,, 2013.
[5] A. Abdolmaleki, R. Lioutikov, J. Peters, N. Lau, Pualo Reis, and G. Neumann. Model- based relative entropy stochastic search. Advances in Neural Information Processing Systems, 28, 2015.
[6] C. Danicl, Neumann, and J. Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pages 273-281. PMLR, 2012.
[7] A. Abdolmaleki, B. Price, N. Lau, L. P. Reis, and G. Neumann. Contextual covariance matrix adaptation evolutionary strategies. In International Joint Conferences on Artificial Intelligence Organization). 2017.
[8] V. Tangkaratt, H. van Hoof, S. Parisi, G. Neumann, J. Peters, and M. Sugiyama. Policy search with high-dimensional context variables. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[9] F. Otto, P. Becker, N. Anh Vien, H. C. Ziesche, and G. Neumann. Differentiable trust region layers for deep reinforcement learning. In International Conference on Learning Representa- tions, 2021.
[10] F. Sehnke, C. Osendorfer, T. Rückstiess, A. Graves, J. Peters, and J. Schmidhuber. Parameter- exploring policy gradients. Neural Networks, 21(4)5551-55)) May 2010. 1o::10.10166f.nemmett 2009.12.004,
[11] D. Wierstra, T. Schaul, Т. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolu- tion strategies. The Journal of Machine Learning Research, 15())9949-98, 2014.
[12] N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strate- gies. Evolutionary computation, 159-115, 2001.
[13] S. Mannor, Y. Rubinstein, and Y. Gat. The cross entropy method for fast policy search. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 512-519, 2003.
[14] A. Abdolmaleki, D. Simões, N. Lau, L. P. Reis, and G. Neumann. Contextual direct policy search. Journal of Intelligent & Robotic Systems, 9612):141 2019.
[15] H. Mania, A. Guy, and B. Recht. Simple random search of static linear policies is competitive for reinforcement learning. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https: http:///poeeeenn..//////// www.eeeiiinnnnnuiis..oo. cttppper/ccpppp/////pppp 2018 htt.sffiiee.eeeeeeeeeeeooooeree.
pdf.
[16] T. Salimans, J. Ho, X. Chen, Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03866 2017.
[17] P. Chrabaszez, Loshchilov, and F. Hutter. Back to basics: Benchmarking canonical evolution strategies for playing atari. In IJCAI, 2018.
[18] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust Region Policy Opti- mization. Proceedings of Machine Learning Research, pages 1889-1897, 2015. URL http://proceedings mlr. www.ppreesvvveee sch.lman15.... html.

---

[19] H. F. Song, A. Abdolmaleki, J. T. Springenberg, A. Clark, H. Soyer, J. W. Rae, S. Noury, A. Ahuja, S. Liu, D. Tirumala, N. Heess, D. Belov, M. Riedmiller, and M. M. Botvinick. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2020. URL https: // //openreview net. net/forruniddSSy1lppll
[20] L. Engstrom, A. Dyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph. and A. Madry. Im- plementation Matters Deep Policy Gradients: A Case Study on PPO and TRPO. In In- ternational Conference Learning Representations, 2020. URL http://aaxts org/abs/
[21] J. Kober and J. Peters. Policy search for motor primitives in robotics. In D. Koller, D. Schu- urmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Sys- tems, volume 21. Curran Associates, Inc.. 2008. URL https: htttps:/pproceeeeee http:///poeeeeeeeeeeeee//eee neurips.
påf.
[22] Stulp and Sigaud. Path integral policy improvement with covariance matrix adapta- tion. In Proceedings of the 29th International Coference on International Conference Ma- chine Learning, ICML'112, page 1547-1554, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312855
[23] Stulp and O. Sigaud. Policy improvement methods: Between black-box optimization and episodic reinforcement learning. 2012.
[24] A. Kupesik, M. P. Deisenroth, J. Peters, L. A. Poha, P. Vadakkepata, and G. Neumann. Model- based contextual policy search for data-efficient generalization of robot skills. Artificial lelligence, 247:415 lincoln.ac. uk/ 22////5774//11 / http///////ssskkkaee AIJ_ _0120005 _a112ikkkAA_1111..____11 pdf. Impact Factor: 3.333.
2017. doi: 10 art.tttttt. 1..001..110000 11.005. URL attp://eprinet
[25] O. Celik, D. Zhou, G. Li, P. Becker, and G. Neumann. Specializing versatile skill libraries using local mixture of experts. In Conference Robot Learning, pages 1423-1433. PMLR, 2022.
[26] S. Bahl, M. Mukadam, A. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensorimotor learning. Advances in Neural Information Processing Systems, 2020. URL https://arxiv awrgg/2s/012.... 02788v1.
[27] R. J. Williams. Simple statistical srudient-followied algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-22) -26,, 1992.
[28] R. Akrour, J. Pajarinen, J. Peters, and G. Neumann. Projections for approximate policy iteration algorithms. Proceedings of Machine Learning Research, pages 181-190, 2019.
[29] S. Schaal. Dynamic Movement Primitives -A Framework for Motor Control in Humans and Humanoid Robotics, pages 261-280. Springer Tokyo, Tokyo, 2006. ISBN 978-4-43-331338116 1007 0-000774-43---188-----
doi: 10011/0/444333311 23. URL https: httt:::://do..rgg///.
[30] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors. Neural Computation, 25(2), 2013.
[31] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement prim- itives. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.. 2013, URL https: httts::/proceeedngg: neurips....... www.eennneee.raaaeeeeeeeeeeeeeee cc/paper/2013/file .dff.
[32] Y. Zhou, J. Gao, and T. Asfour. Learning via-point movement primitives with inter-and ex- trapolation capabilities, In International Conference on Intelligent Robots and Systems), pages 4301-4308. IEEE, 2019.
[33] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019, URL https: / /arxivv. org/ abs/1910 10897.
10

---

[34] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare. Deep rein- forcement learning the edge of the statistical precipice. Advances Neural Information Processing Systems, 2021.
[35] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
A KL-DDvergence Trust Region Projection Layer
As already mentioned in the main text, TRPL.s [9] present a scalable and mathematically sound approach for enforcing trust regions in step-based deep RI. The layer takes the output of a standard Gaussian policy as input in terms of mean and variance Σ and projects it into the trust region the given mean and variance violate their respective bounds. This projection is done for each input state individually. Subsequently, the projected Gaussian policy distribution with parameters ju, Σ is used for any further steps, e. for sampling and/or loss computation. Formally, the layer solves the following two optimization problems for each state
S.I. dmean (jhss (ool Ep and arg min dmeaa (jss (s))),
arg min deww [Eg I(s)), s.t. deav [E, Eata(s) S 6 Σ,
(2)
where fı, and Eg are the optimization variables for input state s and 6p and ey are the trust region bounds for mean and covariance, respectively. Finally, Mold and Eowi are the reference mean and covariance for the trust region and dinam as well as dewv are the similarity metrics for the mean and covariance of decomposable distance or divergence measure. As we only leverage the KL- divergence projection, we will provide only details for this particular projection below. For the other two projections we refer the reader to Otto et al. [9].
Inserting the mean part of the Gaussian KL divergence into Equation yields
arg min (p+ fi)² Eod (μ - ( s.t. (pola
f0)* Dodd (Mold (3) S Ca.
#
After differentiating the dual w.r.t it, we can solve for the projected mean
(Mola 11) End (Mola 11)
WHold
with
CH
leveraging the optimal Lagrange multiplier w. Similarly, we can insert the covariance part of the Gaussian KL divergence into Equation 2, which results in
arg migmints (M~
log
Σ
s.t.
tr (2
log
Eotal
< eg,
Σ
where d' is the number of degrees of freedom (DoF). Once again, differentiating and solving the dual g(j)) for the projected covariance yields
1 + + old
17
with 17" arg min g(n), s.t. 17 2 0.
17" +1
Here, the the optimal Lagrange multiplier 17" cannot be computed in closed form, however, stan- dard numerical optimizer, such as BFGS, is able to efficiently find it. This can be made differentiable by taking the differentials of the KKT conditions of the dual. For more details, refer the orig- inal work [9].
B Environment Details
B.1 Reacher5d
For the Reacher task we modify the original OpenAl gym Reacher-v2 by adding three additional joints, resulting in total of five joints. The task goal is still to minimize the distance between

---

Figure 5: Visualization of the four control tasks box pushing. hopper jumping, beer pong, and table tennis.
the goal point Pgoa! and the end-effector p. We, however, only sample the goal point for y 2 O, e.e. in the first two quadrants, to slightly reduce task complexity while maintaining the increased control complexity. The observation space remains unchanged, unless for the sparse reward where we additionally add the current step value to make learning possible for step-based methods. The context space only contains the coordinates of the goal position. The action space is the 5d equivalent to the original version.
For the reward the original setting leverages the goal distance
Rgoal = |
Pgoal ||2
and the action cost
K Tt Σ (a))
Dense Reward. The dense reward in the 5d setting, hence, stays the and the agent receives in each time step
Rax
Regaal
Sparse Reward. The sparse reward only returns the task reward in the last time step T and addition- ally adds velocity penalty R Σx ( where are the joint velocities, to avoid overshooting
Tt - 200 goal 10 = T.
<<T
Riot
B.2 Box Pushing
The goal of the box pushing task to move box to a specified goal location and orientation using the seven DoF Franka Emika Panda. Hence, the context space for this task is the goal position [0.3, 0.6], y E [-0.45, 0.45) and the goal orientation 0 [0, 2Tr.. the observation space for the step-based algorithms contains the sine and cosine of the joint angles and their velocities as well as position and orientation quaternions for both end-effector and the box. For the action space we use the torques per joint and additionally add gravity compensation in each time step. that does have be learnt. The task is considered successfully solved if the position distance S. 0.05m and the orientation eтror < 0.5rad. For the total reward we consider different sub-rewards. First, the distance the goal
addition to the contexts,
Rgoal
||p Pgoatlls
where p is the box position and Pgont the goal position itself. Second, the rotation distance
Rvotation arceos
Egoat
where r and r gooa are the box orientation and goal orientation in quaternion, respectively. Third, an incentive keep the rod within the box
Riod clip(llp hpos |l. 0.05, 10)
12

---

where poss is the position of the rod tip. Fourth, similar incentive that encourages to maintain the rod in desired rotation
rotatontaiin = clip( arccos | hol, 0.25, 2),
where hrot and ho (0.0, 1.0, 0.0, 0.0) are the current and desired rod orientation in quaternion, respectively. And lastly, we utilize the following error
err(q, q) =
Σ
(1q:1 - 14 +
Σ
(1, - 1451).
JE <31
Here, 9. q, q'. and q" are the robot joint's position and velocity as well as their respective bounds. Additionally, we consider action cost in each time step é
K
7: 5. Σ (a))²
where K = 7 is the number of DoF. In total we consider three different rewards.
Dense Reward. The dense reward provides information about the goal and rotation distance in each time step on top of the utility rewards
Root - Ryod Ryod rotation - T - err(q, q) - 3.5R goal R
Time- Dime-Dependent Sparse Reward. The time-dependent sparse reward is similar to the dense re- ward, but only returns the goal and rotation distance in the last time step T
{
Rood Rrod_rotation rotation - T: er q, q): err( q) 350 goal 200 20o
1 << T. t=
Riot
Ryod
Rood rotation
Time- and Space-Dependen. Sparse Reward. The second sparse reward additionally adds sparsity based on the position and only returns goal and rotation distance in the last time step when the box near the goal location
Ryod rotatonn
err( 4)
Riot
clip Rgo 0, 100) clip 1 0, 100) + 300, 1 1 T and Rgoal S 0.1, Rndd,ootaiion T: enr(q, ()
Redd
else.
B.3
Hopper Jump
In the Hopper jump task the agent has to learn to jump as high as possible and land on a certain goal position at the same time. We consider five basis functions per joint resulting in an 15 dimensional weight space. The context four-din dimensional consisting of the initial joint angles 0 e [-0.5, 0]. Y e [-0.2, 0j. E [0, 0.5) and the goal landing position extends the original observation space from the OpenAI gym Hopper by adding the x-value of the goal position and the x-y-z difference between the goal point and the reference point at the Hopper's foot. The action space is the same as for the original Hopper task. We consider a non- non-Markovian reward function for the episode-based algorithms and step-based reward for PPO, which we have extensively designed to obtain the highest possible jump.
[0.3, 1.35]. The full observation space
Non-Markovian Reward. In each time-step we provide an action cost
7. 10-3
Σ (at)
where K = the number of DoF. In last time-step T of the episode we provide a reward which contains information about the whole episode as
Bheight
10h
Rydist=
foot,T - Pooat 1|
Redist
|ootact Pgoot 2
{
2 if 2T E [0.5, ooland 7, oo)
Rhealthg
clse
13

---

where hmas is the maximum jump height in z-direction of the center of mass reached during the whole episode, P/ Pfoot. is the x-y-z position of the foot's heel time step poottce the foot's heel position when having contact with the ground after the first jump, Pyoal is the goal landing position of the heel. Rhealthy a slightly modified reward of the healthy reward defined in the original hopper task. The hopper is considered as "healthy" if the Z position of the center of mass is within the range [0.5m,, ooj. This encourages the hopper stand at the end of the episode. Note that all states need to be within the range [-100, 100] for Rhealthy Since this is defined in the hopper task from OpenAI already, we haven't mentioned it here. The total reward at the end of an episode is given as
Rtol - Σ To Rheight Rydist + Redist + Rhealthg t=0
Step-Based Reward. We consider step-based alternative reward such that PPO is also able to learn a meaningful behavior on this task. We have tuned the reward such that we can obtain the best performance. The observation space is the same in the original hopper task from OpenAI extended with the goal landing position and the current distance of the foot's heel and the goal landing postion. We again consider the action cost in each time-step
Te = 10-3 " (a))
and additionally consider the rewards
Rheight, = 3h,
R Rydist, = 3) poott Pgout112
1
if 24 [0.5, ooland 0, 7,0 [-00,
Rhealthy,
0
else
where these rewards are now returned to the agent in each time-step 6, resulting in the reward per time-step
(e ae) --T + Rhoight, + Rgdist. + Rhealthy,
B.4
Beer Pong
In the Beer Pong task the K = 7 Degrees of Freedom (DoF) robot has to throw a ball into a cup on big table. The context is defined by the cup's two dimensional position the table which lies in the range x E [-142, 1.42], E [-4.05, 1.225]. For the step-based algorithms we consider cosine and sine of the robot's joint angles, the angle velocities, the ball's 's distance to the bottom of the cup, the ball's distance to the top of the cup, the cup position and the current time step. The action space for the based algorithms is defined as the torques for each joint, the parameter space for the episode-based methods is 15 dimensional which consists of the two weights for the basis functions per joint and the duration of the throwing trajectory, i.e. the ball release time.
We generally consider action penalties
K
Σ (a))
K
consisting of the sum of squared torques per joint. For t < T we consider the reward
(1 0z) = T
with a; 10-2, For T we consider the Mar-Markovi reward
mun(llpe.top 1b.1.T (20 0.5 | bottom - Pb,r
2||pe,bottom
Pb.
Q.
cond. 1 if cond. 2
Rtask
- 4 mmn([lpeec
Pb,1.T
[3) 0.5
1.55|ppe, Pe.bottom Pb,T 2 arT,
2
min( (Pe,top
P..
(2 0.5
Pe.bottom
Pb.T1 OrT,
if cond. 3
Pe.bottom
1b, CTT,,
if cond. 4
14

---

-4 min(l - Pb.1:7 (
0..5 PL.5|P,,ooe
Pebbotomm Pb.21122
Pb, (TTT:
if cond.
Riask
4 min Pc,top Pb,1:T
0.5
Pe.bottom pb,7
OTT,
if cond. 2
2 min( Pe,top Po,1:T
0.5 Pe.bottom
Pb,T
OTT,
if cond. 3
Pe,bottom Pb,T
aTT,
if cond. 4
where Pe.top is the position of the top edge of the cup, Po,bottom is the ground position of the cup, Ph,t is the position of the ball at time point and the squared mean torque over joints during one rollout and aT 10-4. The different conditions are:
cond. 1: The ball had a contact with the ground before having contact with the table.
cond. The ball is not in the cup and had no table contact
cond. 3: The ball is not in the cup and had table contact
cond. 4: The ball is in the cup.
Note that Pb,k is the ball's and the ground's contact position and is only given, if the ball had contact with the ground first.
At time step T we also give information whether the agent's chosen ball release time B was reasonable
-30 - 10 B - Bminn )2 if B < Busn
Brelease
-30 10 B Bynrr if B < Bmaz
where we define Biin 0.1s and Bmaz 1s, such that the agent is encouraged throw the ball within the time range [Bminy Bmaz].
The total return over the whole episode is therefore given as
T-1
Riot
Σ
re(si, ar) + Reask + Rrelease
A throw is considered as successful if the ball is in the cup at the end of an episode.
B.5
Table Tennis
We consider table tennis for the entire table, i.e. incoming balls are anywhere on the side of the robot and goal locations anywhere on the opponents side. The goal is to use the seven degree of freedom robotic arm to hit the incoming ball based on its landing position and return it as close as possible to the specified goal location. As context space we consider the initial ball position [-1.- y E I-0.65, 0.65] and the goal position x E [-1.2, -0.2). y [-0.6, 0.6). The full observation space again contains cosine and sine of the joint space and the joint velocities as well as the ball velocity on top of the above context information. The torques of the joints make up the action space. For this experiment, we do not use any gravity compensation and allow in the episode-based setting to learn the start time to and the trajectory duration I'. The task considered successful if the returned ball lands on the opponent's side of table and within < 0.2m to the goal location. The reward is defined
0,
if cond.
0.2 - 0.2 tanh (min pr - -)
if cond. 2
Ftask
3 2 tanh (min pr 1s|l²) tanh (l]pr- P Pgoat| (
if cond. 3
6 tanh (min) р, pb|l2) tanh [p: Pooat )) if cond. 4
7 tanh (min) PP poll?) - 4 tanh (llp: Pyoat| ) if cond. 5
where P,- is the position of racket center, P& is the position of the ball, Pr is the ball landing position, Pgoat is the target position. The different conditions are
cond. the end of episode is not reached,
cond. 2: the end of episode is reached,
15

---

cond. 3; cond.2 is satisfied and robot did hit the ball,
cond. 4: cond.3 is satisfied and the returned ball landed on the table,
cond. cond.4 is satisfied and the landing position is the opponent's side.
The episode ends when any of the following conditions met
the maximum horizon length is reached
ball did land the floor without hitting
ball did land the floor or table after hitting
For BBRL-PPO and BBRL-TRPL, the whole desired trajectory is obtained ahead of environment interaction, making of this property
collect some samples without physical simulation. The reward function based on this desired trajectory is defined as
Σ (b,)
1931. (i,1) E ((1,)) 1
Ttraj
where 7d is the desired trajectory, the time index, j is the joint index, d² is the joint position upperbound. The desired trajectory is considered as invalid if Ptraj < O, an invalid trajectory will not be executed by robot. The overall reward for BBRL is defined as:
Ttraj+
Teraj < 0 otherwise
Flasks
C Additional Evaluations
Royer Tanp. Hoipl Trgeccey a8gs
Now Syrne
15
0.
DIRI-TRFE
Figure 6: (Left) The improved performance on the Hopper Jump task also demonstrated on jumping profile for fixed context. While BBRL-TRPL jumps once as high possible, PPO con- stantly tries to maximize the height at each time step which leads to several jumps throughout the episode and consequently to a lower maximum height. (Right) Learning curve of SAC for the sparse reward of the 5D Reacher task.
16

---

D Hyperparameters
For all methods, where applicable, we optimized the learning rate, sample size, batch size, number of layers, and the number of epochs. For all BBRL methods and NDP, we additionally optimized the number of basis functions. Moreover, predicted DMP weights, which was hard-coded to 100 in the original code base. However, this value only worked for the meta-world tasks, but for the other tasks, hence adjusted allow for a fair comparison. The population size of ES is always half the number of samples because two function evaluations used per parameter vector.
found that NDP requires tuning of the scale of the
Table 1: Hyperparameters for the modified reacher experiments.
PPO
NDP
TRPL
SAC
CMORE
ES
BBRL-PPO
BBRL-TRPL
numher samples
16000
16000
16000
1000
120
200
GAE A discount factor
0.95
0.95
0.95
0.95
0.99
0.99
0.99
0.99
0.a.
n.a.
D.a.
n.a.
n.a.
n.a.
0.005
D.a.
0.1
n.a.
n.a.
0.05
0.a.
0.0005
0.a
0.0005
optimizer
adam
udam
adam
adam
adam
adam
adam
epochs
10
10
20
1000
a.a.
0.3.
100
100
leurning rate
3e-4
3e-4
3e-4
n.a.
1c-2
3c-4
3o-4
critic
True
True
True
True
False
False
False
False
epochs critic
10
10
10
1000
n.a.
n.a.
n.a.
n.a.
learning critic (and alpha)
3e-4
3c-4
3o-4
3c-4
number minibatches
32
32
64
n.a.
n.a.
n.a.
batch
n.a.
n.a.
n.a.
256
n.a.
n.a.
n.a.
n.a.
buffer size
le6
leurning starts
0
0
10000
0
0
0
polyak weight
5e-3
trust region loss weight
n.a.
n.a.
10
n.a.
n.a.
n.a.
n.a.
10
normalized observations
True
True
True
False False n.a.
False False n.a.
False False n.a.
False False
False
normalized rewards
True
False
observation clip
10.0
10.0
reward clip
10.0
10.0
n.a
nat
n..
ma.
n.a.
n.a.
critic clip importance ratio clip
0.2
n.a.
0.2
0.2
0.2
n.a.
n.a.
n.a.
0.2
hidden layers
[32, 32]
[32, 32]
[32, 32]
[128,128]
n.a.
[32, 32)
[32, 32]
[32, 32)
hidden layers critic hidden activation
[32,
[32,
[128,128]
n.a.
tanh
tanh 1.0
tanh
relu
n.a.
tanh
tanh
tanh
initial
1.0
1.0
1.0
1.0
1.0
1.0
number basis functions number zero basis weight scale
n.a.
n.a.
R...
5
n.a.
1
n.a.
20
n.a.
D.a.
n.a.
D.a.
D.a.
n.a.
17

---

Table 2: Hyperparameters for the box pushing experiments.
PPO
NDP
TRPL
SAC
BBRL-PPO
BBRL-TRPL
numher samples
16000
16000
16000
1000
250
160
160
0.95
0.95
n.a.
n.a.
n.a.
discount factor
0.99
0.99
0.99
0.99
0.005
0.005
n.a.
n.a.
0,00005
n.a.
0.a.
n.a.
0.0005
optimizer
adam
adam
adam
adam
adam
adam
adam
epochs
learning rate
le-4
lc-4
5e-5
Ic-4
Ic-2
lc-4
lc-4
use critic
True
True
True
True
False
True
Trme
epochs critic
1000
100
leurning rate critic (and alpha)
le-4
le-4
2e-4
lc-4
le-4
lc-4
number minibatches
n.a.
n.a.
butch
256
huffer size
n.a.
n.a.
n.a.
leh
n.a.
n.a.
n.a.
leurning starts
polyak_weit weight
5e-3
trust region loss weight
n.a.
n.R.
10
n.d.
n.a.
n.a.
25
normalized observations
True
True
True False n.a.
Fulse n.a.
False n.a.
False False n.a.
Fulse
normalized rewards observation clip
10.0
10.0
n.a.
reward clip critic clip importance ratio clip
10.0
10.0
n.a.
n.a.
n.a.
n.a.
n.a.
0.2
0.2
0.2
0.2
0.2
n.a.
n.a.
n.a.
0.2
n.a.
hidden layers
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[128, 128)
[128, 128]
hidden layers critic
[256. 256]
[256. 256]
[256, 256]
[256, 256]
n.a.
[32, 32]
[32, 32]
hidden activation
tanh
tanh
tanh
relu
tanh
tanh
retu
initial std
1.0
1.0
1.0
1.0
1.0
1.0
1.0
number basis functions
n.a.
5
n.a.
D.S.
D.a.
5
number hasis weight scale
10
n.a.
18

---

Table 3: Hyperparameters for the Meta-Workle experiments.
PPO
NDP
TRPL
ES
BBRL-PPO
BBRL-TRPL
number samples
16000
16000
16000
200
GAE X
0.95
0.95
0.95
discount factor
0.99
0.99
0.99
Sp
n.a.
0.005
n.a.
0.005
n.a.
0.0005
n.a.
0.0005
optimizer
adam
adam
adam
adam
adam
adam
epochs
10
10
20
n.a.
100
100
learning rate
3e-4
3e-4
5e-5
le-2
3e-4
3e-4
use critic
True
True
True
False
False
False
epochs critic
n.a.
n.a.
n.a.
learning rate critic (and alpha)
3e-4
3e-4
3e-4
n.a.
n.a.
n.a.
number minibatches
32
32
n.a.
region loss weight
10.0
normalized observations
True True
True True
True False
False False n.a.
False False
False
normalized rewards
False
observation clip
10.0
10.0
reward clip
10.0
10.0
critic clip importance ratio clip
0.2
0.2
n.a.
0.2
0.2
0.2
n.a.
n.a.
0.2
n.a.
hidden layers
[128, 128)
[128, 128)
[128, 128)
(128, 128) n.a.
[32, 32)
[32, 32]
hidden layers critic
[128, 128]
(128, 128)
[128, 128] tanh
n.a.
n.a.
hidden activation initial
tanh
tanh
tanh 1.0
tanh
relu
1.0
1.0
1.0
10.0
number functions number zero basis weight scale
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
100
19

---

Table 4: Hyperparameters for the hopper jumping experiments.
PPO
TRPL
SAC
CMORE
ES
BBRL-PPO
BBRL-TRPL
number samples
16384
16384
1000
200
64
64
GAE
0.95
0,95
discount factor
0.99
0.99
0.99
eple
0.005
0.1
0.005
en
n.a.
0.00005
n.a.
n.a.
n.a.
n.a.
0.0005
optimizer
adam
adam
adam
adam
adam le-4
adam
epochs
n.a
n.a.
learning
3e-4
5e-5
le-4
0.01
Se-5
use critic
True
True
True
False
False
False
False
epochs critic
10
1000
n.a.
n.a.
n.a.
n.a.
learning critic (and alpha)
3e-4
3e-4
1e-4
number minibatches
32
64
n.a.
n.a.
n.a.
size
n.a.
n.a.
256
n.a.
n.a.
n.a.
n.a.
hufler size
le6
learning starts
0
0
10000
0
0
polyak_weight
n.a.
5e-3
n.a.
n.a.
trust region loss weight
10
25
normalized observations
True True
True False
False False n.a.
False False n.a.
False False n.a.
Fulse False n.a.
False False n.a.
normalized rewards observation clip
10.0
n.a.
reward clip critic clip importance rutio clip
10.0
0.2
n.a.
n.a.
n.a.
n.a.
0.2
n.a.
n.a.
n.a.
n.a.
n.a.
0.2
n.u.
hidden layers
[128, 128]
[128, 128]
[128, 128]
n.a
[128, 128]
[32, 32]
[32, 32)
hidden layers critic hidden activation
[128, 128]
[128, 128]
[128. 128]
tanh 1.0
tanh
relu
tanh
tanh
tanh
initial
1.0
1.0
1.0
1.0
1.0
number basis functions number basis
n.a.
n.a.
n.a.
n.a.
5
B.a.
20

---

Table 5: Hyperparameters for the Beer Pong experiments.
PPO
CMORE
BBRL-PPO
BBRL-TRPL
number samples
16384
60
160
160
GAE X
0.95
discount factor
0.99
n.a.
n.a.
n.a.
n.a.
0.1
n.a.
0.005
n.a.
n.a.
n.a.
0.0005
optimizer
adam
n.a.
adam
adam
epochs
10
n.a.
100
100
learning rate
3e-4
n.a.
lc-4
5e-5
use critic
True
False
False
False
epochs critic
10
n.a.
n.a.
n.a.
learning rate critic (and alpha)
3e-4
n.a.
n.a.
n.a.
number minibatches
32
n.a.
trust region loss weight
n.a.
n.a.
n.a.
25
normalized observations normalized rewards
True True
False False n.a.
False False
False
False
observation clip
10.0
n.a.
n.a.
reward clip
10.0
n.a.
n.a.
n.a.
critic clip importance ratio clip
0.2
n.a.
0.2
n.a.
0.2
n.a.
0.2
n.a.
hidden layers
[128, 128]
n.a.
[32, 32]
[32, 32]
hidden layers critic
[128, 128]
n.a.
n.a.
n.a.
hidden activation
tanh
n.a.
tanh
tanh
initial std
1.0
1.0
1.0
1.0
number basis functions number zero basis
n.a.
n.a.
2
2
21

---

Table 6: Hyperparameters for the Table Tennis experiments.
PPO
TRPL
BBRL-PPO
BBRL-TRPL
number samples
16000
16000
200
200
GAE À
0.95
0.95
discount factor
0.99
0.99
n.a.
n.a.
n.a.
0.0005 0.00005
n.a.
0,0005
6x
n.a.
n.a.
0.00005
optimizer
adam
adam
adam
adam
epochs
10
20
100
100
learning rate
le-4
5e-5
lc-4
3e-4
use critic
True
True
True
True
epochs critic
10
10
100
100
learning rate critic (and alpha)
le-4
le-4
le-4
3e-4
number minibatches
40
40
trust region loss weight
n.a.
10.0
n.a.
25
normalized observations normalized rewards observation clip
True True
True False n.a.
False False
False
False
10.0
n.a.
n.a.
reward clip critic clip importance ratio clip
10.0
n.a.
n.a.
n.a.
0.2
n.a.
0.2
n.a.
0.2
n.a.
0.2
n.a.
hidden layers
[256, 256]
[256, 256]
[256]
[256]
hidden layers critic
[256, 256]
[256, 256]
n.a.
n.a.
hidden activation
tanh 1.0
tanh
tanh
relu
initial std
1.0
1.0
1.0
number basis functions number zero basis
n.a.
n.a.
n.a.
n.a.
22


Reviews:
[{"review_id": "tYlmhQJPeCI", "reviewer": "Reviewer_Rejd", "summary": "The paper presents a novel method for episode-based reinforcement learning (RL), BBRL-TRPL, that uses trust region projection layers (TRPLs) to learn motion parameters, and uses Probabilistic Motion Primitives (ProMP) to generate motor commands. Compared to step-based RL algorithms, BBRL-TRPL shows better performance when the reward is sparse and non-Markovian, and learns more energy-efficient trajectories.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths:**\n\nOverall, I find the writing to be clear and easy to follow. I do have a few suggestions on how to improve presentation, but the paper is overall easy to read. Moreover, the authors did thorough evaluation of their method against existing baselines with detailed analysis.\n\n**Weakness:**\n\nMy major concern for the paper is that it misses an entire family of papers that use evolutionary strategies (ES) for reinforcement learning tasks, such as CMAES [1] and ARS [2]. The authors mentioned that these algorithms may be limited to \"linear adaptations\" (Line 43). It would be great if the authors could clarify on what \"linear\" means here, since these algorithms have been used to find the parameters of neural network policies. Moreover, these algorithms are known to work well across a wide variety of reinforcement learning tasks. Therefore, the authors should at least compare their method some ES baselines. A simple ES baseline could be a single policy that directly takes in the context and the robot states ($c$ and $s_t$ in Fig.1) and outputs motor commands.\n\nMoreover, the major benefits of the proposed algorithm, such as better performance at sparse or non-Markovian rewards and structured exploration, have been known as the benefits of black-box algorithms _in general_ [3]. All these algorithms adapt the policy based on the total _return_ of the entire episode, and are not particularly sensitive to the shape of reward functions. Moreover, they all explore in the parameter space, and outputs smooth actions during training. Therefore, I find that the results of the proposed method does not improve much from existing methods.\n\n[1] The CMA Evolution Strategy: A Tutorial [link](https://arxiv.org/abs/1604.00772)\n\n[2] Simple random search provides a competitive approach to reinforcement learning. [link](https://arxiv.org/abs/1803.07055)\n\n[3] OpenAI Blogpost on ES vs RL. [link](https://openai.com/blog/evolution-strategies/#:~:text=Injecting%20noise%20in%20the%20parameters.&text=However%2C%20RL%20injects%20noise%20in,guess%20and%20check%E2%80%9D%20on%20parameters.)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Updating to weak reject after reading the author's responses. Also reducing confidence from 4 to 3 since I'm not very sure if I'm under-assessing the \"contextual\" part of the paper.\n\n======================Original Content=======================\n\nWhile the paper is well-written, I cannot recommend acceptance of this paper, given that the major result does not show significant difference from existing methods.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "First, I would like the authors to conduct additional results to benchmark their algorithms against ES baselines (see Weakness sections for more details). In addition, I have a few minor comments on writing:\n* Separate Related Works and Background sections.\nI found that Section 2 can be too long right now, and it can be confusing to read related works and the priliminary stuff (TRPL, RL) in the same section. It would be nicer to have a dedicated preliminary section for the problem setup.\n\n* Equation in section 3.1: looks like the definition of the cost function is a bit off? The $\\arg\\max$ should appera in policy optimization, but not in the definition of $J$?"}}, {"review_id": "qW9xZhH9U4b", "reviewer": "Reviewer_3iLp", "summary": "The paper proposes a movement primitive based approach to reinforcement learning - instead of learning a function approximator with state conditioning, the policy is parameterized as a contextual movement primitive that translates a parameter vector to an open loop action trajectory to be deployed in the environment.  The advantage of this kind of approach is that it allows for generating smoother trajectories, and has applicability in situations where it is natural to define sparse or non-markovian rewards. Extensive experiments show the benefit of the approach in non-markovian reward settings and its relevance to energy minimization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper propose a new method based on trust region optimization for movement primitive parameterized policies that allows for successful learning on manipulation tasks using reinforcement learning. The importance of trust region optimization for policy optimization is shown by an ablation.\n2. The method is very relevant to the robotics community where using movement primitives can naturally speed up learning while ensuring more energy efficient and smoother trajectories.\n3. The paper contains extensive experiments in simulated domains that are relevant to robotics. The results on meta-world demonstrate convincingly that the method can learn and achieve a higher asymptotic performance than step-wise RL methods which was surprising. The proposed method also allows to incorporate non-markov or sparse rewards while training which can lead to more aligned and energy efficient behaviors.\n\n**Weaknesses:** \n\n1. Paper/Method clarity: \na. In the method section 3.3, it is not clear how the trajectory generator converts a parameter to a state-based trajectory. Since it forms a integral part of their method it might be important to clarify as a background or in appendix.  \nb. Does the inference of state-based trajectory use any dynamics or inverse-dynamics information? In that case the applicability of the method is quite limited and the baselines compared should incorporate dynamics knowledge.\nc. How is the PD controller tuned? Are the samples used for tuning PD controller counted in the sample efficiency plots?\n2. Baselines:\na. Efficient off-policy baselines like SAC are missing for a fair comparison in the experiments for domains like meta-world.\n    \n    b.  Can the authors give an intutive understanding of why PPO gets stuck in local optima for the meta-world tasks?\n    \n3. Engineering movement-primitives: \na. How does the movement primitive scale to high-dimensional or long horizon environments? When the state space is large the importance ratio used in PPO may suffer from large variance because unlike step-based PPO now we have reweighting of large dimensional states?\n    \n    b. How is the movement primitive decided? The optimal policy’s state visitation should lie in the space of valid movement primitives and cannot be decided without oracle information and is hard to approximate in tasks like locomotion?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is very relevant to the robotics community and presents a method to incorporate movement primitives in the RL framework. The paper has extensive experiments which makes the method convincing but the method writing needs more work and clarity. The ability to incorporate structural prior in policy in order to address sparse reward and non-markovian tasks in nicely addressed in this work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "My concerns can be found in the weaknesses section above.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "Mjh4urBPkcJ", "reviewer": "Reviewer_FPE5", "summary": "This paper presents a deep episode-based reinforcement learning algorithm by adapting an existing step-based RL algorithm (differentiable trust region layer) to instead select a single action per episode, where the action involves selecting the parameters for a parametrized motion primitive which will be executed for the full episode. The authors then demonstrate that the proposed episode-based RL algorithm can more easily learn from sparse rewards for a number of continuous robotic control tasks in simulation. The main contribution of this work appears to be the first deep contextual episode-based RL algorithm and experimental results suggesting the utility of such an algorithm for robotic tasks. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) While the algorithmic innovation with respect to the differentiable trust region layer work appears minimal (all that seems to be required is applying this method for one step and feeding the results to a parameterized motion primitive), the idea to develop a deep episode-based RL algorithm definitely seems to be novel and well motivated for robotic tasks.\n\n(2) The experimental results do clearly indicate that the proposed approach is better able to handle sparse and non-Markovian rewards than step-based RL algorithms and leads to more energy efficient policies as a result across a number of standard simulation environments.\n\n(3) The method and experimental results are very clearly and thoroughly described and easy to understand.\n\nWeaknesses:\n\n(1) My interpretation of this paper was mainly that choosing a more appropriate action space (in this case ProMPs) can lead to more efficient RL. As a result, while comparison to step-based RL algorithms is good, I believe that these comparisons alone are insufficient since they have access to a much less expressive action space than the presented algorithm. To this end, the paper would be significantly strengthened by adding comparisons to hierarchical RL algorithms with similarly parameterized primitives or comparisons where the choice of motion primitive is changed (ie. something different from ProMP).\n\n(2) While I believe that the proposed approach would be useful for practical physical robotic experiments, it would significantly strengthen the paper to evaluate it on physical hardware. \n\n(3) [Minor]: It would be helpful to discuss connections between the proposed approach and ideas from the bandits and contextual bandits community. These papers may be of particular interest: https://arxiv.org/abs/1807.09809, https://arxiv.org/pdf/2206.08921.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I am positive on this paper and like the idea of essentially doing 1-step RL with classical motion primitives for robotic learning. The primary weaknesses of the paper are (1) the lack of comparison to hierarchical RL methods, which would be much fairer and interesting than comparison to purely step-based RL approaches and (2) the lack of physical experiments. Otherwise the contribution and experiments  are very clearly and thoroughly presented, so I am learning towards an accept decision at this time. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "(1) Please add comparisons to (a) hierarchical RL algorithms with similarly parameterized primitives or (b) comparisons where the choice of motion primitive is changed (ie. something different from ProMP) to (a) provide a fairer algorithmic comparison to prior approaches and (b) study the impact of the choice of motion primitive on experimental results\n\n(2) The paper would be much stronger if it is evaluated on physical hardware. RL training need not be performed on physical hardware, but transferring a learned policy from simulation to the real world would be beneficial in making a better case for the relevance of the proposed approach to robotic applications."}}, {"review_id": "6CchxVGvFaG", "reviewer": "Reviewer_xpuk", "summary": "This paper presents an experimental evaluation of episode-based RL (ERL) on a series of simulated tasks, starting from a reacher, box pushing, ball in a cup, to table tennis. The paper starts by proposing a trust-region based ERL algorithm, and evaluates it against step-based approaches on setting with sparse reward functions, and show it performs better than PPO, even with dense rewards.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Well motivated problem setting\n2. A large range of simulated environments used in the evaluation\n\nWeaknesses:\n1. The contributions are minor beyond experimental evaluation in simulation. The paper reformulates and existing approach for an episodic setting, and evaluates it using a motion primitive algorithm.\n2. Since all evaluation is in sim, and most experiments take the order of 10M steps, real-world transfer of the approach is hard to determine.\n3. The experiments are somewhat hard to understand. The states and action spaces are not well-defined, the cost functions are not clear, and the plots are extremely hard to read. For example Figurers 2,3 don't have titles or legends which makes it hard to understand the results. I wrote my own legends on the figures to make sense of the analysis. I would recommend authors to update these figures to make them easier to follow.\n4. Why do authors compare against PPO, but not TRPL? Since PPO does better than BBRL-PPO, I would suspect that TRPL might do better than BBRL-TRPL. Authors should add that comparison, and use it as oracle with dense rewards maybe?\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper's contributions are minor, and the applicability to real robots is questionable due to large number of samples needed. The paper has some really interesting simulation environments though, and I appreciate such a thorough experimentation with competitive baselines.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. It would be good to add an additional baseline of TRPL in the comparisons.\n2. Authors should improve the quality of figures and experimental section\n3. Overall technical contributions are somewhat minor."}}]

Author rebuttal:

Decision: Accept (poster)