Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning
Authors: Ge Li
Hongyi Zhou
Dominik Roth
Serge Thilges
Fabian Otto
Rudolf Lioutikov
Gerhard Neumann
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Trajectory Correlation Modeling
Robot Manipulation
Movement Primitives
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce an efficient Reinforcement Learning (RL) method for robot trajectory generation that effectively captures temporal movement correlations.

Abstract: Current advancements in reinforcement learning (RL) have predominantly focused on learning step-based policies that generate actions for each perceived state. While these methods efficiently leverage step information from environmental interaction, they often ignore the temporal correlation between actions, resulting in inefficient exploration and unsmooth trajectories that are challenging to implement on real hardware. Episodic RL (ERL) seeks to overcome these challenges by exploring in parameters space that capture the correlation of actions. However, these approaches typically compromise data efficiency, as they treat trajectories as opaque black boxes. In this work, we introduce a novel ERL algorithm, Temporally-Correlated Episodic RL (TCE), which effectively utilizes step information in episodic policy updates, opening the 'black box' in existing ERL methods while retaining the smooth and consistent exploration in parameter space. TCE synergistically combines the advantages of step-based and episodic RL, achieving comparable performance to recent ERL methods while maintaining data efficiency akin to state-of-the-art (SoTA) step-based RL.

Published as a conference paper at ICLR 2024
OPEN THE BLACK Box: STEP-BASED POLICY UPDATES FOR CAMPORALLY-CORRELAN EPISODIC REINFORCEMENT LEARNING
Ge Lit Hongyi Zhou' Dominik Roth
Serge Thilges
Fabian Otto
Rudolf Lioutikov!
Gerhard Neumann!
Karlsruhe Institute of Technology, Germany 3 Bosch Center for Artificial Intelligence, Germany
University of Tübingen, Germany
ABSTRACT
Current advancements in reinforcement learning (RL) have predominantly fo- cused on learning step-based policies that generate actions for each perceived state. While these methods efficiently leverage step information from environ- mental interaction, they often ignore the temporal correlation between actions, resulting in inefficient exploration and unsmooth trajectories that are challeng- ing to implement on real hardware. Episodic RL (ERL) seeks overcome these challenges by exploring in parameters space that capture the correlation of ac- tions. However, these approaches typically compromise data efficiency, as they treat trajectories opaque black boxes. In this work, introduce novel ERL algorithm, Temporally-Coree Correlatdd Episodic RL (TCE), which effectively utilizes step information in episodic policy updates, opening the "black box' in existing ERL methods while retaining the smooth and consistent exploration in parameter space. TCE synergistically combines the advantages of step-based and episodic RL, achieving comparable performance to recent ERL methods while maintaining RL. Our code available
data efficiency akin state-of (SoTA) at httttss://githhi ctm wttp://ccccccceeeeeeeece. RL
INTRODUCTION
By considering how policies interact with the environment, reinforcement learning (RL.) methodolo- gies can be classified into two distinct categories: step-based RI (SRL. and episodic RI. (ERL). SRI. predicts actions for each perceived state, while ERL selects an entire behavioral sequence at the start of an episode. Most predominant deep RL methods, such as PPO (Schulman et al. (017)) and SAC (Haarnoja al. 201 8a), fall into the category of SRL. In these methods, the step information comprising state, action, reward, subsequent state, and done signal received by the RL agent at each discrete time step
pivotal for policy updates. This granular data aids in estimating the policy gradient (Williams) 1992, Sutton et al. 1999). approximating state or state-action value functions 201 8a), and assessing advantages (Schulman al. 20156). Although SRL methods have achieved great success in various domains, they often face significant exploration challenges. Exploration in SRL, often based on a stochastic policy like a factorized Gaussian, typically lacks temporal and cross-DoF (degrees of freedom) correlations. This deficiency leads to inconsistent and inefficient exploration across state and action spaces (Raffin et al. 2022, [Schumacher et al. 2023), as shown in Fig. [a Furthermore, the high variance in trajectories generated through such exploration can cause suboptimal convergence and training instability, a phenomenon highlighted by considerable performance differences across various random seeds (Agarwal et al.) 2021).
(Haarnoja et 200
Episodic RI, contrast to SRL,, represents a distinct branch of RI. that emphasizes the maximiza- tion of returns entire episodes (Whitley al. [1993] Igel 2003 Peters & Schaal 22008), rather than focusing on the internal evolution of the environment interaction. This approach shifts the so- lution search from per-step actions to a parameterized trajectory space, employing techniques like
"orresponning author. Email Egeli.brucc.aie gmail.com, ge.li@kit.edu>>

---

Published as a conference paper at ICLR 2024
Neai
Rove
-02
-02
-044
-44
(a) Step-based RL
(b) Traj.bbased, Episodic RL.
(c) TCE (ours)
Figure Illustration of exploration strategies: (a) SRL samples actions by adding noise to the predicted mean, resulting in inconsistent exploration and jerky actions. However, their leverage of step-based information leads efficient policy updates. (b) ERI. samples complete trajectories in a parameter space and generate consistent control signals. Yet, they often treat trajectories single data points and overlook the step-based information during the interaction, causing inefficient policy update. (c) TCE combines the benefits of both, using per-step information for policy update while sampling complete trajectories with broader exploration and high smoothness.
Movement Primitives (MPs) Schaal 2006 Paraschos et al. 2013). Such exploration strategy allows for broader exploration horizons and ensures consistent trajectory smoothness across task episodes, illustrated in Fig. [bb Additionally, it is theoretically capable of capturing temporal correlations and interdependencies among DoF. ERL typically treats entire trajectories as single data points, of- ten overlooking the internal changes in the environment and state transitions. This approach leads to training predominantly using black-box optimization methods (Salimans et al. 2017; (Tankaraat] al., 201 Celik et al. 2022, Otto al. 2022). The term black box in our title reflects this reliance on black-box optimization, which tends to overlook detailed step-based information acquired dur- ing environmental interactions. However, this often results in a lack of attention to the individual contributions of each segment of the trajectory to the overall task success. Consequently, while ERL excels expansive exploration and maintaining trajectory smoothness, it typically requires a larger volume of samples for effective policy training. In contrast, step-based RL methods have demon- strated notable advancements in learning efficiency by utilizing this detailed step-based information.
Open the Black Box. In this paper, our goal to integrate step-based information into the policy update process of ERL. Our proposed method, Cemporally-Correlatee Episodic RL (TCE), moves beyond the traditional approach treating an entire trajectory as single data point. Instead, we transform wrdeectory-wide elements, such as reproducing likelihood and advantage, into their segment-wise counterparts. This enables us to leverage the step-base base- information to recognize and accentuate the unique contributions of each trajectory segment to overall task success. Through this innovative approach, we have opened the black box of ERL, making it more effective while retain- ing its strength. As a further step, we explore the benefits of fully-correlated trajectory ex exploration in deep ERL. We demonstrate that leveraging full covariance matrices for trajectory distributions significantly improves policy quality existing black-box ERL methods like OOto et al. (2022).
Our contributions are summarized as: (a) We propose TCE, a novel RL framework that integrates step-based information into the policy updates ERL, while preserving the broad exploration scope and trajectory smoothness characteristic of ERL. (b) We provide an in-depth analysis of exploration strategies that effectively capture both temporal and degrees of freedom (DoF) correlations, demon- strating their beneficial impact policy quality and trajectory smoothness. (c) We conduct com- prehensive evaluation of our approach on multiple simulated robotic manipulation tasks, comparing its performance against other baseline methods.
2 PRELIMINARIES
2. EPISODIC REINFORCEMENT LEARNING
Markoy Decision Process (MDP). We consider a MDP problem of a policy search defined by a tuple (S, A. T, R. Po. y). We assume the state space S and action space A are continuous and the transition probabilities T S x + [0, 1] describe the state transition probability to s,41, given the current state 8t S and action a; A. The initial state distribution is denoted Po S
2

---

Published as a conference paper at ICLR 2024
[0, 1]. The reward (s,, at) returned by the environment is given by function R S x A - R and y E [0. I] describes the discount factor. The goal of RL in general is to find a policy x that maximizes the expected accumulated reward, namely return, R BT, .o. 1D rr...
Episodic RL (Whitley et al. 1993) focuses on maximizing the return R - E-=E==o''n over a task episode of length T,, irrespective of the state transitions within the episode. This approach typically employs a parameterized trajectory generator, like MPs (Schaal (006), to predict a trajectory param- eter vector This vector is then used to generate complete reference trajectory y(w) = [ The resulting trajectory is executed using a trajectory tracking controller to accomplish the task. In this context, 1/+ R D denotes the trajectory value at time t for system with D DoF, differentiating it from the per-step action a used in SRL. It is important to note that, although ERL predicts an entire action trajectory, still maintains the Markov Property, where the state transition probabil- ity depends only on the given current state and action (Sutton & Barto 2018). In this respect, the action selection process ERL is fundamentally similar to techniques like action repeat (Braylan) et al. 2015) and temporally correlated action selection (Raffin et al., 2022] Eberhard et al. (022). In contrast to SRI., ERI. predicts the trajectory parameters as (( uz 18). which shifts the solution search from the per-step action space A to the parameter space W. Therefore, a trajectory parameterized by a vector w is typically treated as a single data point in W. Consequently, ERL commonly employs
black-box optimization methods for problem-solving (Salimans et al. 2017
Otto et al., 2022 The
general learning objective of ERL is formally expressed as
J To (o( w) V* (s)]dw Euwnoow/is](((s w)),
(1)
where wo represents the policy, parameterized by 0, e.g. using NNs. The initial state s E S characterizes the starting configuration of the environment and the task goal, serving as the input to the policy. The To To(w(s) indicates the likelihood of selecting the trajectory parameter w. The term R(s, w) Er Ei=gl"rrl hyrre] represents the return obtained from executing the trajectory, while (=(s) = (ur=ag(ur(s) [)) denotes the expected return across all possible trajectories under policy тH. Their subtraction is defined as the advantage function A(8, w). which quantifies the ben- efit of selecting specific trajectory. By using parameterized trajectory generators like MPs. ERL benefits from consistent exploration, smooth trajectories, and robustness against local optima, as noted by Otto et al. (2022). However, its policy update strategy incurs trade-off in terms of learn- ing efficiency, as valuable step-based information is overlooked during policy updates. Furthermore, existing method like Bahl et al. (2020); Otto et al. (2022) commonly use factorized Gaussian poli- cies, which inherently limits their capacity to capture all relevant movement correlations.
2.2 USING MOVEMENT PRIMITIVES FOR TRAJECTORY REPRESENTATION
The Movement Primitives (MP), as parameterized trajectory generator, play an important role in ERL and robot learning. This section highlights key MP methodologies and their mathemat- ical foundations, deferring more detailed discussion to Appendix B Schaal (2006) introduced the Dynamic Movement Primitives (DMPs) method, incorporating a force signal into a dynamical system to produce smooth trajectories from given initial robot states. Following this, Paraschos et al. (2033) developed Probabilistic Movement Primitives (ProMPs), which leverages a línear basis function representation to map parameter vectors trajectories and their corresponding distribu- tions. The probability of observing a trajectory [V]]= given a specific weight vector distribution p(w) NN Mw: Ew)) is represented as linear basis function model:
[ur] = $ b: Ey:
(2)
p(MM)-= My Ey) = N I.T T T. Ew PorT + o
(3)
Here, ey is zero-mean white noise with variance of The matrix Po.r houses the basis functions for each time step f. Additionally, p([ye]=-OT; Hy Σn) defines the trajectory distribution coupling the DoF and time steps, mapped from p(w). For D-DoF system with N parameters per DoF and T time steps, the dimensions of the variables in Eq. (2) and
3 are as follows: w. Mw D N; E,,, D N x D N; Po.T D. N x D T; y. My D T: E, D T.
Recently, Li et al. 2023) introduced Probabilistic Dynamic Movement Primitives (ProDMPs), a hy- brid approach that blends the pros of both methods. Similar to ProMP, ProDMPs defines trajectory
3

---

Published as a conference paper at ICLR 2024
Dol'
Dolf 2
1
f1
j1
11.
1,2
1.3
1,4
1.5
11.
1177
01,8
12
H2
02,3
DoF
Select
fg
Ma
3.1
0,2
73.3
33.4
78.0
33.6
8.7
03.8
M1
01.1 01,3
01.0 01,7
and lo
4
03.1 0a,3
f
MA
5..1
66.2
633
66.4
75.0
0G.6
67.7
06.8
M4
05,1
05,3 05.0 17.7
1z
01,1
DoF2 2
ls
j47
01.1 07.2
07.3
07,,4
(77.5
07.6
07.7
07.6
08.1 08,2 68.3 da,4 (8..5
d8,7
Mean
Covariance Matrix (8 x
Sub-Mean Sub-Covariance (4
Figure 2: Reduce the trajectory distribution dimensions using two time steps eL al. 2023 shown in an element-wise format. Here, the trajectory has two DoF and four time steps, with D Left: The 8-dim mean vector and the 8 x 8-dim covariance matrix of the original trajectory distri- bution, capture correlations across both DoF and time steps. Right: Randomly selecting two time points, e. g. I and la, yields reduced distribution while still capturing the movement correlations.
8.
as v() = T() c1 (1) + 02V2(1). The added terms 61 (1) c2)2(() are included to ensure accurate trajectory initialization. This formulation combines the distributional modeling benefits of ProMP with the precision in trajectory initiation offered by DMP.
2.3 REPRESENTATION OF TRAJECTORY DISTRIBUTION AND LIKELIHOOD
Computing the trajectory distribution and reconstruction likelihood is crucial for policy updates in ERL. Previous methods like Bahl et al. (220)): Otto et al. (2022) represented the trajectory distri- bution using the parameter distribution p( w) and the likelihood of a sampled trajectory y² with its parameter vector as p(w" I I ) However, this approach treats entire trajectory singular data point and fails to efficiently utilize step-based information. In contrast, research imitation learning, including works by Paraschos et al. (201)) Gomez-Gonzalez al. (2016), maps parameter distributions to trajectory space and allows the exploitation trajectory-specific information. Yet, the likelihood computation in this space is computationally intensive, primarily due to the need to invert a high-dimensionall covariance matrix, a process with an O((D time complexity. Recent studies, like those by (Seker al. 2019 Akbulut et al. 2021 Przystupa al. 2023), advocates for directly modeling the trajectory distribution using neural networks. These methods typically employ factorized Gaussian distribution. NV ( 3 instead of full Gaussian distribution N(y Myy En) that accounts for both the DoF and time steps. This choice mitigates the computational burden of likelihood calculations, but comes the cost of neglecting key temporal correlations and interactions between different DoF. To address these challenges, Li et al.] (2023) introduced a novel approach for estimating the trajectory likelihood with set of paired time points (t&, t,,, 1,... K, as
K
(gg)(()))=00
R
K
Σ
log MogN(Wcanss E(e,))))
As shown in Fig. 2 this method scales down the dimensions of trajectory distribution from D.T T to a more manageable D 2. Through the use of batched, randomly selected time pairs during training, the method is proved efficiently capture correlations while reducing computational cost.
2.4 USING TRUST REGIONS FOR STABLE POLICY UPDATE
In ERL, the parameter space W typically exhibits higher dimensionality compared to the action space A. This complexity presents unique challenges in maintaining stable policy updates. Trust regions methods (Schulman et al. 2015a 2017) has long been recognized as effective technique for ensuring the stability and convergence of policy gradient methods. While popular methods such PPO approximate trust regions using surrogate cost functions, they lack the capacity for exact enforcement. To tackle this issue, Otto et al. (202T) introduced trust region projection layer (TRPL), mathematically rigorous and scalable technique that precisely enforces trust regions in deep RI.
4

---

Published as a conference paper at ICLR 2024
RL Policy me for Trajectory Parameter Selection
ProDMI' Compute Traj. [=00
Environment
Observation
Policy Nots
Predict
Sample Traj. Parameter
Interaction
2n
N D
Σ
w
[ t0:22
Rohot Initial State you Yo
Trajj Segments
ProDMPs
tjk
Compuie Segment Mean and Cov.
Policy Update
M (tniit 1 (
tox C
Maximize Segment
Compute Segment Likelibood. Eq. 7
Mty b/
Advantage-wcigheed
Likelihood Eq.
p(
Segment Evaluation
Segment states and rewards
Segment Values
V-fune. Net
Policy Gradient Update 0 0 + VeJ,
Compute Segment Advantage, Fq.
[[
V (( V(aig)
Stg 86
Constraint to Trust Region of Mm and Ew
the Cm
Segment return Rs
Figure 3: The TCE framework. The entire learning framework can be divided into three main parts. The first part, shown in green arrows, involves trajectory sampling, generation, and exeeu- tion, detailing how the robot is controlled to complete a given task. The second part, indicated in blue arrows, focuses on estimating the likelihood of selecting a particular segment the sampled trajectory. The third part, marked by red arrows, deals with segment evaluation and advantage com- putation, assessing how much each segment contributes to the successful task completion.
algorithms. By incorporating differentiable convex optimization layers (Agrawal et al. 2019). this method not only allows for trust region enforcement for each input state, but also demonstrates significant effectiveness and stability in high-dim parameter space, as validated in method like BBRL Otto et al. (2022). The TRPL takes standard outputs of a Gaussian policy-namely, the mean vector and covariance matrix Σ and applies a state-specific projection operation to maintain trust regions. The adjusted Gaussian policy, parameterized by μı and Σ, forms the basis for subsequent computations. Let dynaan and deov be the dissimilarity measures, e. g. KL-divergence, for mean and covariance, bounded by en and ex respectively. The optimization for each state s is formulated as:
s.t. dissnn (fte Mold (s)) S Eus and arg min dmsaan (iss ((s)), n.
(5)
arg min door (Ee. E(s)). s.t. doov (E.. Eolo(s) S Σ,
3 USE STEP-BASED INFORMATION FOR ERL POLICY UPDATES
We introduce an innovative framework of ERL that builds on traditional ERL foundations, aiming facilitate efficient policy update mecha- nism while preserving the intrinsic benefits of ERL. The key innovation lies in redefining the role of trajectories in the policy update process. In contrast to previous methods which consider entire trajectory as a sin- gle data point, our approach breaks down the trajectory into individual segments. Each segment is evaluated and weighted based its distinct contribution to the task success. This method allows for more effective of based information in ERL. The comprehensive structure of this framework is depicted in Figure
Veg
Figure 4: Divide a trajec- tory into K segments
Trajectory Prediction and Generation. As highlighted by green arrows in Fig, ture similar to previous ERL works, such as the one described by Otto et al. (022).. However, this part distinguish itself by using the most recent ProDMPs for trajectory generation and distribution
adopt struc-
5

---

Published as a conference paper at ICLR 2024
modeling, due the improved support for trajectory initialization. Additionally, we enhance the previous framework by using a full covariance matrix policy л (w]ss) s) = ww Ew) as opposed to factorized Gaussian policy, to capture a broader range of movement correlations.
Trajectory Likelihood Representation. In RL, the likelihood of previously sampled actions, along with their associated returns, is often used adjust the chance of these actions being selected in fu- ture policies. In previous ERL methods, this process typically involves the probability of choosing an entire trajectory. However, our framework adopts a different strategy, as shown in blue arrows in Fig. 3 Using the techniques in Sections 2.2 and 2.3 our approach begins by selecting K paired time steps. We then transform the parameter likelihood into trajectory likelihood, which cal- effectively divides the whole trajectory into K distinct segments, with each segment defined by a pair of time steps. In essence, this method allows us to break down the overall trajectory likelihood into individual segment likelihoods, offering a more detailed view of the trajectory's contribution to task success.
culated using these K pairwise likelihoods. This approach, depicted in Figure 4
Trajectory to Segments:
P(VA))orr (s) A (v((vol-eaat (asea11))) [s)]x-I..KK
(6)
Local Representation:
p Plyvoletaat (s) A p(yc yi; (2) ()
Definition of Segment Advantages. Similar to standard SRL methods, we leverage the advantage function to evaluate the benefits of executing individual segments within a sampled trajectory. When being at state St, and following the trajectory segment [yr]r=tastt) the segment- advantage function A A(st. [y0] ) quantifies the difference between the actual return obtained by executing this sampled trajectory segment and the expected return from a randomly chosen segment, as
t-te--
A (wll==ea)))
Σ 1mmt
Te ri + 742 VTa Tau
V (81x);
(8)
where V (st,) denotes the value function of the current policy. In our method, the estimation
of Viaal (st,) is consistent with the approaches commonly used in SRL and is independent of the design choice of segment selections. We use NNs to estimate (*)(s) V)((s which is fitted on
targets of true return or obtained by general advantage estimation (GAE) (Schulman et al. 2015b).
Learning Objective. By replacing the trajectory likelihood and advantage with their regment-based counterparts in the original ERL learning objective stated in Eq. objective of our method as follows
propose the learning
1
K
Peomee ([Vele=euee (s) A ATaM (Stur (vn)==on:)) Prou (1) ( (s)
((0) = Errss
K
k=1
Here, s denotes the initial state of the episode, used for selecting the parameter vector w. and 8t, is the state of the system time f.. The learning objective takes the Importance Sampling update policies using data from previous policies (Schulman et al., 2015a [017] Otto et al.] (021). Our method retains the advantages of exploration in parameter space and generating smooth trajectories. This enables us to enhance the likelihood of segments with high advantage and reduce the likelihood of less rewarding ones during policy updates. To ensure a stable update for the full covariance Gaussian policy To w/ s) = [ E.,), we deploy differentiable Trust Region Projection step (Oto et al. 202T) after each policy update iteration as previously discussed Section 2.4
RELATED WORKS
Improve Exploration and Smoothness in Step-based RI.. SRL methods, such as PPO and SAC, interact with the environment by performing a sampled action at each time-step. This strategy often results in a control signal with high-frequency noise, making it unsuitable for direct use in robotic systems (Raffin et al. 2022). A prevalent solution is to reduce the sampling frequency, technique commonly known as frame skip (Braylan et al. 2015). Here, the agent only samples actions every k-th time step and replicates this action for the skipped steps. Similar approaches decide whether to repeat the last action or to sample a new action in every time step (Biedenkapp et al.) 2021 Yu et al. 2021). This concept is also echoed in works such general State Dependent Exploration

---

Published as a conference paper at ICLR 2024
(gSDE) (Raffin al. 2022 Rückstieeß et al. 2008 Chiappa et al. 2023 where the applied noise is sampled in a state dependennn fashion; leading to smooth changes of the disturbance between consecutive steps. However, while these methods improved the smoothness in small segments, they struggled to model long-horizon correlations. Another area of concern is the utilization of white noise during sampling, which fails to consider the temporal correlations between time steps and results in a random walk with suboptimal exploration. To mitigate this, previous research, such as Lillicrap et al. (2015) and Eberhard et al. (2022), have integrated colored noise into the RL policy, aiming to foster exploration that is correlated across time steps. While these methods have shown advantages over white noise approaches, they neither improve the trajectory's ' smoothness, nor adequately capture the cross-DoF correlations.
Episodic RI.. The early E approaches used black-box optimization to evolve parameterized poli- cies, e.g., small NN (Whitley et al., 1993 Igel 2003 Gomez et al. 2008). However, these early works were limited to tasks with low- dimensiooon action space, for instance, the Cart Pole. Although recent works (Salimans et al. 2017 Mania et al. 2018) have shown that, with more computing, these methods can achieve comparable asymptotic performance with step-based algorithms some lo- comotion tasks, none of these methods deal with tasks with context variations (e.g., changing goals). Another research line in ERL works with more compact policy representation. [Peters & Schaal (2008): Kober & Peters (2008) first combined ERL with MPs, reducing the dimension of searching space from NN parameter space to MPs parameter space with the extra benefits of smooth trajectories generation. Abdolmaleki al. (2015) proposed a mosel-based method improve the sample efficiency. Notably, although those methods can already handle some complex manipulation tasks such as robot baseball (Peters & Schaal 2008), none of them can deal with contextual tasks. To deal with that problem, (Abdolmaleki et al., 2017) further extends that utilizes linear policy con- ditioned on the context. Another recent work from this research line (Celik et al. (022) proposed using a Mixture of Experts (MoB) to learn versatile skills under the ERL framework.
BBRL. As the first method that utilizes non-linear adaptation to contextual ERI., Deep Black Box Reinforcement Learning (BBRL.) (Otto et al., 2022) is the most related work to our method. BBRI. applics trust-region-coostrannee policy optimization to learn a weight adaptation policy for MPs. Despite demonstrating great success in learning tasks with sparse and non-Markovian rewards, it requires significantly more samples to converge compared to SoTA SRL methods. This could be attributed to its black- box nature, where the trajectory from the entire episode is treated as a single data point, and the trajectory return is calculated by summing up all step rewards within the episode.
EXPERIMENTS
We evaluate the effectiveness of our model through experiments on a variety of simulated robot ma- nipulation tasks. The performance of TCE compared against well-known deep RL algorithms as well as methods specifically designed for correlated actions and consistent trajectories. The evalu- ation is designed to answer the following questions: (a) Can our model effectively train the policy across diverse tasks, incorporating various robot types, control paradigms (task and joint space), and reward configurations?
(b) Does the incorporation of movement correlations lead to higher task success rates? (c) Are there limitations trade-offs when adopting our proposed learning strategy?
For the comparative evaluation, we select the following methods: PPO, SAC, TRPL, gSDE, PINK (Eberhard et al. 2022) and BBRL. Descriptions, seperr-araaeterss and references to the used code bases of these methods can be found in Appendix [C.]] We use step-based methods (PPO, SAC, TRPL. gSD,, and PINK) predict the lower-level actions for each task. On the other hand, for episodic methods like BBRL. and TCF, we predict position and velocity trajectories and then employ PD controller to compute the lower-level control commands. Across all experiments, we measure task success in terms of the number of environment interactions required. Each algorithm is evalu- ated using 20 distinct random seeds. Results are quantified using the Interquartile Mean (IQM) and are accompanied by a 95% stratified bootstrap confidence interval (Agarwal et al. 2021).
5.1 LARGE SCALE ROBOT MANIPULATION BENCHMARK USING METAWORLD
We begin our evaluation using the Metaworld benchmark YYu et al. 2020), a comprehensive testbed that includes 50 manipulation tasks of varying complexity. Control is executed in a 3-DOF task
7

---

Published as a conference paper at ICLR 2024
space along with the finger closeness, and dense reward signal is employed. In contrast to the original evaluation protocol, we introduce a more stringent framework. Specifically, each episode is initialized with randomly generated context, rather than fixed one. Additionally, we tighten the success criteria to only consider a task successfully completed if the objective is maintained until the final time step. This adjustment mitigates scenarios where transient successes are followed by erratic agent behavior. While we train separate policies for each task, the hyperparameters remain constant across all 50 tasks. For each method, we report the overall success rate as measured by the IQM across the 50 sub-tasks in Fig. 5a. The performance profiles are presented in Fig. [5
In both metries, our method sig- nificantly outperforms the base- lines in achieving task success, BBRL. exhibits the second-best performance in terms of over- all consistency across tasks but
TCE (ours)
BBRL
PPO
TRPL
SAC
ESDE
PINK
Ruis with Sacice Rac R-t
Sucasittrr IQM
0.3
01.5
0.6
lags in training speed compared to step-based methods. We attribute this difference to the of per-step dense rewards, which enables faster policy up- dates in step-based approaches. TCE leverages the advantages of both paradigms, surpassing other algorithms after approxi- mately 10" environment interac- tions. Notably, the off-policy methods SAC and PINK were trained with fewer samples than used for on-policy methods due to their limitations in parallel environment utilization. PINK achieved superior final performance but at the expense of sample efficiency compared to SAC. In Appendices C.2 and C,3
0.6
0.4
0.4
0.2
0.4
0.6
0.8
Number Enviromenn Interactions (x107)
(a) Success Rate
(b) Performance Profilc
Figure 5: Metaworld Evaluation. (a) Overall Success Rate across all 50 tasks, reported using Interquartile Mean (IQM) (Agarwal jet al. 202T). (b) Performance profile, illustrating the fraction of runs that exceed the threshold specified on the x-axis.
we provide
the results for 50 tasks and performance profile analysis of TCE.
5.2 JOINT SPACE CONTROL WITH MULTI TASK OBJECTIVES
Next, we investigate the advantages of modeling complete movement correlations and the utility of intermediate feedback for policy optimization. To this end, we enhance the BBRL algorithm by expanding its factorized Gaussian policy to accommodate full covariance (BBRL Cov.), thereby capturing movement correlations. Both the original and augmented versions of BBRL included in the subsequent task evaluations. We evaluate the methods on a customized Hopper Jump task, sourced from OpenAl Gym (Brockman et al., 2016). This 3-DoF task primarily focuses maxi- mizing jump height while also accounting for precise landing at a designated location. Control is executed in joint space. We report the max jump height as the main metric of success in Fig. 6a Our method exhibits quick learning and excels in maximizing jump height. Both BBRL versions exhibit similar performance, while BBRL Cov. demonstrates marginal improvements over the orig- inal. However, they both fall behind TCE in training speed, highlighting the efficiency gains we achieve through intermediate based policy updates. Step-based methods like PPO and TRPL tend to converge to sub-optimal policies. The only exception is gSDE. As augmented step-based method, it enables smoother and more consistent exploration but exhibits significant sensitivity to model initialization (random seeds), evident from the wide confidence intervals.
5.3 CONTACT-RICH MANIPULATION WITH DENSE AND SPARSE REWARD SETTINGS
We further turn to a 7-DoF robot box- nox-pushing task adapted from (Oto et al. 2022). The task requires the robot's 's end-effector, equipped with a rod, to maneuver box to a specified target position and orientation. The difficulty lies in the need for continuous, correlated movements to both position and orient the box accurately. To amplify the complexity, the initial pose of the box is randomized. We test two reward settings: dense and sparse. The dense setting offers intermediate rewards inversely proportional to the current distance between the box and its target pose, while the sparse setting only allocates rewards at the episode's 's end based on the final task state. Performance metrics for both settings are shown in Fig. 6b In either case, TCE and gSDE exhibit superior performance but
8

---

Published as a conference paper at ICLR 2024
TCE (ours)
BBRL Cov,
BBRI.
PPO
TRPL.
SAC
gSDE
PINK
0.8
Mao Jump ltegtt, NQM
Sacoas Ratt, 10M
0.66
Succes Ruit, 3QM
00
0.4
0.4
1.6
0.2
0.-
1.5
0.6
Benation
-107
Nunber Enviooneen Interactions (1207)
Number Esvironannt Interactions (x110)
(a) Hopper Jump, Max Height
(b) Box Pushing, Dense
(c) Box Pushing, Sparse
Figure 6: Task Evaluation of (a) Hopper Jump Max Height. (b) Box Pushing success rate in dense reward, and (c) Box Pushing success rate in sparse reward setting.
with TCE demonstrating greater consistency across different random seeds. The augmented BBRL version outperforms its original counterpart, emphasizing the need for fully correlated movements in tasks that demand consistent object manipulation. The other step-based methods struggle to learn the task effectively, even when dense rewards are provided. This further highlights the advantages of modeling the movement trajectory unified action, opposed to step-by-step approach.
5.4 HITTING TASK WITH HIGH SPARSITY REWARD SETTING
In our last experiment, we assess the limitations of our method using 7-DoF robot table tennis task, originally from (Celik
TCE (ours)
BBRL BBRL Cos,
TRPL.
SAC
OSDE
PINK
et al. 2022). The robot aims to return randomly incoming ball to desired target on the opponent's ' court. To enhance the task's realism, we randomize the robot's 's initial state instead of using fixed starting pose. This task is distinct due to its one-shot na- ture: the robot has only one chance to hit the ball and loses con- trol over the ball's trajectory thereafter. The need for diverse hit- ting strategies like forehand and backhand adds complexity and increases the number of samples required for training. Perfor- mance metrics are presented in Fig. 7 The BBRL Cov. emerges as the leader, achieving a 20% higher success rate than other methods. It is followed by TCE and the original BBRL, with TCE displaying intermediate learning speeds between the two BBRL versions. Step-based methods, led by TRPL mere 15% task success, struggle notably in this setting. We attribute the underperformance of TCE and step-based methods to the task's 's reward sparsity, which compli- cates the value function learning of SRL and TCE. Despite these challenges, TCE maintains its edge over other baselines, further attesting to its robustness, even under stringent conditions.
0.8
Rote
0.1
0.5
Namber Envitonmnnt Incerations (x110)
Figure 7: Table Tennis with High reward sparsity.
CONCLUSION
We introduced TCE that synergizes the exploration advantages of ERL with the sample efficiency of SRI. Empirical evaluation showcases that TCE matches the sample efficiency of SRL and consis- tently delivers competitive asymptotic performance across various tasks. Furthermore, we demon- strated both the sample efficiency and policy performance of episodic policies can be further im- proved by incorporating proper correlation modeling. Despite the promise, several opening ques- tions remain for future work. Firstly, TCE yields moderate results for tasks characterized by partic- ularly sparse reward settings, as observed in scenarios like table tennis. Secondly, ERL approaches often need low-level tracking controller, which might not be feasible for certain task types, such as locomotion. Additionally, the current open-loop control setup lacks the adaptability needed for complex control problems in dynamic environments where immediate feedback and swift adaptation are crucial. These issues will be at the forefront of our future work.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank our colleagues Onur Celik, Maximilian Xiling Li, Vaisakh Kumar Shaj, and Balázs Gyenes at KIT for the valuable discussion, technical support and proofreading. We thank the anonymous reviewers for their insightful feedback which greatly improved the quality of this paper.
The research presented in this paper was funded by the Deutsche Forschungsgemeinscaaff (DFG, German Research Foundation) 448648559, and was supported part by the Helmholtz. Associa- tion of German Research Centers. Gerhard Neumann was supported in part by Carl Zeiss Foundation through the Project JuBot (Jung Bleiben mit Robotern.. The authors acknowledge support by the state of Baden- Württemberg through bwHPC, and the HoreKa supercomputer.
REFERENCES
Abbas Abdolmaleki, Rudolf Lioutikov, Jan Peters, Nuno Lau, Luis Pualo Reis, and Gerhard Neumann. Model-based relative entropy stochastic search. Advances in Neural Information Pro- cessing Systems, 28, 2015.
Abbas Abdolmaleki, Bob Price, Nuno Lau, Luis Paulo Reis, and Gerhard Neumann. Contextual covariance matrix adaptation evolutionary strategies. International Joint Conferences Artificial Intelligence Organization (ICCAI). 2017.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Informa- tion Processing Systems, 2021.
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. Advances in neural information processing systems, 32, 2019.
Mete Akbulut, Erhan Oztop, Muhammet Yunus Seker, X Hh, Ahmet Tekden, and Emre Ugur. Ac- nmp: Skill transfer and task extrapolation through learning from demonstration and reinforcement learning via representation sharing. In Conference Robot Learning, pp. 1896-1907. PMLR, 2021.
Heni Ben Amor, Gerhard Neumann, Sanket Kamthe, Oliver Kroemer, and Jan Peters. Interaction primitives for human-robot cooperation tasks. In 2014 IEFE international conference on robotics and automation (ICRA). pp. 2831-28311 IEEE, 2014.
Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta, and Deepak Pathak. Neural dynamic policies for end-to-end sensorimotor learning. Advances in Neural Information Processing Systems, 33: 5058-5069, 2020.
Lars Berscheid and Torsten Kröger. Jerk-limited real-time trajectory generation with arbitrary target states. arXiv preprint arXiv:2105.04830 2021.
André Biedenkapp, Raghu Rajan, Frank Hutter, and Marius Lindauer. Temporl: I Learning when to act. In International Conference on Machine Learning, pp. 914-924. PMLR, 2021.
Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Mükkulainen. Frame skip is a pow- erful parameter for learning to play atari. In Workshops at the twenty-ninth AAAI conference on artificial intelligence, 2015.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Sylvain Calinon. A tutorial task-parameterizeed movement learning and retrieval. Intelligent service robotics, 9:1-29, 2016.
Sylvain Calinon, Zhibin Li, Tohid Alizadeh, Nikos G Tsagarakis. and Darwin G Caldwell. Statistical dynamical systems for skills acquisition in humanoids. In 2012 12th JEEE-RAS International Conference Humanoid Robots (Humanoids 20/2), pp. 323-329. IEEE, 2012.
10

---

Published as a conference paper at ICLR 2024
Onur Celik, Dongzhuoran Zhou, Ge Li, Philipp Becker, and Gerhard Neumann. Specializing ver- satile skill libraries using local mixture of experts. In Conference on Robot Learning, pp. 1423- 1433. PMLR, 2022.
Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Zixiang Huang, and Alexander Mathis. La- tent exploration for reinforcement learning, Advances in Neural Information Processing Systems (NeurIPS), 2023.
Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
Faustino Gomez, Jürgen Schmidhuber, Risto Mikkkulainen, and Melanie Mitchell. Accelerated neu- ral evolution through cooperatively coevolved synapses. Journal of Machine Learning Research, 9(5), 2008.
Sebastian
Gerhard Neumann, Bernhard Schölkopf. and Jan Peters. Using prob- abilistic movement primitives for striking movements. In 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), pp. 502-508. IEEE, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: arXiv:1812.05905.. 2018b.
Neville Hogan and Dagmar Sternad. Sensitivity of smoothness measures to movement duration, amplitude, and arrests. Journal of motor behavior, 41(6):529-553 534, 2009.
Christian Igel. Neuroevolution for reinforcement learning using evolution strategies. In The 2003 Congress on Evolutionary Computation, 2003. CEC'O3,, volume 4, pp. 2588 2598-5595. IEEE, 2003.
Auke Jan ljspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2): 328-373, 2013.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21, 2008.
Friedrich Lange and Michael Suppa. Trajectory generation for immediate path-accurate jerk-limited stopping of industrial robots. 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 2021-2026. IEEE, 2015.
Ge Li, Zeqi Jin, Michael Volpp, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann. Prodmp: A unified perspective dynamic and probabilistic movement primitives. IEEE Robotics and Automation Letters, 8(4):2325-2332, 2023.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.022 02977, 2015.
Guilherme Maeda, Marco Ewerton, Rudolf Lioutikov, Heni Ben Amor, Jan Peters, and Gerhard Neumann. Learning interaction for collaborative tasks with probabilistic movement primitives. In 2014 IEEE-RAS International Conference Humanoid Robots. pp. 527-534. IEEE, 2014.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Franziska Meier and Stefan Schaal. A probabilistic representation for dynamic movement primi- tives. arXiv preprint arXiv: 1612: 2016.

---

Published as a conference paper at ICLR 2024
Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann. Differ- entiable trust region layers for deep reinforcement learning. International Conference on Learning Representations, 2021.
Fabian Otto, Onur Celik, Hongyi Zhou, Hanna Ziesche, Vien Anh Ngo, and Gerhard Neumann. Deep black-box reinforcement learning with movement primitives. In Conference Robot Learning, pp. 1244-1265. PMLR, 2022.
Fabian Otto, Hongyi Zhou, Onur Celik, Ge Li. Rudolf Lioutikov, and Gerhard Neumann. Mp3: Movement primitive-baser (re-) planning policy. arXiv preprint arXiv:2306. 127299, 2023.
Rok Pahič, Barry Ridge, Andrej Gams, Jun Morimoto, and Aleš Ude. Training deep neural networks for the generation of dynamic movement primitives. Neural Networks, 2020.
Alexandros Paraschos, Christian Daniel, Jan Peters, and Gerhard Neumann. Probabilistic movement primitives. Advances in neural information processing systems, 26, 2013.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 001-68266772222 2008.
Michael Przystupa, Faezeh Haghverd, Martin Jagersand, and Samuele Tosatto. Deep probabilistic movement primitives with a bayesian aggregator. arXiv preprint arXiv:2307.05 2023.
Antonin Rafin,, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. basble-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http::/jmlz http:///ini....gpppppee wttp:ppppppp//222/22.
20- 13.41.34....
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on Robot Learning, pp. 1634-1644. PMLR, 2022.
Leonel Rozo and Vedant Dave. Orientation probabilistic movement primitives on riemannian man- ifolds. In Conference Robot Learning, pp. 373-383, PMLR, 2022.
Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for pol- icy gradient methods. In Walter Daclemans. Bart Goethals, and Katharina Morik (eds.). Ma- chine Learning and Knowledge Discovery in Databases, pp. 234-249, Berlin, Heidelberg, 2008 Springer Berlin Heidelberg. ISBN 978-3-540-87481-2.
Thomas Rückstiess, Frank Sehnke, Tom Schaul, Daan Wierstra, Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforcement learning. Paladyn, 1:14-24, 2010.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.
Stefan Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid robotics. In Adaptive motion of animals and machines, pp. 261-280. Springer, 2006.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 006.02 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.000 2017.
Pierre Schumacher, Daniel F.B. Haeufle, Dieter Büchler, Syn Schmitt, and Georg Martius. Dep-rl: Embodied exploration for reinforcement learning in overactuated and musculoskeletal systems. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), May 2023. URL htttpps///penrevieev..n
d= 1aai-xa______ x
12

---

Published as a conference paper at ICLR 2024
Muhammet Yunus Seker, Mert Imre, Justus H Piater, and Emre Ugur. Conditional neural movement primitives. In Robotics: Science and Systems, volume 10, 2019.
RB Ashith Shyam, Peter Lightbody, Gautham Das, Pengcheng Liu, Sebastian Gomez-Gonzalez, and Gerhard Neumann. Improving local trajectory optimisation using probabilistic movement primitives. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2666-26666 IEEE, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Voot Tangkaratt, Herke Van Hoof, Simone Parisi, Gerhard Neumann. Jan Peters, and Masashi Sugiyama. Policy search with high-dimensional context variables. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Jens Timmer and Michel Koenig, On generating power law noise. Astronomy and Astrophysics, 300:707, 1995.
Darrell Whitley, Stephen Dominic, Rajarshi Das, and Charles W Anderson. Genetic reinforcement learning for neurocontrol problems. Machine Learning, 13:259-285 284, 1993.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Michael Wininger, Nam-Hun Kim, and William Craelius. Spatial resolution of spontaneous accel- erations in reaching tasks. Journal of Biomechanics, 42(1))29-34, 2009.
Chenguang Yang, Chuize Chen, Wei He, Rongxin Cui, and Zhijun Li. Robot learning system based adaptive neural control and dynamic movement primitives. IEEE transactions neural networks and learning systems, 30(3) 2018.
Haonan Yu, Wei Xu, and Haichao Zhang. Taac: Temporally abstract actor-critic for continuous control. Advances in Neural Information Processing Systems, 34:29021-29033, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-wol world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
You Zhou, Jianfeng Gao, and Tamim Asfour. Learning via-point movement primitives with inter- and extrapolation capabilities. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4301-4308. IEEE, 2019.
13

---

Published as a conference paper at ICLR 2024
List of Content in Appendix
A. Algorithm box complementary to Fig.
B. Mathematical formulations of MP methods used for trajectory generation.
C. Experiment settings as complementary to Section
D. Additional evaluation and metrics to prove the effectiveness of TCE.
E. Hyper-parameters selection and sweeping.
A ALGORITHM BOX
Algorithm TTemporally-Correlatee Episodic RI. (TCH)
1: Initialize policy parameters 0 and value function parameters &
2: for iteration = 1, 2, ... do
3: Get the initial state 80
4: Predict the mean P.w. covariance Ew. and sample w*
5: Generate the trajectory y using Eq. (2) and execute it in the environment
6:
Collect step-based information through the execution
7:
Update é use true returm or GAE style return (Schulman et 2015b)
8:
9:
Select K time-pairs, e.g. choose every 10 steps along the trajectory
10:
Compute the segment-wise likelihood {ph fk=1:K using Eq. (6) and under old for update epoch 1, 2, ... do
11:
12:
Make prediction of the mean new covariance Enew under the latest policy new Enforce Trust Region by projecting fo and new through TRPL using Eq. (
13:
14:
Get projected policy =eew represented by jene and Snew
-w
15:
Compute the segment-wise likelihood (prow Ik=1K using Eq. (6 and under new
16:
Update by taking gradient step w.r.t. ((0 in Eq. 99
end for
18:
end for
14

---

Published as a conference paper at ICLR 2024
B MATHEMATICAL FORMULATIONS OF MOVEMENT PRIMITIVES.
In this section, we provide a concise overview of the mathematical formulations of movement prim- itives utilized in this paper. We begin with the fundamentals of DMPs and ProMPs, followed by detailed presentation of ProDMPs. This includes a focus on trajectory computation and the mapping between parameter distributions and trajectory distributions. For clarity, we begin with single DoF system and then present the full trajectory distribution using a multi-DoF systems.
B.1 DYNAMIC MOVEMENT PRIMITIVES (DMPS)
Schaal (006);; : [jspeert et al. (013) describe single movement trajectory [ye]t=o.r, which is governed by second-order linear dynamical system with a non-linear forcing function f. The mathematical representation is given by
-- = = - v) - -D) f(x), f(z) - E Ex(()) - xpI.u Ex((z)
(10)
where = y(t), y - dy/dt, y = / denote the position, velocity, and acceleration of the system at a specific time I, respectively. Constants a and B are spring-damper parameters, signifies goal attractor, and is time constant that modulates the speed of trajectory execution. To ensure convergence towards the goal, DMPs employ a forcing function governed by an exponentially decaying phase variable xtt) exp(-ax/Tii foreing term. The trajectory's shape as it approaches the goal is determined by the weight parameters wy E w. for = 1, I,,,, N. The trajectory [3o]-= is typically computed by numerically integrating the dynamical system from the start to the end point (Pahic et al.] 2020 Bahl et al.] (020). However, this numerical process is computationally intensive, and complicates a directly translation between a parameter distribution p(w) to its corresponding trajectory distribution p(y) (Amor et al. 2014
Here, (x) represents the basis functions for the
Meier& & Schaal 2016). Previous method, such as GMM/GMR (Calinon et al. 2012 Calinon 2016
Yang et al. (0088) used Gaussian Mixture Models to cover the trajectories' domain. However, this
neither captures temporal correlation nor provide a generative model for the trajectories.
B.2 PROBABILISTIC MOVEMENT PRIMITIVES (PROMPS)
Paraschos et al. (013) introduced framework for modeling MPs using trajectory distributions,
capturing both temporal and inter-dimensional correlations. Unlike DMPs that use a forcing term,
ProMPs directly model the intended trajectory. The probability of observing 1-DoF trajectory [yc]==::T given a specific weight vector distribution p(w) N(wlww Ew) is represented as a linear basis function model:
Linear basis function:
[ve]==0:T = DT. E
(11)
Mapping distribution:
P([u))==0T: My: En) = N Lw o..).
(12)
Here, Ey is zero-mean white noise with variance o² The matrix Po.T houses the basis functions for each time step t. Similar to DMPs, these basis functions can be defined in terms of phase variable instead of time. ProMPs allows for flexible manipulation of MP trajectories through probabilistic operators applied to p(w) w)) such as conditioning, combination, and blending (Maeda et al. 2014 Gonez--Gnzalezz et al., 22166) (Shyam et al. 2019 Rozo & Dave, 2022 Zhou et | 2019). However, ProMPs lack an intrinsic dynamic system, which means they cannot guarantee a smooth transition from the robot's initial state or between different generated trajectories.
B.3 PROBABILISTIC DYNAMIC MOVEMENT PRIMITIVES (PRODMPs)
Solving the ODE underlying DMPs Li Liet al. (023) noted that the governing equation of DMPs, as specified in Eq. (10). admits an analytical solution. This is because is second-order linear non-
homogeneous ODE with constant coefficients. The original ODE and its homogeneous counterpart
can be expressed in standard form as follows:
a
f(x)
a3 F(x.9), 729
Non-homo. ODE:
+
- +
72 y
(13)
72
a
Homo. ODE: 5
ーリ 72
0.
(14)
15

---

Published as a conference paper at ICLR 2024
The solution to this ODE is essentially the position trajectory, and its time derivative yields the velocity trajectory. These are formulated as:
w
y = IymP2 M/P1 1/92
9191]
+ 01811 + C292
(15)
9
w
ý = 1b2P2 V1P1
82292
+ crúi + C2//2-
(16)
9
Here, the learnable parameters and g which control the shape of the trajectory, separable from the remaining terms. Time-dependent functions y1, V2 P1, P2. 91, O2 in the remaining terms offer the basic support to generate the trajectory. The functions y1, 322 are the complementary solutions to the homogeneous ODE presented in Eq. (14). and U1, 322 their time derivatives respectively. These time-dependent functions take the form as:
(t) = exp
(-2r1 2T
a
t):
(2(1) texp t).
α 2T (-2T"
(17)
Pi() (1) 72 I/' l' eeex
1
( a
()(l()))pdd 2T
pz(()= 72 0
1
exp
(18)
2T
(t)
(
a 1
1) exp
a 2T 1)++ +1, +1
ga(1)
a 2
exp
(19)
It's worth noting that the p1 and Pa cannot be analytically derived due to the complex nature of the forcing basis terms 92.- As result, they need to be computed numerically. Despite this, isolating the learnable parameters, namely ur and g. allows for the reuse of the remaining terms across all generated trajectories. These residual terms
be more specifically identified as the position and velocity basis functions, denoted as (l) and ((), respectively. When both w and g are included in a concatenated vector, represented as wg. the expressions for position and velocity trajectories can be formulated in manner akin to that employed by ProMPs:
Position: y(t) = (tt w + C101 (t) (2)2(t),
(20)
Velocity:
i({ w,,, + (1 (t))
(21)
In the main paper, for simplicity and notation convenience, we use instead of wig to describe the parameters and goal of ProDMPs.
Smooth trajectory transition The coefficients C1 and C2 serve as solutions to the initial value problem delineated by the Eq. (1) et al.] propose utilizing the robot's initial state or the replanning state, characterized by the robot's position and velocity (yh. (n)) to ensure a smooth com- mencement or transition from a previously generated trajectory. Denote the values of the comple- mentary functions and their derivatives at the condition time to as Va, 32, V1, and j/2, Similarly, denote the values of the position and veloeity basis functions at this time as Fy and $ respectively. Using these notations, Cj and C2 can be calculated as follows:
$2., yo 22, d0 S1, V2, 22, U1
U2, E 02 T
VI, V2, 22, Vin W7g
c1
(22)
C2
20 21, 2h
+
vI, 02, V2, Vi, u7
Big 82, yo, 81,
Substituting Eq. (2 into Eq. 20 and Eq. (I).. the position and velocity trajectories take the form as
y = Ei + Eagn + [Sx t + E, + +b) Tegg.
(23)
ý E. + E2g + [&s P + E + + Tuy
(24)
Here, & for k E (1, serve as intermediate terms that are derived from the complementary functions and the initial conditions. The formations of these terms are elaborated below. To find their derivatives &x. one can simply replace yı, 32 with their time derivatives 91, d2 in the equations.
(4)
Ua, V1 Ur, U2
Ea(()
V1, U2, V1
y1, v2, 22, $1
yı, 1/2, y2, 91,
(t)
Vh.12 122 V1
Et())
y2,y1
y1, 92
3ie Sz. U2g U1.
yr, U2,
U2, U1,
16

---

Published as a conference paper at ICLR 2024
Extend to High DoF system Both ProMPs and ProDMPs can be generalized accommodate high-DoF systems. This allows for the capture of both temporal correlations and interactions among various DoF. Such generalization is implemented through modifications to matrix structures and vector concatenations, as illustrated in Paarascoss et al. (2013) Li et al. (2023). To be specific, the basis functions Φ 4 along with their values at the condition time &s Pb. are extended to block- diagonal matrices V Wr 4P, and ,, respectively. This extension is executed by tiling the existing basis function matrices D times along their diagonal, where D is the number of DoF. Additionally, the robot initial conditions for each DoF are concatenated into one vectors. For instance, the initial positions yb .... y/8 are unified into a single vector yb and velocity trajectories are extended as
[ yb PT. In this way, the position
y Eiyb + Ezgb + KK v + E4 R + Wg
(25)
ý E.g + Ezúr + 1s3 v E. + Tegg
(26)
Parameter distribution to trajectory distribution In manner analogous to the description pro- vided for ProMPs from Equation Eq. to Equation Eq. (3) ProDMPs also exhibits a comparable architecture framework. This similarity is particularly evident in the structure of the learnable pa- rameters, denoted as wg which follow linear basis function format. Consequently, is reasonable delineate the trajectory distribution for ProDMPs in fashion akin that of ProMPs. Given that the parameter distribution w follows Gaussian distribution Wy ~ NNw Hww Ew,) and adhering to the probabilistic formulation analogous to ProMPs indicated in Eq. (3) the trajectory distribution for ProDMPs can be expressed as:
p((n)e=o::: Py: En) = N(((ee)=0:T) My. Ey),
(27)
where
By E1yb + E2gb + H.. Hor = Es v + L. + Nor,
Σy Ew HoT + I &. = (E)
In this context, the trajectory mean, denoted as By constitutes vector dimension DT, whereas the trajectory covariance, represented by Σy is DT x DT dimensional matrix. These quantities serve to integrate the trajectory values across all degrees of freedom (DoF) and temporal steps, encapsulating them within single distribution. This multi-DoF ProDMPs representation can be seen as an enhancement of the ProMPs framework, augmented by the inclusion of initial condition terms. This ensures that the trajectories sampled under this distribution start from the specified initial state. Additionally, the time range = 0 : T is flexible and can be replaced by any set of discrete time points. For instance, in the TCE method, a pair of time points 4, and l can define a trajectory segment, allowing for down-sampling of the trajectory distribution to specific trajectroy segment.
17

---

Published as a conference paper at ICLR 2024
C EXPERIMENT DETAILS
C.I
DETAILS OF METHODS IMPLEMENTATION
PPO Proximal Policy Optimization (PPO) (Schulman al. 2017) prominent on-policy step- based RL algorithm that refines the policy gradient objective, ensuring policy updates remain close the behavior policy. PPO branches into two main variants: PPO-Penalty, which incorporates KL-divergence term into the objective for regularization, and PPO-Clip, which employs clipped surrogate objective. In this study, we focus our comparisons on PPO-Clip due to its prevalent use in the field. Our implementation of PPO is based the implementation of Raffin et al. (2021)
SAC Soft Actor-Critic (SAC) (Haarnoja et al. 2018a employs a stochastic step-baset policy in off-policy setting and utilizes double Q-networks mitigate the overestimation of Q-values for stable updates. By integrating entropy regularization into the learning objective, SAC balances between expected returns and policy entropy, preventing the policy from premature convergence. Our implementation of SAC is based on the implementation of Raffin et al. (2021).
Trust Region Projection Layers (TRPL) (Otto et al. 2021), akin to PPO, addresses the problem of stabilizing the on-policy policy gradient by constraining the learning policy staying close to the behavior policy. TRPL formulates the constrained optimization problem as a projection problem, providing mathematically rigorous and scalable technique that precisely enforces trust regions on each state, leading to stable and efficient on-policy updates. We evaluated its performance based on the implementation of the original work.
TRPL
gSDE
Generalized State Dependent Exploration (gSDE) (Rffinn et al. 2022 Rückstieß et al., 2008 Rückstiess al., (2000) is an exploration method designed to address issues with traditional step-based exploration techniques and aims to provide smoother and more efficient exploration in the context of robotic reinforcement learning, reducing jerky motion patterns and potential damage to robot motors while maintaining competitive performance in learning tasks.
To achieve this, gSDE replaces the traditional approach of independently sampling from a Gaussian noise at each time step with a more structured exploration strategy, that samples in a state-dependent manner. The generated samples not only depend on parameter of the Gaussian distribution μ & E, but also on the activations of the policy network's 's last hidden layer (s). We generate disturbances e using the equation
E = O.s, where 0 ~ Nd (0, E).
The exploration matrix 0, composed vectors of length (im that were drawn from the Gaus- sian distribution we want gSDE to follow. The vector describes how this set of pre-computed exploration vectors are mixed. The exploration matrix is resampled at regular intervals, as guided by the 'sde sampling frequency' (ssf), occurring every n-th step if is our ssf.
gSDE versatile, applicable as a substitute for the Gaussian Noise source in numerous on- and off- off-policy algorithms. We evaluated its performance in an on-policy setting using PPO by utilizing the reference implementation for gSDE from Raffin et al. (2022). In order for training with gSDE to remain stable and reach high performance the usage of linear schedule over the clip range had to be used for some environments.
We utilize SAC to evaluate the effectiveness of pink noise for efficient exploration. Eber- hard et al. (2022) propose to replace the independent action noise of
PINK
a. = te + oe
with correlated noise from particular random processes, whose power spectral density fol- low a power law. In particular, the use of pink noise, with the exponent B = in S(f) [2) x f-8 should be considered (Eberhard et al. 2022).
We follow the reference implementation and sample chunks of Gaussian pink noise using the in- verse Fast Fourier Transform method proposed by Timmer & Koenig (1995). These noise variables are used for SAC's exploration but the the actor and critic updates sample the independent action distribution without pink noise. Each action dimension uses independent noise process which
18

---

Published as a conference paper at ICLR 2024
causes temporal correlation within each dimension but not across dimensions. Furthermore, we fix the chunk size and maximum period to 10000 which avoids frequent jumps of chunk borders and increases relative power of low frequencies.
BBRL-Cov/Std Black-Box Reinforcement Learning (BBRL) OOto et al. 2022 2023) is a recent developed episodic reinforcement learning method. By utiliziig ProMPs (aaasschss al. 2013) as the trajectory generator, BBRL learns policy that explores at the trajectory level. The method can effectively handle sparse and -orrkiioo rewards by perceiving an entire trajectory as a unified data point, neglecting the temporal structure within sampled trajectories. However, on the other hand, BBRL suffers from relatively low sample efficiency due its black-box nature. Moreover, the original BBRL employs degenerate Gaussian policy with diagonal covariance. In this study, we extend BBRL to learn Gaussian policy with full covariance to build more competitive baseline. For clarity, refer to the original method as BBRL-Std and the full covariance version BBRL- Cov. We integrate BBRL with ProDMPs (Li et al. 2023), aiming to isolate the effects attributable different MP approaches.
C.2 METAWORLD PERFORMANCE PROFILE ANALYSIS
The distribution of success rates, reported in the performance profile in Fig. 5b may seem to contra- dict the nearly perfect IQM of TCE but in reality provide insight into the consistency of TCE. Nearly two thirds of runs exceed 99% success rate and are therefore able to perfectly solve the task with this seed. The individual performances reported Appendix O show that only very few tasks, e.g., Disassemble and Hammer, have a significant fraction of unsuccessful seeds. This consistency per task is also visible in the profile, as only two percent of runs fall between zero and sixty percent success rate which is visible by the near solve a small set of tasks and therefore show a gap in the profile. This does not contradict the very high IQM success rate of TCE, as the IQM trims the upper and lower 25% of results. The commonly reported median effectively trims 50% and would result in even higher values. Due to the smaller and later decline in the profile compared the other methods, the intersection between 75% of runs and the profile is located at a success rate of 90%. Therefore, only small fraction of runs, roughly ten percent, fall within the 25% trim but only slightly decrease the value of the IQM due their high success rate. Other methods have larger fraction of imperfect runs with lower success rate within the quartiles.
slope in this range. All methods are entirely unable to
19

---

Published as a conference paper at ICLR 2024
C.3 PERFORMANCE ON INDIVIDUAL METAWORLD TASKS
We report the Interquartile Mean (IQM) of success rates for each Metaworld task. The plots clearly illustrate the varying levels of difficulty across different tasks.
TCE (ours)
BBRL
PPO
TRPL
SAC
gSDE
PINK
Assembly
PickOutOfHole
1.0
1.0
1.0
Suctess rate IQM
Success ratn IQM
Suctess rate 10M
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Number
to
Number Interaction PlatoSlideSide
Number Env
PlateSlidBBack
BackkShdeBackSide
1.0
1.0
Success rate 1QM
Success 10M
Success rate 10M
0.8
0.8
0.6-
0.6-
0.6
0.4
0.4-
0.4
0.2
0.2
0.2
0.0
0.0
Number of Env Interaction
Number Env Interaction
Number of Env Interaction
BinPicking
Hammer
Sweepinto
1.0
1.0
Suoness rate (QM
1QM
Suctews rate 1OM
0.8
0.8
0.8
0.6
0.6
0
1
2
Number of Env Interaction
Number Env Interaction
Number of Env Interaction
BoxClose
BattonPress
ButtonPreseWWll
1.0
1.0
1.0
Surousss rate 10M
Success rain 1QM
Suroesss rate 10M
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2-
0.2
0.2
0.0
0.0
0.0
3
Number
Number Interaction
Number
Button PressTopdoww
PuttonPPesss PraesnnPrrooosowwoeee Wall
CoffeeBution
1.0
1.0
1.0
Sucress rate 1QM
Success pale 10M
Sucvess Tale 10M
0.8-
0.8
0.8
0.6-
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Number of Env Interaction
Numhor of Env Interaction
Numhor of Env Interaction
Figure 8: Success Rate IQM of each individual Metaworld tasks.
20

---

Published as a conference paper at ICLR 2024
TCE (ours)
BBRL
PPO
TRPL
SAC
gSDE
PINK
CoffeePull
CoffeePush
DialTurn
1.0
1.0
Succoss rota: 10M
rala 10M
Success mite 10M
0.8
0.8
Number of Env Interaction
Number of Env Interaction
Number of Eny Interaction
Disassemble
DoorClose
DoorLock
1.0
1.0
1.0-
Suroess ratn 10M
Success rata 10M
Sucoess rate 10M
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
2
3
Interaction
Number Env Interaction
Number Interaction 1et
DoorOpen
DoorUnlock
Handlnsert
1.0
1.0
Sucress rate 10M
Saccess 10M
Sucress rale 10M
0.8
0.8
0.6-
0.6
0.4-
0.4
0.4
0.2
0.2
0.2
0.0
0.0
Number of Env Interaction
Number of Env Interaction
Number of Env Interaction
DrawerClose
DrawerOpen
FaucetOpen
1.0
1.0
Suotess rate 10M
Succres rate 10M
Suctess rate 10M
0.8
0.8
0.6
0.6
0.4
0.2
2
0
Number of Env Interaction
Number Env Interaction
Number of Env Interaction
FaucriClose
HandlePressSide
HandlePress
1.0
1.0
1.0
Surcesss rate 10M
Success ratn 10M
Surcess rate 10M
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
2
3
Number Interaction
Env Interaction
Number
HandloPullSide
HandlePull
LeverPull
1.0
1.0
1.0
Sucross rate 1QM
Success rale 10M
Sucpess rale I0M
0.8
0.8
0.8
0.6-
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
3
Number of Env Interaction
Numhor of Env Interaction
Numhor of Env Interaction
Figure 9: Success Rate IQM of each individual Metaworld tasks.
21

---

Published as a conference paper at ICLR 2024
TCE (ours)
BBRL
PPO
TRPL
SAC
gSDE
PINK
PeginserSiid
PickPlaceWall
Reach
1.0
1.0
Success rate 10M
10M
Success mln 10M
0.8
0.8
2
Number of Env Interaction
Number of Env Interaction
Number of Env Interaction
PushBack
Push
PickPlace
1.0
1.0-
1.0-
Suroass ratn 10M
Surcess 10M
Sucoess ratn 10M
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
2
3
Interaction
Number Interaction
Number
tet
PogUnplugSide
Soccer
StickPush
1.0
1.0
Sucress rate 10M
Saccess 10M
Sucress rale 10M
0.8
0.8
0.6-
0.6
0.4-
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Number of Env Interaction
Number of Env Interaction
Number of Env Interaction
StickPull
PushWall
ReachWall
1.0
1.0
Suotess rate 10M
Succee 10M
Suctess rate 10M
0.8
0.8
0.6
0.6
0.4
0.4
2
Number of Env Interaction
Number Env Interaction
Number of Env Interaction
ShelfPlace
Sweep
WindowOpen
1.0
1.0
1.0
Suroesss rate 10M
Successs ratn 10M
Surcesss rate 10M
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
2
3
Number
100
Number Interaction
WindowClose
Basketball
10M
0.6
0.6
0.4
0.4
0.2
0.2
0.0
Number Env Interaction
Number Env Interaction
Figure 10: Success Rate IQM of each individual Metaworld tasks.
22

---

Published as a conference paper at ICLR 2024
C.4
HOPPER JUMP
TCE (ours)
BBRI Cov.
BBRL.
PPO
TRPL.
SAC
gSDE
PINK
leegt, TQM
Dinance to Coall (ul..Q00 (nn]
Eptcode KQM
15
1.6
022
1.5
107
107
107
(a) Max Jump Height
(b) Distance to Goal
(c) Overall, Height Dist.
Figure 11: Hopper Jump
As addition the main paper, we provide more details on the Hopper Jump task. We look at both the main goal of maximizing jump height and the desired position. These are shown along with
secondary goal of landing the overall episode reward in Fig. [I Our method shows quick learning and does well in achieving high jump height, consistent with what reported earlier. While it's not as strong in landing accuracy, it still ranks high in overall performance. Both versions of BBRL have similar results. However, they train more slowly compared to TCE, highlighting the speed advantage of method due the of intermediate states for policy updates. Looking other methods, step- based ones like PPO and TRPL focus too much on landing distance and miss out on jump height, leading less effective policies. On the other hand, gSDE performs well but is sensitive to the initial setup, as shown by the wide confidence ranges in the results. Lastly, SAC and PINK shows inconsistent results jump height, indicating the limitations of using pink noise for exploration, especially when compared to gSDE.
C.5
Box PUSHING
The goal of the box-pushing task to move a box to a specified goal location and orientation using the 7-DoFs Franka Emika Panda (Otto et al.) 2022). To make the environment more challenging, we extend the environment from a fixed initial box position and orientation orientation. The range of both initial and target box pose varies from æ [0.3, 0.6]. y E [-0.45, 0.45), A2
randomized initial position and
2Tr). Success defined as a positional distance error of less than 5 em and a z-axis orientation error of less than 0.5 rad. We refer to original paper for the observation and action spaces definition and the reward function.
C.6 TABLE TENNIS
The goal of table tennis environment to use the 7-DoFs robotic arm to hit the incoming ball and return it close possible to the specified goal
location. We adapt the table tennis environment from Celik et al. (2022): ;
Otto al. (202)) and extend it to randomized initial robot joint conligura- tion. As context space we consider the initial ball position & 1- y [-0.65, 0.65) and the goal position E [-1.2, 0.2), y ( 0.6.. The task is considered successful if the returned ball lands on the opponent's side of the table and within < 0.2m to the goal location. refer to the
original paper for the observation and action spaces definition and the reward function.
23

---

Published as a conference paper at ICLR 2024
TCE (ours)
BBRL Cov,
BBRL.
PPO
TRPL.
SAC
ESDE
PINK
Eqioodi T-tal Rowand
0.6
Sacoss Rote
01.6
0.4
0.4
0.2
0-
Number Ensinumentt Interaction (x10))
Nanber Envircament Ixteractionn (x
Number Esviromment Interactioo (1)
(a) Success Rate
(b) Hit Rate
(c) Episode Reward
Figure 14: Robot Table Tennis Rand2Rand
D ADDITIONAL EVALUATION AND ABLATION STUDY
D.I TRAJECTORY SMOOTINESS METRIC
We compared the trajectory smoothness all methods in Table To ensure fair comparison, all methods were trained using the fixed start box pushing dense reward setting as originally reported Oito et al. (2022). where each method achieved a minimum 50% success rate. Trajectories for evaluation were generated using the mean prediction of the policy. The smoothness was assessed using three metrics: maximum jerk, mean squared jerk (Wininger et al., (009), and dimensionless jenk (Hogan & Sternad, (2009). The first two metrics standard in robot trajectory generation (Berscheid & Kröger 2 & Suppa 2015). while the last is proposed as more equitable measure of smoothness, eliminating the effects of motion magnitude and time scaling. TCE and BBRL Cov outperformed all other methods in smoothness, followed by the original BBRL. This performance disparity likely stems from the original BBRL's inability to model inter-DoF movement correlations. In contrast, all step-based methods exhibited lower smoothness, attributable to their inherent per-step action selection approach.
Table Trajectory Smoothness, Mean (Std) Three Metries over 400 Trajectories.
Metrie
TCE
BBRL Cov
BBRL
PPO
TRPL
BSDE
SAC
PINK
Maximum Jerk, x 110rmd//"
3.4 (1.9)
3.5 (1.5)
4.3 (1.4)
9.1 (3.3)
12,9 (4.8)
6.9 (2.2)
9.3 (4.2)
6.5 (1.7)
Mean Sq. Jerk, x x10 rad? rodd//" /
(0.2)
(0.3)
0.6 (0.6)
1.3 (0.9)
5.5 (8.6)
0.8 (0.6)
3.9 (1.1)
1.7 (0.7)
Dimensionless x10
(73)
(56)
(67)
(83)
506 (262)
(127)
24

---

Published as a conference paper at ICLR 2024
D.2 ACTION CORRELATIONS PREDICTED BY TRAINED POLICIES
We plot the action correlation coupling DoF and time steps in Fig. 16 All policies were trained under the box-pushing task with dense reward setting. The action outputs for TCE, BBRL, and BBRL Cov are the positions of the robot joints, while step-based methods, such as PPO, predict actions the torque space. TCE and BBRL Cov demonstrate the ability to predict actions correlated both temporally and across DoF, as indicated by the non-zero off-diagonal elements in their correlation matrices. In contrast, the original BBRL translates factorized weight distribution into block-diagonal action correlation matrix, capturing variance within individual DoF but not between them. Similarly, PINK constrained to modeling intra-DoF correlations, which depend solely on the time difference. This limitation arises from the wide-sense stationarity of the noise, resulting in a constant value along each diagonal. gSDE, however, models temporal correlation but only over few consecutive time steps, observable along the diagonal elements. Actions predicted by PPO. TRPL, and SAC lack both temporal and DoF correlation, resulting correlation matrices resembling identity matrices. Interestingly, for methods that only capture intra-DoF correlations, these correlations are uniformly positive. This trend may relate to the control cost in the reward function, promoting consistent movement within each DoF over time. On the other hand, TCE and BBRL Cov are unique in their ability to capture negative correlations, both between and within DoF, enhancing their flexibility in trajectory sampling.
DNFI
DaF2
DUPS
IVF
DUFS
DURS
DVFT
DNFI
DJF2
DVFS
DVF
DIPS
DVPS
TNFT
DVF
DJF3
DyF
DeP4
DVFS
DuFs
DVF7
DaT1
DVF
Da56
DvPG
Da77
D477
Du57
(a) TCE
(b) BBRL Cov
(c) BBRL. (Original, Std)
DoF1
DoF2
DoF3 DuF4 DoFs
DoFS DoF7
De71
02F2
Dx72
DM62
Do73
DV55
1A774
DsF4
140-
DaF5
DV55
DxF6
D=F7
(d) PINK
gSDE
PPO TRPL/ SAC
Figure 16: This figure presents predicted actions" correlation across 7 DoF and 100 steps, visualized a 700x700 correlation matrix. Each two DoF during these steps. Correlation values range from -1 (negative correlation, depicted in blue) to (positive correlation, depicted in red), with white areas indicating correlation. The action outputs for TCE, BBRL. and BBRL Cov are the positions of the robot joints, whereas step-based methods predict actions in the torque space. TCE and BBRL Cov exhibit to higher capacity movement correlations. The original BBRL and PINK only model correlations within each DoF. gSDE models correlations over a few consecutive time steps. We show only one representative matrix for PPO, TRPL, and SAC, as their results are visually identical, typically resulting in matrices resembling the identity matrix.
100 square demonstrates movement correlation hetween
25

---

Published as a conference paper at ICLR 2024
D.3 ABLATION: SAC + MOTION PRIMITIVES-BAN BASED METHOD
Training movement primitive-based methods using standard RL techniques, such as PPO and SAC, generally poses challenges due to the complex, higher-dimensional trajectory parameter space. In the study by Otto et al. (022)) ablation study employing PPO-style trust region (likelihood clip- ping) for training BBRL demonstrated inferior performance compared to the use of differentiable trust region projection layer.
we present an additional ablation study where SAC is used to learn the trajectory parameters of movement primitives. This study compares the performance of SAC with that of the original BBRL and BBRL Cov, leading to relatively poorer performance. The SAC model selected for reporting was the best performer among 40 different combinations of hyperparameters. The hyperparameters adjusted include the output action scaling factor (necessary because the SAC action space bounded by [-1, 1). policy/critic learning rate, batch size, and the size of the policy/critic network. The relatively shorter training curve of SAC can be attributed to its higher computational cost in policy update (Haarnoja et al. 2018b).
In Figure 17
BBRL Cov.
BBRL
SAC + MP
TQM
0.2
Eastoremeer Interactions
Figure 17: By employing the standard SAC method learn trajectory parameters w, we compared its perfor- mance with that the original BBRL and BBRL Cov methods under the box pushing dense reward setting. The ablated method, using SAC, showed relatively poorer performance.
D.4 ABLATION: USING PPO STYLE TRUST REGIONS FOR TCE METHOD
We developed an ablated version of method, incorporating the PPO-style trust region via likeli- hood clipping. We tuned the clipping ratio e between 0.05 and 0.2. As illustrated Figure [8] this version's performance falls between the original TCE and the standard PPO. The movement primi- tives' high-dimennional parameter space limits the effectiveness of likelihood clipping in precisely maintaining the trust region during policy updates. This limitation likely accounts for the perfor- mance gap between TCE and its PPO variant. Nonetheless, the ablated model still demonstrates notable advantage over standard PPO, further substantiating our model's effectiveness temporally correlated trajectory prediction.
TCE (ours)
TCE (PPO)
PPO
Secess Rat. ToM
0G
0.4
Nanber Esvistemnnt Interation (x307)
Figure 18: Training TCE with a PPO-style trust region, employing likelihood clipping with 0.1, yields suboptimal performance the box pushing dense reward setting. Nevertheless, its superior performance com- pared standard PPO underscores our method's 'ss effectiveness episodic trajectory modeling.
26

---

Published as a conference paper at ICLR 2024
D.5 ABLATION: SELECTION OF THE AMOUNT OF SEGMENTS K
We conducted an ablation study to evaluate the effect of varying the number of segments (k) on model performance. The number of segments tested ranged from 2 to 100. Our experiments in- volved training in both dense and sparse box-pushing environments. The results revealed a greater sensitivity the number of segments in the sparse reward environment compared to the dense envi- romment. We attribute this to the challenges associated with the value function approximation under sparse reward settings. However, within an not overly sensitive compared to other optimal range, such as 10-25 segments, this parameter is Consequently, we have adopted k=25 for
all experiments in this paper.
segments=2
segments=5
sogments=100
5ogments=25
segments=50
segments= 100
10M
Succes (tan, 1OM
0.6
0.4
0.4
0.2
0.2
0-
Number Fredeonment Interctioos (x10F)
Number Envinement Internctions 103
(a) Box Pushing, Dense
(b) Box Pushing, Sparse
Figure 19: Study of the number of segment per trajectory.
D.6 TCE Cov vs. STD
We conducted an ablation study to assess the impact of employing a full covariance policy in the TCE framework. This involved comparing the standard TCE with its variant, TCE Std, which utilizes factorized Gaussian policy N(w Mww, o.). The comparison was conducted in scenarios involving both dense and sparse reward settings box pushing tasks. The findings revealed that the ablated version, TCE Std, consistently underperformed compared to the full covariance version. This underperformance is attributed to the limited correlation capacity of the factorized Gaussian policy.
Furthermore, it is important to note that while the factorized Gaussian distribution results in a rela- tively lower computational load in the parameter space. it does not offer marked advantage when translated into trajectory space. As illustrated in Fig. [6]c) of Section D.2, a factorized parame- ter distribution ultimately converts into a blocked diagonal trajectory distribution. Although this format is visually simpler compared to a full trajectory covariance matrix, both share same time complexity in terms of likelihood computation. This computational process is significantly more resource-intensive than that for a purely diagonal matrix. Therefore, we utilize the techniques in Li al. (2023) to apply likelihood estimation and reduce the computational cost.
TCE Cov (ours)
TCD Std (oblatio))
Bui, 10M
Saccen Bate. n2M
0.6
0.4
0.4
0.2
0.2
Number Ervitonment Interactioss (x100)
Nomber Envinomene Internctions 107
(a) Box Pushing, Dense
(b) Box Pushing, Sparse
Figure 20: Study of the TCE Cov vs. TCE Std.
27

---

Published as a conference paper at ICLR 2024
E HYPER PARAMETERS
We executed a large-scale grid search to fine-tune key hyperparameters for each baseline method. For other hyperparameters, relied on the values specified in their respective original papers. Below is a list summarizing the parameters we swept through during this process.
BBRL: Policy net size, critic net size, policy learning rate, critic learning rate, samples per itera- tion, trust region dissimilarity bounds, number of parameters movement DoF.
TCE: Same types of paper-parameters listed in BBRL, plus the number of segments per trajectory. A learning rate decaying scheduler is applied stabilize the training in the end.
Policy network size, critic network size, policy learning rate, critic learning rate, batch size, samples per iteration.
PPO:
TRPL: Policy network size, critic network size, policy learning rate, critic learning rate, batch size, samples per iteration, trust region dissimilarity bounds.
gSDE: Same types of hyper-parameters listed in PPO, together with the state dependent explo- ration sampling frequency (Raffin et al. 2022).
SAC:
Policy network size, critic network size, policy learning rate, critic learning rate, alpha learn- ing rate, batch size, Update-To- To--ata (UTD) ratio.
PINK:
Same types of
listed in SAC.
The detailed hyper parameters used are listed in the following tables. Unless stated other- wise, the notation lin x refers to linear schedule. It interpolates linearly from x 0 during training. The ERL methods (TCE, BBRL) take an entire trajectory as a sample where the SRL methods take time step sample. In this way, one sample in ERL is equivlent T sample of SRL, where T is the length of one task episode.
28

---

Published as a conference paper at ICLR 2024
Table 2: Hyperparameters for the Meta- World experiments. Episode Length T 500
PPO
gSDE
TRPL
SAC
PINK
TCE
BBRL
number samples
16000
1000
GAE A
0.95
0.95
0.95
0.95
discount factor
0.99
0.99
0.99
0.99
0.99
0.a.
n.A.
0.005
n.a.
n.a.
0.005
0.005
n.a.
n.a.
0.0005
n.a.
n.a.
0.0005
0.0005
trust rogion loss coef.
0.a.
0.a.
0.a.
R.a.
optimizer
adam
adam
odam
adam
adam
adam
epochs
10
10
1000
1
50
100
learning rate
3e-4
le.3
5e-5
3e.4
3e-4
3e-4
3e-4
use critic
True
True
Tru
True
True
True
Troe
epachs critic
10
10
1000
50
100
learning rale critic
3e-4
le-3
3e-4
3e-4
3e-4
3e-4
3e-4
minihatches
32
n.a.
n.a.
n.a
n.a.
batch size
500
256
512
buller size
1e6
2e6
learning starts
0
0
10000
le5
0
0
polyak. weight
n.a.
n.a.
Se.3
Se.3
n.a.
n.a.
SDE sampling froquency
D.a.
n.a.
n.a.
n.a.
n.a.
n.D.
entropy cocfticient
0
auto
30te
normalized observations
True
True
False
False
True
False
normalized rewards
True
True
False
False
False
False
False
observation clip
10.0
n.a.
n.a.
n.a.
n.a.
reward clip
10.0
10,0
n.a.
12...
12.0.
n.a.
1....
critic clip
0.2
lin_0.3
n.a.
n.D.
1.S.
1....
11.0.
importance ratio clip
0.2
lin,0.3
n.a.
n.a.
R...
n.a.
n...
hidden layers
[128, 128]
[128, 128]
256]
[256,
[128, 128]
hidlen critic
[128, 128]
[128, 128]
[128, 128]
[256, 256]
[256, 256]
[128, 128]
32]
hidden activation
tanh
tanh
tanh
relu
relu
relu
relu
orthogonal initialization
Yes
No
Yes
fanin
funin
Yes
Yes
initial std
1.0
0.5
1.0
1.0
1.0
1.0
1.0
Movement Primitive (MP) type
B.a.
n.a.
n.a.
n.a.
n.a.
ProDMPs
ProDMPs
number basis functions
B.8.
B.a.
weight scale
0.0.
D.a.
0.1
0.1
goal scale
0.1
0.1
I Lincar Schedule from 0.3 to 0.01 during the first 25% of the training. Then continued with 0.01.
29

---

Published as a conference paper at ICLR 2024
Table 3: Hyperparameters Box Pushing Dense, Episode Length 100
PPO
gSDE
TRPL
SAC
PINK
TCB
BBRL
BBRL Cov.
number samples
48000
80000
48000
152
152
GAE A
0.95
0.95
1123.
n.a.
discount factor
1.0
1.0
0.99
0.99
1.0
fy
11a0.
1.a.
0.005
m.a.
1.3.
0.05
0.1
0.05
n.a.
u.a.
0.00005
n.a.
n.a.
0.000S
0.00025
0.0005
trust region loss coef.
n.a.
n.a.
10
n.a.
n.a.
optimizer
adam
adam
adam
adam
adam
adam
adam
adam
epochs
10
10
20
50
20
20
learning rate
5e-5
le-4
5e-5
3e-4
3e-4
3e-4
3e-4
3e-4
use critic
True
True
True
True
True
True
True
True
epochs critic
10
10
10
50
learning rate critic
1c-4
le-4
lo-4
30-4
3e-4
1e-3
3e-4
3e-4
minibatches
40
n.a.
batch size
2000
512
512
buffer size
2e6
266
learning starts
0
0
lc5
lc5
0
polyak, weight
n.a.
n.a.
Se:3
Se.3
n.a.
n.a.
n.a.
SDE sampling frequency
n.a.
n.a.
n..
n.a.
n.a.
n.a.
n.a.
entropy coefficient
auto
auto
normalized observations
True
True
True
False
False
True
False
False
normalized rewards
True
True
False
False
False
False
False
False
observation clip
10.0
n.a.
n.a.
n.a.
reward clip
10.0
10.0
n.a.
n...
n.a.
R....
D.a.
n.a.
critic clip
0.2
n.d.
n.a.
n.a.
n.a.
n.a.
n.a.
importance ratio clip
0.2
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
hidden layers
[512,
[256, 256]
512]
[256,
[256, 256]
[128, 128]
[128, 128]
[128, 128]
hidden layers critic
[512. 512)
[256. 256]
[512, 512)
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[256. 256]
hidden activation
tanh
tanh
tanh
tanh
tanh
leaky.relu
leaky_relu
leaky_relu
orthogonal initialization
Yes
No
Yes
fanin
fanin
Yes
Yes
Yes
initial std
1.0
0.05
1.0
1.0
1.0
1.0
1.0
1.0
type
n.S..
D.a.
ProDMPs
ProDMPs
ProDMPs
number basis functions
8
weight scale
n.A.
D.A.
n.a.
0.3
0.3
0.3
goal scale
n.a.
D.a.
n.a.
0.3
0.3
0.3
30

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameters for Box Pushing Sparse, Episode Length 100
PPO
gSDE
TRPL
SAC
PINK
TCB
BBRL
BBRL Cov.
number samples
48000
80000
48000
76
GAE A
0.95
0.95
1123.
0.95
n.a.
discount factor
1.0
1.0
0.99
0.99
1.0
fy
11a0.
1.a.
0.005
m.a.
1.3.
0.05
0.1
0.05
n.a.
u.a.
0.00005
n.a.
n.a.
0.000S
0.00025
0.001
trust region loss coef.
n.a.
n.a.
10
n.a.
n.a.
optimizer
adam
adam
adam
adam
adam
adam
adam
adam
epochs
10
10
20
50
20
20
learning rate
5c-4
le-4
5c-5
3e-4
3e-4
3e-4
3e-4
3e-4
use critic
True
True
True
True
True
True
True
True
epochs critic
10
10
10
50
learning rate critic
1c-4
le-4
lo-4
30-4
3e-4
3e-4
3e-4
3e-4
minibatches
40
n.a.
batch size
2000
512
512
buffer size
2e6
266
learning starts
0
0
lc5
lc5
0
polyak, weight
n.a.
n.a.
5e:3
Se.3
n.a.
n.a.
n.a.
SDE sampling frequency
n.a.
n.a.
n..
n.a.
n.a.
n.a.
n.a.
entropy coefficient
auto
auto
normalized observations
True
True
True
False
False
True
False
False
normalized rewards
True
True
False
False
False
False
False
False
observation clip
10.0
n.a.
n.a.
n.a.
reward clip
10.0
10.0
n.a.
n...
n.a.
R....
D.a.
n.a.
critic clip
0.2
n.d.
n.a.
n.a.
n.a.
n.a.
n.a.
importance ratio clip
0.2
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
hidden layers
[512,
[256, 256]
512]
[256,
[256, 256]
[128, 128]
[128, 128]
[128, 128]
hidden layers critic
[512. 512)
[256. 256]
[512, 512)
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[256. 256]
hidden activation
tanh
tanh
tanh
tanh
tanh
leaky.relu
leaky_relu
leaky_relu
orthogonal initialization
Yes
No
Yes
fanin
fanin
Yes
Yes
Yes
initial std
1.0
0.05
1.0
1.0
1.0
1.0
1.0
1.0
type
n.S..
D.a.
ProDMPs
ProDMPs
ProDMPs
number basis functions
8
weight scale
n.A.
D.A.
n.a.
0.3
0.3
0.3
goal scale
n.a.
D.a.
n.a.
0.3
0.3
0.3
31

---

Published as a conference paper at ICLR 2024
Table 5: Hyperparameters for the Hopper Jump, Episode Length T = 250
PPO
8SDE
TRPI.
SAC
PINK
TCE
BBRI.
BBRI Cov.
number samples
8000
8192
8000
1000
64
0.95
0.99
0.95
R.a.
D.a.
0.95
n.a.
n.a.
discount factor
1.0
0.999
1.0
0.99
0.99
1.0
1.0
1.0
0.05
n.a.
n.a.
0.1
0.005
0.0005
0.02
0.00005
trust region loss coef.
10
10
optimizer
udam
adarn
adam
adamm
adam
adam
adam
adam
epechs
10
10
20
1000
50
100
100
learning rate
3c-4
9.5c-5
3e-4
lc-4
2c-4
lc-4
lc-4
lc-4
critic
Truc
True
True
True
True
Truc
True
Truc
epochs critic
10
10
10
1000
50
100
100
leamnigg critic
3e-4
9.5e-5
3e-4
le-4
2e-4
le-4
1e-4
1e-4
number minibatches
40
40
128
n.a
256
256
n.a.
n.a.
buffer size
n.a.
n.a.
n.d.
le6
n.u.
n.a.
n.a.
learning starts
0
10000
le5
0
polyak_weight
n.a.
n.a.
n.a.
5c-3
5e-3
n.a.
n.a.
n.a.
SDE sampling frequeney
n.a.
8
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
entropy coefficient
0
0.0025
0
auto
auto
0
0
normalized obscrvations
True
False
True
False
False
Truc
Falsc
False
normalized rewards
True
False
False
False
False
False
False
False
observation clip
10.0
n.a
n.a.
nhd..
n.a.
11.1.
n.l.
n.d..
reward clip
10.0
10.0
n.a.
R.a.
n.a.
R.B.
n.a.
R.a.
critic clip
0.2
1.4
T.B.
n.l.
n.a.
TLI.
n.a.
T..l.
importance ratio clip
0.2
lin
n.a.
n.a.
n.a.
ILD.
n.a.
n.a.
hidden layers
[32, 32]
[256, 256]
[32, 32]
[256, 256]
[32, 32]
[128, 128]
[32, 32]
[32, 32)
hidden layers critic
[32, 32]
[256,
[32. 32]
[256. 256]
[32,
[128. 128]
[32. 32]
hidden activation
tanh
tanh
tanh
relu
selu
reaky
tanh
tanh
orthogonal initialization
Yes
No
Yes
famin
fanin
Yes
initial
1.0
0.1
1.0
1.0
1.0
1.0
MP type
11..L
T1...
D.a..
ProDMPs
ProDMPs
ProDMPs
number basis functions
D.a.
3
3
3
weight scale
1
goal scale
1
32

---

Published as a conference paper at ICLR 2024
Table 6: Hyperparameters for the Table Tennis, Episode Length T 300
PPO
8SDE
SAC
BBRL Cov.
number samples
48000
24000
48000
8
76
GAE A
0.95
0.95
0.95
0.95
discount factor
1.0
1.0
1.0
0.99
0.99
1.0
1.0
1.0
0.3.
11...
0.005
1133.
R.3.
0.005
0.004
0.005
65:
0.a.
I...
0,0005
n.a.
R.a.
0.00025
0.000025
0.001
trust region loss
n.a.
n.a.
optimizer
adam
adam
adam
adam
adam
adam
adam
adam
epochs
10
10
20
1
50
100
100
learning rate
5c-5
le-4
5e-5
3e-4
3e-4
3o-4
3e-4
3e-4
use critic
True
True
True
True
Truc
True
True
True
epochs critic
50
learning rate
le-4
3e-4
minihatches
n.a.
n.a.
batch size
4000
512
512
buffer size
4e6
4e6
learning starts
0
0
le5
165
0
polyak, weight
n.a.
n.a.
n.a
Se:3
5e:3
n.a.
n.a.
n.a.
SDE sampling frequency
n.a.
n.a.
na.
n.a.
n.a.
n.a.
n.a.
entropy cnefficient
0
auto
auto
0
normalized observations
True
True
True
False
False
Truc
False
False
normalized rewards
True
True
False
False
False
False
False
False
observation clip
10.0
reward clip
10.0
10.0
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
critic clip
0.2
0.2
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
importance ratio clip
0.2
0.2
D.a.
n..
n.a.
n.a.
n.a.
n.a.
hidden layers
[512, 512]
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[256]
[256]
[256]
hidden layers critic
[12)
[256. 256]
[512, 512]
[256,
(256, 256]
[256, 256]
[256)
[256)
hidden activation
tanh
tanh
tanh
tanh
tanh
tanh
tanh
tanh
orthogonal initialization
Yes
Yes
Yes
fanin
fanin
Yes
Yes
Yes
initial std
1.0
0.1
1.0
1.0
1.0
1.0
1.0
1.0
MP type
n.a.
n.a.
n.a.
n.a.
n.a.
ProDMPs
ProDMPs
PmDMPs
number basis functions
D.8.
R....
D.3.
n.a.
n.a.
weight scale
0.3.
0.3.
0.7
goal scale
0.1
33


Reviews:
[{"review_id": "6vPZLpHCql", "reviewer": "Reviewer_dLeV", "summary": "The paper proposes a framework called Temporally-Correlated Eposodic RL, which is able to generate smooth trajectories and capture the movement correlation. In particular, the contributions are:\nA. Change the action space to the parameter space of the smooth trajectories.\nB. Reconstruct the temporal correlation using pair information.\nC. Complete trajectory sampling.\nD. Exact likelihood computation for trajectory segments.", "questions": "1. What does black box in the title stand for? Can you help me accurately define it? Does it mean the dynamics is black-box, or the policy approximator (neural network) is black-box? Though this term is repeated several times in the paper, there is no accurate definition in the paper. Furthermore, since the title says 'open the black box', is your algorithm a white-box algorithm?\n\n2. Is Figure 1 a real experiment, or just an illustrative explanation?\n\n3. It seems that Section 3.1 is just a description of whole-trajectory sampling. Is there any novelty in terms of the exploration here? Doesn't normal RL actor also sample using their own stochastic policy?\n\n4. \n> Notably, the off-policy methods SAC and PINK were trained with less samples than used for on-policy methods due to their limitations in parallel environment utilization.\n\n      What does the limitation refer to here? Why cannot SAC and PINK be deployed to parallel training? (Also, 'fewer' samples, not 'less' samples).\n\n5. In terms of the ablation studies:\n\n- a. The PPO and SAC in this paper are only used to generate low-level actions. However, both of them are general RL frameworks, which should also be applicable even if the action space is changed to the parameter space of the trajectory. And these versions in the parameter space of trajectory are missing as the baselines. You can still claim that these new baselines are just some naive version of TCE since they do not really calculate per-segment credit assignment and there's no exact probability calculation. But, showing how these frameworks work in this new parameter space is still important, since now we can evaluate how this different action space could improve the sample efficiency by itself.\n\n- b. Smoothness comparison. Define a metric for smoothness, and evaluate how all the final trajectories for all the methods perform under this metric. Since you claim that TCE helps the smoothness, then only showing the success rate is not sufficient.\n\n- c. The significance of the exact probability calculation. If you abandon this component and just use the normal p(w) instead of p(y), how would the performance change empirically? \n\n- d. One thing I am wondering is how this temporal correlation helps the sampling complexity. Can you provide the final Sigma for each DoF? Ideally, this matrix should be far from the diagonal matrix, otherwise, we can just use the step-wise algorithm.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. The new parameterization of the trajectory seems to guarantee smoothness by construction.", "weaknesses": "1. The novelty of the algorithm is incremental. It is a direct application of Li et al. 2023 to the RL regime, by (1) changing the action space to the parameter space of trajectories, and (2) enlarging the timestep to the segmentation scale.\n\n2. Many terms in the paper are described with words, which makes it hard to understand. I strongly suggest the authors provide mathematical definitions. For example, \n\n- a. Since this is still an RL approach, define the Markov Decision Process first. What is the action space? What is the state space? Now you can explain how your approach differs from traditional RL more clearly, by providing us the details in this different MDP. A tricky question here is that since MDP is Markovian, why do you need the temporal correlation?\n\n- b. It would be more helpful to define the dimensions of $y$, $w$, $\\Sigma$ and $s$.\n\n- c. What are d_mean and d_cov in Equation (4)? Are they Frobenius distances?\n\n3. As a reader, I feel many unimportant texts are taking too many places, which makes it not easy to understand the paper. For example, Sections 3.1, 3.2, and 3.4 don't need so many places to mention, since they are merely (a) complete-trajectory sampling, (b) GAE, and (c) TRPO-like. Instead, providing the pseudo-code for the whole learning process would be more valuable as a complement to Figure 3. Currently, there are too many technical things and curves in Figure 3, which makes it hard to understand the learning procedure from this single figure.\n\n4. The result is not sufficient to show the significance of the proposed method. In particular,\n\n- a. After checking Appendix C.2, I found that the baselines perform generally well when TCE works well. However, there exist some situations where TCE significantly fails and the other baselines still work well, and these situations are not rare, including LeverPull, StickPull, Hammer, ButtonPressTopdown, and ButtonPressTopdownWall. \n\n- b. Furthermore, the curves for SAC and PINK are terminated after 10M and 27M interactions respectively, whereas the other methods terminate at 50M interactions, which is an unfair comparison. It is hard to show the efficiency of this algorithm unless these baselines are finished. The reason is: in many cases, these baselines would raise their success rates nonlinearly during the training process, and we cannot conclude by only inspecting their intermediate performances.\n\n- c. Several key ablation studies are missing to test the soundness of each proposed component. See the questions.\n\nFinally, there are some typos in the paper. For example, in Equation 5, the left curly brace is missing. In Equation 6, the (|s) looks strange, it might be (s) instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DIUaLteQ1v", "reviewer": "Reviewer_ALmz", "summary": "By using parametric trajectory generators such as Movement Primitives (MP), episodic reinforcement learning techniques tackle robot trajectory generation issues by rephrasing them as contextual optimization. Although these methods are efficient in producing smooth trajectories and detecting certain movement correlations, they do not make use of the temporal structure present in trajectories, leading to less sample efficiency. To overcome these problems, the authors provide the Temporally-Correlated Episodic RL (TCE) technique. By sampling multi-second trajectories in a parameterized space, TCE improves exploration efficiency and ensures high-order trajectory smoothness and movement correlation capture. TCE divides the whole trajectory into smaller parts for policy changes, assessing each part according to its unique benefits.", "questions": "The authors are suggested to experiment with some real scenes. The simulation environments now set by the authors can not provide a big enough challenge to verify the robustness of the algorithm.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Figure 3 efficiently summarizes the entire learning framework.\n\n2. The code base and config files are provided for double-blind review, which enhances the credibility of the study.", "weaknesses": "1. I suggest the authors to spend more time re-organizing this paper. The overall description is very messy at present. There is no good connection between the various parts, making it difficult to follow the storyline.\n\n2. Only previous works and their limitations are discussed in the BACKGROUND AND RELATED WORKS section without explaining how they relate to your own work. What's more, there are a lot of claims or conclusions without adding corresponding references, especially in the EPISODIC REINFORCEMENT LEARNING section.\n\n3. The writing of this paper should be improved. The current expression is too redundant, especially in the LIKELIHOOD COMPUTATION OF A SAMPLED TRAJECTORY SEGMENTS section. The author should reasonably remove some content to refine it.\n\n4. The simulation experiment results do not verify the effectiveness of the method. Compared with strong step-based methods, the proposed method does not demonstrate the obvious final performance improvement. Although the variance of the proposed method is the lowest in the CONTACT-RICH MANIPULATION section, the gSDE approach achieves better performance in 2 of 3 tasks. Moreover, the BBRL Cov. algorithm achieves a 20% higher success rate with faster convergence in the HITTING TASK.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFn01HHdOK", "reviewer": "Reviewer_psBV", "summary": "This is an empirical paper that proposes a segment (sub-trajectory) based reinforcement learning method Temporally-Correlated Episodic RL (TCE).\n\nThis method adopts the trajectory-based policy representation using Probabilistic Dynamic Movement Primitives (ProDMP) [Li et. al 2023]. It parameterizes the trajectory as a full Gaussian distribution, whose parameters are predicted by the policy network. This enables temporally coherent exploration by sampling trajectories from the Gaussian distribution.\n\nSubsequently, they divide the trajectory into segments, and run trust-region policy gradient based on the segment-wise advantage function estimate and the projection technique in [Otto et al. 2021], to update the trajectory parameter. To simplify the computation of the likelihood, they utilize the pairwise technique from Li et. al (2023). \n\nThe efficacy of this approach is demonstrated through a number of robotics manipulation tasks.", "questions": "Is there a feasible approach to directly represent each segment, rather than representing the entire trajectory?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Good empirical performance across a variety of tasks and included negative cases like the table tennis with sparse rewards\n- The approach is straight-forward and bridges the gap between the extremes of step-based and trajectory-based reinforcement learning.", "weaknesses": "1. The terminology “episodic RL” could potentially lead to confusion, as it bears similarity to the concept of “episodic task” contrasted with “continuing task” [Sutton and Barto, 2018]. A more fitting term, such as “sub-trajectory based RL,” may better capture the essence of the work.\n2. The paper would benefit from a more organized exposition of its contributions. It appears that applying policy gradient to segments is the primary novel component. And this work integrates techniques from Li et al. (2023) and Otto et al. (2021). A clearer differentiation from these prior works would enhance the clarity of the paper.\n3. While hyperparameters are listed in the appendix, the rationale behind their selection remains opaque. Expanding on this would add depth to the methodology.\n4. I feel that the current experimental setup does not sufficiently justify certain design choices. Consider incorporating ablation studies to address the following:  \n    a. The necessity of the differentiable policy projection, as compared to alternatives like those used in PPO.  \n    b. The benefits of employing full covariance over diagonal variance in the trajectory representation, especially in light of the increased computational cost. \n5. Formatting:\n    Certain pages in the main pdf are images, making it impossible to do a text search in the pdf and hindering readability\n6. In Section 2.2, the claim \"The most effective episodic RL approaches directly explore in parameter space\" is presented without supporting references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EgFSpoMCac", "reviewer": "Reviewer_DXh7", "summary": "This paper proposes an approach for episodic reinforcement learning. In such a setting, the policy outputs a high-level trajectory instead of \na single action per state. This idea has been extensively explored (e.g., hierarchical reinforcement learning). The main technical innovations of this work are (1) optimizing over trajectory chunks instead of the whole trajectory and (2) modeling the correlation between states observed during the chunks. The paper is extensively evaluated in multiple benchmarks.", "questions": "I would like to see a detailed derivation of the objective in the proposed multi-step setup. In addition, I would appreciate a discussion and possibly experiments on the value of K and a more in-depth analysis to justify the experimental setup.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The approach is technically sound. While the change from previous work (Otto et al. 202; Li et al. 2023) is relatively limited, the proposed innovations are novel. \nThe extensive experiments show both the advantages and limitations of the approach.", "weaknesses": "The main weaknesses I see in the approach are:\n1. Integrating over multiple timesteps breaks the markov assumption from which the learning objective in Eq. 8 is derived. This is because, given the multi-state parametrization, a state s_t depends on more than s_{t-1}. This is true not only for the proposed approach but also more generally for every method that introduces a dependency between states (e.g. smoothing). Previous work even noticed that such smoothing can hurt performance (see Smith et al., A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning).\n2. The dependence on a hand-picked segment length K. While having some parts of the algorithm heuristically tuned is generally not a problem, I find this parameter to possibly be challenging to pick (due to its dependence on the task) and time-varying. My intuition is that the problem does not appear in the selected experiments due to their simplicity. I guess it will be much more challenging as soon as the time derivative of the reward drastically changes over time.  A cue in this direction is the failure case on the table tennis task, which shows these characteristics (when the ball is close to the end-effector, the robot needs very high-resolution updates, while a longer horizon is sufficient in the waiting times. This problem is not unusual in most robot tasks: for example, in visual navigation or locomotion, the interdependence between states is rarely constant (e.g., when a sudden obstacle appears, one needs to react fast). I think the paper would benefit from studying this aspect in more detail. Could there be a connection between the derivative of the value function and the segment length K? Can it be learned with the task? \n\nA second limitation is in the experimental setup. Specifically:\na) I find some of the results to be difficult to interpret. In Fig. 6b-c, all methods appear to perform similarly in the dense and sparse setting, which I find surprising. Even more so, almost all approaches (including gSDE, which achieves the best performance) perform better with sparse rewards than dense ones. Why is this case? At least for step-wise methods, dense should do better. Why does this happen? \nb) I think that results should be reported for a longer number of interactions. In Fig. 5,6, and 7, the methods do not appear to be fully converged. Showing results up to convergence (as in Fig. 5a) would give a better intuition about the approach's strength in comparison to the baselines.\n\nOther relatively minor limitations:\n1. The baselines are not defined (e.g., what paper is BBRL?)\n2. The comparison of trajectory-based to state-based is not entirely apple to apple since the trajectory methods require an extra low-level controller, possibly specifically tuned to the robot/task. Their data efficiency could be attributed to this prior. In addition, such a low-level controller is difficult to get in some cases, and a simple PD controller won't do (e.g., locomotion). I would mention this disclaimer to the experimental setup.\n3. There is a typo in Eq. 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)