TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning
Authors: Ge Li
Dong Tian
Hongyi Zhou
Xinkai Jiang
Rudolf Lioutikov
Gerhard Neumann
Venue: ICLR 2025 Spotlight (2025)
Keywords: Value of sequences of actions
Reinforcement Learning
Transformer
Robot Manipulation
Movement Primitives.
Primary area: reinforcement learning
TL;DR: We propose a novel transformer-based RL method that learns values of consecutive actions.

Abstract: This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), a novel algorithm that enables off-policy updates in the ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single actions at every time step. These trajectories are typically parameterized by trajectory generators such as  Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing high-level temporal correlations. However, ERL methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for entire action sequences, limiting their sample efficiency and preventing the use of more efficient off-policy architectures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-based critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms state-of-the-art RL methods. Thorough ablation studies additionally show the impact of key design choices on the model performance.

Published as a conference paper at ICLR 2025
TOP-ERL: TRANSFORMER-BASE OFF-POLICY EPISODIC REINFORCEMENT LEARNING
Ge Li
Dong Tian
Hongyi Zhou
Xinkai Jiang
Rudolf Lioutikov
Gerhard Neumann
Karlsruhe Institute of Technology, Germany
ABSTRACT
This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), novel algorithm that enables off-policy updates in ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single per-step actions. These trajectories are typically parame- terized by trajectory generators such as Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing temporal cor- relations. However, ERL. methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for action sequences, limiting their sample efficiency and preventing the use of more efficient off policy architec- tures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-basee critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms RL methods. Thorough ablation studies additionally show the im- pact of key design choices on the model performance. Our code is available here
INTRODUCTION
This work proposes novel off- sofico Reinforcement Learning (RL) algorithm that utilizes trans- former architecture for predicting the values for action sequences. These returns are effectively used update the policy that predicts smooth trajectory instead of single action in each decision step. Predicting a whole trajectory of actions commonly done in episodic RL (ERL) (Kober & Peters, 2008) and differs conceptually from conventional step-based RL (SRL) methods like SAC (Haarnoja et al. 2015 Sa) where an action is sampled in each time step. The action selection concept in ERL is promising as shown in recent works in RL (Otto et al. 2022 Li et al.) 2024). Similar insights have been made in the field of Imitation Learning, where predicting action sequences instead of single ac- tions has led to great success (Zhao et al.) 2023 Reuss et al., 2024). Additionally, decision-maker in ERL aligns with the human's 's decision-making strategy, where the human generally does not de- cide in each single time step but rather performs a whole sequence of actions to complete a task for instance, swinging arm to play tennis without overthinking each per-step movement.
Episodic RL is a distinct family RL that emphasizes the maximization of returns over entire episodes, rather than optimizing the intermediate states during environment interactions (Whitley et al. 1993 Igel 2003 Peters & Schaal 2008). Unlike SRL., ERL shifts the solution search from per-step actions parameterized trajectory space, leveraging techniques like Movement Primitives (MPs) (Schaal) 2006 Paraschos et al. 2013) for generating action sequences. This approach enables broader exploration horizon (Kober & Peters 2008), captures temporal and degrees of freedom (DoF) correlations (Li et al. (024)). and ensures smooth transitions between re-planning phases (Otto)
al. 2023). Recent advances have integrated ERL with deep learning architectures, demonstrating significant potential in areas such as versatile skill acquisition (Celik et al.) (0024) and safe robot reinforcement learning (Kicki et al., 2024). However, despite their advantages, ERL methods often suffer from low update efficiency. Nearly all ERL approaches to date remain constrained to an
'Aceepted Spotlight at ICLR 2025. Email <geli.bruce..it gmail.com, ge.li kit.edu>>

---

Published as a conference paper at ICLR 2025
on-policy training paradigm, limiting their ability to exploit more efficient off-po update rules, where an action-value function, or critic, is explicitly learned to guide policy updates and action selection. The primary challenge is that prominent off-policy methods, such SAC (Haarnoja et al.] 2018a), rely on temporal difference (TD) error (Sutton) 1988) to update the critic, which implicitly assumes that actions are selected based each perceived state, rather than sequence of actions predicted at the start of the episode, as in ERL approaches. In this paper, we address this limitation by predicting the N-step return (Sutton & Barto 2018) for a sequence of actions using a Transformer architecture, enabling the learning of sequence values within an off-policy framework.
Transformer in RL. Over the past few years, the Transformer architecture [Vaswani 2017) has emerged as one of the most powerful models for sequence data. It has been been integrated into RL across various domains, capitalizing on their strengths in sequence pattern recognition from static datasets and functioning as a memory-based architecture, which aids in task understanding and credit assignment. Applications of Transformers in RL. include offline RI. (Chehotar et al. 2023 Yamagata
ct al.
2023 Wu et al. 2024), oflline-to-online fine-tuning (Zheng et al. 2002 Ma & Li [2224] Zhang
ct al. 2023). handling partially observable states (Parisotto et al. 2020; Ni et al. 2024; Lu et al.
2024 J. and model-based RL (Lin et al.) 2023). However, the use of Transformers within model-free online RL framework, specifically for sequence action prediction and evaluation, remains largely unexplored (Yuan al. 2024). This is noteworthy, similar techniques, such action chunking
(Bharadhwaj et al. (2024). 4. have already proven successful in other domains like imitation learning.
In this paper, propose Transformer-basee Off-Policy ERL (TOP-ERL), which leverages the Transformer as a critic to predict the value of action sequences. Given trajectory from ERL, we split it into smaller segments and input them into the Transformer for value prediction. We adapt off-policy update rules for action sequences, using the N-step TD error for critic updates. The policy then selects action sequences based on the preferences of the Transformer critic, similar to SAC. Compared to existing ERL and SRL methods, we show that TOP-ERL improves both policy quality and sample efficiency, outperforming them in several simulated robot manipulation tasks. Our contributions are: (a) A novel off-policy RL method that integrates the Transformer
critic (b) The use of N-step return as the learning objective for the Transformer critic. (c) Comprehensive evaluation on simulated robotic manipulation tasks, demonstrating superior performance against baselines. (d) Analysis of different critic update rules, design choices, and the impact of segment length on model performance.
for action sequences in a model-free, online RL framework.
2 RELATED WORKS
Episodic RL. The study of ERL approaches dates back the 1990s. Early approaches employed black-box optimization techniques to update parameters of policies, such as small MLPs (Whitley jtt al.] 1993 Igel 2003 Gomez et al.] (208). Due to the substantial data requirements of black- algorithms and the limited computational resources available at the time, these approaches were constrained to low-dimensional tasks like Pendulum and Cart Pole. Subsequent works (Salimans) eet al. [017: Mania et al. 2018) demonstrated that, given sufficient computational resources, FRI. methods can also achieve comparable performance to step-based RL on challenge locomotion tasks, such as Ant and Humanoid, at the cost of more samples for convergence. Another line of research in ERL focuses on more compact policy representations. Peters & Schaal (2008) first proposed using movement primitives (MPs) as parameterized policies for ERL, reducing the search space from the high-dimensional neural network parameter space to the MP weight space, which typically ranges from 20 to 50 dimensions, resulting in less samples required for convergence. Using MPs as policies also provides additional benefits, such as smooth trajectory generation and more consistent exploration (Li et al. 2024). MP-based ERL approaches have demonstrated the ability master complex manipulation tasks such as robot baseball (Peters & Schaal 2008) and juggling (Ploeger al. 2021). To further improve sample efficiency, Abdolmaleki et al. (2015) introduced a model- based method to enable more sample-efficient black-box searching. However, these methods are limited in handling tasks with contextual variations, e.g., changing goals. To address this limitation, Abdolmaleki et al. (017)) and Celik et al. (2022) extend MP-based ERL by using linear policies conditioned on context. Otto et al. (2022) enhanced contextual MPRL by employing neural network policies and trust-region regularized policy update. Despite these advances, existing ERL methods generally treat the episodic trajectory as a black box. While this approach allows them to handle sparse and even non-Markovian rewards, ignoring the temporal structure within each episode leads
2

---

Published as a conference paper at ICLR 2025
to lower sample efficiency compared to step-based methods, especially in settings with dense wards. To address this issue, a most recently proposed method, Temporally-Coree Correlated ERL (TCE) (Li et al. 2024) introduced more efficient update scheme that "opens the black-box" and utilizes sub-segment information for policy update while retaining the benefit of episodic exploration. Al- though TCE improves the sample efficiency of contextual ERL. methods, it still relies on-policy policy gradient updates, which are considered sample-inefficiett To the best of our knowledge, TOP-ERL the first off-policy ERL algorithm capable of handling contextual tasks.
Transformers in model-free RL. Inspired by the success of Transformers in domains requir- ing sequence reasoning, the study incorporating Transformers in RL to solve tasks that require long-horizon memory emerged. However, using standard Transformers in RL could results in performance comparable to random policy (Parisotto et al., 2020). To address this issue, Gated Crnnsformer-XL(GTXXXXI (Parisotto et al. 2020) augmented Transformer-X. with GRU-style gat- ing layers between multi-head self-attention layers, stabilizing the training of deep Transformer networks (up to 12 layers) with online RL. Another research line focuses on utilizing Transformers to enhance offline RL, where the learning process is based on a fixed dataset collected by arbitrary behavior policies. Decision Transformers (Chen et al.] (021) were the first to formulate offline RL as a sequence modeling problem. Subsequent works extended this approach by incorporating dy- namic history length adjustment (Wu et al. 2024), Q-learning (Yamagata al. 2023), and replacing the Transformer with a more efficient state-space model OOta [024). Online Decision Transform- ers Zheng al.] 2022) further advanced Decision Transformer by introducing online fine-tuning. In contrast to these studies, which primarily focus on offline RL or fine-tuning pre-trained models, TOP-ERL is designed for online RL and does not rely on offline training. Additionally, TOP-ERL is not designed to solve tasks that require long-horizon memory. Instead, it focuses on using a Transformer- based critic to improve multi-step TD learning within the ERL framework.
PRELIMINARIES
3.1 OFF-POLICY REINFORCEMENT LEARNING
Markoy decision process (MDP). RL. learns policies that maximize cumulative rewards in a given environment, modeled as an MDP. Formally, we consider an MDP defined by tuple (S, where both state S and action spaces A are continuous. Here, P(s'ls, a) denotes the state transition probability, r(s, a) is the reward functionn, and 7 E [0, 1] is the discount factor. The goal of RL is to find policy T(als) that maximizes the expected return, which is the sum of discounted future rewards as G,(st, de) = E. 7're+i-
In off-policy RL, the agent learns a policy x using data generated by different behavior policy (als)). This enables off-policy methods to reuse past experiences, significantly improving sample efficiency against on- on-policy methods. A common approach in off off-policy RL is to use a critic, which estimates the veaiue function Q3T (s, a) and is updated using a temporal difference (TD) error
Q" (s, a) E,, [G. s; s, a = a], & = rr nQ (St+1, at+1)= Q" (sc. at), (I)
where the TD error 5, estimates the difference between the current Q-value and the target Q-value. While the above single-step TD error is useful, can suffer from high bias and slow convergence, especially in environments with delayed rewards. To address this, N-step returns (Sutton 1988) are often used to provide a better balance between bias and variance.
The N-step return extends the single-step TD return by incorporating multiple future time-steps into the target. Unlike bootstrapping after a single time step, the N-step return accumulates rewards over N steps before using the current value estimate for bootstrapping. These estimates are typically less biased than the 1-step return, but also contain more variance. In off-policy settings, the N-step return typically involves importance sampling (Sutton& & Barto, 2018). as the selection of the future action path used to accumulate rewards differs from the current policy π (alss) seen as:
N-1
N-1 P Pets r NM (S++N,04N))
c C (s)
a1) =
Σ
II 1=
PLts
Arees
x(a/مم)) where P: 7,(0,))7) is the importance sampling ratio, ensuring that updates remain unbiased even when using trajectories generated by different policy.
3

---

Published as a conference paper at ICLR 2025
Despite this mathematical correction, applying N-step returns in off- learning can face dif- ficulties, particularly for long sequences. The product of importance ratios can become highly volatile, leading to either exploding or vanishing values over extended trajectories, which in turn can cause high variance in the value estimates and destabilize the learning process. In TOP-ERI, however, we employ N-step return for computing the target value of a sequence of actions, e.e. G
(st, ae, at+1..... 4++N), ) where N-step actions are determined in a sequence read from the re- play buffer, rather than sampled from the policy. Therefore, the resulting formulation does not contain the importance weights. We will further discuss the details in Sec.
3.2 EPISODIC REINFORCEMENT LEARNING (ERL)
Episodic RL (Whitley et al. 1993 Kober& Peters 2008) focuses on predicting an entire sequence
of actions to complete task, optmmiing the cumu laive return without explicitly considering de- tailed state transitions within the episode. Typically, ERI methods utilize a parameterized trajectory generator, such as motion primitives (MP) (Schaal 2006; Paraschos et al., 2013), which predicts trajectory parameter vector w. This vector is then mapped to a full action trajectory a(w) = [ar]:=0- where 7 is the trajectory length. Here, a, E Rº denotes the action at time step t, and D represents the dimensionality of the action space, such as the degrees of freedom (DOF) in a robotic system. In this framework, intelligent agent-such directly as motor commands or follows the trajectory using a tracking controller.
babot-executer the predicted action sequence
Although ERL predicts entire action trajectory, it still adheres to the Markov property, where the state transition probability depends only on the current state and action (Sutton & Barto] (2018)
Thus, while the action sequence in ERL spans multiple time steps, the underlying process remains consistent with the MDP formalism. This approach is conceptually related to techniques such as action repeat (Braylan al., 2013) and temporally correlated action selection (Ratlin et al. 2022 Eberhard et al. 2022), which also incorporate temporal dependencies into action selection.
Movement Primitives (MP), as parameterized trajectory generators, play a crucial role in ERL. We briefly highlight key MP methodologies and their mathematical foundations used in this work, with a more detailed discussion in Appendix [Schaal (2006) introduced Dynamic Movement Primitives (DMPs), which incorporates forcing term into dynamical system to generate smooth trajectories from a given initial condition, such as a robot's position and velocity at particular time")
Bi==($$0 - 9) - TÚ) f(x), f(x) = E Σ (((())
D Ex(()) xPI.u
(3)
where y y(t,, y = dy/dt, y ddd d denote the position, velocity, and acceleration of the system at time I, respectively. Constants a and ,3 are spring-damper parameters, with g as the goal attractor and as a time constant modulating the speed of trajectory execution. The functions (p)()) represents the basis functions for the forcing term, and the trajectory's 's shape is determined by the weight parameters uy E w. for I, L.,..N NN. The trajectory [yle==TTT is typically computed by nu- merically integrating the dynamical system from the start to the end. Building on the same concepts, Li et al. (2023) proposed Probabilistic Dynamic Movement Primitives (ProDMPs), which directly uses the closed- fore solution of (3). ProDMP employs a linear basis function representation to directly map a parameter vector w to its corresponding trajectory [We]ro
ytt + (t) + c1g1(t) + 0232(t).
(4)
Here, the terms C131 (t) + C232 (t) ensure precise trajectory initialization, with the constants C1 C2 calculated based the initial condition yh. yb time t.. The term @(t) denotes the integral form of the basis functions & used in the E3. Unlike DMP, ProDMP benefits from the closed-form solution of the dynamic system, enabling Taster computation and probabilistic modeling without the burden for numerical integration. This allows for flexible trajectory generation and precise initial condition enforcement. In TOP- we leverage ProDMP's fast initial condition enforcement to compute accurate target values for the Transformer critic, thereby reducing bias in policy learning.
ERL Learning Objectives. A key distinction between ERL and step-based RL (SRL) lies in the action space. ERL shifts the solution search from the per-step action space A to parameterized
An initial condition in mathematics refers to the value of function or its derivatives at starting point, which can he specified any time and is necessarily f O.
4

---

Published as a conference paper at ICLR 2025
trajectory space W. predicting the trajectory parameters sr(w)s s). As result, trajectory param- eterized by w is treated as a single data point in W. This often leads ERL to employ black-box optimization methods for trajectory optimization (Salimans al. 2017). The learning objective in
ERL is often formulated using an importance sampling ratio, such as in BBRL. (Otto et al. 2022)
Tnew (w||s
Update using trajectory parameter:
GTTM u))
(5)
Told (w))(
where represents the policy parameterized by 0. typically using neural network. The terms new and old refer to the current policy being optimized and the policy used for data collection, respectively. The initial state serving as input to the policy. The policy TT (w]ss) determines the likelihood of selecting trajectory parameters w. The term Gm (s, w) Σi= y'r, represents the return accumulated by executing the trajectory under an old policy, where 7 is the discount factor and r is the reward at time step t. By leveraging parameterized trajectory generators like MPs, ERL benefits from consistent exploration, smooth action trajectories, and improved robustness against local optima, as highlighted by Otto eet al.] (2022). To further enhance learning efficiency, recent work TCE (Li et al. 2024) proposes hybrid update strategy that decomposes the trajectory parameter-wise update into the segment-wise updates, incorporating per-step information into ERL's learning objective. This approach divides
S defines the starting configuration and objective of the task,
the longer action trajectory into smaller segments, calculating the return of each segment. The new learning objective adapts Eq. (5), with the maximization of segment- returns as
Naw (Ia") le=o: Is) GFOu ( (a)l==o (6) 1
Update using segments: J (1a((1)(s))
K
proa (raa([a))) (a))==
where K and L represent the number and length of the trajectory segments, respectively, with K 25 in the original paper and k = 1, 1,.., K denotes the segment index. In this expression, p" denotes the likelihood of reproducing the segment, calculated using the parameterized policy To (w) and G(si, (af]==o:z.)) represents the return of executing the k-th action sequence segment [af] from the segment's 'ss starting state st. It is worth noting, despite the usage of importance sampling, both Eq. (5) and Eq. (6) still remain within the on-policy RL framework. In TOP-ERL, we employ similar strategy in splitting long action trajectory into smaller segments, and use these segments for efficient critic and policy updates, under an off-policy framework.
4 TEANSFORMER-BASHE OFF-POLICY ERL
Ht 0:T
Replay Buffer
In this section, we present TOP-ERI, an innovative off-policy so- lution for ERL that leverages a Transformer for action sequence evaluation. The section is structured as follows: Section 4.1 in- troduces the Gaussian policy modeling and action trajectory gen- eration, followed by the design of the transformer critic in Section The learning objectives for the critic and policy detailed in Section 4.3 and Section 4.4, respectively, with additional technical details. Lastly, summarize other design choices in Section 4.5 The main contributions of our model are described from Section 4.2 to Section 4.44 while the remaining sections cover techniques adopted from the literature.
Ir 0:7
[a:]az2
Gaussian Policy
Movement Primitives
4.2
Σ un" ~ (W E
Figure 1: Trajectory genera- tion and environment rollout.
4.1
TRAJECTORY GENERATION: TECHNIQUES ADOPTED FROM ERL LITER ATURE
TOP-ERL adopts policy structure similar to previous ERL approaches, such as BBRL Otto et al. 2022). As shown in Fig. [. our policy is modeled as Gaussian distribution, To ( = N(u Mu: Ew), where s defines the initial observation and the task objective, and represents the parameters of the movement primitive (MPs). In TOP-ERL, we employ ProDMPs (Li et al.) to help correct the target computation via enforcing the initial condition of the MP, as discussed later in Section 1. Given an initial task state the policy predicts the Gaussian parameters and samples a parameter vector w". This vector is then passed into the movement primitive to generate the action trajectory [a.]==07.. The agent then executes the action trajectory in the environment until the end of the episode. During the rollout, both the state trajectory and the reward trajectory are recorded. These, along with the action trajectory, subsequently stored in the replay buffer B for later
5

---

Published as a conference paper at ICLR 2025
Trajectory Segmentation
Segment Sequence Value Predictions
Encoding
so
$0
Volag)
Positional Encoding
dg
af
a
ag
PE
Transformer
Qolot. al)
Linear Output
a
1-1
Action Encoder
Qolobi a alj Qolst aB at ab)
The
Replay Buffer
Qu(s a at att.a...... ab....
a..,,,,,.)))
Spltt segments
Figure Architecture overview of the Transformer critic, as described Sec. 4.2
4.2 TRANSFORMERS AS VALUE PREDICTOR FOR ACTION SEQUENCES
An architectural overview of our Transformer critic is depicted Fig. 2 At each iteration, we sample batch B of trajectories from the replay buffer and split each trajectory into K segments, where each segment is time steps long. An ablation on how to select the segment length L. can be found in Sec. 5.3 The transformer-based critic has L+1 input tokens that given by each action in the segment aa I==0: and the starting state & of the corresponding segment. These tokens are first processed by corresponding state and action encoders, each modeled by single linear layer. Positional information is added to the processed tokens through trainable positional encoding, with s) and the first action token af sharing the same positional encoding (both at 0). The tokens are subsequently fed into decoder-only Transformer, followed by linear output layer, producing I, output tokens. The first output represents the state value V(5)) for the starting state, while the remaining outputs correspond to the state-action values for the subsequent action sequence. For example, Q an af ab) represents the value of executing the actions a af a sequentially from the starting state si and subsequently following policy
causal mask is applied in the Transformer to ensure that actions do not attend to future steps.
4.3 N-STEP RETURNS AS THE TARGET FOR TRANSFORMER CRITIC
For each predicted state-action value Q(50, a ..., ....aa - utilize the N-step return as its target. The objective to update the parameters & of the critic the N-step squared TD errof"
Critic loss:
[(0) - N=1
Σ
(s ab; a - ( a 1
Predicted value actions
Target using N-step
Vo( Predicted state value
E Egi (-1ss) Q (0/ ao
Target of new actions using
N-1
N-step return: G (N)) ab a x)-1)==
Vilue (SN)-
(8)
1=0
Future after N-stop
N-step rewards
represents the number of actions in a sub- starting from s The term Qoy (sf, ag. ..., á ) in 7 denotes the target value of Vo(at) with actions a ..., a generated by new MP parameters sampled from the current policy. v Te(- [s). The term Vo, (8x) in Eq. (8) represents the future return after N steps. Both Q and Van. are predicted by a target critic (Mnih al.) 2013), with a delayed update rate p = 0.005. Please note that Q and Vo, are the same transformer network, with and without action tokens.
Here, N E [1,
Forr simplicity, we omit the expectation over buffer B and average over segment number in Eq.

---

Published as a conference paper at ICLR 2025
In off- off-policy RL literature, there are several alternatives to replace Vow (s.) in Eq.48) However, we find that this choice alone performs well in our experiments. In other words, TOP-ERL does not necessarily rely some common off- off-policy techniques, such the clipped double-Q (Fujimoto et al. 2018), to be stable and effective. We attribute this to the usage of the N-step returns, which help reduce value estimation bias. In Sec. [..]] show that model can be further improved using these augmentations, though at cost of additional computation.
Unlike Eq.(2), our N-step return targets (()) .j.................... aN in Eq. do not include impor- lance sampling as the the action sequence a......c a is directly used as input tokens for the Q-function. Hence, the actions are fixed and we do not require compute any expectations over the current policy's action selection. Hence, using the fixed action sequence in (8) as input to the Q-Function eliminates the need for importance sampling. thus avoiding the high variance typically introduced by it in off-policy methods, as discussed in Sec. 3.1
4.3.1 ENFORCE INITIAL CONDITION FOR NEWLY PREDICTED ACTION SEQUENCE
When calculating the target value Qéru (s), a a in Eq new parameter vector is sampled from the current policy t0 no generating a new action trajectory [ā.]==:TT with [a (w) (= as a sub-sequence. However, this sequence is not necessarily guaranteed to pass through the segment's ss starting state s6. which creates a mismatch between the state and corresponding action sequence when querying the target in Eq. To address this issue, we append the old reference po- sition to s/. and then leverage the dynamic system formulation inherent in ProDMPs by setting the initial condition of the new action sequence to match the old reference at sp. as illustrated Fig. 3 The result- ing action sequence lā =0 is therefore depending on both the MP parameters (s) and the initial condition si. This approach mathematically equivalent to resetting the initial conditions of an dinary differential equation (ODE), ensuring consistency between the state and action sequences. Further mathematical details and illustration are provided in Appendix B.3 and B.4
[a (
$0
troj.
New action segment
Enforee init. condition
Figure 3: Enforce action initial condition
4.4 POLICY UPDATES USING THE TRANSFORMER CRITIC
We utilize the transformer critic to guide the training of our policy, using the reparameterization trick similar to that introduced by SAC (Haarnoja et al. 200 8ap. The learning objective is to maximize the expected value the averaged action sequence over varying lengths, defined as:
K L-1 3E N
(a+) (
Policy Objective: J(0) = Eg B. (-13))
(9)
KL
where (ātk==:N denotes the action sequence generated by the new MP parameters we (-)s).). This learning objective allows the policy Te(w|s) to be trained based on the value prefer- ences provided by the Transformer critic. We refer Appendix A.T for more detailed discussion.
4.5 ADDITIONAL DESIGN CHOICES FROM LITERATURE FOR STABLE LEARNING
We summarize the key learning steps in Algorithm I To effectively capture broader range of corre- lations in both temporal and DoF movements, we utilize a full covariance matrix Σ,, in the Gaussian policy (Li al. (024).. Since the Gaussian policy over MP parameters typically high-dimensional, we employ the Trust Region Projection Layer (TRPL) (Otto et al. 2021) for stable policy updates, following the design of previous ERL methods (Otto et al.) 2022: et al. 2024). More discussions are in Appendix A.2 For the Transformer critic, we apply Layer Normalization sole data normalization technique, while disabling dropout, as we found it detrimental to perfor- mance. In our experiments, results were achieved by randomly sampling L at each update iteration, which we attribute to the Transformer critic's ability to attend to different time horizons, resulting in more robust outcomes.
(Ba) 2016) as the
identified the segment length L as a key hyperparameter. The best

---

Published as a conference paper at ICLR 2025
TOP-ERL (ours)
TCE
BBRL
PPO
(TPXL(PPO)
BSDE
SAC
PINK
Man Jwwp www.pappppgggaaee (n)) iol.
Stecos Rote, 10M
10M
0.6
0.8
112
02-
044
0.5
03
Gevavement Interaction (HUP))
Entreeraoe (x
Ex Eprioreec Enterattoo (K217)
(a) Metaworld Agg. of 50 tasks
(b) Hopper Jump
(c) Box Pushing Dense Reward
(d) Box Pushing Sparse Reward
Max Height
Figure 4: Task Evaluation of (a) Metaworld success rate of 50 tasks aggregation. (b) Hopper Jump Max Height. (c) Box Pushing success rate in dense reward, and (d) sparse reward settings.
EXPERIMENTS
Algorithm TOP-ERL
1: Initialize critic Φ: target critic 6ur + ó
Our experiments focus on the following
2: Initialize policy 0 and replay buffer B
questions: I) Can TOP-ERL improve sample
3: repeat
efficiency in classical ERL tasks featured by challenging exploration problems? II) How does TOP-ERL perform in large-scale, gen- eral manipulation benchmarks? III) How do key design choices affect the performance of TOP-ERL? We compare TOP-ERL against a set of strong baselines. For the ERI.compan compar- isons, we select BBRL and TCE as SOTA
4: Reset environment and get initial task state 8
5: Predict the policy mean Mwo and covariance E.w
6: Sample was and generate action trajectory aalorr
7: Execute the action trajectory till task ends.
Store visited states [a]o:r. rewards [r]orr, and the action [a) Jorr trajectories in replay buffer B for each update step do
9:
10:
From B, sample batch of 8, a, trajectories. Split them into segments, each time steps. Compute N N-step return targets as in (8)
ERL methods. For step-based RL, we use PPO (Schulman et al.) 2017) (on-policy)
12:
13:
Update transformer critic, using Eq.
and SAC (off-policy) as established base- lines. Additionally, we employed gSDE and PINK. two step-baste based RL methods that aug- ment with consistent exploration techniques, to test the impact of exploration strategies.
14:
Update policy, using Eq. end for
15:
Update target critic Chu f- (1 p) onr + P until converged
17:
To assess the impact of using Transformer- basedd architectures in RL, include GTrXL base- line for online RL with Transformers architecture. It is worth noting that in the original work, GTrXL trained using VMPO. However, since the original code not open-sourced, we used the implementation from Liang et al.] (2018), where GTrXL is trained with PPO instead. For all ERLs, the trajectories are generated using ProDMPs with the same hyperparameters and tracked with PD-controllers (or P-controller for Meta Worlds); for all the SRLs, the action outputs are torque (or delta position for MetaWorlds). An overview of the baselines can be find in Table 3 regarding the implementation and hyperparameters are provided in the Appendix
and details
The evaluation of TOP-ERL are structured in three phases. First, we demonstrated that TOP-ERL significantly improve the sample efficiency over state-of- offe-offthheart art ERL methods, showcasing its ability to better handle the challenges of sparse rewards and difficult exploration scenarios (Li al 2024). Next, we evaluate TOP-ER ERL on the Meta-War World MT50 (Yu et al.] (0200) benchmark, a large-scale scale suite of general manipulation tasks. this setting, TOP-ERL consistently outperform all baselines, demonstrated TOP-ERL's 's ability to generalize across wide range of manipulation tasks. Finally, we conduct a comprehensive ablation study to analysis which ingredient accounts for the strong performance of TOR- The results confirm that theses components strong performance observed with TOP-ERL. To ensure a robust evaluation, all empirical results reported using Interquartile Mean (IQM), accompanied by 95% stratified bootstrap confidence interval (Agarwal et al. 2021) across 8 random seeds.
essential to achieving the
5.1 IMPROVING SAMPLE EFFICIENCY IN TASKS WITH CHALLENGING EXPLORATION
ERL methods are renowned for their superior exploration abilitics, which often give them an ad- vantage over step- methods in environments with exploration challenges. However, ERL al-
8

---

Published as a conference paper at ICLR 2025
gorithms are also notoriously sample inefficient, limiting their applicability in scenarios where ob- taining samples expensive. In this evaluation, we investigate whether TOP-ERL can address this limitation by comparing it with basclines on three challenging tasks from Li et al. (2024): Hopper- Jump, a sparse-reward environment where the objective is to maximize the jump height within an episode, and two variants of contact-rich Box Pushing task. We evaluate the Box Pushing task under both dense and sparse reward settings. Further details about the environments and rewards can be found in Appendix C The results of these experiments, shown in Fig. A demonstrate that TOP-ERL achieved the highest final performance across all three tasks. Notably, in the reward Box Pushing task, TOP-ERL reached an 80% success rate after just 10 million samples, while the second-best method, TCE, only reaches 60% success after 50 million samples. Similar results is observed in the sparse-reward Box Pushing task, where TOP-ERL reaches 70% success rate with 14 million environment interactions, while TCE and gSDE require 50 million samples to reach 60% success. GTrXL performs moderately in the dense-reward setting, achieving a 50% success rate, but fails completely in the sparse-rewart environment. Step-based methods like SAC, PINK and PPO failed in both cases, underscoring the difficulty of these tasks. Among the step-based algo- rithms, only gSDE achieved comparable performance in compare with ERL methods in these three environments, which we attribute to its state-dependent exploration strategy.
5.2 CONSISTENT PERFORMANCE IN LARGE-SCALE MANIPULATION BENCHMARKS
In the previous evaluation, we demonstrated that TOP-ERL significantly improves sample efficiency compared to sfate-of-the-ae art ERL baselines, while maintaining strong performance tasks with challenge exploration. In this evaluation, focus on answering the second question: How does TOP-ERI. perform on standard manipulation benchmarks with dense rewards? We conducted ex- periments on the Worad benchmark(Yu et al.] (0200), reporting the aggregated success rate across 50 tasks in the MT50 task set. To ensure a fair comparison, we followed the same evaluation protocol described in Otto et al. (202)) and Li et al. (2024), where an episode is only considered successful if the success criterion is met at the end of the episode, a more rigours measure than the original setting where success any time step counts. The results in Fig. 4aj show that TOP-ERL achieved highest asymptotic success rate (98%) after 10 million samples. TCE was able to achieve the same success rate but required 20 million interactions. SAC also converged after million samples but with significantly lower success rate of 85%. BBRL and other step-based methods achieved moderate success rate but required significantly more samples.
5.3 ABLATION STUDY AND DISCUSSION
Single Q-Network leads to stable and efficient training. We compare four common design choices for targets calculation in Q-function update in Eq. [): 1) V- Target which uses a single V target network, 2) Qarrgtt which employs single Q target network, 3)V-Emmem,l, which consists of an ensemble of predictions from two V target networks, 4) V-Clip, which takes the minimum of two V target networks. Detailed description of these design choices can be found in Appendix A.3 Fig. presents the learning curves for TOP-ERL in dense-reward (5a) and sparse-reward (5b) Box Pushing, while Table presents the numerical success rate and computation times per update. The results demonstrate that using a single V target network yields performance comparable to approaches that rely on two target networks, with additionally benefit of significantly reduced computation time (approximately 50% faster). We attribute the stable performance with single target network to the use of N-step Bellman equation in target calculation, discussed in Sec.
Key Components Ablation. We evaluate the impact of five key components on the performance of TOP-ERL: trust region constraints in policy updates, enforcing the initial condition each segment, the presence of layer normalization, fixed vs. random segment lengths, and the inclusion of dropout in Transformer layers. These evaluations were conducted in both dense-reward and sparse-reward Box Pushing environments using 8 random seeds. The results, presented in Fig. 5] dashed lines, show performance for TOP-ERL with corresponding component been added or removed. The results indicate that the random segment length has the most significant effect on TOP-ERL's performance. When using fixed 25 segments the success rate dropped from 80% to 35% in the dense-rewwe setting, and from 70% to 20% in the pparse-reward setting. Layer normalization, trust region con- straints, and enforcing initial conditions also contributed positively to the performance. Interestingly, adding even small dropout rate (0.05 in the ablation) had negative impacts the performance in
9

---

Published as a conference paper at ICLR 2025
0..
Quaget
V-lagg
Table 1: Quantitative performance and update time of different critic update strategies. With additional computational cost, TOP-ERI. can be further enhanced.
N0M
06-
Swves Rate XQM
Eusemble
Clipped
TR
No lax Cord.
07-
NolN
Fined Sog. Dripour
Resbomenn Iekenccion
Envesment (x
+ Time
Dense
Sparse Success, %
Variant
critie
s/iter
Success,
(a) Box Pushing Dense Reward
(b) Box Pushing
Sparse Reward
V-Target
(defouit)
Figure 5: Performance of different critic update strategies (solid lines) and model ablations (dashed lines), using Box pushing dense and sparse reward settings respectively.
Q-Target
3
2.44
86.1 :: 2.7
69.1 #.5
V-Ensem. 2
2.49
83.8
75.7 1: 4.4
V-Clip
2 2.49
86.0
75.5 3.7
both tasks. We hypothesize that this effect may be attributed to the use of a relatively small replay buffer combined with a higher buffer update ratio (0.1% in our setting), which likely mitigates the risk of overfitting in Q-function learning, thereby diminishing the benefit of dropout.
Impact of Random Segment Lengths. As shown in the pre- vious ablation study, random segment length played a cru- cial role in the strong performance of TOP-ERL. To further examine whether this conclusion holds for different segment lengths, we evaluated the dense-reward Box Pushing task with segment lengths ranging from 5% of the episode length to 100% (i.e., no segmentation). The results in Fig. 6 indicate that fixed-length segmentation leads to significant performance variation depending on the segment length. In contrast, ran- dom segment lengths consistently achieve faster convergence and higher asymptotic performance. We infer that using a vari- ety of action sequence lengths regularizes the training of the critic network. For example, in Eq. 77. the expectation of the Q-value over L actions is used as the target for the V-function's prediction. When J. varies across update iterations, the V-function is trained on Q-values derived from different amounts of actions. Additionally, random segmentation simplifies hyperparameter tuning, making it a practical choice. Therefore, we adopt random segmentation length as the default setting for TOP-ERL. To further illustrate the impact of random segment lengths, we provide a visualization of action correlations under different segmentation strategies in Appendix
Randon
100%.
Secoee Rak IOM
99%.
5%
2
10
Eestenment Ex 10%s
Figure 6: Random or Fixed segment length
6 CONCLUSION
This work introduced Transformer-based Off-Policy Episodic RL (TOP-ERL), a novel off-policy ERL method that leverages Transformers for N-steps return learning. By integrating ERL with an off-policy update scheme, TOP-ERL significantly improves the sample efficiency of ERL methods while retaining their advantages in exploration. The of Transformer-based critic architecture allows TOP-ERL to bypass the need for importance sampling in N-steps target calculation, stabi- lizing training while enjoying the benefit of low-bias value estimation provided by N-steps return. TOP-ERL has demonstrated superior performance compared to state-of-the-art ERL approaches and step-based RL methods augmented with exploration mechanism across 53 challenging tasks, pro- viding strong evidence for its broader applicability to wide range of problems. The ablation studies reveal the reasons behind design choices and components, providing insights into the factors con- tributing to the strong performance of FRI-
Limitations and Future Works. Despite all the advantages, TOP-ERL shares limitation common to ERL methods: it generates trajectories only at the start of each episode, making it incapable of handling tasks involving dynamic or target changes within an episode. A promising future research direction would be to incorporate replanning capabilities into TOP-ERL. Additionally, although TOP-ERL uses Transformers as critic, is not designed to address POMDPs, as the Transformer is used for action-to-ga processing in Q- Q-function learning, rather than incorporating state sequences as input. Merging these two paradigms and enhancing TOP-ERL with the ability to handle POMDPs presents another avenue for future investigation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank our friends and colleagues Juan Li, Onur Celik, Aleksandar Taranovic, Tai Hoang and Zuzhao Ye for their valuable discussion and technical support. We thank the anonymous reviewers for their insightful feedback which greatly improved the quality of this paper.
The research presented in this paper funded by the Deutsche Forschungsgemeinschatt (DFG, Genman Research Foundation) 448648559 and 471687388 and was supported in part by the Helmholtz Association of German Research Centers. Gerhard Neumann was supported in part by Carl Zeiss Foundation through the Project JuBot (Jung Bleiben mit Robotern). The authors acknowl- edge support by the state of Maden-Württember through bwHPC, and the HoreKa supercomputer.
ETHICS STATEMENT
No human participants were involved in this study. All data used in this work was generated through simulations. As such, there are no privacy or security concerns related to personal or sensitive information. We acknowledge the importance of fairness in Al and have taken care to ensure that our methodology does not introduce bias in simulated environments, though broader fairness issues in real-world applications of such models should be considered in future work. There are no conflicts of interest or sponsorship concerns associated with this research, and all practices adhere to legal and ethical standards.
8 REPRODUCIBILITY STATEMENT
The considerable efforts were made to ensure that our work is fully reproducible. All relevant code, including the implementation of the proposed algorithms, simulation environments, and trained models, will be made available an GitHub repository provided in the main paper. Detailed de- scriptions of the experimental setup, including hyperparameter configurations can be found in the appendix.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Abbas Abdolmalcki, Rudolf Lioutikov, Jan R Peters, Nuno Lau, Luis Pualo Reis, and Gerhard Neumann. Model-based relative entropy stochastic search. Advances Neural Information Pro- cessing Systems, 28, 2015.
Abbas Abdolmaleki, Bob Price, Nuno Lau, Luis Paulo Reis, and Gerhard Neumann. Contextual co- variance matrix adaptation evolutionary strategies. In IJCAI 2017. International Joint Conferences on Artificial Intelligence Organization (UCAI), 2017.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Informa- fion Processing Systems, 2021.
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Differentiable convex optimization layers. Advances in neural information processing systems, 32, 2019.
Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.06-453 2016.
Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta, and Deepak Pathak. Neural dynamic policies for end-to-end sensorimotor learning. Advances in Neural Information Processing Systems, 33: 5058-5069, 2020.
Homanga Bharadhwaj.. Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmenta- tions and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4788 11888-795... IEEE, 2024.
Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Mükkuulaienn. Frame skip is a pow- erful parameter for learning to play atari. In Workshops at the twenty-ninth AAAI conference on artificial intelligence, 2015.
Onur Celik, Dongzhuoran Zhou, Ge Li, Philipp Becker, and Gerhard Neumann. Specializing satile skill libraries using local mixture of experts. In Conference on Robot Learning, pp. 1423- 1433. PMLR, 2022.
Onur Celik, Aleksandar Taranovic, and Gerhard Neumann. Acquiring diverse skills using currieu- lum reinforcement learning with mixture of experts. In Forty-first International Conference Machine Learning. 2024. URL httpp:://ppaneeeew..rr
forum ?id 92KUFSw1UH
Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive fufunctions. In Conference on Robot Learning, pp. 3909-3928. PMLR, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 508445 2021.
Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Faustino Gomez, Jürgen Schmidhuber, Risto Mikkuulainen, and Melanie Mitchell. Accelerated neu- ral evolution through cooperatively coevolved synapses. Journal of Machine Learning Research, 9(5), 2008.
Sebastian
Gerhard Neumann. Bernhard Schölkopf, and Jan Peters. Using prob- abilistic movement primitives for striking movements. In 2016 IEEE-RAS 16th International Conference Humanoid Robots (Humanoids), pp. 502-508. IEFE, 2016.
12

---

Published as a conference paper at ICLR 2025
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishck Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: 18/2.05905, 2018b.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022.
Christian Igel. Neuroevolution for reinforcement learning using evolution strategies. In The 2003 Congress on Evolutionary Computation, 2003. CEC'O3., volume 4, pp. 2588-25998 IEEE, 2003.
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2): 328-373, 2013.
Piotr Kicki, Davide Tateo, Puze Liu, Jonas Güinter, Jan Peters, and Krzysztof Walas. Bridging the gap between learning-to-plie motion primitives and safe reinforcement learning. 8th An- nual Conference on Robot Learning, 2024. URL httpthttttp/poeeeeeeeeee ZdgaF B BOco
net. forum?id-
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21, 2008.
Ge Li, Zeqi Jin, Michael Volpp, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann. Prodmp: A unified perspective on dynamic and probabilistic movement primitives. IEEE Robotics and Automation Letters, 8(4):2325 233322 2023.
Ge Li, Hongyi Zhou, Dominik Roth, Serge Thilges, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann. Open the black box: Step-based policy updates for temporally correeltee episodic reinforcement learning. In The Twelfth International Conference on Learning Representations, WtRL/tttss///peeeeeiiie http::////pppeeeee..eeoom net/ http://wwweeee..etoooorro..eeeea
2024. URL|tttps:
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and lon Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pp. 3053-3062. PMLR, 2018.
Haoxin Lin, Yihao Sun, Jiaji Zhang, and Yang Yu. Model- based reinforcement learning with multi- step plan value estimation. ECAI 2023. pp. 1481-1488. IOS Press, 2023.
Chenhao Lu, Ruizhe Shi, Yuyao Liu, Kaizhe Hu, Simon Shaolei Du, and Huazhe Xu. Rethinking transformers in solving POMDPs. In Forty-first International Conference on Machine Learning, 2024. URL /ttttp:://penreviiie httpp//wwww...eeeeeee.eoeooaaaaa
Xiao Ma and Wu-Jun Li. Weighting online decision transformer with episodic memory for offline- to-online reinforeement learning. In 2024 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 10793-10799, IEEE, 2024.
Guilherme Maeda, Marco Ewerton, Rudolf Lioutikov, Heni Ben Amor, Jan Peters, and Gerhard Neumann. Learning interaction for collaborative tasks with probabilistic movement primitives. In 2014 IEEE-RAS International Conference on Humanoid Robots, pp. 527-534. IEEE, 2014.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5177544))559-533 2015.
13

---

Published as a conference paper at ICLR 2025
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Lue Bacon. When do transformers shine in rl? decoupling memory from credit assignment. Advances in Neural Information Processing Systems, 36, 2024.
Toshihiro Ota. Decision mamba: Reinforcement learning via sequence modeling with selective state spaces. arXiv preprini arXiv:2403. 79925, 2024.
Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann. Differ- entiable trust region layers for deep reinforcement learning. International Conference on Learning Representations, 2021.
Fabian Otto, Onur Celik, Hongyi Zhou, Hanna Ziesche, Vien Anh Ngo, and Gerhard Neumann. Deep black-box reinforcement learning with movement primitives. In Conference Robot Learning, pp. 1244-1265. PMLR, 2022.
Fabian Otto, Hongyi Zhou, Onur Celik, Ge Li, Rudolf Lioutikov, and Gerhard Neumann. Mp3:
Movement primitive-bast baased (re-) planning policy. arXiv preprint arXiv:2306.122 12729, 2023.
Rok Pahič, Barry Ridge, Andrej Gams, Jun Morimoto, and Aleš Ude. Training of deep neural networks for the generation of dynamic movement primitives. Neural Networks, 2020.
Alexandros Paraschos, Christian Daniel, Jan Peters, and Gerhard Neumann. Probabilistic movement primitives. Advances in neural information processing systems, 26, 2013.
Emilio Parisotto, Francis Song, Jack Rac, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.
Kai Ploeger, Michael Lutter, and Jan Peters. High acceleration reinforcement learning for real-world juggling with binary rewards. In Conference on Robot Learning, pp. 642-653. PMLR, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah
Dormann. Sabble-bsselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http:://jml:: http:///mmmor..r./.eee// .rg. patggpppprs//22/
20 1364. 3tml htmml
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on Robot Learning, pp. 1634-1644. PMLR, 2022.
Moritz Reuss, Ömer Erding Yağmurlu, Fabian Wenzel, and Rudolf Lioutikov. Multimodal diffu- sion transformer: Learning versatile behavior from multimodal goals. In Robotics: Science and Systems, 2024.
Leonel Rozo and Vedant Dave. Orientation probabilistic movement primitives riemannian man- ifolds. In Conference on Robot Learning, pp. 373-383. PMLR, 2022.
Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State stateddependent exploration for pol- icy gradient methods. In Walter Daclemans, Bart Goethals, and Katharina Morik (eds.), Ma- chine Learning and Knowledge Discovery in Databases, pp. 234-249, Berlin, Heidelberg, 2008 Springer Berlin Heidelberg. ISBN 978-3-540-87481.
Thomas Rückstiess,, Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforcement learning. Paladyn, 1:14-24, 2010.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative reinforcement learning. arXiv preprint arXiv:1003.03864 2017.
Stefan Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid roboties. In Adaptive motion of animals and machines, pp. 261-280. Springer, 2006.
14

---

Published as a conference paper at ICLR 2025
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-18888888 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
RB Ashith Shyam, Peter Lightbody, Gautham Das, Pengcheng Liu, Sebastian Gomez-Gonzalez, and Gerhard Neumann. Improving local trajectory optimisation using probabilistic movement primitives. In 2019 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pp. 2666-2671. IEEE, 2019.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine learning, -:9-44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jens Timmer and Michel Koenig. generating power noise. Astronomy and Astrophysics, 300:707, 1995.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Darrell Whitley, Stephen Dominic, Rajarshi Das, and Charles W Anderson. Genetic reinforcement learning for neurocontrol problems. Machine Learning, 13:259-284, 1993.
Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.
Taku Yamagata, Ahmed Khalil, and Raul Santos- Rodriguez Q-learning decision transformer:
Leveraging dynamic programming for conditional sequence modelling in offline rl. In Inter- national Conference on Machine Learning, pp. 09989939007.. PMLR, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1108 100. PMLR, 2020.
Weilin Yuan, Jiaxing Chen, Shaofei Chen, Dawei Feng, Zhenzhen Hu, Peng Li, and Weiwei Zhao Transformer in reinforcement learning for mecision-making: a survey. Frontiers of Information Technology & Electronic Engineering, 25(6):763-790, 2024.
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging fifline-to-onlng reinforce- ment learning. arXiv preprint arXiv:2302.00993: 2023.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint 1317:2200. 2023.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pp. 27042-27057 PMLR, 2022.
You Zhou, Jianfeng Gao, and Tamim Asfour. Learning via-point movement primitives with inter- and extrapolation capabilities. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4301-4308. IEEE, 2019.
15

---

Published as a conference paper at ICLR 2025
List of Content in Appendix
A. Further technical details of TOP-ERL.
B. Mathematical formulations of MP trajectories and initial condition enforcement.
C. Experiment settings and details complementary to Sec.
D. Action Correlation Visualization.
E. Hyper-parameters selection and sweeping.
A FURTHER TECHNICAL DETAILS OF TOP-ERL
A.1 POLICY TRAINING USING SAC-STYLE REPARAMETERIZATION TRICK
We utilize the transformer critic to guide the training of our policy, using the reparameterization trick similar to that introduced by SAC (Haarnoja et al. 2018a). Given a task initial state 8, the current
policy To Te(w]s) (w)s) ~ ~N (Wwlpw Ew) predicts the Gaussian parameters of the MP's and samples w as follows:
Sample MP parameter vector:
+ Mw + Lwc, c~ C(
(10)
Here, Lu is the Cholesky decomposition of the covariance matrix Ew, where L. Ew. This parameterization technique commonly used for predicting full covariance Gaussian policies. The term is a Gaussian white noise vector with the same dimensionality as w. Eq. (IO) represents the full covariance extension of the reparameterization trick typically used in RL, known as (a + Jac, (0,
The sampled w is then used to compute the new trajectory segments. The resulting trajectory seg- ments computed using the linear basis function expression in Eq. (4) from Section 3.2, where the coefficients Cj and C2 are determined by the initial conditions and solved using Eq. [2 in Appendix
B.3
Compute action segment: a(t) = ( c1 - c2)2(1)
(4)
Here, = 0 N represents the time interval from the beginning to the end of the k-th segment. There two design choices for the initial conditions: the first is always the task initial state s for all segments, while the second is to use the state s specific to each segment. While the second choice aligns with the techniques described in Section 4.3.1, first choice leads to better performance. To the best of our knowledge, we attribute this interesting finding to the following reason. Although the second design choice ensures better consistency in the input to the value function, the per-segment initial conditions were not provided to the policy for action selection, which introduced challenges in updating the policy. We believe this phenomenon is worth further investigation in future research.
empirical results show that the
We evaluate and maximize the value of these action segments, using their expectation the policy's 's learning objective, as shown in Eq. (9) in Section 4.4:
L-1 Σ Σ
SAC style Objective:
(()) Eg Be (
Q0(
[at]
KL
Since (4), (9), (22) and 10) are all differentiable. the policy neural network parameters O can be trained using gradient ascent. Compared to the technique introduced in SAC, TOP-ERL adds only one additional step: computing the action sequence [ā])==ox from the sampled MP parameter tö.
16

---

Published as a conference paper at ICLR 2025
A.2 UTILIZING TRPL FOR STABLE FULL-COVARIANC GAUSSIAN POLICY TRAINING
In Episodic Reinforcement Learning (ERL), the parameter space W generally has higher dimen- sionality than the action space A, creating distinct challenges for achieving stable policy updates. Trust region methods (Schulman et al. 2015; 2017) are widely regarded as reliable techniques for ensuring convergence and stability in policy gradient algorithms.
Although methods like PPO approximate trust regions using surrogate objectives, they lack the ability to enforce trust regions precisely. To address this limitation, Otto et al. (202T) proposed trust region projection layer (TRPL), mathematically rigorous and scalable approach for exact trust region enforcement deep RL algorithms. Leveraging differentiable convex optimization layers (Agrawal et al. 2019), trust region projection layer (TRPL) enforces trust regions the per- state level and has demonstrated robustness and stability in high-dimensional parameter spaces, as evidenced in methods like BBRL (Otto et al. 2022) and TCE (Li et al. 2024).
TRPL operates on the standard outputs matrix E-and enforces trust regions through state-specific projection operation. The adjusted Gaussian policy, represented by jı and Σ, serves as the foundation for subsequent computations. The dissimilarity measures for the mean and covariance, denoted as dmean and deov (c.g.. KL-divergence), bounded by thresholds en and eg, respectively. The optimization problem for each state is expressed as:
Gaussian policy-the mean vector μ and covariance
arg min dmmaa (iss (())), s.t. dinann (ptrr Hold (s)) S Cus and n,
(11)
arg min dowr I(s): s.t. dowr (I.. Iola(s) SE..
the unconstrained, newly predicted per-state Gaussian parameters (u()) and E(s) exceed the trust region bounds defined by çμ and eg, respectively, TRPL projects them back the trust region boundary, ensuring stable update steps. In TOP-ERL, the old Gaussian parameters, Mold(s) and (s), can be derived either from the behavior policy that interacted with the environment or from an exponentially moving averaged (EMA) policy, which serves as delayed version of the current policy. This approach is analogous to the concept employed in the target critic network.
A.3 TARGET OPTIONS
Table 2: Options for the future return used in Eq.
Option
Math
Description
V-turget
(*w(s)
State value after N steps
Q-target
Q (aN, ax....
Action value after N steps
Clipped
Min(.
Minimum of target crities
Ensemble
Avg.(s-)
Mean of turget critics
B MATHEMATICAL FORMULATIONS OF MOVEMENT PRIMITIVES.
In this section, provide overview of the movement primitive formulations used in this paper. We begin with the basics of DMPs and ProMPs, followed by a detailed explanation ProDMPs. For clarity, start with single DoF system and then expand to multi-DoF systems.
B.1 DYNAMIC MOVEMENT PRIMITIVES
Schaal 2006); ; Ijspeert et al. 2013) describe a single movement as trajectory [Vd]=ooT which governed by second-order linear dynamical system with a non-linear forcing function f. The mathematical representation is given by
-2 = = - -)) -() ) f(x), f(x) = Eei(z)e I Ex((z)
(12)
17

---

Published as a conference paper at ICLR 2025
where y y(t,, y dy/dt, y denote the position, velocity, and acceleration of the system at a specific time t. respectively. Constants a and 3 are spring parameters, signifies goal attractor, and is time constant that modulates the speed of trajectory execution. To ensure convergence towards the goal, DMPs employ a forcing function governed by an exponentially decaying phase variable stt) exp(-ax//o t). Here, (a(()) represents the basis functions for the forcing term. The trajectory's 's shape as it approaches the goal is determined by the weight parameters wi E w. for I, ...., .N. The trajectory [We] is typically computed by numerically integrating the dynamical system from the start to the end point (Pahié et al.] 2020 Bahl et al., 2020). However, this numerical process is computationally intensive. For example, to compute the trajectory segment in the end of an episode, DMP must integrate the system from the very beginning till the start of the segment.
B.2 PROBABILISTIC MOVEMENT PRIMITIVES
Paraschos et al. (013)) introduced framework for modeling MPs using trajectory distributions, capturing both temporal and inter-dimensional correlations. Unlike DMPs that use a forcing term,
ProMPs directly model the intended trajectory. The probability of observing 1-DoF trajectory
[yl given a specific weight vector distribution p(w) N(wlww Ew) is represented as a linear basis function model:
Linear basis function:
[uole-UT = U. + Cy.
(13)
Mapping distribution:
P(yv]]=O:T My: Eny = -N( O.r
OOT + o)
(14)
Here, cy is zero-mean white noise with variance of The matrix uo: houses the basis functions for each time step I. Similar to DMPs, these basis functions can be defined in terms of phase variable instead of time. ProMPs allows for flexible manipulation of MP trajectories through probabilistic operators applied to p( w)), such as conditioning, combination, and blending (Maeda et al. 2014
Jomez-Gonzales et al. 2016, Shyam et al. 2019 Rozo Dave, (022 Zhou et al., 2019). However, ProMPs lack intrinsic dynamic system, which means they cannot guarantee smooth transition from the robot's initial state or between different generated trajectories.
B.3 PROBABILISTIC DYNAMIC MOVEMENT PRIMITIVES
Solving the ODE underlying DMPs Li et al. 2023) noted that the governing equation of DMPs, as specified in Eq. [12. admits an analytical solution. This is because is a second-order linear non-
homogeneous ODE with constant coefficients. The original ODE and its homogeneous counterpart
can be expressed in standard form as follows:
a3
f(x)
ag 7 g- =FF((.g)),
Non-homo. ODE:
+
72 マリ= 72
(15)
Homo. ODE: 5 デ - -リ The solution to this ODE is essentially the position trajectory, and its time derivative yields the velocity trajectory. These are formulated as:
a3
(16)
y = [U2P2 V1P1
3292
91
w
+ C191 + C29/
(17)
9
ý == 1inP2 WIP1
Ú292
w
+ +181 + C292-
(18)
9
Here, the learnable parameters w and g which control the shape of the trajectory, are separable from the remaining terms. Time-dependent functions y1, P1. P2, n1 G2 in the remaining terms offer the basic support to generate the trajectory. The functions y1, 1/2 are the complementary solutions
to the homogeneous ODE presented in equation 16 and j1, 22 their time derivatives respectively. These time-dependent functions take the form as:
(t) = exp
(2())= = t texp
(19)
2T
2T
(t) 72 S
1
exe
a (())((t')pddd 2T
px(t)= 2
exp
(20)
(1)) = 2T
a
1)
exp
a
+1,
(2(1) 27
a
α exp
-
(21)
2T
18

---

and Published as a conference paper at ICLR 2025
Old Traj
New Traj, w/o init cond.
0.2
Seg-O with init cond.
Seg-1 with init cand.
Position of the 1st istjoint [rad]
Seg-2 with init cand.
0.1
Seg-3 with init cond.
Seg-4 with init cond.
0.0
-0.1
-02
0.00
0.25
0.50
0.75
1.00 Time [s]
1.25
1.50
1.75
2.00
Figure 7: TOP-ERL leverages the initial condition enforcement techniques of dynamic system to ensure that the new action trajectory starts from the corresponding old state. These action trajecto- ries are taken from the first DoF of the robot in the box pushing task.
It's worth noting that the Pı and p2 cannot be analytically derived due the complex nature of the forcing basis terms Sz. As result, they need to be computed numerically. Despite this, isolating the learnable parameters, namely w and g. allows for the reuse of the remaining terms across all generated trajectories. These residual terms can be more specifically identified as the position and velocity basis functions, denoted as (() and (L), respectively. When both w and g included in a concatenated vector, represented as wg. the expressions for position and velocity trajectories can be formulated in manner akin to that employed by ProMPs:
Position: y(t) = (t) wy + c181 (t) (2))
(22)
Velocity: (t)) (t) + (t) 02)2(().
(23)
In the main paper, for simplicity and notation convenience, parameters and goal ProDMPs.
instead of wy to describe the
Intial Condition Enforcement The coefficients C1 and 02 problem delineated by the Eq, (3). Li et al.] propose utilizing the robot's initial state or the replanning state, characterized by the robot's position and velocity (yb- (b)) to ensure smooth com- mencement or transition from a previously generated trajectory. Denote the values of the comple- mentary functions and their derivatives the condition time
solutions to the initial value
V1o 1/2, V1, and ja.- Similarly, denote the values of the position and velocity basis functions at this time as Py and & respectively. Using these notations, Cj and C2 can be calculated as follows:
d0g Un, in. Big U2g U2, U1
V2, + Un. T
Wg
C1
91, Un, 22, vi,
(24)
C2
du 21, 26 B1g 92, Vig
51, +I T
21, 92, 22, Vi, Wy
Despite the complex form used in the initial condition enforcement, the solutions conducted above only rely solving several linear equations and can be casily implemented in therefore computationally efficient, normally < ms.
B.4 ENFORCE TRAJECTORY SEGMENTS' INITIAL CONDITION IN TOP-ERL.
In Figure 7 illustrate how TOP-ERL leverages this mechanism. The figure is based the
motion trajectory of the first degree freedom of the robot in the box-pushing task. the critic update, five segments example.
19

---

Published as a conference paper at ICLR 2025
ProDMP, trajectory generator, models the trajectory dynamic system. In TOP-ERL, the RL policy predicts ProDMP parameters, which are used to generate a force signal applied to the dynamic system. The system evolves its state based this force signal and the given initial conditions, such as the robot's position and velocity at a specific time. The resulting evolution trajectory, shown as the black curve in Fig. 7 can be computed in closed form and used to control the robot.
When the policy is updated and predicts a new force signal for the same task, a new action trajectory generated, depicted as the red dashed curve, which gradually deviates from the old trajectory. However, by utilizing the dynamic system's 's features, we can set the initial condition of each segment of the new trajectory to the corresponding old state. This ensures that the new action sequence used in the target computation in Eq.(7), can start from the old state, as shown across the five segments in the figure. Therefore, we matched the old state and new actions by eliminating the gap between them, as previously discussed in Section 4.3.1.
From our empirical results, we found that this enforcement is highly beneficial for value function learning. Interestingly, for policy updates, it introduces challenges since these conditions are not used as inputs to the policy. Therefore, we apply this technique only during value function training.
C EXPERIMENT DETAILS
C.I DETAILS OF METHODS IMPLEMENTATION
Table 3: Baseline methods categorized by type (ERL or SRI.) and update rules (On- or Off-policy).
Method
Category
Description
BBRL
Otto et al.
2022
ERL, On
Black Box Optimization style ERL, policy scarch in parameter space
TCE
Li et
2024
ERL, On
Extend BBRL per-step info for efficient policy update
PPO
Schulman et al.
2017
SRI., On
Standard on-policy method with simplified Trust Region enforcement
RSDE
Raffin et al.
2022
SRL. On
Consecutive exploration noise for NN parameters of the policy
GTXXL
Parisotto et al.
2020
SRL., On
canssormmr-augmeetee SRL with multiple state as history Standard off aff-policy method with entropy bonus for better exploration
SAC
Haarnoja et al.
2018a
SRL. Off
PINK
Eberhard
2022
SRL, Off
Use temporal correlated pink noise for better exploration
PPO
Proximal Policy Optimization (PPO) (Schulman al. 2017 prominent on-policy step-
based RL algorithm that refines the policy gradient objective, ensuring policy updates remain close to the behavior policy. PPO branches into two main variants: PPO-Penalty, which incorporates a KI--divereence term into the objective for regularization, and PPO-Clip, which employs clipped surrogate objective. In this study, we focus our comparisons on PPO-Clip due to its prevalent use in the field. Our implementation of PPO is based the implementation Raffin et al. (2211.
SAC Soft Actor-Critis (SAC) (Haarnoja et al. 2018a employs a stochastic step- based policy in off-policy setting and utilizes double Q-networks mitigate the overestimation of Q-values for stable updates. By integrating entropy regularization into the learning objective, SAC balances between expected returns and policy entropy, preventing the policy from premature convergence. Our implementation of SAC is based on the implementation of Raffif et al. (021)).
Gated TransformerXL (GTrXL) (Parisotto al., 2020 design to stabilize the training of Transformers in online RI, offers an easy-to-train, simple-to- implement but substantially more expressive architectural alternative to standard RNNs used for RL. agents in POMDPs. Our implementation of GTrXI. is based on the implementation of PPO GTrXL from Liang et al. (2018). We augmented the implementation with minibatch advantage normalization and state-independent log standard deviation as suggested in Huang et al. (022).
GTrXL
Transformer architecture that
SSDE
Generalized State Dependent Exploration (gSDE) Raffin et al. 2022 RückstieB et al,
2008
Rückstiess al. 2010) is exploration method designed to address issues with traditional
20

---

Published as a conference paper at ICLR 2025
step-based exploration techniques and aims provide smoother and more efficient exploration in the context of robotic reinforcement learning, reducing jerky motion patterns and potential damage to robot motors while maintaining competitive performance in learning tasks.
To achieve this, gSDE replaces the traditional approach of independently sampling from a Gaussian noise at each time step with more structured exploration strategy, that samples in depeedde manner. The generated samples not only depend on parameter of the Gaussian distribution μ & E, but also on the activations of the policy network's last hidden layer (s). We generate disturbances er using the equation
6 = 0,8, where 0, ~ Nad (0, 5)
The exploration matrix 6e is composed of vectors of length (im that were drawn from the Gaus- sian distribution we want gSDE to follow. The vector describes how this set of pre-computed exploration vectors are mixed. The exploration matrix is resampled at regular intervals, as guided by the 'sde sampling frequency' (ssf), occurring every n-th step if n is our ssf.
gSDE is versatile, applicable as substitute for the Gaussian Noise source in numerous and off-policy algorithms. We evaluated its performance in an on-policy setting using PPO by utilizing the reference implementation for gSDE from Raflin et al. (2022). In order for training with gSDE to remain stable and reach high performance the usage of a linear schedule over the clip range had to be used for some environments.
PINK We utilize SAC to evaluate the effectiveness of pink noise for efficient exploration. Eber- hard al. (2022) propose replace the independent action noise 6 of
a₂ Me of
with correlated noise from particular random processes, whose power spectral density fol- low a power law. In particular. the use of jinkkk noise, with the exponent 3 = in s(f) [F(c(() x f-0, should be considered (Eberhard et al. 2022).
We follow the reference implementation and sample chunks of Gaussian pink noise using the in- verse Fast Fourier Transform method proposed by Timmer & Koenig (1995). These noise variables are used for SAC's exploration but the the actor and critic updates sample the independent action distribution without pink noise. Each action dimension uses an independent noise process which causes temporal correlation within each dimension but not across dimensions. Furthermore, we fix the chunk size and maximum period to 10000 which avoids frequent jumps of chunk borders and increases relative power of low frequencies.
Black-Bou Reinforcement Learning (BBRL) OOto et al. 2022 2023 is recent developed episodic reinforeement learning method. By utilizing ProMPs (Paraschos et al. (2013) the trajec- tory generator, BBRL learns a policy that explores at the trajectory level. The method can effectively handle sparse and non-Markovian rewards by perceiving entire trajectory
BBRL
unified data point, neglecting the temporal structure within sampled trajectories. However, on the other hand, BBRL suffers from relatively low sample efficiency due its black-box nature. Moreover, the original BBRL employs degenerate Gaussian policy with diagonal covariance. In this study, we extend BBRL to learn Gaussian policy with full covariance build a more competitive baseline. For clar- ity, we refer to the original method as BBRL-Std and the full covariance version as BBRL-Cov. We integrate BBRL with ProDMPs (Li al. 2023), aiming to isolate the effects attributable to different MP approaches.
TCE Temporally-Corre Correlated Episodic RL (TCE) (Li al.) (2024) is an innovative ERL algorithm that leverages seeel information in episodic policy updates, shedding light on the "black box' of current ERL methods while preserving smooth and consistent exploration within the parameter space. TCE integrates the strengths of both dep-based and episodic RL, offering performance par with recent ERL approaches, while matching the data efficiency of state-of-ther (SoTA) step- based RL methods.
C.2 METAWORLD
World Yu et al. 2020 is an open-source simulated benchmark specifically designed for meta- reinforcement learning and multi-task learning in robotic manipulation. It features 50 distinct ma-
21

---

To ensure fair comparison, followed the evaluation protocol described in Otto al. (2022) and Li et al. (2024), where an episode is considered successful only if the success criterion is met at the end of the episode. This is equivalent requiring the robot complete the task and maintain Published as a conference paper at ICLR 2025
nipulation tasks, each presenting unique challenges that require robots learn wide range of skills, such as grasping. pushing, and object placement. Unlike benchmarks that focus on narrow task dis- tributions, MetaWorld provides broader range of tasks, making it ideal platform for developing algorithms that can generalize across different behaviors.
its success state until the episode ends, which a more rigorous measure than the original setting, where success at any time step sufficient.
We reported each individual Metaworld task in Fig. 8 and Fig. These tasks cover a wide range of types and complexities.
22

---

Published as a conference paper at ICLR 2025
TOP-ERL (ours)
TCE
BBRL
PPO
(PPL
BSDE
SAC
PINK
PrckDatCGitdd
1.0
1.0
1.0-
1.0
9.8
Sudecco rate TON
0.88
10N4
0.6
6.6-
0.6-
0.6
0.4
0.4-
0.4-
Success
0.4
15
Number of Eav Interaction
Namber of Eay Internction
Number of Eay Interaction
Number of Eun Interaation
Plate BlldcBockSile
BluPicking
Hammer
Swesplate
0..8
S00000 s0r0 50ge
8.8
D0M
0.8
8..8
3.6
0.6
0.5
0.6
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.5
1.0
2.0
0.5
1.0
1.5
2/0
0.0
0.5
1.0
1.5
2
0.5
1.0
1.5
2.0
BoaClloee
ButtonPress
Assemibly
BustonPressWiil
3.4
1.0
1.0
1.0
Succes rade D0M
2.0
0.0
0.4
0.4
0.4
0.4
0.2
0.2-
0.2-
0.22
0.0
0.5
1.0
1.
2.0
0.0
0.5
1G0
1.5
2.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
0.5
2.0
Number of Eas Interaction
Namber of Eay Interaction
Number
Eay Interaction
Number of Eus Interoction
1.0
1.0-
1.0
1.0
Spctcoo cate 10NE
10M
0.6
8.66
0.8-
8.8
0.4
0.4-
0.4-
0.4
0.5
1/0
015
1.5
Number of Eav Interaction
Namber Eay Interaction
Number of Eay Interaction
Number of Eus Interaation
CodleePush
DeafTurn
Disassemble
DoceClose
8.8
Baooogg page Toye
8.88
TOM
0.88
0.8
9.6
0.6-
0.6
0.0
0.4
0.4
0.2
0.0
0.0
0.0
0.0
155
DoneLock
DoeeOpen
DowcUnlook
Handinsert
3.0
1.0
1.0-
1.0
3.8
Spo0cg pate 500e
8.8.
2004
0.88
0.8
0.4
0.4
8.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.5
1.0
15
200
0.0
0.3
1.0
1'3
2.0
0.0
0.5
1.0
1.5
0.0
0.3
1.0
1.5
2.0
Number Interaction
Number Internction
Number
Interaction
Number
Interaction
DrawerClose
RaucetClose
2.0
1.0
1.0
1.0
Success ogto 10Mt
8.88
0.6.
0.6.
0.6
0.4
0.8
8.4
0.2
0.2
1.5
1.0
1.5
0'5
1.3
1.5
Number of Eav Interaction
Number of Tay Intectationn
Numbec of Das Intervction
Number of Enw Interattion
Figure 8: Success Rate IQM of each individual Metaworld tasks.
23

---

Published as a conference paper at ICLR 2025
(ours)
BBRL
BSDE
HesdicPeosssidee
HandiePress
Handi:PulSide
HandlePul all
1.0
1.0
1.0
1.41
Sucocs 20te DQM
2.n
ren
0.2
0.2
0.3
0.0
0.5
1,0
1.5
2,0
0.0
0.5
180
1.5
2.0
0.0
0.5
1.0
1.5
2
0.0
0.5
1.0
1.5
2,0
Number
Interaction
Namber Internction
Number Tiee Interartion
Number Internntion
PeglosertSide
3.0.
1.0-
1.0-
1.0
Sy0005 rate 2004
TOM
0.6
8.6
0.6
9.6
0.4
0.4-
0.4
9.4
Number Eav Interaction
Namber Eey Interaction
Number Eay interaction
Number Interactios
Pushifack
Push
PirkPlace
PegtinplugSid Ride
2.8
S0000s pate 1004
8.8-
Succees TOM
0.8
8.8
0.6-
0.5
0.0
0.0
0.0
0.0
0.0
2.0
1'5
0.0
1.0
3.5
Number
Soccec
StickPush
StikPall
PusiWalll
3.0
1.0
1.0
1.0
0.8
1QM
8.8
8.8
0.88
0.2
0.2
0.2
0.5
0.5
1.0
2.0
0.0
0.5
1.0
0.0
0.5
1.0
2.0
0.0
0.5
1.b
2.0
Number Interaation
Number Enve Interwation
Number Bive Interwation
Number Bov Intereation
ReachWWlll
SheltPlace
WindowOpen
1.0
1.0
1.0
1.0
Supoae tate 1044
Sucocso rate 2004
0.0
2.4
8.4
0.6
0.2
0.5
0.5
1.0
1.5
0.5
3.0
15
Number of Eav Interactuns
Number of Say Interactios
Numbbr of Say Interediao
Number of Bit Interactio
WindowCloes
Backethall
ToM
Specess cate TONE
0.8-
8.8
0.4-
8.4
Numbles
Figure 9: Success Rate IQM of each individual Metaworld tasks.
24

---

Published as a conference paper at ICLR 2025
C.3 HOPPER JUMP
As an addition to the main paper, we provide more details on the Hopper Jump task. We look both the main goal of maximizing jump height and the secondary goal of landing on a desired position. Our method shows quick learning and does well in achieving high jump height, consistent with what we reported carlier. While it's not as strong in landing accu- racy, still ranks high in overall performance. Both versions of BBRL have similar results. However, they train more slowly compared to TCE, highlighting the speed advantage our method due to use of inter- mediate states for policy updates. Looking other methods, step-based ones like PPO and TRPL focus too much on landing distance and miss out on jump height, leading to less effective policies. On the other hand, gSDE performs well but is sensitive to the initial setup, as shown by the wide confidence ranges in the results. Lastly, SAC and PINK shows inconsistent results in jump height, indicating the limitations of using pink noise for exploration, especially when compared to gSDE.
Figure 10: Hopper Jump
C.4 Box PUSHING
The goal of the box-pushing task is to move box to specified goal location and orientation using the 7-DoFs Franka Emika Panda (Otto et al.) 2022). To make the environment more challenging, the environment from a fixed initial box position and orientation to randomized initial position and orientation. The range of both initial and target box pose varies from x E [0.3, 0.6]. E [-0.45, 0.45). 0g E positional distance error of less than 5
extend
[0, 2T). Success is defined cm and a z-axis orientation error of less than 0.5 rad. We refer to the original paper for the observation and action spaces definition and the reward function.
Figure 11: Box Pushing
D ACTION CORRELATION WITH SEGMENTATION
DoF4
DoF5
DoF6
DoF7
DoF4
DoF5
DoF6
DoF7
DoF4
DoF4
DoF4
DoF5
DoF5
DoF5
DoF6
DoF6
DoF6
DoF7
DoF7
DoF7
(a) No Segmentation (BBRL)
(b) Random Segmentation (Ours)
(c) Fixed Segmentation (TCE)
Figure 12: This figure presents predicted actions' correlation across DoF and 100 time steps, vi- sualized in 400 400 correlation matrix. Each 100 100 square tile demonstrates the movement correlation between two DoF during these steps. Correlation values range from -1 (negative cor- relation, depicted in blue) to I (positive correlation, depicted in red), with white areas indicating no correlation. BBRL treats the entire trajectory thus, the correlation broadcasts smoothly across time steps, as shown in (a). On the contrary, TCE uses segmentation with fixed length, constraining the correlation learning within fixed segments, resulting in sudden correlation changes at each segment's boundary, as presented (c). TOP-ERL. utilizes randomly sampled segment length and positions itself between the two paradigms, being able to learn the smooth correlation while retaining the benefits of higher sample efficiency by using segmentation.
whole and does not have any segmentation;
25

---

Published as a conference paper at ICLR 2025
E HYPER PARAMETERS
We executed a large-scale grid search to fine-tune key hyperparameters for each baseline method. For other hyperparameters, relied on the values specified in their respective original papers. Below is a list summarizing the parameters we swept through during this process.
BBRL: Policy net size, critic net size, policy learning rate, critic learning rate, samples per itera- tion, trust region dissimilarity bounds, number of parameters movement DoF.
TCE: Same types of paper-parameters listed in BBRL, plus the number of segments per trajectory. A learning rate decaying scheduler is applied stabilize the training in the end.
Policy network size, critic network size, policy learning rate, critic learning rate, batch size, samples per iteration.
PPO:
gSDE: Same types of hyper-parameters listed in PPO, together with the state dependent explo- ration sampling frequency (Raffin et al. 20022..
SAC: Policy network size, critic network size, policy learning rate, critic learning rate, alpha learn- ing rate, batch size, Update-To- To-ta (UTD) ratio.
PINK: Same types of peper-parameters listed in SAC.
Number of multi-head attention layers, number of heads, dims per head, importance- sampling ratio clip, value function clip, grad clip, and same hyperparameters listed in PPO
GTrXL:
TOP-ERL: Number of multi-head attention layers, number of heads, dims per head, learning rates. The other movement primitives hyper-parameters are taken from TCE.
The detailed hyper parameters used listed in the following tables. Unless stated otherwise, the notation lin_x refers to a linear schedule. It interpolates linearly from to 0 during training. The ERL methods (TCE, BBRL) take an entire trajectory as a sample where the SRL methods take one time step as a sample. In this way, one sample in ERL is equivlent to T sample of SRL, where T is the length of one task episode.
26

---

Published as a conference paper at ICLR 2025
Table Hyperparameters for the Meta-World experiments. Episode Length T' 500
PPO
BSDE
GTXXL
SAC
PINK
TCE
BBRL
TOP-ERL.
number samples
16000
16000
19000
1000
4
16
16
n.a.
n.a.
discount factor
0.99
0.99
0.99
0.99
0.99
1.0
0.a.
n.a.
n.a.
11..
0.0055
0.005
0.005
ex:
B.a.
n.a.
n.a.
trast region loss coef.
n.a.
10
1.0
optimizer
adam
adam
adam
adam
adom
adam
adam
adam
epochs
leamiing rate
3c-4
1o-3
2c-4
3o-4
3o-4
3c-4
3e-4
1o-3
use critic
Tree
Trae
True
epochs critic
1000
50
100
leaming rate aritic
3e-4
1e-3
2e.4
3e-4
3e.4
3e-4
3e-4
5e-5
number minibatches
B.a.
batch size
a.a.
500
1024
256
512
n.a.
n.a.
256
butTer
m.a.
n.a.
le6
n.a.
n.a.
leamning starts
10000
165
0
0
polyak_weigg weight
0.a.
n.a.
n.a.
5e-3
56-3
n.a.
n.a.
SDE sampling trequency
entropy coefficient
auto
auto
n.a.
normalized observations
True
True
False
Folse
False
True
False
normalized rewards
True
True
0.05
False
False
False
False
False
observation clip
10.0
n.a.
n.a.
n..
na.
n...
n.a.
n...
reward clip
10.0
10.0
10.0
critic clip
0.2
10.0
n.a.
n.a.
1.S.
n.a.
U.a.
importance ratio clip
0.3
n.a.
n.a.
hidden layers
[128, 128]
[128, 128]
[256,
[256, 256]
[128, 128]
[128,
hidden layers critic
[128, 128)
[128, 128]
n.a
[256, 256]
[256, 256]
[128, 128)
[32, 32]
n.a.
hidden activation
retu
Meaky.relu
orthogonal initialization
Yes
No
xavier
fanin
fanin
Yes
Yes
Yes
initial
number of beads
4
dims per head
16
number attention layers
max sequence longth
5
1024
I Linear Schedule from 0.3 0.01 during the first 25% of the training. Then continued with 0.01.
27

---

Published as a conference paper at ICLR 2025
Table 5: Hyperparameters for the Box Pushing Dense, Episode Length = 100
PPO
gSDE
GTXXL
SAC
PINK
TCE
BBRL
TOP-ERL
number samples
48000
80000
8000
8
8
152
152
4
GAE
0.95
0.95
0.95
118..
D.a.
0.95
n.a.
n.a.
discount factor
1.0
1.0
0.99
0.99
0.99
1.0
1.0
1.0
0.3
m.a.
n.a.
0.05
0.1
0.005
6x:
n.a.
0.00025
0.0005
trust region loss coef.
n.a.
10
1.0
optimizer
adam
adam
adamm
adam
adam
adam
adam
adam
epochs
10
20
15
learning rate
5c-5
le-4
2c-4
3e-4
3o-4
3o-4
3e-4
3o-4
critic
True
True
True
True
True
epochs critic
10
10
1
50
10
30
learning rate
le-4
3e-4
5e-5
numher minihatches
40
n.a.
n.a
n.a.
batch size
n.a.
2000
1000
512
512
n.a.
n.a.
512
buffer size
266
2c6
7000
learning starts
le5
le5
8000
polyak weight
Se.3
5e:3
Se:3
SDE sampling frequency
n.a.
n.a.
n.a.
D.a.
n.a.
n.a.
n.a.
entropy coefficient
auto
normalized observations
True
True
False
False
False
True
False
False
normalized rewards
True
True
False
False
False
False
False
observation clip
10.0
reward clip
10.0
10.0
10.
m.a.
n.a.
n.a.
n.a
n.a.
critic clip
0.2
importance ratio clip
0.2
0.2
0.1
n.a.
n.a.
n.a.
n.a.
n.a.
hidden layers
[512, 512)
[256. 256]
n.a.
[256. 256]
[256, 256]
[128, 128]
[128, 128]
[256, 256]
hidden layers critic
[512, 512)
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[256, 256]
hidden activation
Lanh
tanh
relu
tanh
tanh fanin 1.0
leaky relu
leaky_relu
leaky_relu
orthogonal initialization
Yes
No
xavier
fanin
Yes
Yes
Yes
initial std
1.0
0.05
1.0
1.0
1.0
1.0
1.0
number
dims per head
16
16
number attention layers max sequence length
5
1024
Movement Prinitive (MP) type
113.
11...
value
1.a.
D.d.
PruDMPs
ProDMPs
ProDMPs
number basis functions
value
8
weight scale
0.3.
n.a.
value
n.a.
n.a.
0.3
0.3
0.3
goal scale
D.a.
0.3
28

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameters for the Box Pushing Sparse, Episode Length T 100
PPO
ESDE
GTrXL
TCE
BBRL
TOP-ERL
number samples
48000
80000
8000
76
76
4
GAE
0.95
0.95
0.95
n.a.
n.a.
0.95
D.a.
n.a.
discount factor
1.0
1.0
1.0
1.0
1.0
G
n.a.
n.a.
0.005
0.0005
0.00025
0.0005
trust region loss coef.
n.a.
n.a.
n.a.
D.d.
D.a.
10
1.0
optimizer
adam
adam
adam
adam
adam
adam
adam
adam
epochs
10
10
50
20
15
learning rate
5e-4
1c-4
2e-4
3e-4
3e-4
3e-4
3e-4
critic
True
True
True
True
True
True
True
True
epochs critic
10
10
50
10
30
leamning rate critic
le-4
3e-4
3e-4
Se:5
number minibutches
40
n.a.
n.a.
n.a.
n.a.
batch size
n.a.
2000
1000
512
512
n.a.
n.a.
512
n.a
n.a.
n.a.
n.a.
n.a
7000
leamiing starts
0
0
le5
le5
0
400
polyak. weight
n.a.
n.a.
0
5o-3
5e-3
n.a.
n.a.
5o-3
SDE sampling frequency
n.a.
n.a.
n.a.
n.a.
n.a.
n.a.
entropy coefficient
0.01
0
normalized observations
True
True
False
Fulse
False
True
False
False
normalized rewards
True
True
0.1
False
False
False
False
False
observation clip
10.0
n.a.
False
D.a.
n.a
n.a.
D.a.
n.a.
reward clip
n.a.
n.a.
n.a.
n.a.
critic clip
0.2
0.2
10.0
D.a.
importance ratio clip
0.2
0.2
0.1
n.a.
n.a.
n.D.
n.a.
n.a.
hidden layers
[512, 512]
[256, 256]
n.a.
[256, 256]
[256, 256]
[128, 128]
[128, 128]
[256, 256]
hidden layers critic
[512, 512]
[256, 256]
n.a.
[256, 256)
[256, 256]
[256, 256]
[256, 256]
n.a
hidden activation
tanh
tanh
relu
tanh
leaky relu
reau
leaky relu
orthogonal initialization
Yes
No
xavier
funin
fanin
Yes
Yes
Yes
initial std
1.0
0.05
1.0
1.0
1.0
1.0
1.0
1.0
number of beads dims head
8
16
16
number of attention layers
4
2
max sequence length
1024
type
D.a.
11.0.
value
n.a.
n.a.
ProDMPs
ProDMPs
ProDMPs
number basis functions
value
8
8
weight scale
n.a.
value
0.3
0.3
0.3
goal scale
0.a.
n.a..
value
n.a.
n.a.
0.3
0.3
0.3
29

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameters for the Hopper Jump, Episode Length T 250
PPO
gSDE
GTrXI.
SAC
PINK
TCR
BBRI.
TOP-ERI.
number samples
8000
8192
10000
1000
64
GAE A
0.95
0.99
0.95
n.a.
0.95
discount factor
1.0
0.999
1.0
0.99
0.99
1.0
1.0
1.0
Sp
D.a.
n.a..
0.8.
n.a.
1...
n.a.
<z trust region loss coef.
n.a.
n.d.
n.a.
n.a.
n...
n.a.
1.0
optimizer
adam
adam
adam
adam
adam
adam
adam
adam
epochs
10
10
10
1000
50
100
10
learning rate
3c-4
9.5c-5
5c-4
1o-4
2c-4
lc-4
True
True
True
True
True
True
epochs critic
10
10
1000
100
20
learning rate critic
3c-4
9.5c-5
5c-4
lc-4
2c-4
le-4
lc-4
5c-5
number minibatches
40
n.a.
n.a.
n.a.
n.a.
batch size
n.a.
128
1024
256
256
n.a.
n.a.
256
buffer
n.a.
n.a.
n.a.
lc6
n.a.
n.a.
1000
learning starts
polyak weight
5o-3
So-3
5e-3
SDE sampling frequency
8
entropy coefficient
0
0.0025
0.
auto
auto
normalized observations
True
Fulse
False
False
False
True
False
False
normalized rewards
True
False
False
False
False
False
False
False
observation clip
10.0
False
n.a.
reward clip
10.0
10.0
10.
n.a.
critic clip
0.2
lin.0.4
1.
n.a.
importance ratin clip
0.2
lin_0.4
0.2
n.a.
n.a.
n.a.
n.a.
n.a.
hidden layers
[32, 32]
[256, 256]
n.a.
[256, 256]
[32, 32]
[128, 128]
[32, 32]
[128, 128]
hidden layers critie
[32,
[256, 256]
(256. 256]
[32,
[128, 128]
[32, 32]
hidden activation
lanh
tunh
relu
relu
relu
leuky relu
tunh
leaky_relu
orthogonal initialization
Yes
No
xavier
fanin
fanin
Yes
Yes
Yes
initial std
1.0
0.1
1.0
1.0
1.0
1.0
1.0
1.0
number of heads
4
dims per head
numher of attention layers
sequence length
1024
MP
D.a.
value
n.a.
T...
ProDMPs
ProDMPs
ProDMPs
number basis functions
n.a.
n.a.
value
n.a.
n.a.
3
weight scale
n.a.
n.a.
value
n.a.
n.a.
goal scale
D.a.
n.a.
value
n.a.
n.H.
30


Reviews:
[{"review_id": "i1KK7xcU94", "reviewer": "Reviewer_gDNi", "summary": "This paper presents TOP-ERL, Transformer-based Off-Policy Episodic RL, which leverages the Transformer as a critic to predict the value of action sequences, splits it into smaller segments and inputs them into the Transformer for value prediction, with an off-policy update scheme.", "questions": "1. The paper claims to be \"the first off-policy episodic reinforcement learning algorithm,\" but this seems to be an overclaim. To my knowledge, there are already several related works, such as:\nLiang D, Zhang Y, Liu Y. \"Episodic Reinforcement Learning with Expanded State-reward Space.\" arXiv preprint arXiv:2401.10516, 2024.\n\n2. Regarding the statement about \"enforcing action initial conditions,\" can this approach effectively address the mismatch between the state and the corresponding action sequence? In addition to the ODE explanation, could you design experiments to validate this claim?\n\n3. Why does using random segments yield better results than most fixed segments? Could you provide a more intuitive explanation? The correlation presented in Appendix D does not seem to clarify this adequately.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is easy to follow. Given that research on episodic RL is relatively limited, TOP-ERL shows its strengths in sample efficiency and overall performance.", "weaknesses": "1. The code repository is incomplete, lacking the environment configuration files and key training files. Additionally, several important functions, such as ValueFunction, have not been implemented.\n\n2. Figure 4(a) only presents the average results for the 50 tasks, without showing the individual results for each task. While I understand that the main text has space constraints, I strongly recommend including the results for all 50 tasks in the appendix.\n\n3. The methodology seems to lack novelty, as important components follow the design of previous episodic RL methods. I did not observe other core contributions aside from the reported strong performance. In the experimental section, it would be beneficial to see experiments and analyses related to the gaussian policy and movement primitives module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B6gKm7Dv13", "reviewer": "Reviewer_bGrR", "summary": "The paper proposes a new off-policy episodic reinforcement learning framework that, especially, improves sampling efficiency over traditional episodic-based and step-based baselines. A main novelty of the work is using a Transformer-based architecture to train a critic network that predicts value and Q-functions for a sequence of actions from an initial state. This critic network is then adapted and applied in an SAC-like algorithm for off-policy episodic reinforcement learning training.", "questions": "Comments:\n\n1.\tIn Eq. (8), the future return after N-step is modeled by the critic network during training? It is interesting that if this is the case, the training is stable.\n2.\tIn the reviewer’s view, it is hard to grasp the need for the enforcement of the initial condition in section 4.3.1 as s_0^k is already a condition to generate the trajectory in the policy. Some additional intuition about how this is related to the division in segments, or because of the variable length used, or the sampling of trajectories would clarify this part to the reader.\n3.\tAnother critical part is Section 4.4.  It is not clear to the reviewer how SAC is adapted to the new policy objective during training. \n4.\tIn Fig.4, why for (c) and (d), the TOP-ERL training is stopped before the other baselines?\n5.\tThe impact of choosing random segments seems to be crucial to performance. Any intuition regarding this result?\n6.\tGiven the resemblance of the critic training to the prediction of rewards that can be used for more effective credit assignment compared to dense and sparse rewards, the reviewer thinks what is the relation between the proposed methodology and using other networks that predict future rewards for more effective credit assignment in performance for the robotic tasks in Section 5.\n\nMinor Comments:\n\n1.\tLine 62: Typo “Transformer”\n2.\tLines 124-130: The writing in these lines is very confusing. It could be rewritten more clearly.\n3.\tTitle Section 4.5: Is it “summarize” the right word here?\n4.\tLine 425: Typo “empirical”\n5.\tLine 439: Typo “Noteably”\n6.\tLine 471: Typo “sprase”", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tA critic network for action sequences using a traditional Transformer architecture is proposed to model N-step returns.\n2.\tA SAC-like algorithm is adapted for off-policy episodic reinforcement learning using the trained critic network.\n3.\tThe proposed methodology is shown to improve sampling efficiency and also is shown to stabilize training over other baseline methods.", "weaknesses": "1.\tIn the current writing, it is hard to identify the novel parts that are proposed by the author and parts that are kept from other works.\n2.\tSection 4.4 seems very important to understand how to adapt SAC to episodic reinforcement learning but it seems not clear.\n3.\tGiven the resemblance of the critic training to the prediction of rewards that can be used for more effective credit assignment compared to dense and sparse rewards, additional ablation studies or baselines could be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TrAcqYgI4s", "reviewer": "Reviewer_iE4K", "summary": "The paper introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), an interesting combination of episodic RL with off-policy updates and the transformer architecture. \nThe main contribution is the way to use the transformer architecture in this setting to learn an off-policy critic.\nThe authors leverage N-step returns, Trust Region Projection Layer, Probabilistic Movement Primitives and Target Networks. \nEmpirical results on robot learning environments (Meta-World, etc.) show that TOP-ERL outperforms state-of-the-art RL methods, both in terms of policy quality and sample efficiency. The paper also includes thorough ablation studies to analyze the impact of key design choices.", "questions": "1. It is not completely clear to me if the transformers can attend only the current observation s^k_0 or also the previous ones during training?\n\n2. I am wondering how important are the probabilistic MP in the performance, what if the policy was also a transformer?\n\n3. I am also wondering if the positional encoding of two different states s^k_0 and s'^k_0 would always be the same (even if one is more in the future that the other one)?\n\n4. Is a target network necessary at all?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I find the paper very well written, also well structured, motivated and explained.\n\nThe idea of using a transformer such that s_0 maps to V(s), a_0 maps to Q(s, a_0), a_1 maps to Q(s, a_1), ... is very natural and creative.\nTo the best of my knowledge, it is the first approach that proposes to do it this way.\n\nI also find it interesting that V and Q can be modeled by the same unique transformer architecture without having dedicated weight just for V or just for Q.\n\nThe contributions of this paper are good for the field of RL with transformers, although I believe ERL is slightly less used than standard RL.", "weaknesses": "The authors claim that off-policy algorithms are often more sample efficient than on-policy counterparts. However, PPO is still often the way to go when you face a new RL problems because of better stability and the need for less hyperparameter tuning. I am not fully convinced this approach won't have the same issue.\n\nI believe the explanations of why importance sampling is not necessary could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v9lXfOqMFy", "reviewer": "Reviewer_L7MP", "summary": "This paper presented a transformer-based episodic reinforcement learning method for enabling online updates in an off-policy RL framework. Combined by a trajectory segmentation technique, the transformer architecture is employed to construct a critic that evaluates the values of segments of state-action trajectories. As a result, the proposed method achieves an off-policy update, improving update efficiency compared to previous episodic RL methods using on-policy training. Empirical evaluations on control tasks show that the proposed method improves performance.", "questions": "-  Designing additional experiments to verify the individual contribution of the transformer-based critic network, such as TOP-ERL without the transformer, is suggested. However, I have no idea about the specific details about the exp.  \n- Explaining why layer normalization and trust region constraints are critical in Section 5.3 are suggested.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well motivated.\n- The writing is quite good. \n- The main experiments are conducted on the large-scale RL benchmark.\n- Analysis of many design choices are presented.", "weaknesses": "- Some used techniques, such as trust region constraints, layer normalization, greatly affect the performance of the proposed algorithm (see Fig. 5). The ablations of TOP-ERL, which do not use these techniques, actually achieved worse performance than previous methods.  Someone may conclude that the performance improvement of TOP-ERL over baselines is achieved by using all these techniques, rather than the usage of the transformer architecture or others. \n\n- Layer normalization and trust region constraints is important for TOP-ERL achieving good performance. But the paper lacks a detailed discussion about the importance of these two components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)