POLTER: Policy Trajectory Ensemble Regularization for Unsupervised Reinforcement Learning
Authors: Frederik Schubert
Carolin Benjamins
Sebastian Döhler
Bodo Rosenhahn
Marius Lindauer
Venue: Accepted by TMLR ()

Abstract: The goal of Unsupervised Reinforcement Learning (URL) is to find a reward-agnostic prior policy on a task domain, such that the sample-efficiency on supervised downstream tasks is improved. Although agents initialized with such a prior policy can achieve a significantly higher reward with fewer samples when finetuned on the downstream task, it is still an open question how an optimal pretrained prior policy can be achieved in practice. In this work, we present POLTER (Policy Trajectory Ensemble Regularization) – a general method to regularize the pretraining that can be applied to any URL algorithm and is especially useful on data- and knowledge-based URL algorithms. It utilizes an ensemble of policies that are discovered during pretraining and moves the policy of the URL algorithm closer to its optimal prior. Our method is based on a theoretical framework, and we analyze its practical effects on a white-box benchmark, allowing us to study POLTER with full control. In our main experiments, we evaluate POLTER on the Unsupervised Reinforcement Learning Benchmark (URLB), which consists of 12 tasks in 3 domains. We demonstrate the generality of our approach by improving the performance of a diverse set of data- and knowledge-based URL algorithms by 19% on average and up to 40% in the best case. Under a fair comparison with tuned baselines and tuned POLTER, we establish a new state-of-the-art for model-free methods on the URLB.

Published in Transactions on Machine Learning Research (04/2023)
POLTER: Policy Trajectory Ensemble Regularization for Unsupervised Reinforcement Learning
Frederik Schubert
schabert www.::tnww.uiiinoooooo.e.o
Institute for Information Processing Leibniz Universily Hannover
Carolin Benjamins
Instibute of Artificial Intellgence Leibniz University Hannover
Sebastian Döhler
dochler wnw.ummoenn-eeeeeoe....de
Institute for Information Processing Lesbniz University Hannover
Bodo Rosenhahn
rosenhahn Stnt, Sennt.uniihhmmower..d
Institute for Information. Processing Leibniz University Hannover
Marius Lindauer
m. m.lindauer@ais www.eaao....ieeeemme..e...
Institate of Artificial Intellignnce Leibniz University Hannover
Reviewed OpenReview: https: openrreveee, net/ forum? id= Hnr23kn2fy
Abstract
The goal of Unsupervised Reinforcement Learning (URL) is to find reward-agnostec prior policy on a task domain, such that the sample-efficinccy on supervised downstream tasks is improved. Although agents initialized with such prior policy can achieve significantly higher reward with fewer samples when finetuned on the downstream lask, is still an open question how an optimal pretrained prior policy can be achieved in practice, work, we present POLTER (Policy Trajectory Ensemble Regularization) to regularize the pretraining that can be applied to any URL algorithm and is especially useful data- and knowledge- based URL algorithms. Tt utilizes an ensemble of policies that are discovered during pretraining and moves the policy of the URL algorithm closer to its optimal prior. Our mothod is based on a thcoretical framework, and we analyze its practical effects on a white-box benchmark, allowing us lo study POLTER with full control. In our main experiments, we evaluate POLTER on the Unsupervised Reinforcement Learning Benchmark (URLB), which consists of 12 tasks in 3 domains. We demonstrate the generality of our approach by improving the performance of diverse set of data- and knowledge- based URL algorithms by 19% average and up to 40% in the best case. Under fair comparison with tuned baselines and tuned POLTER, we establish a new state-of-the-art for model-free hots on the URLB.
this general method
1 Introduction
Reinforcement Learning (RL) has shown many successes in recent years (Mnih et al., 2015; Silver et al., 2017; OpenAI al., 2019) and is starting to have an impact on real-world applications (Bellemare et al., 2020; Mirhoseini al., 2021; Degrave al., 2022). However, all these applications require detailed knowledge of the task to train the agents in sufficiently close simulation. Implementing each and every task and

---

Published in Transactions on Machine Learning Research (04/2023)
Pretraining
t
P1
p
p
P2
P3
PT
Po
Finetuning
Figure 1: Optimization trajectory in the induced state- marginall space p(s) of a policy ((s) that optimizes an Unsupervised Reinforcement Learning objective with Policy Trajectory Ensemble Regularization (POLTER) regularization. The regularized policy's state distribution P((s) is pulled towards the ensemble state distribution 7(s) during pretraining. The ensemble policy #ss) approximates the optimal prior (s) with its induced statenmarginal (s) for finetuning
specific task T. Note that other tasks + might lie further away from the average. However, shown by Eysenbach et al. (2022), the average still minimizes the distance without knowledge about the specific downstream task. The state distribution trajectory of a policy without POLTER regularization indicated by dashed lines O
training new agent is time-consuming and inefficient. In some cases, the task might be difficult to model in simulation, or it unknown beforehand, which makes sample-efficieccy even more important. Unsupervised Reinforcement Learning (URL), i.e., reward-free pretraining a task domain (Srinivas and Abbeel, 2021), has shown to improve the sample-efficiecyy of finetuning RL algorithms on downstream tasks. This setup only requires a simulation of the agent's domain without having to model task-specific components in detail. Although URL results in higher sample-efficiency wide range of tasks. For example, Laskin et al. (2021)
applications, pretraining policy still does
not guarantee improvement in performance observed that longer pretraining often degrades the performance of many URL algorithms. Also, many finetuning steps are still needed to achieve the optimal return, and some algorithms fail to learn a sufficient policy at all.
In this work, we introduce POLTER (Policy Trajectory Ensemble Regularization) a general method improve the performance of URL algorithms via novel regularization term during pretraining that results in better prior policies for finetuning specific tasks. Our thcorctical motivation builds on the geometric interpretation of URL by Eysenbach al. (2022) who connect the optimal prior policy to specific state distribution. The prior state distribution is optimal in the sense that it minimizes the distance to the state distribution of an adversarially chosen downstream task. This problem statement can be seen an instance of the information-theone try to approximate the optimal prior state distribution by learning separate skills during pretraining that are combined to initialize the finetuning policy. POLTER borrows this idca for URL algorithms that do not learn explicit skills, e.g., data- and knowledge-based URL algorithms. This is based the insight that each point during pretraining encodes implicit skill that depends the pretraining objective. In mutshell, our main idea is to minimize the distance of the current policy to ensemble policy during pretraining, which acts as proxy for the optimal prior. As shown in Figure 1, the pretraining trajectory of the policy and hence the induced state distribution affected by our regularization and attracted to the ensemble of earlier pretraining policies, each potentially encoding a different skill.
framework bounded rationality (Ortega and Lee, 2014). Some URL algorithms
We demonstrate the effect of POLTER on the simplistic PointMass environment and tasks in the Pendulum domain, and provide empirical evidence that method reduces the distance to the optimal prior policy and
2

---

Published in Transactions on Machine Learning Research (04/2023)
improves the downstream performance. Extensive experiments on the Unsupervised Reinforcement Learning Benchmark (URLB) (Laskin et al., 2021) show that our method improves the Interquartile Mean (TQM) (Agarwal et al., 2021) performance of data- and knowledge-based URL algorithms on average by 19%. A sensitivity analysis reveals that POLTER is a robust method with a reasonable default setting but can further be tuned on some domains and algo Additional experiments on the siate-visitation entropy provide empirical evidence that our method improves the performance by finding better priors in contrast to an improved exploration. In summary, our contributions are:
1. We introduce POLTER, a novel regularization method for URL algorithms that uses an ensemble of pretraining policies to reduce the deviation from the optimal prior policy.
2. Our method is empirically validated the white-box PointMass benchmark and demonstrated in the Pendulum domain.
3. An extensive evaluation on the URLB shows the effectiveness of POLTER for broad range of URL algorithms with pixel- and state-based observations.
With our sensitivity amalysis, we demonstrate that our approach is robust with a good default and can be further boosted via a task-specific strength of the regularization and achieves a new state-of-the-art for model-free algorithms on the URLB
2 Preliminaries
2.1 Notation in Reinforcement Learning (RL)
RL relies on the notion of a Markov Decision Process (MDP) (Sutton and Barto, 1998). An MDP is a tuple (S, A. P, r) which describes the states S, actions a E A, transition dynamics p(8 st, s1,44) P, initial state distribution po(so) ~ P, reward function r(ss) and discount factor (0, 1] of an environment. At each time step 1, an agent observes the state of the environment and chooses an action de using a policy [a; st) to transition into a new state St+1- The objective is to maximize the expected discounted return by finding a policy x (s) that induces the optimal discounted state occupancy mcasure or state-marginal (*()) = (1- - Eio 2Pp (s), where Pr (s) is the probability of being in state 8 at time step when following policy n². As we focus on state-based reward functions r(s), this optimal discounted state occupancy measure corresponds the maximal discounted return of the optimal policy.
2.2 Related Work
Unsupervised Reinforcement Learning describes algorithms that train an RL agent without a learning signal, i.e., without lask-specific reward function. It encompasses two main research branches: The first is Unsupervised Representation Learning (van den Oord et al., 2019; Ebert et al., 2018; Ha and Schmidhuber, 2018; Laskin et al., 2020; Schwarzer et al., 2021a; Stooke et al., 2021) which focuses on learning representation of the states or dynamics from noisy or incomplete observations. This line of work includes algorithms that learn world models to train an agent with imagined trajectories to reduce the number of environment interactions (Dockhorn and Apeldoorn, 2018; Hansen et al., 2022; Xu et al., 2022; Rajeswar et al., 2022; Seo et al., 2022). The second branch is called Unsupervised Behavioral Learning.
Unsupervised Behavioral Learning is a variant of the RL problem where the agent can interact with the environment in reward-free setting before being assigned the downstream task T defined by an external reward function (Oudeyer et al., 2007; Jin et al,, 2020). During pretraining, intrinsic rewards are used to train a policy to become an optimal prior and initialization for the subsequent finetuning task. We denote the policy after finetuning as गे and the corresponding oracle policy that acts optimally on lask T as म.. Algorithms for Unsupervised Behavioral Learning can be categorized into knowledge, data- and competence-basee approaches (Srinivas and Abbeel, 2021). Knowledgo-basee algorithms define a self-supervised task by making predictions on some aspect of the environment (Pathak et al., 2017; Chen et al., 2022a;b). The approach based methods follow is to maximize the state visitation entropy to explore
3

---

Published in Transactions on Machine Learning Research (04/2023)
the environment (Zhang et al., 2020; Guo et al., 2021; Mutti et al., 2021; 2022; Hazan et al., 2019; Zhao et al., 2022; Jacq et al., 2022; Nedergaard and Cook, 2022). A recent approach (He et al., 2022) maximizes the Wasserstein distance of the induced state-marginals instead. Competence-baser algorithms learn skills that maximize the mutual information between the trajectories and a space of skills (Mohamed and Rezende, 2015; Gregor et al., 2017; Baumli et al., 2021; Jiang et al., 2022; Zeng et al., 2022; Shafiullah and Pinto, 2022; Cho et al., 2022). Some algorithms relax the requirement of a task-agnostic prior by selecting the most promising skill before finetuning on the task (RHIM et al., 2022; Laskin et al., 2022). All categories have one thing in common: They traverse the policy space during pretraining and potentially encounter behavior close to the optimal prior policy for the finetuning tasks. There are also algorithms that show great performance by combining both branches of URL by simultaneously learning the dynamics and possible behaviors in a domain (Sharma et al., 2020; Yuan et al., 2022). In this work, however, we focus on model-free approaches.
3 Method
In the following, we first discuss the motivation of our approach by linking it to the theoretical results of Eysenbach et al. (2022). Based on that, we derive the regularization strategy of POLTER using a mixture ensemble policy and discuss the concrete implementation of our method. To ground our theoretical motivation, we show exemplary behaviour of CRL algorithms with and without POLTER on simple control tasks.
3.1 Motivation
The adaptation objective by Eysenbach el al. (2022) connects the policy prior after pretraining (s) with the finetuned policy that has been adapted to arbitrary downstream task m7(s). As the reward function be identified with state-occupancy measure and because there are multiple policies that induce the same state-occupancy measure, the adaptation objective is defined in terms of these mtati-marginals:
min max E (M)
[r()]]
E (mpp((s)
[r()]] + DKL. p()))
(1)
(+)EC P
worst-case regret
informatioo cost.
The first term in Equation (1) represents the regret that a finetuned policy (s) achieves with respect to an oracle policy (s) for the given task T. The second term defines the information cost between क (s) and the prior after pretraining =ss) via their induced maagi-mas (s((s and p(s). The set of all feasible state-marginals in an environment is denoted as C.
Competence-base based URL algorithms learn set of skills 2 E Z during pretraining by maximizing the mutual information =(p(s)=)) between these skills and the state-marginal." We use the non-standard notation ((p()):)) to differentiate between the mutual information of different state-marginals For these algorithms,, Eysenbach et al. demonstrated that the state-marginal averaged over all skills minimizes the distance to the furthest skill. Note, that this skill is generally unknown and depends on the geometric structure of the state-marginal space of the domain:
Lemma (Lemma 6.5 in Eysenbach et al. (2022), Theorem 13.1.1 in Cover and Thomas (2006)). Let p(s | 2) be given. Marimizing the muttual information is equivalent to minimizing the divergence between the average stogi- p(s) and the state-marginal of the furthest possible skill Z:
max 1 (p(s);z)= min max (xi(p(s 22) p(s)) p(2)
(2)
p(s)
Thus, the average state-marginal inducing policy minimizes the worst-case information cost of adapting the policy after pretraining to the downstream task T. Competence- based algorithms estimate this optimal prior policy by conditioning their policy on the average of the skill space that they have learned during pretraining.?
In practice, this achieved by conditioning the policy on vector that represents the skill. For example, if the skill space lies within unit hypereube, the vector in the center of the hypercube chosen.
4

---

Published in Transactions on Machine Learning Research (04/2023)
Data- and knowledge-basee based algorithms, however, do not try to estimate the optimal prior and thus perform worse in the URL setting (Laskin et al., 2022). Additionally, these algorithms explore the policy space of the domain on a random trajectory that might be close to the optimal prior at one point, but can deviate from it arbitrarily everywhere clsc. These disadvantages motivate our key idea to improve the samplo-cfficiency of data- and knowledge-base URL algorithms.
3.2 POLTER: Policy Trajectory Ensemble Regularization
In this section, we adapt the results from Section 3.1 for data- and knowlodge-basee URL algorithms and derive regularization POLTER. The mutual information in Lemma can be decomposed in the entropy of the state-marginal and conditional entropy term:
I (p(s):2) H (p(s)) - H (p(p(s) =))
Data-based algorithms maximize the first term in Equation (3), i.. they maximize an upper bound of the mutual information by maximizing the entropy of the state-marginal distribution. For knowledge-based algorithms this connection is not explicit. However, our experiments in Appendix C show that they also achieve similar state-margh margina entropy. Both categories of URL algorithms use mcchanism M (e.g. the prediction error of dynamics model) that encodes their objective is trying to maximize during pretraining. To apply Lemma 1 to these algorithms, we have to show that the conditional entropy vanishes.
intrinsic reward that the RL agent
Our assumptions can be expressed as the latent vari- able model in Figure where everything is observed except for the skill z which determines the intrinsie return Tint. At different points during the pretrain- ing steps 0 < t < NPT, observe different intrinsic returns for taking an action in a state Thus, the underlying skill that the agent is supposed to learn must have changed. This leads us to the following result:
2
8
S'
Pint
a
NPT
Proposition 1. Let a mechanism M be given that provides an intrinsic return. At a fized point during pretraining, this mechanism defines a reward func- tion re(s). Each reward function can be identified with lalent skill zi, i.e. there erists mapping between the set of reward functions and the set of skills. Let a policy T((s) with state-marginal pi(s) that macimizes this reward function be given. Then the conditional entropy of the optimal state-marginal and the skill (p(p(()) 2)) equal to O.
Figure 2: Probabilistic graphical model that underlies the assumption of POLTER for approximating the optimal policy using an ensemble of pretraining policies. The latent skill & determines the intrinsic return Fint via the pretraining objective, and the actions a of the agent's policy are implicitly conditioned on it.
Proof Skelch. To maximize the intrinsic return for a given skill zi, the optimal policy T((s) has to place all of the probability mass on the optimal itate-marginal i.e. H (p(p((s) 24)) E|| log p(p(s(s 24] -[[ log 1] = 0. If there policy that induces different conditional stato-marginal distribution with higher return, then the policy (s) is not optimal and there is contradiction.
Thus, under Proposition and given agent that learns the optimal policy for the current state of the intrinsic reward mechanism M, the conditional entropy in Equation (3) is minimized. Given these assumptions, the data- and bnosledge-based URL algorithms also maximize the mutual information between the stale-marginal and the latent skill and we can apply Lemma to them. This suggests that estimating the policy which induces the average state distribution is possible by selecting policies during pretraining that maximize different reward functions, i.c. learn different latent skills. Therefore, we can approximate the optimal prior policy by ensemble consisting of set of policies from different points of pretraining.
5

---

Published in Transactions on Machine Learning Research (04/2023)
Implementation The policy loss LURI. of the RL agent is augmented by our approach POLTER with the following regularization term:
CHOLTER (x) (T)
( (x) aDki T)
(4)
where a(s) = Ex (s) is the mixture ensemble policy consisting of previous policics from the URL trajectory. The mixture components ΦR = paramelrize calegorical distribution to construct the non- markovian average ensemble policy. At the start of each episode, the ensemble policy samples the index of an ensemble member with probability and follows this policy aftorwards. In the limit, this ensemble policy induces the average slate-marginal (see Appendix A.2 in Eysenbach el al. (2022)).
Algorithm URL+POLTER
Require: Initialize URL algorithm, policy *,,o, replay buffer D, pretraining steps NPT Require: Empty ensemble E = B, ensemble snapshot time steps TE, regularization weight 1: for ( O... NpT do if Beginning of episode then Obscrve initial state 8t pu(s))
URL init b POLTER init
b. Unsupervised pretraining
If E TE then
Update ensemble policy
Extend ensemble E + FU
Update ensemble policy
7:
Choose action + (0r [a))
Observe next state $14-1 p(8 oz)
Add transition to replay buffer D DU (u (s,,
10:
Sample minibatch B~ D
11:
Compute loss (POLTER = (U,L aDRL ( mo,e)
12:
Update policy Na, with EPOLTER
13:
> Supervised finetuning on task T
In Figure 1, illustrate the progression of the optimization trajectory in the induced state-marginal space p(s) of a policy T(s) under the URL objective with POLTER regularization. Along its trajectory, the policy (s)) is attracted to the ensemble policy # and, thus, closer to the optimal prior गे for a specific finetuning task T. This results in a reduced information cost to adapt the pretraining policy to the task and improves the sample-eficiency of the regularized URL algorithm.
We outline the combination of URL with our regularization POLTER in Algorithm 1. We fix the ensemble policy to finite set of states to stabilize training, i.e., each time add member to the ensemble (Line 5); condition the ensemble policy divergence (KI-divergence) loss and add it to the policy loss of the URL algorithm (Line 11). The memory overhead is dominated by the requirement to store the ensemble policies. For the computational overhead, we only have to consider the forward passes for conditioning the ensemble policy on a state when adding a new member. 3
the next state. Finally,
compute the
3.3 Demonstration on PointMass and Pendulum
PointMass To ground our theoretical motivation in empirical evidence, we evaluate the effect of our regularization on Random Network Distillation (RND) (Burda et al., 2019), a knowledge-based URL algorithm, and ProtoRL (ProtoRI.) (Yarats al., 2021), a data-based URL. algorithm, in the simplistic PointMass environment (Tassa al., 2018; 2020). In PointMass, we can define the optimal prior #*(s) that maximizes the average performance for the ream finetuning tasks. This optimal prior policy moves the ball into the center of the area, which minimizes the distance to any possible target location that represent finetuning task. With the optimal prior, we can compute the KL-divergence between the optimal prior policy and the current pretraining policy ( T). With POLTER, the KL-divergence to the optimal prior policy nे is generally lower throughout the whole pretraining phase.
II our experiments, each policy takes neglected amount of around 4MB of storage and the wall-clock time increases by 12%.

---

Published in Transactions on Machine Learning Research (04/2023)
35.0
RND RND+POLTER
22.5
ProtoAL ProtoRL
32.5
20.0
KL-divergence
30.0
17.5
27.5
25.0
15.0
KL
12.5
20.0
10.0
17.5
7.5
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.25
0.50
0.75
1.50
1.75
2,00
Pretraining Steps
1e6
Pretraining Steps
166
(a) RND
(b) ProtoRL
Figure 3: Average KL-divergence of RND/ PPottoo (dashed) and AND-PPOLTERRPPOOOR-PPPOO (solid) between the pretraining policy π (80) and the optimal pretraining policy m7)))) the Paint domain during reward-free pretraining. The shaded area indicates the standard error over seeds and the dots are the pretraining checkpoints that were used for the ensemble.
Pendulum As described in Figure 1, the effect of POLTER on the performance on a specific finetuning task in given domain depends on the optimal policy for the given task. Thus, even though POLTER improves the average performance
all possible tasks, it might have or even detrimental effect on single downstream tasks. To test this hypothesis, downstream tasks. The first one is the known SwingUp task, where the agent has to balance the pole in an upright position. The second task is Propeller task, which rewards the maximization of the angular velocity of the pole. This task's optimal policy induces a state-marginal that is further away from the average, which is reflected in Figure 4 in a slightly decreased performance when applying POLTER to RND. However, POLTER consistently increases the performance on the regular SwingUp task. Further results for Contrastive Intrinsic Control (CIC) (Laskin al., 2022) and ProtoRL (Yarats al., 2021) in this experiment can be found in Appendix B.
train POLTER on the Pendulum domain for two different
800
600
4000
500
400
3000
300
2000
200
1000
DDPG&1004 DDPG@7002
AND+POLTER
(a) SwingUp
(b) Propeller
Figure 4: Finetuning performance of RND and RND+POLTER in the Pendulum domain averaged over 10 seeds after 100k k of pretraining. The baseline of Deep Deterministic Policy Gradient (DDPG) is trained for 100 k or 200 k steps without pretraining. The error bars indicate the standard error over 10 seeds.

---

Published in Transactions on Machine Learning Research (04/2023)
IQM
Median
Mean
Optimality Gap
ProtoRL
Competence Knowledge Data
PratoRL+POLTER
APT
APT+POLTER
ICM+POLTER
Disagreement
Disagreement+POLTER
RND
PND+POLTER
CIC
CIC+POLTER
APS
PPSTRO
DIAYN
PILTNR
SMM
SMM+POLTER
DDPG
0.50
0.75
0.3
0.6
0.50
0.75
0.25
0.50
Figure 5: Aggregate statistics of applying POLTER to several URL algorithms after pretraining for 2M steps. POLTER improves the performance most algorithms substantially, for ProtoRL even by 40%. Each algorithm is tested on the URLB with its 12 tasks from domains using 10 independent seeds, resulting in 120 runs per algorithm. The error bars indicate the 95% bootstrap confidence intervals. DDPG is the bascline without pretraining.
4 Experiments
With our theorctical motivation and our prospect on the toy environments PointMass and Pendulum, we now address the following empirical questions at larger scale: (Q1) How does POLTER affect the performance after pretraining, and does the performance differ across URT algorithm categories? (Q2) Does POLTER (Q3) How does the performance vary with an increasing number of pretraining steps? (Q4) How does the strength of the regularization affect the performance of different algorithms in different domains? (Q5) How to select the checkpoints for the ensemble policy in POLTER? (Q6) Do the results also hold for pixel-based observations?
improve the sample-eficiency for finetuning?
To answer these questions, we evaluate several different URL algorithms with and without our POLTER regularization on the Unsupervised Reinforcement Learning Benchmark (Laskin et al., 2021). We use the provided code from Laskin et al. (2021) to aid reproducibility and provide our source code in the supplementary material. Knowledge-basee algorithms, such as RND (Burda al., 2019), Disagreement (Pathak al., 2019), and Intrinsic Curiosity Module (ICM) (Pathak et al., 2017), use the norm or variance of their prediction errors of some aspect of the environment Pre-training (APT) (Schwarzer et al., 2021b), and ProtoRL (Yarats et al., 2021), are data-based algorithms, that try to maximize the entropy of the state-visitation distribution to explore the environment. The last category is competence-base based algorithms that try to learn set of skills by maximizing the mutual information between a Icarned skill vector and some features of the obscrved states. Besides the three algorithms that were used by Laskin et al. (2021), Active Pre-training with Successor Features (APS) (Liu and Abbeel, 2021), )), State Marginal Matching (SMM) (Lee al., 2020), and "Diversity is All You Need" (DIAYN) (Eysenbach al., 2019), we evaluate our method with newer method from this category, called CIC (Laskin al., 2022).
learning signal. The second category consisting of Active
5 Results
Evaluation For evaluation, the agent performs pretraining reward-free setting for total of NeT 2M steps in one of the three domains using one of the described algorithms with without our POLTER
8

---

Published in Transactions on Machine Learning Research (04/2023)
regularization. The performance is then measured as the average return over 10 episodes after finetuning for 100 steps given external rewards that describe the downstream task. We follow the exact evaluation protocol of the URLB and use DDPG (Lillicrap et al., 2016) on state-based observations. For its hyperparameters and the setup of POLTER, see Appendix E. We follow Agarwal et al. (2021) in their evaluation using the IQM as our main metrie.* 4 The expert performance is provided by Laskin et al. (2021). Each algorithm is tested on 12 tasks from domains using 10 independent sceds, resulting in 120 runs per algorithm.
Q1: How does POLTER affect the performance of different algorithm categories?
We report the finetuning performance of each URL algorithm after 2M pretraining steps with and without POLTER Figure 5. With our method, we see an increase in performance for data- and knowledge- based algorithms by 19% TQM on average. For the data-based method ProtoRL, pretraining with POLTER increases the performance even by 0.22 IQM (40%). This result supports the argument in Section 3.2, as both data-based algorithms directly maximize the entropy of the state-margina distribution and with it the upper bound of the mutual information between the state marginal and the latent skill. The improvement due to POLTER also depends on the exploration capabilities of the URL algorithm. This can be seen for Disagreement POLTER, which performs worst of all snowledge-based algorithms and is due to the limited exploration of the state space, as has recently been shown by Yarats et. al. (2022). As noted by Laskin et al. (2021), the baseline performances notably differ among themselves, and most competence- based methods are worse than the supervised DDPG bascline without pretraining. The application of POLTER has no noticcable effect on their overall performance. This is due to the similar effect of their skill-averaging and the ensemble, which both pull the policy to the optimal prior. However, in Section 5, show that POLTER can be beneficial even for these algorithms. For more detailed results see Appendix F and Section 5.
Walker
Quadruped
Jaco
Normalizer Return (%)
POOTORL+POLTER
80
ProtoRL
60
APT
APT
40
20
25k
50k
75k
100k
0
25k 50k
75k
100k
25k
50k
75k
100k
Finetuning Steps
Finetuning Steps
Finetuning Steps
Figure 6: Data-based URL algorithms with (solid) and without POLTER (dashed) regularization after 2 M steps of pretraining. The shaded area indicates the standard error. POLTER speeds up finetuning by R 40% on Walker and Quadruped and Rs 10% on Jaco.
Q2: Does POLTER improve the sample-efficiency for finetuning?
As doscribed in Section 3, try to reduce the information cost of finetuning prior policy on specific task to improve the sample-efficinnyy Because the effect of our method is most pronounced in the data-based algorithm category, in Figure 6 we show the expert normalized return of finetuning APT and ProtoRL with and without POLTER after 2 steps of pretraining. The improvement in sample-efficinncy is clearly visible, especially in the locomotion domains. Here, POLTER achieves a speed-up of z 40% compared to the unregularized URT hms. It should be noted that the initial performance directly after pretraining is similar for all variants, and they all begin to improve after the same number of finetuning steps. This implics that exploration during finetuning is not the deciding factor for the performance differences.
4 AAthhughh used the provided code by Laskin al. (2021; 2022) for all baselines, our reproduced results slightly differ from their reported performance.
9

---

Published in Transactions on Machine Learning Research (04/2023)
Q3: How does the performance vary with an increasing number of pretraining steps?
their original publication, Laskin et al. (2021) noted that the performance of most URL algorithms decrcased with an increasing number of pretraining stops. To evaluate the effect of applying POLTER during pretraining,
observe the performance at different numbers of pretraining frames for the three domains of URLB. In Figure POLTER (solid) improves performance at all steps over the bascline (dashed) during pretraining, especially in the locomotion domains. Jaco is a special case because the arm is in fixed location, and the return is dominated by the ability to bring the arm into a specific position. Thus, the domain relies much less on deeper exploration compared to the locomotion environments, and the initial amount pretraining for 100 k steps is sufficient to improve on the DDPG baseline without pretraining.
present results for knowledge- and data-based methods. Applying
Walker
Quadruped
Normalzzdd Return
70
Competence
Competence-
60
Data
50-
DDPG
40
Knowledge
Knowledge-PROLTER
30
100000
500000
1000000
2000000
100000
500000
1000000 2000000
100000
500000 Pretraining Steps
1000000 2000000
Pretraining Steps
Pretraining Steps
Figure Normalized return per task and algorithm category with (solid) and without (dashed) POLTER regularization after finetuning from different pretraining snapshots. DDPG is the baseline without pretraining.
The results in Figure 7 also show an increase of the effect of POLTER on the downstream performance with an increasing number of protraining steps. This leads to the question of what specific effect POLTER has the URL algorithm and in particular Figure 8, computed the variance of the actor gradients ProtoRL with and without POLTER. The results support the findings of Laskin et al. that longer pretraining is often not beneficial for the downstream performance, the variance of the gradients and thus, the changes to the network weights decrease substantially for both baselines. With POLTER, however, the variance of the gradients does not decrease as much. is also apparent that the difference between POLTER and the bascline is much more pronounced for ProtoRL than for RND. This result is also in line with the detailed analysis in Figure 18.
the actor network where the regularisation term is applied. In
window of 5 gradient steps for RND and
Quadruped
Jaco
10"5
13-4
Goutentrarrnn unus
Gra
10
13-1
30+1
MND
5MD-+70.TER
10-1
0.00
0.25
250
0.75
1.0
1.25
150
1.75
2.00
0.00
0.25
0.30
0.75
1.00
1.25
150
1.75
250
0.00
0.25
6.50
0.75
1.00
3.25
1.75
200
anrgg Staps
Pectoringg Stens
Pecraring Stors
Figure 8: Variance of the actor gradients for ProtoRL and RND with and without POLTER regularization on the URLB averaged over 10 seeds with sliding window size of 5. The shaded area indicates the standard error of the mean.
10

---

Published in Transactions on Machine Learning Research (04/2023)
Q4: How does the strength of the regularization affect the performance?
A hyperparameter of our method is the strength of our regularization, see Equation (4). We set this to 1.0 in our preceding evaluations as a natural choice. To study the sensitivity of this hyperparameter, we evaluated POLTER with different scttings on a set of representative URL algorithms. In Figure 9, we present.
ProtORL+POLTER
RND+POLTER
CIC+POLTER
Nomalized Return
60
40
Walker
Quadruped
Jaco
Walker
Quadruped
Jaco
Walker
Quadruped
Jaco
Figure 9: Average normalized return of ProtoRL+ RND+POLTER and CIC+POLTER after 2M steps of pretraining and 100 finetuning for different. values of the regularization strength a.
the performance of the POLTER regularized variants of ProtoRT, RND and CTC different values of a. The results show robust performance of POLTER with its default 1.0, i.e. POLTER performs better or is on par compared to not using POLTER (a = 0.0). On average, = 1.0 seems to be the best choice. Nevertheless, the performance of our method can be further improved if is tuned depending on the task and algorithm. the Jaco domain, a higher regularization helps RND but might be detrimental for ProtoRL. We hypothesize that this is due to increasingly extreme pretraining policies RND that are dampened via POLTER, which is helpful for the tasks in this domain. For CIC, the regularization of POLTER has a minor effect even degrades the performance in the locomotion domains. As Laskin et al. (2022) noted, locomotion requires high-entropy pretraining policy that explores the environment well. POLTER is indifferent to the skill vector used by the CIC policy in each episode. So the ensemble contains different skill vectors that are likely to suggest different actions, which partly cancel each other and reduce the exploration.
IQM
Median
Mean
Optimality Gap
ProtoRL ProtoRL+POLTERS
RND
POL+TR
CIC
CIC+POLTER*
0.6
0.7
0.8
0.60
0.75
0.90
0.6
0.7
0.8
0.2
0.3
0.4
0.5
Figure 10: Aggregate statistics of three representative algorithms from each URL category after 2 steps of pretraining and 100 finetuning with tuned regularization strength in Equation (4).
However, a stronger regularization up certain value significantly improves the performance on Jaco, it does not require much exploration to perform well. This demonstrates the trade-off between good prior for finetuning and attenuated exploration if the regularization is too strong, It also shows that POLTER should be tuned to the domain.
If we apply the insights from Figure 9, we can improve the performance of the three evaluated algorithms even further by tuning a to the locomotion and manipulation domains.3 For the results in Figure 10,
TTuning here means selecting the best performing a from the previously evaluated grid sweep.
11

---

Published in Transactions on Machine Learning Research (04/2023)
set for ProtoRL to 1.0/2.0 [Iocomotoon/maniipaaattoo for RND to 2.0/8.0 and for CIC to 0.0/4.0. RND-POLTTR** and CIC+POLTER* Note that our regularization complements the discriminator loss of CIC, which is set to 0.9/0.0 on the two domain categories. These optimized settings lead to a new state-of-the-an in the URLB of 0.84 IQM. We note that previous stale-of-the-arl methods such as CIC are also luned based on the domain of the task and that this experiment. allows for a fair comparison.
We call the tuned variants
Q5: How to select the checkpoints for the ensemble policy in POLTER?
With POLTER, we introduce the hyperparameter TE that determines which checkpoints should be added to the ensemble. As described in Appendix E, we use the same setting for all our experiments by aligning this hyperparameter with the sign-changes of the second derivative the intrinsic return during pretraining of RND in the Quadruped domain. Following our derivation in Section 3.2, this indicates a change in the latent skill because the actions that the agent takes are now being rated differently by the intrinsic reward mechanism.
IQM
Median
Mean
Optimality Gap
ProtoRL ProToRL+POLTER (200k)-
ProtORL+POLTER (100k)
ProtRLLPOLTEER (50k)
ProtoRL+POLTER
0.72 0.80
0.64
0.32
0.48
Figure 11: Aggregate statistics on the URLB of different linear and logarithmic checkpoint schedules when applying POLTER to ProtoRL.
To analyze the sensitivity of our regularization to this hyperparameter, Figure 11 we evaluated ProtoRL on the URLB with three different ensemble schedules. Each schedule adds checkpoint every {50k, 100 200 k} steps. The results in Figure 11 indicate that POLTER consistently improves the performance of ProtoRL for several settings of this hyperparameter. However, using logarithmic schedule that aligns with the intrinsic reward dynamics to select the checkpoints further boosts the performance impact of the regularization. Thus, a better understanding of the latent skill space of the domain and algorithm would enable more sophisticated schedules and increase the performance even further.
Q6: Do the results also hold for pixel- based observations?
IQM
Median
Mean
Optimality Gap
ProtoRL ProtoRL+ POLTER RND-
RND+POLTER
CIC+POLTER DDPG
0.15
0.30
0.45
0.15
0.30
0.45
0.15
0.30
0.45
0.60
0.75
0.90
Figure 12: Aggregate statistics of applying POLTER to several URL algorithms after pretraining for 2M steps. Fach algorithm is tested on the URLB with its 12 tasks from 3 domains using 10 independent seeds with pixcl-based observations, resulting in 120 runs per algorithm. The error bars indicate the 95% bootstrap confidence intervals. DDPG is the baseline without pretraining.
Laskin et al. (2021) showed that the performance pixel- and state-based URL algorithms differs substantially To demonstrate the generality our approach, we evaluate the effect of POLTER on ProtoRL, RND and CIC on the URLB with pixel-based observations. The results Figure 12 show that the improvements in TQM are approximately the same for RND and ProtoRI. as in state-based environments. For CIC also
12

---

Published in Transactions on Machine Learning Research (04/2023)
see the same effect of a slightly reduced performance. Note, however, that CIC does not work well for the tasks in the Jaco domain."
6 Limitations and Future Work
In this work, we focused our experiments on DDPG to compare our performance on the URLB. This, however, limits our scope to continuous control problems. Therefore, extending URL to environments with discrete action spaces a possible next step. Moreover, a deeper theoretical understanding of the pretraining processes and the connection to the optimal prior policy would enable more sophisticated mixture policies and a better approximation of the optimal prior in the future. As noted in Section 3, the space of policies well explored by the URL algorithm, using the average policy is optional under Proposition 1. However, future research direction is the analysis of the skills that the agent learns during pretraining and incorporate this knowledge into the mixture prior.' 7 Finally, POLTER. is not adapted to the category of the URL algorithm it is applied to. So its performance with competence- based URL algorithms leaves room for improvement by taking the observed explicit skills into account.
7 Conclusion
In this work, we introduced POLTER (Policy Trajectory Ensemble Regularization) a general method to improve the performance of URL algorithms. URL is method to increase the samplo-cfficiency of RL algorithms on of lasks by pretraining the policy with an intrinsie reward in the task's domain. Our regularization pulls the pretraining policy closer to an ensemble of pretraining policies seen during pretraining that correspond to a set of explicit or implicit skills. We demonstrated the effect of our method in the PointMass and Pendulum domain and extensively evaluated baseline algorithms regularized with POLTER on the URLB and established its effectiveness our main experiments. For data- and knowledge-basee URL methods, we improved performance on average by 19% and up to 40% (IQM). We argue that our method is a suitable regularization for those algorithms,, in contrast to competence- methods where the effect of POLTER is highly algoritam-deppedent. Finally, we showed that our regularization strength can be tuned to the domain and algorithm at hand for further performance improvements, achieving new state-of-the- art performance with CIC+POLTER* In future work, this manual tuning could be automated utilizing AutoRL (Parker-Holes Holder et al., 2022). With POLTER's easy implementation and negligible computational requirements, we hope it finds its way into more URL algorithms and spurs further research on how we can learn general priors for arbitrary tasks.
Acknowledgements
Carolin Benjamins and Marius Lindauer ackuowledge funding by the German Research Foundation (DFG) under LI 2801/4-1. Additionally, this work was supported by the Federal Ministry of Education and Research (BMBF), Germany under the project LeibnizKlLabbr (grant no. 01DD20003) and the AT service center KISSKT (grant no. 01T822093C), the Center for Digital Innovations (ZDIN) and the Deutsche Forschunggemminnchafff (DFG) under Germany's Excellence Strategy within the Cluster of Excellence PhoenixD (EXC 2122).
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Mare' Aurclio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jemnifer Wortman Vaughan, editors, Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, Virtual, pages 29304-29320, 2021.
6This prompted Laskin et al. in the original publication set the weight of their discriminator loss 0 in the Jaco domain. We explored exponentially decaying mixture policies with ProtoRL but found the IQM improvement be slightly worse. For decay factor 0.9, ProtoRI. achieved TQM of 0.72 compared to the 0.77 of the uniform mixture.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Kate Baumli, David Marde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. In Thirly-Fifth A AAAT Conference on Artificial Intelligence, AAAT 2021, Thirly- Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, the Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 6732-6740. AAAI
Press, 2021.
Mare G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77 82, December 2020. ISSN 0028-0836, 1476-4687. doi:
(0.1038/441586-00--2222
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview. net, 2019.
Chao Chen, Zijian Gao, Kele Xu, Sen Yang, Yiying Li, Bo Ding, Dawei Feng, and Huaimin Wang. Nuclear Norm Maximization Based Curiosity-Driven Learning, May 2022a.
Chao Chen, Tianjiao Wan, Peichang Shi, Bo Ding, Zijian Gao, and Dawei Feng. Uncertainty Estimation based Intrinsic Reward For Efficient Reinforcement Learning. In 2022 IEEE International Conference on Joint Cloud Computing (JCC), pages 8, Fremont, CA, USA, August 2022b. IEEE. ISBN 978-1-66544--8853. doi: D0.1109JJCCan
2022.00008.
Daesol Cho, Jigang Kim, and H. Jin Kim. Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery. IEEE Robotics and Automation Letters, 7(3):7455-7755) July 2022. ISSN 2377-3766,
2377-3774. doi: 10.1109/110. (01109/LRA.200 21A.......221111999991
Thomas M Cover and Joy A Thomas. Elements of Information Theory. Viley-Interscienee July 2006.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackic Kay, Antoine Merle, Jcan-Mare Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602((897):414 419, February 2022. ISSN 0028-0836, 1476-4687. doi: 10.1038/ 10.1038/s41586
Alexander Dockhorn and Daan Apeldoorn. Forward model approximation for general video game learning, In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8, 2018. doi: 10.1109/ RIG.2011.8499///
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv:1812.00566 [es], December 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You Need: Learning Skills without Reward Function. In 7th International Conference on Learning Regresentalions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The Information Geometry of Unsupervised Reinforcement Learning. In International Conference Learning Representations, 2022.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational Intrinsic Control. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview .ect, 2017.
Zhaohan Danicl Guo, Mohammad Gheshlaghi Azar, Alaa Saade, Shantanu Thakoor, Bilal Piot, Bernardo Avila Pires, Michal Valko, Thomas Mesnard, Tor Lattimore, and Rémi Munos. Geometric Entropic Exploration, January 2021.
14

---

Published in Transactions on Machine Learning Research (04/2023)
David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnelt, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2455-2467, 2018.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML, 2022, 17-23 July 2022, Ballimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 8387 8406. PMLR, 2022.
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably Efficient Maximum Entropy Exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th Inter- national Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Leurning Reseorch, pages 2681 2691. PMLR, 2019.
Shuncheng He, Yuhang Jiang, Hongchang Zhang, Jianzhun Shao, and Xiangyang Ji. Wasserstein Unsupervised Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, (884-6688, June 2022. ISSN 2374-3468. doi: 20.1609/anai.v333 w3t...66666666556
Alexis Jacq, Manu Orsini, Gabriel Duloc-Arnold, Olivier Pietquin, Matthieu Geist, and Olivier Bachem. C3PO: Learning to Achieve Arbitrary Goals via Massively Entropic Pretraining, November 2022.
Zheyuan Jiang, Jingyue Gao, and Jianyu Chen. Unsupervised Skill Discovery via Recurrent Skill Training. In Advances in Neural Information Processing Systems, October 2022.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-Free Exploration for Rein- forcement. Learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4870-4879 PMLR, 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive Unsupervised Representations for Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, 1CML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 6639-5650 PMLR, 2020.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised Reinforcement Learning Benchmark. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, Virtual, 2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CTC: Contrastive Intrinsic Control for Unsupervised Skill Discovery. arXiv:2202.001 00161 fes), February 2022.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Fric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient Exploration via State Marginal Matching. arXiv:1906.0522 [es, statf, February 2020.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. In Yoshua. Bengio and Yann LeCun, editors, 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerlo Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Hao Liu and Pieter Abbeel. APS: Active Pretraining with Successor Features. In Marina. Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1736-6747, 2021.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young- Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, Jiwoo Pak, Andy Tong, Kavya Srinivasa, William Hang, Emre Tuncer, Quoe V. Lc, James Laudon, Richard Ho, Roger Carpenter, and Jeff Dean. A graph placement methodology for fast chip design. Nature, 594(7862):207-2111 June 2021. ISSN 1476-4687. doi: 10.
15

---

Published in Transactions on Machine Learning Research (04/2023)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-lovel control through deep reinforcement learning. Nature, 518(7540)) 533, February 2015. ISSN 0028-0836, 1476-4687. doi: 10.1038/ maturel44236
Shakir Mohamed and Danilo Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2125-2133, 2015.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimate. Proceedings of the AAAT Conference on Artificial Intelligence, 35 (10):9028 9036, May 2021. ISSN 2374-3468, 2159-5399.
doi:
Mirco Mutti, Mattia Mancassola, and Marcello Restelli. Unsupervised reinforcement learning in multiple environments. In Thirly-Sicth AAAT Conference on Arlificial Intelligence, AAAT 2022, Thirty-Fourty Conference Innovative Applications of Artificial Intelligence, IAAI 2022, the Twelveth Symposium Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 7850-7858. AAAI Press, 2022.
Alexander Nedergaard and Matthew Cook. K-Means Maximum Entropy Exploration, June 2022.
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glemn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving Rubik's Cube with a Robot Hand. arXiv:1910007113 [es, statf, October 2019.
Pedro Ortega and Daniel Lee. An Adversarial Interpretation of cittmmmttion-TThoreeee Bounded Rationality. Proceedings of the AAAI Conference Arlificial Intelligence, 28(1), June 2014. ISSN 2374-3468, 2159-5399. doi: 10.1609 aaai aaaai.228i... v2811.9071.
Pierre- Yves Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic Motivation Systems for Autonomous Mental Development. IEEE Transactions on Evolutionary Computation, 11(2):265-28) 286, April 2007. ISSN 1089-778X. doi:
Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Bicdenkapp, Yingjic Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, and Marius Lindauer. Automated reinforcement learning (AutoRL): A survey and open problems. Journal of Artificial Intelligence Research, 74:517 568, 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven Exploration by Self- supervised Prediction. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML, 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 7778-2787 PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-Supervised Exploration via Disagroement. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5062 5071. PMLR, 2019.
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Unsupervised model-based pre-training for data-cfficient reinforcement learning from pixels. In Decision Awareness in Reinforeement Learning Workshop al ICML 2022, 2022.
JUNGSUB RHIM, Eunscok Yang. and Tacsup Kim. Efficient task adaptation by mixing discovered skills. In First. Workshop on Pre-Traine Training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022.
16

---

Published in Transactions on Machine Learning Research (04/2023)
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-Fificiiet Reinforcement Learning with Self-Predcctive Representations ions. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nn .nt,,
net, 2021a.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining Representations for Data-Eiffiienn Rein- forcement. Learning. In Marc' Aurelio Ranzato, Alina, Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, Virlual, pages 11686-12699, 2021b.
Younggyo Seo, Kimin Lee, Stephen L. James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from vidcos. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages PMLR, 2022.
Nur Muhammad Shafiullah and Lerrel Pinto. One After Another: Learning Incremental Skills for a Changing World, March 2022.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsuper- vised discovery of skills. In 8th International Conference Learning Representations, 1CLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nee net, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering Chess and Shogi by Self-Play with General Reinforcement Learning Algorithm. arXiw:1712.01811 [es), December 2017.
Aravind Srinivas and Pieter Abbeel. Unsupervised Learning for Reinforcement Learning, 2021.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling Representation Learning from Reinforcement Learning, May 2021.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learnigg: An Introduction. IEEE Trans. Neural Nelworks, 9(5):1054-1055 1998. doi:
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Frez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite. ar 0rXiv:1801.0060 [cs], Jamuary 2018.
Yuval Tassa, Saran Tunaauuvuakool, Alistair Muldal, Yotam Doron, Piotr Trochim, Siqi Liu, Steven Bohez, Josh Merel, Tom Frez, Timothy Lillicrap, and Nicolas Heess. Dm control: Software and Tasks for Continuous Control. Software Impacts, 6:100022, November 2020. ISSN 26659638. doi: 10.1016/j.simpa. 2020.100022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, January 2019.
Yingchen Xu, Jack Parker-Holder Aldo Pacchiano, Philip J. Ball, Olch Rybkin, Stephen J. Roberts, Tim Rocktäschel, and Edward Grefenstette. Learning General World Models in a Handful of Reward-Free Deployments, October 2022.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In Marina Mcila and Tong Zhang, oditors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine PMLR, 2021.
Learning Research, pages
17

---

Published in Transactions on Machine Learning Research (04/2023)
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Picter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Jinyi Liu, Yingleng Chen, and Changjie Fan. EUCLID: Towards efficient unsupervisod reinforcement learning with multi-choice dynamics model. CoRR, abs/2210.00499, 2022. doi: 10.48550/arXiv. wtt..iv.222100099999
Kailin Zeng, QiYuan Zhang, Bin Chen, Bin Liang, and Jun Yang. APD: Learning Diverse Behaviors for Reinforcement Learning Through Unsupervised Active Pre- Pre-Tiainnn.. IEEE Robolics and Aulomalion Letters, 7(4):12251-12258 October 2022. ISSN 2377-3766, 2377-3774. doi:
Chuheng Zhang, Yuanying Cai, Longbo Huang, and Jian Li. Exploration by Maximizing R) 'enyi Entropy for Reward-Free RL Framework, December 2020.
Andrew Zhao, Matthieu Gaetan Lin, Yangguang Li, Yong-Jin Liu, and Gao Huang, A mixture of surprises for unsupervised reinforcement learning. In Advances in Newral Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, December 2022.
18

---

Published in Transactions on Machine Learning Research (04/2023)
A Extended Demonstration on PointMass
35.0
DDPG
32.5
800
RND+POLTER
RND
(1-divergence
30.0
RND+POLTER*
600
27.5
25.0
400
22.5
200
0.00
0.50
0.75
1.25
1.50
1.75
2.00
20000
60000
80000
Pretraining Steps
1e6
Finetuning Steps
(a) Pretraining (2 M steps)
(b) Finetuning (100 k steps)
RND RND+POLTER
26
800
AND
RND+POOLTER RND+PPOTTRR
KL divergeence
24
600
22
20
200
18-
0-
20000
40000
60000
80000
100000
25000
50000
100000 Pretraining Steps
150000
200000
Finetuning Steps
(d) Finetuning (100 k steps)
(c) Pretraining (200 k steps)
Figure 13: (a), (c) Average KL-divergence of RND (dashed) and RND+POLTER (solid) between the pretraining policy (550) and the optimal pretraining policy (s0) in the PointMass domain during reward- free pretraining. (a) is trained for 2M steps, and (c) is trained for 200 k. Each policy is evaluated every 100 k steps 20 initial states over 10 seeds. The black dots indicate the steps snapshot is added to the ensemble. (b), (d) Return during finetuning after M and 200 pretraining steps, where the target is placed at fixed random position for cach of the 10 seeds. We also provide a DDPG bascline without pretraining and RND+POLTER* using the optimal policy instead of the ensemble. The shaded area indicates the standard error.
We evaluate the effect of our regularization RND (Burda et al., 2019), well-known knowledge-based URL algorithm, in the simplistic PointMass environment (Tassa et al., 2018; 2020). In this environment, the agent has to apply a force to a mass (green in Figure 13b) a 2D plane to reach a target (red) and observes its position and speed. We pretrain DDPG agent using RND with and without POLTER for 2 steps and evaluate the policies every 100k steps, accordance with the URL benchmark evaluation protocol. Because PointMass much simpler environment than the contained in URLB and to gain a better resolution for the start of the pretraining, repeat the experiment with 200 k pretraining and 100k finetuning steps. We use 10 seeds.
In our whitebox benchmark PointMass, we can train the optimal prior policy and compute the KI-divergence between the optimal prior policy and the current pretraining policy DKL :T). Comparing the KL-divergence during pretraining with and without POLTER Figure 13a shows that with our method, the KL-divergence the optimal prior policy पम is generally lower, which is also apparent for the short pretraining (Figure 13c). POLTER keeps the KT-ddvergence bay, whereas RND without POLTER diverges
19

---

Published in Transactions on Machine Learning Research (04/2023)
much more, which is consistent with prior work showing that URL tends to diverge with too many pretraining steps.
A.1 Effect of POLTER on Finetuning
The effect of this improved prior is also apparent during finetuning in Figure 13b. Here, we see that using an ensemble is good proxy for the optimal prior due to the similar sample-effciincy. In addition, we observe a speed-up of approximately 25 % (2M pretraining steps) and 12.5 % (200 pretraining steps) when using RND+POLTER in reaching the final performance of DDPG.
A.2 Visualization of State Distributions
In Figure 14a, we can see the stale distribution changes during pretraining with and without POLTER. With POLTER, the state space coverage is less, and the trajectories seem more ordered. RND without POLTER also scems to visit the edges often at the end. When using the POLTER regularization, pretraining checkpoint is visiting dilferent states, as indicated by the visibility of the previous checkpoint's 's state visitations. When not using POLTER, we can see that the visitations overlap. Figures 14b and 14c show the discretized position and speed the agent explores throughout pretraining, Especially the discretized speed (Figure 14c) demonstrates the tradeoff between dampening the exploration with POLTER and finding better prior policies because with POLTER, fewer states are frequented, and the states are less extreme.
see that cach
B Extended Experiments on Pendulum
In this experiment, shed further light the properties of POLTER and its effect under different finetuning tasks. For example, in the PendulummPrpelllrr task, the goal is to maximize the angular velocity of the pole. However, this requires rather extreme policy, whereas, in the PenduhumSwingT) task, the agent has to balance the pole and keep it upright.
The SwingUp task demonstrates that POLTER can consistently improve the performance of the baseline URL agent. However, for the Propeller task, the results are mixed. For ProtoRL, POLTER again shows a great improvement. over the baseline. This supports the performance that observed in the main experiments. But for CIC and RND, POLTER decreases performance. The extreme policy of achieving a high angular velocity is further away from the average, applying POLTER has a detrimental effect. Nevertheless, note that regardless of this failure case, on average, the tasks will lie closer to the average state distribution, and thus, POLTER will increase the performance.
C State Viste-Visitation Entropy
To gain further insights into POLTER, we conduct an experiment following Hazan al. (2019). We discretize the state space of the Walker environment and compute the stato-visitation entropy during reward-free pretraining. In Table 1, we that the entropy of the distribution of POLTER regularized algorithms is lower than that of their counterpart. This effect is most pronounced in data-based algorithms, such as ProtoRL. and APT, where the performance is also improved the most (see Table 3).
Table 1: State-visitation entropy of the evaluated URL algorithm categorics in the Walker domain during pretraining. Averaged over 10 seeds with 50 k states each al pretraining sleps 100 k, 500 k, M and 2 M.
POLTER
Data
Knowledge
Competence
x
0.2772 + 0.0188
0.2863
0.0036
0.2540 + 0.0429
0.2545 + 0.0493
0.2848 0.0054 0.2511 + 0.0422
Knowiedge-based algorithms also benefit from the regularization but have a slightly roduced entropy. Because competence-
based based algorithms already average over of skills found during pretraining, the effect of POLTER is the smallest.
20

---

Published in Transactions on Machine Learning Research (04/2023)
State distribution p(s) of several checkpoints.
DSD EENEET
(b) Discretized state histogram of the positions. Looking closely, see that the edges of the 2D plane are very often visited. This is due to policies constantly applying the force and thus reaching the edge of the 2D plane.
(c) Discretized state histogram of the speeds. Note that the prominent horizontal and vertical lines result from moving at the edges and being stuck the corners of the 2D plane.
Figure 14: State distribution and histogram of RND (always left column) and RND+POLTER (right column) during pretraining on the MaintMass environment.
These results imply that POLTER does not lead better state-space exploration. Instead, its performance gains the result of an improved prior as indicated by the reduced KT-ddvergence between the policy and
21

---

Published in Transactions on Machine Learning Research (04/2023)
30
Figure 15: Average return over 10 seeds on two tasks in the Pendulum domain with different characteristics. Algorithms from all three URL categorics compared to DDPG baseline that is trained for cither 100 k 200 k steps without pretraining.
the optimal pretraining policy on PointMass (Section 3.3). This experiment showed that a good exploration of the state-space required but sufficient to achiove good performance with URL algorithms.
D Environments in the Unsupervised Reinforcement Learning Benchmark
The Unsupervisod Reinforcement Learning Benchmark Laskin et al. (2021) contains three domains with the topics of locomotion and manipulation. Walker, Quadruped and Jaco, are used to explore the effects of different URL algorithms. It has specific training and evaluation protocol which we also follow in this work. The Walker domain contains planar walker constrained to 2D vertical plane, with an 18-dimensional observation space and 6-dimensional action space with three actuators for each leg. The associated tasks are stand, walk, run and flip. The walker domain provides a challenging start for the agent since it needs to learn balancing and locomotion skills to be able to adapt to the given tasks. The next domain is Quadruped, which expands to the 3D space. has a much larger stale space 56 dimensions and a 12-dimensional action space with three actuators for each leg. The tasks in this environment are stand, walk, jump and run. The last environment used is the Jaco Arm, which is robotic arm with 6-DOF and three-finger gripper. This domain is very different from the other two, its setting is manipulation and not locomotion. The tasks are Reach top left, Reach top right, Reach bottom left and Reach bottom right.
E Hyperparameters and Resources
POLTER Hyperparameters During pretraining, we construct the mixture ensemble policy म with k 7 members at specific time steps Tg- For adding each member we choose the ensemble snapshot time steps Tg = k, k, 50k. k. 100 k, 200k, 400 k, 800 k, 1.6 The steps were chosen according to initial experiments of applying RND in the Quadruped domain, where there are large changes the intrinsic reward at the beginning, which become progressively smaller over time. We the regularization strength a and use the same hyperparamoters for each of the three domains unless specified otherwise.
Baseline Hyperparameters The hyperparameters for our bascline algorithms follow Laskin et al. (2021) and Laskin et al. (2022). The hyperparameters for the DDPG baseline agent are described in Table 2.
Compute Resources All experiments were rum on our internal compute cluster on NVIDIA RTX 1080 Ti and NVIDIA RTX 2080 Ti GPUs and had 64GB of RAM and 10 CPU cores. total, we trained over 12000 models and performed 3 500 200 000 environment. steps.
22

---

Published in Transactions on Machine Learning Research (04/2023)
Table Hyperparameters for the DDPG algorithm.
Hyperparameter
Value
Replay buffer capacity
106
Action repeat
1
Seed frames
4000
n-step returns
3
Batch size
1024
Discount factor 7
0.99
Optimizer
Adam
Learning rate
10- 4
Agent update frequency
2
Critic target EMA rate
0.01
Feature size
1024
Hidden size
1024
Exploration noise std clip
0.3
Exploration noise value
0.2
Pretraining frames
106
Finetuning frames
x 105
F Detailed Results on Unsupervised Reinforcement Learning Benchmark
This section provides additional results for our experiments on Unsupervised Reinforcement Learning Benchmark. In the supplementary, we provide the raw scores. The statistics comparing URL algorithms with and without POLTER aggregatod for finctuning on 12 tasks across 10 sceds can be found in Table In addition show aggregate statisties of the absolute improvement in expert performance in Figure 16 and the performance profiles (Agarwal et al., 2021) per URL algorithm category in Figure 17. As before, large improvement for data- and knowledge- based algorithms and a small negative for competence-basee algorithms. The improvement sometimes varies strongly across seeds and tasks. Also, we show the normalized return after finetuning from different pretraining snapshots for each domain and URL category in Figure 18. In the Jaco domain, URL algorithms with and without POLTER mostly deteriorate with an increasing number of pretraining steps. Each category shows a different trend in each domain. Interestingly, the competence-bant based algorithms SMM and DIAYN fail during pretraining in the Jaco domain. In Figure 19 we see the normalized return over finetuning steps. POLTER is mostly par speeds up compared to the CRL algorithm without POLTER. In total, most algorithms do not converge yet after 100k k steps.
IQM
Median
Mean
ProtoRL
APT
ICM
Disagreement
RND
CIC
APS
DIAYN
SMM
0.0
0.2
0.0
0.2
0.0
0.2
Figure 16: Aggregate statistics of the absolute improvement with POLTER per URL category.
23

---

Published in Transactions on Machine Learning Research (04/2023)
Table Raw aggregate statistics following Agarwal et al. (2021) of evaluated URL algorithms with and without. POLTER regularization. The results marked with POLTER* were obtained by tuning the regularization strength to the task domain of locomotion (Walker and Quadruped) and manipulation (Jaco).
IQM
Mcan 1
Median
Optimality Gap
POLTER IQM
Improvement
Algorithm
ProtoRL
0.56
0.55
0.52
0.45
ProTORL+POLTER
0.77
0.71
0.65
0.29
+40%
PrOTERL+POLTER+
0.79
0.76
0.80
0.24
+41%
APT
0.59
0.61
0.56
0.39
APT+POLTER
0.69
0.68
0.66
0.32
+17%
RND
0.70
0.71
0.67
0.30
RND POLTER
0.77
0.75
0.74
0.26
+10%
RND PPLTTRT
0.77
0.76
0.71
0.24
+10%
ICM
0.54
0.52
0.59
0)48
ICM+POLTER
0.63
0.60
0.65
0.40
+17%
Disagreement
0.68
0.69
0.66
0.31
Disagremmen-POLTERR
0.69
0.70
0.69
0.31
+1%
CIC
0.78
0.76
0.74
0.24
CIC+POLTER
0.76
0.74
0.77
0.26
-2%
CIC+POLTER*
0.84
0.81
0.86
0.20
+7%
DIAYN
0.36
0.39
0.42
0.61
DIAYN+POLTER
0.39
0.42
0.42
0.58
+8%
SMM
0.36
0.42
0.30
0.58
SMM+POLTER
0.36
0.41
0.30
0.59
+ 0%
APS
0.56
0.58
0.55
0.42
APS+POLTER
0.52
0.53
0.54
0.47
-7%
DDPG
0.55
0.54
0.56
0.46
24

---

Published in Transactions on Machine Learning Research (04/2023)
1.00
Frcction cfrun with zore
0.75
score
0.50
0.25
0.25
0.00
0.00
0.0
02
0.4
0.6
0.8
1.0
0.0
0.A
0.8
Expert Normal and Score
Expert Normalzzd Score
0.50
0.00
0.6
Expert Normalized Score
Figure 17: Performance profiles after finetuning of the different algorithms averaged over 10 seeds where the shaded region indicates the standard error. Variants without POLTER are dashed.
Whiter
Quedruped
Nomma lond Return
30-
Pronani
HOOOLL PPOU
500000
2005000
350000 Pretraiting
100220
1000003 2000033
Pretmaniing
Pretraining Shepe
Parkr
Quadruped
JICo
Nomaltae fatum (50
Disggammant
KOR POUTON
10000c sc06ce 2000033 2030003
100000 500000 1000000 2000000
100000
500000
1100000 2000000
Pa Stape
Pratraining Stess
Pratrening Staaa
Quadruped
Wommskzed Rotumm Rotum (5)
OC
OC-POOTE
APS
FOITER
100020
505200 Protraining Stops
1000200 2030005
100020
1000200 3022000 Pretraining Stops
5302c0
100023
500200 1002200 2030000
Pretraining Stops
Figure 18: Finetuning from different pretraining snapshots of data-, knowledge- and competenco-based algorithms. The error bars indicate the standard error of the mean.
25

---

Published in Transactions on Machine Learning Research (04/2023)
Walker
Quedruped
Jaco
FOLTER
Protont.
+POTTE
02-
Normal Rectan
20-
20300
35000
10003
00000
100005
20030
53000
00000
00030
100032
20035
a35c0
60000
00035
100035
Sretuning
Fintuning Staps
Roetuning
Walker
Quadruped
Jaco
Dewapreeeent
Deougnes POLTER
RND
+N
22-
Nemalaze
Tietuning
finetuning
finetnnin
Walker
Quadruped
Jaco
CRC
CI POLTER
70-
ATS
POLTER
20-
Nemalaze
60-
10-
Tinttning
finetniing
finetninn
Figure 19: Finctuning curves of data-, knowledge- and competenco-based algorithms after pretraining for 2 M steps. The shaded area indicates the standard error.
26


Reviews:
[{"review_id": "E4wWKMIcYt", "reviewer": "Reviewer_hyqK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I quite like this paper, the proposed technique is well-grounded theoretically and is shown to have good empirical results. I especially like the simplicity of the idea and can see this as a new standard addition to future URL algorithms, my comments are as follows:\n- The proposed method is intuitive and has theoretical support from previous work. The idea that each point during pre-training encodes an implicit skill, averaging these results can result in a proxy to the average state marginal of skills is a very natural and direct application of the insights from Eysenbach et al. (2022).\n- The proposed approach is straightforward to implement and widely applicable to a whole range of URL algorithms.\n- I believe the main strength of this paper is its experiments. The set of experiments is quite thorough and provides an abundance of empirical evidence of both the effectiveness and wide-adaptability of the technique. The authors also did a good job of analyzing their approach for different categories and sensitivity to hyperparameters. I also really like the discussion on state-visitation entropy in Appendix C which provides empirical evidence on the relationship between space space exploration on POLTER.\n- While the paper is mostly clear, I do think some additional clarifications could make the paper more self-contained. For example, some concrete examples/descriptions on data/knowledge/skill-based algorithms and how POLTER works in each case would help visualize the author's ideas better.\n- I recommend defining $\\pi(s)$ in section 3.2 and adding some clarification on what is meant by an \"optimal oracle policy\", this relates back to the point of making the paper more self-contained.\n- I think it is fair to present the results for an $\\alpha$ tuned for each domain and task since the authors did explicitly note this and that the authors did also present results of the untuned variant. However, I think the way the term \"tuned variants\" is used on page 10 might be a little misleading as it seems imply some kind of auto-tuning approach to finding $\\alpha$.\n- Not asking this as a requested change but related to the previous point, do you think there is any way to adaptively tune $\\alpha$? \n- Could you elaborate on some of the challenges and difficulties of extending POLTER to pixel-based environments? On a high level, what modifications do you envision you have to make?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a regularization technique for Unsupervised Reinforcement Learning (URL) which involves minimizing the KL divergence to an ensemble of policies discovered during the pre-training process. This approach can be easily applied to large set of URL algorithms and is shown to improve the performance of a set of data and knowledge-based URL algorithms on the URL benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See previous section", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fIvesuK3PA", "reviewer": "Reviewer_tT3Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The proposed POLTER is very simple to implement, which only requires storing pertaining policy checkpoints for an ensemble policy and adding KL-divergence regularization term between the ensemble policy and the learning policy to the objective function.\n- It is empirically shown that POLTER archives the policy that yields a smaller divergence to the optimal prior policy in the PointMass domain, which coincides with the main motivation of the paper.\n- In the URLB benchmark, POLTER generally improves the performance of the existing baseline algorithms, especially for data-based and knowledge-based methods.\n- Various ablation experiments are provided, which is great for a better understanding of when/how POLTER works well or not.\n\n[Weaknesses]\n- Theoretical analysis is lacking. Beyond the intuition level, it is unclear how minimizing the KL divergence to the ensemble policy can lead to an optimal prior policy. Since the policy and its state occupancy measure have a non-linear relationship, it is unclear whether an average of multiple policies induces a state-marginal distribution defined in terms of a convex combination of each policy's state marginal. It would be great to provide some theoretical justification for the provided method.\n- In Figure 3, only RND results are provided. Besides RND, it would be great to see the results of other URL baselines: whether adding POLTER obtains a resulting policy that is consistently closer to the optimal prior policy.\n- In addition to $\\alpha$, POLTER has an additional hyperparameter, $\\mathcal{T}_E$. How often should we add policy checkpoints to the ensemble, and how sensitive is the performance of POLTER depending on this choice?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Policy Trajectory Ensemble Regularization (POLTER), a method to improve the performance of unsupervised RL algorithms. POLTER introduces a KL regularization term between the learning policy and the ensemble of pretraining policies in the policy learning objective function. In the PointMass domain, they showed that POLTER obtains the lower KL-divergence to the optimal pretraining policy. In the experiments, POLTER is shown to be effective to improve the existing URL algorithms, especially data-based and knowledge-based algorithms.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be great to provide (1) a theoretical justification for POLTER. (2) More baseline experimental results for PointMass/Pendulum (currently only RND is being considered). (3) Explanation of how to choose the checkpoint intervals for the ensemble policy and experimental results on how sensitive POLTER's performance is depending on this choice.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YlXCUpQPrD", "reviewer": "Reviewer_xtTv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed regularizer is simple to implement and can be easily plugged into various unsupervised RL methods;\n- The empirical analysis shows that POLTER improves the performance of some data-based and knowledge-based methods, while it never hurts performance in the presented tasks.\n\nWeaknesses\n- The motivation for the proposed regularizer is somewhat weak: The paper does not clearly describe why POLTER should benefit the pre-training performance in general.\n\nThis paper is tackling a relevant problem, which is how to pre-train a policy with unsupervised interactions such that to improve the sample-efficiency of learning downstream tasks. In this context, the proposed regularizer seems to provide interesting benefits with little implementation and computation overhead. However, the current version of the paper fails to provide a full understanding on why the regularization actually improves performance of the tested methods, which makes unclear whether we can expect similar benefits in different domains. I report below some more detailed comments and suggestions to improve the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses policy pre-training through unsupervised RL. Especially, it proposes a novel regularization method (POLTER) that can be combined with any unsupervised RL algorithm. The regularizer pushes the pre-trained policy closer to a uniform mixture of the previous policies. The paper first motivates the regularization term through the geometric framework of Eysenbach et al. (2022), then provide an illustrative validation in point-mass and pendulum domains, and finally empirically evaluate POLTER in combination of various unsupervised RL methods over the URLB tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(Motivation) The paper seems to rely too much on the derivations by Eysenbach et al. (2022) to motivate the proposed regularizer. The adaptation objective in equation (1) is not completely clear to me (what is the policy $\\pi$ inducing distribution $\\rho (s)$? $\\pi_T^*$ is the pre-trained policy or the policy after fine-tuning? Which are the actual optimization variables?), and I have some doubts over the generality of the claim \"The optimal prior policy is the one inducing the average state marginal distribution of the skills\" (which skills?). Thus, I suggest the authors to expand Section 3.1 to include a more formal motivation of the regularizer and adaptation objective.\n\n(Regularization improves stability) It is known that knowledge-based and data-based methods can suffer from stability issues, as they are usually optimizing an ever changing reward, which sometimes exacerbates the inherent instability of value predictions. The proposed regularizer seems to help the general stability of those methods, as it provides a penalty for deviating from the previous policies. Thus, I am wondering whether the actual motivation of the benefits guaranteed by POLTER is the result of an improved stability of the pre-training.\n\n(State-based evaluation) Whereas the choice to evaluate the proposed regularizer in the state-based version of the URLB is somewhat reasonable, it is not straightforward to say whether POLTER would help in pixel-based tasks as well. The authors are rightly saying that POLTER can be combined with pixel-based methods, but Laskin et al. (2021) show that the performance of the algorithms can change significantly in the two settings (see Figure 3 in Laskin et al.).\n\n(Uniform mixture) The authors are claiming that the theoretical framework in (Eysenbach et al., 2022) supports the use of a uniform mixture of the previous policies. However, it would be interesting to see whether a different choice of $\\tilde{\\pi}$ can provide even better results. Did the authors tried with an exponentially-decaying mixture that keeps the current policy close to the recent policies?\n\n(Related work) The quickly-growing unsupervised RL literature is now including numbers of methods in the knowledge, data, competence-based categories. The paper is only citing the few works that are most relevant for the subsequent empirical analysis, but I believe the related work section could at least mention the previous literature.", "broader_impact_concerns": "I believe this paper can be labelled as fundamental research, and thus I think it does not need to address specific ethical implications in a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers find the work to be useful contributions about Unsupervised Reinforcement Learning, with good supports. The revised manuscript addresses earlier feedback with a number of improvements. We recommended acceptance, with a few minor suggestions for the final version:

* page 4: in equation 2, the domain Z is still unclear. Is it over all (infinitely many) possible skills, or a finite set of skills? How is it determined (human-defined, or automatically discovered, or else)? A sentence or two will make the paper more self-contained.
* page 4: typo “an dynamics model”
* page 5: mark the end of Proposition 1’s proof sketch.
* The notation of mutual information is convenient, but not standard (in my opinion). MI is a function of two random variables, not of a (distribution, random variable) pair. If the authors prefer to use the current notation, please add a sentence to explain the choice. It also avoids confusion (e.g., when readers refer to the relevant papers like Eysenbach et al. that uses the standard notation).


Decision: Accept (poster)