EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model
Authors: Yifu Yuan
Jianye HAO
Fei Ni
Yao Mu
YAN ZHENG
Yujing Hu
Jinyi Liu
Yingfeng Chen
Changjie Fan
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Unsupervised RL
Model-based RL

Abstract: Unsupervised reinforcement learning (URL) poses a promising paradigm to learn useful behaviors in a task-agnostic environment without the guidance of extrinsic rewards to facilitate the fast adaptation of various downstream tasks. Previous works focused on the pre-training in a model-free manner while lacking the study of transition dynamics modeling that leaves a large space for the improvement of sample efficiency in downstream tasks. To this end, we propose an Efficient Unsupervised Reinforcement Learning Framework with Multi-choice Dynamics model (EUCLID), which introduces a novel model-fused paradigm to jointly pre-train the dynamics model and unsupervised exploration policy in the pre-training phase, thus better leveraging the environmental samples and improving the downstream task sampling efficiency. However, constructing a generalizable model which captures the local dynamics under different behaviors remains a challenging problem. We introduce the multi-choice dynamics model that covers different local dynamics under different behaviors concurrently, which uses different heads to learn the state transition under different behaviors during unsupervised pre-training and selects the most appropriate head for prediction in the downstream task. Experimental results in the manipulation and locomotion domains demonstrate that EUCLID achieves state-of-the-art performance with high sample efficiency, basically solving the state-based URLB benchmark and reaching a mean normalized score of 104.0±1.2% in downstream tasks with 100k fine-tuning steps, which is equivalent to DDPG’s performance at 2M interactive steps with 20× more data. More visualization videos are released on our homepage.

Published as a conference paper at ICLR 2023
EUCLID: TOWARDS EFFICIENT UNSUPERVISED RE-
INFORCEMENT LEARNING WITH MULTI-CHOICE DY- NAMICS MODEL
Yifu Yuan-, Jianye Hao Fei Ni Yao Mu Yan Zheng', Yujing Hu², Jinyi Liu', Yingfeng Chen', Changjie Fan 'College of Intelligence and Computing, Tianjin University, Fuui AI Lab, Netease, Inc., Hangzhou, China, 'The University of Hong Kong
ABSTRACT
Unsupervised reinforcement learning (URL) poses a promising paradigm learn useful behaviors in a task-agnostic environment without the guidance of extrin- sic rewards facilitate the fast adaptation of various downstream tasks. Previous works focused on the pre-training in a model-free manner while lacking the study of transition dynamics modeling that leaves a large space for the improvement of sample efficiency in downstream tasks. To this end, we propose an Efficient Unsupervised reinforCement Learning framework with multi-cholce Dynamics model (EUCLID), which introduces novel model-fused paradigm to jointly pre- train the dynamics model and unsupervised exploration policy in the pre-training phase, thus better leveraging the environmental samples and improving the down- stream task sampling efficiency. However, constructing generalizable model which captures the local dynamics under different behaviors remains a challeng- ing problem. We introduce the multi-choice dynamics model that covers different local dynamics under different behaviors concurrently, which uses different heads to learn the state transition under different behaviors during unsupervised pre- training and selects the most appropriate head for prediction in the downstream task. Experimental results in the manipulation and locomotion domains demon- strate that EUCLID achieves state-of-the-art performance with high sample effi- ciency, basically solving the state-based URLB benchmark and reaching a mean normalized score of 104.0+ 1.22% in downstream tasks with 100k fine-tuning steps, which is equivalent to DDPG's performance at 2M interactive steps with 20x more data. More visualization videos are released on our homepage.
INTRODUCTION
Reinforcement learning (RL) has shown promising capabilities in many practical scenarios (Li ct al., 2022b; Ni et al., 2021; Shen et al., 2020; Zheng et al., 2019). However, RL typically requires substantial interaction data and task-specific rewards for the policy learning without using any prior knowledge, resulting in low sample efficiency (Yarats et al., 2021c) and making it hard to generalize quickly new downstream tasks (Zhang et al., 2018; Mu et al., 2022). For this, unsupervised reinforcement learning (URL) emerges and suggests new paradigm: pre-training policies in an unsupervised way, and reusing them as prior for fast adapting the specific downstream task (Li et al., 2020; Peng et al., 2022: Seo et al., 2022), shedding a promising way to further promote RL to solve complex real-world problems (filled with various unseen tasks).
Most URL approaches focus on pre-train a policy with diverse skills via exploring the environment guided by the designed unsupervised signal instead of the task-specific reward signal (Hansen al., 2020; Liu & Abbeel, 2021a). However, such a pre-training procedure may not always benefit down- stream policy learning. As shown in Fig. we pre-train policy for 100k, 500k, 2M steps in the robotic arm control benchmark Jaco, respectively, and use them as the prior for the downstream pol- icy learning see how pre-training promotes the learning. Surprisingly, long-hour pre-training does not always bring benefits and sometimes deteriorates the downstream learning (500k vs 2M in the
Corresponding authors: Jianye Hao wwanneehaa@@ra.ddu.c.m

---

Published as a conference paper at ICLR 2023
Pre-baining Frume
Downstream
Jaauo
Nom 0 Fetum (%)
100
EUCLID
Mismatch
M ismatch
Match
Pre training (Disegrement
100k
500k
Pre-triining Framnees (Log (Log-scale)
500k
Figure I: A motivation example.
orange line). We visualize three pre-trained policies on the left of Fig. I, and find they learn different skills (i.e., each covering a different state space). Evidently, only one policy (pre-trained with 500k) where the red brick exists.
is beneficial for downstream learning as it happens to focus the This finding reveals that the downstream policy learning could heavily rely on the pre-trained policy, and poses a potential limitation in existing URL approaches: Only pre-training policy via diverse exploration not enough for guaranteeing to facilitate downstream learning. Specifically, most mainstream URL approaches pre-train the policy in a model-free manner (Pathak et al., 2019; 2017; Campos et al., 2020), meaning that the skill discovered later in the pre-training, will more or less suppress the earlier ones (like the catastrophic forgetting). This could result in an unpredictable skill that most likely not the one required for solving the downstream task (2M vs. 500k). We refer to this as the mismatch issue which could make pre-training even less effective than randomized policy in the downstream learning. Similarly, Laskin al. (2021) also found that simply increasing pre-training steps sometimes brings no monotonic improvement but oscillation in performance.
To alleviate above issue, we propose the Efficient Unsupervised reinforCement Learning frame- work with multi-cholce Dynamic model (EUCLID), introducing the model-based RL paradigm achieve rapid downstream task adaption and higher sample efficiency. First, in the pre-training phase, EUCLID proposes to pre-train the environment dynamics model, which barely suffers from the mismatch issue as the upstream and downstream tasks in most time share the same environment dynamics. Notably, the pre-training dynamics model is also orthogonal to the pre- pre-training policy, thus EUCLID pre-trains them together and achieves the best performance (see Fig. 1). In practice, EUCLID requires merely no additional sampling burden as the transition collected during the policy pre-training can also be used for the dynamics model pre-training. On the other hand, in the fine- tuning phase, EUCLID leverages the pre-trained dynamics model for planning. which is guided by the pre-trained policy. Such combination could eliminate the negative impact caused by the mis- match issue and gain fast adaptation performance. More importantly, EUCLID can monotonically benefit from an accurate dynamics model through a longer pre-training.
Another practical challenge is that, due the model capacity, pre-training one single dynamics model is hard to accurately model all the environment dynamics. The inaccuracy can be further exacerbated in complex environments with huge state space, and thus deteriorates the downstream learning performance. Inspired by multi-choice learning, EUCLID proposes multi-headed dynam- ies model with each head pre-trained with separate transition data. Each head focuses a different region of the environment, and is combined to predict the entire environment dynamics accurately. phase, EUCLID could select the most appropriate head (sharing a similar
As such, in the dynamics to the downstream task) to achieve a fast adaptation.
Our contributions are fold: (I) extend the mainstream URL paradigm by innovative intro- ducing the dynamics model in the pre-training phase, so that model-based planning can be leveraged in the fine-tuning phase to alleviate the mismatch issue and further boost the downstream policy learning performance; (2) We propose multi- dynamics model to achieve a fine-grained and more accurate prediction, which promotes effective model planning in solving downstream tasks; (3) We empirically study the performance of EUCLID by comparing different mainstream URL mechanisms or designs, and comprehensively analyze how each part of EUCLID affect the ultimate performance; (4) Extensive comparisons on diverse continuous control tasks are conducted and the results demonstrate significant superiority of EUCLID in performance and sample efficiency, espe- cially in challenging environments. Our approach basically solves the state-based URLB, achieving mtte-offttheaan performance with a normalized score of 104.0 1.2% and outperforming the prior leading method by 1.35x, which is equivalent to DDPG with 20x more data.
2

---

Published as a conference paper at ICLR 2023
2 BACKGROUND
Markov Decision Process (MDP) is widely used for formulating a continuous control task, defined of the state space S, action space A, reward function R[s, a), transition
as tuple (8, A, RR
Po. 7) Po, probability P ( s' S, a), initial state distribution po and discounting factor y (Sutton & Barto, 1998).
The objective is to learn the optimal policy a, To (- (s)) that maximizes the expected discounted
return E (St.oaaass.oo........
ΣT-1 - " (st, a.) where T is the variable episode length.
Unsupervised reinforcement learning (URL) poses a promising approach to learning useful pri- ors from past experience and accelerating the learning of downstream tasks (Schwarzer et al., 2021). URLB (Laskin al., 2021) split the whole learning phase into two parts, pre-training (PT) and fine- tuning (FT).At every timestep in the PT phase, the agents can only interact with the task-agnostic reward-free environment to obtain intrinsic rewards learned through contrast, in the FT phase, agents need to adapt quickly to downstream tasks with task-specific ex- trinsic rewards provided by the environment. Specifically, URL algorithms can be generalized into three categories, including knowledge-based, data-based and competence-basee methods (Oudeyer al., 2007; Srinivas & Abbeel, 2021). The goal of the knowledge- methods is increase knowledge of the world by maximizing prediction errors (Pathak et al., 2017; 2019; Burda et al., 2019) while data-based methods aim to maximize the entropy of the state of the agents (Liu & Abheel, b021b; Yarats et al., 2021a; Hazan et al., 2019). Competence-band based methods learn an ex- plicit skill vector by maximizing the mutual information between the observation and skills (Campos et al., 2020; Eysenbach et al., 2019; Gregor et al., 2017).
sef--supervised manner. In
Model-based reinforcement learning (MBRL) leverages a learned dynamic model of the envi- ronment to plan a sequence of actions in advance which augment the data (Sutton, 1991; Janner et al., 2019; Pan et al., 2020; Mu et al., 2020; Peng et al., 2021) or obtain the desired behavior through planning (Chua et al., 2018; Hafner et al., 2019; Lowrey al., 2019: Mu et al., 2021: Chen et al., 2022). However, training the world model requires large number of samples (Polydoros & Nalpantidis, 2017: Plaat et al., 2021), and an imprecise model can lead to low-quality decisions for imaginary planning (Freeman et al., 2019). Thus constructing a reliable world model by pre-training can greatly accelerate the learning process of downstream tasks (Chebotar et al., 2021: Seo al., 2022). A concurrent work of ours is Rajeswar et al. (2022), which also focuses on promoting URL performance through world model, but only considers the incorporation of simple single dynam- ics model without realizing the low accuracy caused by the mismatch of the pre-training policies and the optimal policies for the downstream tasks. Our work designs multi-choice learning and policy constraints to explore how to construct a generic pre-trained dynamics model to cover different local dynamics under different behaviors.
Multi-choice learning (MCL) learns ensemble models that produce multiple independent but di- verse predictions and selects the most accurate predictions to optimize the model, better reduce training loss and improve stability (Guzmán-Riven) et al., 2012: 2014; Dey et al., 2015; Lee et al., 2017). Multi-choice learning has been widely used, such as in vision computing (Cuevas et al., 2011; Tian et al., 2019: Lee et al.. 2017) and visual question answer (Lei et al., 2020; Lu al., 2022), but most works focus on supervised learning. T-MCL (Seo et al., 2020) and DOMINO (Mu et al., 2022) bring MCL into the meta RL, which approximates multi-modal distribution with context and models the dynamics of changing environments. Inspired by this, EUCLID builds a multi-headed dynamics model cover specialized prediction regions corresponding to different downstream tasks.
METHODOLOOGY
In this work, we propose an Efficient Unsupervised reinforCement Learning framework with multi- cholce Dynamic model (EUCLID) to further improve the ability of mainstream URL paradigm in fast adapting to various downstream tasks. As a starter, EUCLID adopts the task-oriented la- tent dynamic model as the backbone for environment modeling, and contains two key parts: model-fused URL paradigm that innovatively integrates the world model into the pre-training and fine-tuning for facilitating downstream tasks learning, and captures different enviro dynamics separately for an accurate prediction of the entire envi- ronment. In this way, EUCLID can achieve fast downstream tasks adaptation by leveraging the accurate pre-trained environment model for an effective model planning in the downstream fine- tuning. The detail pseudo code is given in Algorithm
multi-headed dynamics model that
3

---

Published as a conference paper at ICLR 2023
Pre-training with Intrinsic Rew ands
Fine-tuning Downstream esm Tesks
Self supervisod Tuck
Intrisic Reward
Pro- tmined
Guide
inct
Weight
Task aguosac Enviroment
State
World Model
World Model
Downstreen Tisk
Explontion Palicy
Platnee
Action & Intriisic Reward
Extrinsic Reward
Figure 2: Overview of the model-fused URL paradigm. In the pre- training phase (left). we jointly update the world models and policy using exploring samples collected from the environment inter- action. In the fine-tuning phase (right), we reuse the pre-trained weights to initialize the downstream world models and the policy, perform policy guided planning via dynamies model for fast adaption.
3.1 BACKBONE FOR ENVIRONMENT MODELING
We build world models for representation learning and planning based on the Task-Oriented Latent Dynamie Model (TOLD) of TDMPC (Hansen et al., 2022). EUCLID learns world models that compresses the history of observations into compact feature space variable and enables policy learning and planning in this space (Zhang al., 2019; Hafner et al., 2019). EUCLID'' world model components include three major model components: (i) the representation model that encodes state St model state Zt and characterizes spatial constraints by latent consistency loss, (ii) the latent dynamics model that predicts future model states žr+1 without reconstruction and (iii) the reward predictor allows the model encoder task-relevant information compact potential space. The model can be summarized as follow:
Z4 = F.g (s:) Reward predictor: f: ~ Ro (2,,ar)
Representation:
Latent dynamics: 2++1 Do (Zz, ar)
(1)
where s, z denote a state, action and latent state representation. Then, for effective value-based learning guidance planning, we built actor and critic based on DDPG (Lillicrap al., 2016):
Value: c = Qo (Zr, ax)
Policy: å, ~ Te (2:)
(2)
All model parameters 0 except actor are jointly optimized by minimizing the temporally objective:
C. (0, D) = B. (.
c1 |Ro (z,, a.) - rull +02 DDo (Z,, a.) F (51+1)1
Reward prediction
Lstent stile consistency
(3)
+03 [Qa (2i, (ra + QQ (zi+1, no (2 1)
Value prediction
while actor Te maximizes Qo approximated cumulative discounted returns. The reward term pre- diets single-step task reward, the state transition term aims predict future latent states accurately and the value term optimized through TD-leaniig (Haarnoja et al., 2018; Lillicrap et al., 2016). Alype--paaamterr
adjusted to balance the multi-source loss function and trajectory of length H is sampled from the replay buffer D.
3.2 MODEL-FUSED UNSUPERVISED REINFOR LEARNING PARADIGM
Overall, EUCLID introduces the environment modeling into the pre-training (PT) and fine-tuning (FT) phases, formulating a new model-fused URL paradigm. As shown in Fig. (left), EUCLID adopts reward-free exploration policy to collect tano-tgi transitions for pre-training the dy- namics model and then uses them boost downstream FT. In the following, we describe three important detail designs in EUCLID throughout the PT and FT processes.
Firstly, in the PT phase, we expect as diverse data as possible to pre-train the dynamics model for an accurate environment prediction. As EUCLID can be easily combined with almost any mainstream exploration methods (Hao al., 2023), we empirically evaluate the knowledge-based method (Disagreement, Pathak et al. (2019)), data-based method (APT, Liu & Abbeel (2021b)) and competence-basee method (DIAYN, Eysenbach et al. (2019)) (see in Appendix B). We find
4

---

Published as a conference paper at ICLR 2023
O Explore diverse specialiied regions
① Learn dynamics from experitnce
Act in the downstrcam tasks
Head
Maximum Zero-shot
Head
Reward
Head
Head
Bockbonz Net
Head
Head
Plamiing
Downstream Task
Illustrations of multi-choice learning in EUCLID. We design a multi-headed dynam- ics model to predict different environment dynamies, training each predictor head separately using diverse data, and select the most beneficial predictor head for the downstream task.
Figure 3:
the knowledge-based method (i.e., Disagreement) synthetically performs the best, and hypothesize this happens because it explores via maximizing the prediction uncertainty, allowing discovery more unpredictable dynamics.
Secondly, in the PT phase, how select the loss function for the dynamics model pre-training is challenging. We use latent consistency loss (Schwarzer et al., 2020; 2021; Hansen et al., 2022) learn the dynamics model in latent space to directly predict the feature of future states without reconstruction (Hafner et al., 2020b;a; Ha & Schmidhuber, 20.8). This improves the generalization of the dynamics model and avoids capturing task-irrelevant details between PT and FT phase while the reconstruction loss forces to model everything in the environment in such a huge state space.
Lastly, shown in Fig. 2 (right), we utilize the pre-trained dynamics model to rollout trajectories for both planning and policy gradient in the FT for downstream tasks. An important reason is that model-based planning can yield stable and efficient performance due to the similarity of the envi- ronment dynamics between upstream and downstream tasks, thus an accurate pre-trained dynamics model could benefit downstream learning. This can avoid the above-mentione mismatch issue, caused by utilizing a less effective pre-trained policy for the downstream tasks. For this, we adopt representative model predictive control method (Williams et al., 2015) to select the best action based on the imaged rollouts (via dynamics model). Meanwhile, we also notice that the pre-trained policy can master some simple skills like rolling, swinging, and crawling, which is useful and can be utilized in the model planning further boost downstream learning. Therefore, we additionally mix the trajectories generated by the pre-trained policy interacting with the dynamics model for planning. This would further speed up the planning performance and effectiveness, especially in the early stage of fine-tuning (Wang& & Ba, 2020). Another side benefit is that the Q-value function (in the critic) in the pre-trained policy could speed up the long-term planning via bootstrapping (Sikchi et al., 2021) 1) (see details in Appendix H).
3.3 MULTI-CHOICE LEARNING VIA MULTI-HEADED DYNAMICS MODEL
Prior works (Wu al.. 2019; Seo al., 2020) reveal that training one single dynamics model to accurately predict the whole environment dynamics is difficult in complex environments. Besides, pre-training one single dynamics model to predict the entire state space in the environment is ineffec- tive, as downstream tasks may involve only limited local state distributions (left in Fig. 3). Therefore, EUCLID proposes multi-choice learning mechanism via multi-heades dynamics model in the PT stage for achieving a more accurate environment prediction and to select one of the heads for the FT stage to further boost downstream learning.
As shown in Fig. (middle), we design a multi-heades dynamics model to predict different environ- ment dynamics (i.e., state distributions), encouraging each predictor head to enjoy better prediction accuracy in its own specialized prediction region. The model contains backbone parameterized by 0, sharing the high-level representations of the first few layers of the neural network, while the latter layers build H prediction heads parameterized by (Ohead | The output is:
2++1 = { DDo (ze, az; 0 0. )
We do not use separate ensemble models with completely different parameters because we empir- ically observe that there may also be large number of shared dynamics between different down- stream tasks which can be captured by the shared backbone to leverage environmental samples efficiently and also avoid actively reducing the information of acquired environment samples.
5

---

Published as a conference paper at ICLR 2023
To encourage each head to focus on separate region possible, we assign different exploration policies for each head, denoted by [= (z), ..... T ( To avoid region overlap, we design a diversity-encouragen term as an additional regularization term as follows:
(0, D) ERNND (Qo (Zt, (z.) DKL. (T (Zt) I| (2,)),,
(5)
where To (z,) = E.= Tú /h. In this way, each policy is optimized by its own intrinsic for exploration and policies are encouraged to move away from each other (see more in Appendix G).
In the beginning FT phase, we select one head in the pre-trained nulti-headed dynamics model to construct a single-head dynamic model that can benefit the downstream learning the most (right in Fig. 3). To figure out the optimal head h" (that covers the most appropriate state region) for solving the downstream tasks, inspired by Seo et al. (2020), we evaluate the zero-shot performance of each head and pick one with the highest zero-shot performance in downstream tasks. After that, only optimal head is used in the subsequent fine-tuning for adapting to downstream tasks.
Overall, empirically show that different prediction heads indeed cover different state regions with respective closely related tasks and the multi-choice learning improves the performance by selecting the appropriate specialized prediction head (see Fig. 7).
4
EXPERIMENTS
We conduct experiments on various tasks to study the following research questions (RQs): Combination (RQ1): Which category of URL method works best in combination with EUCLID?
Performance (RQ2): How does BUCLID compare to existing URL methods?
Monotonism (RQ3): Can EUCLID monotonically benefit from longer pre-training steps?
Specialization (RQ4): Does multi-choice learning delineate multiple specialized prediction region?
Ablation (RQ5): How each module in EUCLID facilitates the downstream policy learning?
4.1 EXPERIMENTAL SETUP
Benchmarks: We evaluate our approach on tasks from URLB (Laskin et al., 2021), which con- sists of three domains (walker, quadruped, jaco) and twelve challenging continuous control down- stream tasks. Besides, we extend the URLB benchmark plex humanoid domain and three corresponding downstream tasks based on the DeepMind Control Suite (DMC) (Tunyasuvunakool et al., 2020) to further demonstrate the efficiency improvement of EUCLID more challenging environments. Environment details can be found in Appendix A.
(xRLB-Extension by adding a more com-
The proposed EUCLID is general framework that can be easily combined with any unsupervised RI. algorithms. Therefore we make comprehensive investigation of EUCLID com- bined with popular unsupervised RL algorithms of three exploration paradigms. Specifically, we choose the most representative algorithms from each of the three categories including Disagree- ment XKnowledge-basee (Pathak et al., 2019), APT (Data-based) (Liu & Abbeel, 2021b) and DI- AYN (Competence-based) (Eysenbach et al., 2019). In addition, we compare our method to the pre- vious state-of-the-ar method CIC (Laskin et al., 2022) in the URLB, which is hybrid data-based and competence-based method. More detailed information about the baselines is in Appendix B.
Baselines:
Evaluation: In the PT phase, all model components are pre-trained under unsupervised setting based on the sample collected by interacting 2M steps in the reward-free environment. Then in the FT phase, steps, which is moderated to 1 150k steps for the humanoid domain due to special difficulties. All statistics are obtained based on 5 independent runs per downstream task, and report the average with 95% confidence regions. For fairness of comparison, all model-fres baselines and the policy optimization part of based methods opt DDPG (Lillicrap et al., 2016) agent as backbone, maintaining the same setting as Laskin et (2021). To better compare the average performance of multiple downstream tasks, we choose the expert score of DDPG learning from scratch with 2M steps in the URLB as normalized score. Note that expert agents run for 2M steps with extrinsic reward, which is 20x more than the budget of the agent used for evaluation steps in the FT phase.
fine-tune each agent in downstream tasks with the extrinsic reward for only 100k
Implementation details: EUCLID employs both model planning and policy optimization in the FT phase but uses only policy optimization in the PT phase. During planning, we leverage Model Predictive Path Integral (MPPI) (Williams et al., 2015) control and choose planning horizon of

---

Published as a conference paper at ICLR 2023
Walber
quddrupe
1000
Oagal
Nomslleo Retum
S 5
Dssigrement
APT EUCLIDAPTT
DIAYN
CIC TOWPO 100K
EUC,
EUCLDIIAAN)
Figure 4: We combine EUCLID with three types of exploration algorithms to show the synergistic benefit of URL and MBRL, methods for 12 downstream tasks on URLB. The normalized scores are the average performance after pre-training 2M steps and then fine-tuning 100k steps with 5 indepen- dent seeds. EUCLID achieves significant improvement over the corresponding exploration backbone and empirically shows the combination with the knowledge-based method is most effective.
SEDE TOAPPEgESSe Claggmement
80
Eptende Runur
Epikoda Rutuen
Epicoda Return
60
200
1000
100
30
60
90
120
150
0
30
60
90
120
150
0
30
60
90
120
130
Environment Stops [x10
Er Ervironment Staps Ix10%)
Enviiomment Stops (x110)
(a) Humanoid-Stann
(b) Humanoid Walk
(c) Humanoid --nnonnddRnnn
Figure 5: Learning curves of BUCLID and three baselines on downstream tasks of the humanoid domain. Curves show the mean and 95% confidence intervals of performance across 5 independent seeds. The dashed reference lines are the asymptotic performance of the Disagreement algorithm with 2M PT steps and 2M FT steps. These results show that EUCLID has better sample efficiency and large performance gains on tasks with complex dynamics.
5. Notably, we still retain seed steps for warm-start in the FT phase despite sufficient pre- training steps to alleviate the extreme case of strong mismatch between the pre-training policies and the optimal policies for the downstream tasks. For multi-choice learning, we build multi- headed dynamics model maintaining heads of H = 4 by default, each optimized by independent samples. To avoid unfair comparison, we keep the same training settings for all methods.
4.2 COMBINATION (RQ1)
To answer RQI, we show the synergistic benefit EUCLID unsupervised RL methods and model-based architecture. As shown in Fig. 4, We build EUCLID on different URL backbone named EUCLID (Disgrremee//APTIDAAYY) the light line denotes the corresponding vanilla URL backbone results in Laskin et al. (2021). For the purpose of simplicity, we do not use the multi-choice learning mechanism here. We show that we can combine all three types of exploration baselines and obtain significant performance gains in all environments. In particular, the previous competence-basee because of the weak discriminator, while the EUCLID (DIAYN) greatly improves fast adaptation downstream tasks through pre-trained world models. Besides, we empirically find that EUCLID is better combined with the knowledge-based approaches with the best performance and stability. Unless specified, we choose Disagreement the exploration backbone in the PT stage by default.
where the dark line denotes its normalized score and
approaches under-perfonee in URLB
4.3 PERFORMANCE (RQ2)
Comparative evaluation on URLB. To answer RQ2, we evaluate the performance of EUCLID and other baseline methods in the URLB and URLB-EAL B-L-e--einiion environment. The results in Table show that EUCLID outperforms all other mainstream baselines all domains and basically solves the twelve downstream tasks of the state- URLB in 100k steps, obtaining a performance com- parable the expert scores while the expert agent trained 2M steps with extrinsic reward. It is worth noting that EUCLID significantly improves performance in reward-sparse robotic arm control tasks, such jaco, which is challenging for previous URL methods. Moreover, the results of EU-
7

---

Published as a conference paper at ICLR 2023
Table 1: Performance of EUCLID and EUCLID w/ multi headed structure URLB after 2M reward-free pre-training steps and finetuning for 100k steps with extrinsic rewards. All baseline runs with 5 seeds. We refer Appendix C for full results of all baselines.
Domain
Tack
Disagreement
CIC
TDMPC 100k
EUCLID
EUCLID
Flip
49TE21
631234
750g4 930E28
971E1
969:2
Run
444121
486425
765110
770,9
Walker
907+15
Walk
782433
885428
967=2
96724
972=1
Normalized score
72.5+2.6
82.2+3,4
101 4+1.6
104,3+0,5
104.6:10.44
Jamp
140=13
Run
461+12
505144
465+83
651+35
735+16
Quadriped
Stand
840+33
761+54
768+92
953+6
958+5
Walk
721+56
723443
710:77
874+-2
925s1
Normalized
73,5+3.5
72,545.2
74.7+9.8
98.1+2.7
Reach bottom left
13488
13849
168=7
214+5
221H-33
Reach bottom right
12264
145&7
183+11
20549
212+2
Inco
Reach left
225+5
Reach top right
140247
16344
172+24
21947
229
Normalized score
63.4+8.6
74.033
86.956.4
103.345
109.7+2.0
Normalized
69.8
87.746.0
104,0=1.2
Qustrped
Dvert
Norre fretamm
104
Norrod Eteesar
15
18
8 33
Puhiig Fizmns
Pre-zuiiing Framus
5XC
Prottaiiin Frames
1804 Pre-taiiin Frames
Figure 6: We show the tine-tuning performance of EUCLID with different pre-training steps, in- cluding 100k, 500k, and 2M steps. EUCLID can benefit from longer pre-training steps to achieve monotonic performance gains, while previous URL algorithms fail.
CLID also show significant improvement over TDMPC 100k learned from scratch, especially with greater gains in the Quadruped domain where is challenging to predict accurately. This suggests that unsupervised pre- tre-training can significantly reduce model error.
Effects of multi-headed dynamics model. To verify the importance of multi-choice learning, we compared the performance of EUCLID and EUCLID w/o multi-headed dynamics model. Ta- ble 1 demonstrates that EUCLID achieved almost all of the 12 downstream tasks' average perfor- mance improvement and significantly reduced the variance, reaching the highest normalized score of 104.0=1.22% This indicates that multi-choice learning reduces the prediction error of the down- stream tasks corresponding local dynamics and obtains better results.
experiments. As shown Fig. 5, we evaluate the agents in the humanoid do- main which is the most difficult to learn locomotion skills due to high-dimensional action spaces. We found that the previous model-free URL method is difficult to improve the learning effi- ciency of downstream tasks in such complex environment, while the previous model-based meth- ods (TDMPC) still make some initial learning progress. In contrast, EUCLID is able to consistently improve performance in all three downstream tasks with small variance. Also, we compare EUCLID with the naive pre-traine pre-training scheme, which uses a random exploration policy collect data in the PT phase to train the world models. As the task difficulty increases, we observe that EUCLID w/ naive PT struggles achieve competitive performance because of the lack of intrinsic unsupervised goal.
4.4 MONOTONISM (RQ3)
Intuitively, a longer pre-training phase is expected allow for a more efficient fine-tuning phase, but the empirical evidence of Laskin et al. (2021) demonstrated that longer pre-training is not always beneficial or even worse than random initialization when applying the previous URL algorithm, be- cause it is difficult to capture such rich environmental information with only pre-trained policies. This is a major drawback that makes the application URL pre- pre-training paradigm fraught with un- certainty. However, EUCLID better leverage environmental information to build additional world models. Fig. 6 shows that our methods achieve monotonically improving performance by varying pre-training steps on most of the tasks. All results are better than random initialization, except for the quadruped domain with 100k pre-training steps.
8

---

Published as a conference paper at ICLR 2023
115
Walber
Head
Head
Head 5
Head 4
Mutio e mosels
110
Muti cholte mode)(uur))
Hlpp
267
200
183
264
Run
66
205
309
100
Stand
410
408
38B
432
Wilk
178
239
2108
186
95
Quadruped
Head
Head
Head 3
Head 4
90
Jump
290
326
242
309
85
Run
215
288
138
228
80
Stand
461
325
550
353
Walk
217
266
173
161
75
Walkr
Quadruupd
Jaco
Ovaral
(a) Specialized region of cach predict head
(b) Importance multi heati- model
Figure 7: (a) Zero-shot performance of the pre-trained multi-headed dynamics model for different tasks on Walker and Quadruped domain. We highlight the top-2 prediction heads which are suitable for Corresponding tasks. (b) The average performance of the multi-headed dynamics model used by EUCLID and multiple dynamics models across 12 downstream tasks for 2M pre-training steps.
4.5 SPECIALIZATION (RQ4)
Prediction heads corresponding the specialized regions. To investigate the ability of EU- CLID to learn specialized prediction heads, we visualized how to assign heads to downstream tasks Fig. 7(a), we can observe that different prediction heads are better adapted different downstream tasks, with their corresponding specialized region. i.e., head 2 works best for Run tasks while head for Jump tasks in the Quadruped environment. No one prediction head can perform optimally for all downstream tasks. See our homepage for more visualization analysis.
Comparison of multiple models and multi-headed models. Fig. 7(b) shows the comparison of the multi-headed dynamics model used by EUCLID and multiple dynamics models without a shared backbone. We find that multiple models are less effec- tive than multi-headed models in all domains, especially in the quadruped domain which requires a large amount of sample to model complex dynamics. This also demonstrates that sharing the underlying dynamics of agents is beneficial to improve the sample efficiency for downstream tasks.
molf laves
(50
Nomaltzd
-00
70
Figure 8: Ablation different pre-trained components for FT.
4.6 ABLATION (RQ5)
Roles for each module. As shown Fig. 8, conducted ablation experiments with 2M steps pretraining on the EUCLID, reusing different subsets of pre-trained components in the FT phase. To ensure the generality of the conclusions, we conduct experiments based on different URL methods and take the average results. From the overall results, our default settings, reusing all components of the world models (encoder, dynamics model, reward model) and policy (actor and critic) work best. We show the detailed results for each domain in Appendix with in-depth analysis.
5 CONCLUSION
In this work, we propose EUCLID which introduces the MBRL methods into the URL domain to boost the fast adaption and learning performance. We formulate novel model-fused URL paradigm to alleviate the mismatch issue between the upstream and downstream tasks and propose the multi- choice learning mechanism for the dynamics model achieve more accurate predictions and further enhance downstream learning. The results demonstrate that EUCLID achieves offiee----heaan- per- formance with high sample efficiency. Our framework points novel and promising paradigm for URL to improve the sample efficiency and may be further improved by the more fine-grained dynamics divided by skill prior (Pertsch et al., 2020), the combination of offline RL (Yu al., 2022). In addition, extending EUCLID in the multiagent RL (Li et al., 2022a: Hao et al., 2022; Zheng al., 2018; 2021) ) is also promising direction, which leave future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Grant No.62106177 the "New Generation of Artificial Intelligence" Major Project of Science & Technology 2030 (Grant No.2022ZD01 16402), and the Science and Technology on Information Systems Engineering Labo- ratory (Grant No. ww..DDZ2202335500000,, No. wWD.220000555400)))
REFERENCES
Alberto Bemporad and Manfred Morari. Control of systems integrating logic, dynamics, and con- straints. Aulomatica, 35(3):407-427, 1999.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, 2019.
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceed- ings of the 37th International Conference Machine Learning, 2020.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Ir- pan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In Proceedings of the 38th Interna- tional Conference on Machine Learning, 2021.
Xiaoyu Chen, Yao Mark Mu, Ping Luo, Shengbo Li, and Jianyu Chen. Flow-based recurrent belief state learning for pomdps. International Conference on Machine Learning, pp. 3444-3468. PMLR, 2022.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 3/, 2018.
Erik Cuevas, Daniel Zaldivar, and Marco Pérez-Cisneros. Seeking multi-thresholds for image seg- mentation with learning automata. Machine Vision and Applications, 22(5):805-818, 2011.
Debadeepta Dey, Varun Ramakrishna, Martial Hebert, and J. Andrew Bagnell. Predicting multiple structured visual interpretations. In 2015 IEEE International Conference on Computer Vision, 2015.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In 7th International Conference on Learning Repre- sentations, 2019.
C. Daniel Freeman, David Ha, and Luke Metz. Learning to predict without looking ahead: World models without forward prediction. In Advances in Neural Information Processing Systems 32, 2019.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In 5th International Conference Learning Representations, 2017.
Abner Guzmán Guzman-Rivera, Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning to produce multiple structured outputs. In Advances in Neural Information Processing Systems 25, 2012.
Abner Guzmán Rivera, Pushmeet Kohli, Dhruv Batra, and Rob A. Rutenbar. Efficiently enforcing diversity in multi-output structured prediction. Proceedings of 17th International Conference on Artificial Intelligence and Statistics, 2014.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: /0122, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, 15s/1812.05905. 2018
10

---

Published as a conference paper at ICLR 2023
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. Proceedings of the 36th International Conference Machine Learning, 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020a.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learn- ing behaviors by latent imagination. In 8th International Conference on Learning Representa- lions, 2020b.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04900 2022.
Steven Hansen, Will Dabney, André Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In 8th In- ternational Conference on Learning Representations, 2020.
Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent multiagent domain. IEEE Transactions on Neural Networks and Learming Systems, 2023.
Xiaotian Hao, Weixun Wang, Hangyu Mao, Yaodong Yang, Dong Li, Yan Zheng, Zhen Wang, and Jianye Hao. Api: Boosting multi-agent reinforcement learning via ggent-permutatioon-nvaaaaa networks. arXiv preprint arXiv:2203.05285. 2022.
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems 32, 2019.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Ler- rel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 10. 15191, 2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: contrastive intrinsic control for unsupervised skill discovery. CoRR, 0/0b//202.00161,1 2022.
Kimin Lee, Changho Hwang, KyoungSoo Park, and Jinwoo Shin. Confident multiple choice learn- ing. In Proceedings of the 34th International Conference Muchine Learning, 2017.
Chenyi Lei, Lei Wu, Dong Liu, Zhao Li, Guoxin Wang, Haihong Tang, and Houqiang Li. Multi- question learning for visual question answering. In 34th AAAI Conference on Artificial Intelli- gence, 2020.
Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Li, H. Tang, T. Yang, X. Hao, T. Sang, Y. Zheng, J. Hao, M. E. Taylor, W. Tao, and Z. Wang. PMIC: improving multi-agent reinforcement learning with progressive mutual information col- laboration. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- limore, Maryland, USA, Proceedings of Machine Learning Research, pp. 12979-12997, 2022a.
Pengyi Li, Hongyao Tang. Jianye Hao, Yan Zheng, Xian Fu, and Zhaopeng Meng. Erl-re 2: Effi- cient evolutionary reinforcement learning with shared state representation and individual policy representation. arXiv preprint arXiv:2210. 2022b.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference Learning Representations, 2016.
11

---

Published as a conference paper at ICLR 2023
Hao Liu and Pieter Abbeel. APS: active pretraining with successor features. In Proceedings of the 38th International Conference on Machine Learning, 202
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems 34, 2021b.
Kendall Lowrey, Aravind Rajeswaran, Sham M. Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In 7th International Conference on Learning Representations, 2019.
Jiaying Lu, Xin Ye, Yi Ren, and Yezhou Yang. Good, better, best: Textual distractors generation for multiple-choice visual question answering via reinforcement learning. IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, 2022.
Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-driven exploration via latent bayesian surprise. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Yao Mu, Baiyu Peng, Ziqing Gu, Shengbo Eben Li, Chang Liu, Bingbing Nie, Jianfeng Zheng, and Bo Zhang. Mixed reinforcement learning for efficient policy optimization in stochastic environ- ments. In 2020 20th International Conference on Control, Automation and Systems (ICCAS), pp. 1212-1219. IEEE, 2020.
Yao Mu. Yuzheng Zhuang, Bin Wang. Guangxiang Zhu, Wulong Liu, Jianyu Chen, Ping Luo, Shengho Li, Chongjie Zhang, and Jianye Hao. Model-based reinforcement learning via imagina- tion with derived memory. Advances in Neural Information Processing Systems, 34:9493-9505, 2021.
Yao Mu, Yuzheng Zhuang, Fei Ni, Bin Wang, Jianyu Chen, HAO Jianye, and Ping Luo. Domino: Decomposed mutual information optimization for generalized context in meta-reinforremmnnt learning. In Advances in Neural Information Processing Systems, 2022.
F. Ni, J. Hao, J. Lu, X. Tong, M. Yuan, J. Duan, Y. Ma, and K. He. A multi cilti-graphhho attributed reinforcement learning based optimization algorithm for large-scale hybrid flow shop scheduling problem. In KDD, pp. 3441-3451, 2021.
Pierre- Yves Oudeyer. Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):25-2866, 2007.
Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the model when it is confident: Masked model-based actor-critic. In Advances in Neural Information Processing Systems 33, 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference Machine Learning, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Baiyu Peng, Yao Mu, Yang Guan, Shengbo Eben Li, Yuming Yin, and Jianyu Chen. Model-based actor-critic with chance constraint for stochastic system. In 2021 60th IEÉE Conference on Deci- sion and Control (CDC), 4694-4700. IEEE, 2021.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4):9:1194:11, 2022.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In 4th Conference on Robot Learning, 2020.
Aske Plaat, Walter A. Kosters, and Mike Preuss. High-accuracy model-based reinforcement learn- ing, a survey. CoRR, 0is/22107.08441, 2021.
Athanasios S. Polydoros and Lazaros Nalpantidis. Survey of model-based reinforcement learning: Applications robotics. J. Intell. Robotic Syst., 86(2):153-173, 2017.
12

---

Published as a conference paper at ICLR 2023
Sai Rajeswar, Pietro Mazzaglia. Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste, Unsupervised model-based pre-training for data-efficient reinforcement learning from pixels. In Decision Awareness in Reinforcement Learning Workshop al 2022, 2022.
Reuven Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1)89-112, 1997.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self seff-predictive representations. arXiv preprint arXiv:2007.05999, 2020.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data- efficient reinforcement learning. In Advances in Neural Information Processing Systems 34, 2021.
Younggyo Seo, Kimin Lee, Ignasi Clavera Gilaberte, Thanard Kurutach, Jinwoo Shin, and Pieter Abbeel. Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning. In Advances in Neural Information Processing Systems 33, 2020.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State en- tropy maximization with random encoders for efficient exploration. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Younggyo Seo, Kimin Lee, Stephen L.. James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In Proceedings of the 39th International Conference Machine Learning, 2022.
Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan. and Yang Liu. Generating jehavior-diverse game ais with evolutionary multi-objective deep reinforcement learning. In Christian Bessiere (ed.). Proceedings of the Twenty-Ninth International Joint Con- ference Artificial Intelligence, IJCAI 2020, pp. 3371-3377. ijcai.org, 2020.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off off-policy with online planning. In Conference on Robot Learning, 2021.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- est neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):301-321, 2003.
Aravind Srinivas and Pieter Abbeel. Unsupervised learning for reinforcement learning. 2021. URL https:/ /mcml.... cc cc/medial / 202112221111111 hatp//0884__HaBBBAAAA QHHH pdf.
Richard S. Sutton. Dyna, integrated architecture for learning, planning, and reacting. SIGART Bull., 164):160- 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 1998.
Kai Tian, Yi Xu, Shuigeng Zhou, and Jihong Guan. Versatile multiple choice learning and its ap- plication to vision computing. In IEEE Conference Computer Vision and Pattern Recognition, 2019.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts. 2020.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In 8th Inter- national Conference on Learning Representations, 2020.
Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control using covariance variable importance sampling. CoRR, abs/1509.00 149, 2015.
Bohan Wu, Jayesh K. Gupta, and Mykel J. Kochenderfer. Model primitive hierarchical lifelong reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, 2019.
13

---

Published as a conference paper at ICLR 2023
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Letrel Pinto. Reinforcement learning with proto- typical representations. In Pruceedings of the 38th International Conference on Machine Learn- ing. 2021a.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint vrXiv:2107.0964 2021b.
Denis Yarats, Amy Zhang. Ilya Kostrikov, Brandon Amos, Joelle Pincau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In 35th AAAI Confer- ence on Artificial Intelligence, 2021c.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine.
How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, 2022.
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. CoRR, abs/1 hes/1804406893, 2018.
Marvin Zhang, Sharad Vikram, Laura M. Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. SOLAR: deep structured representations for model-based reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjic Fan. A deep bayesian policy reuse approach against non-stationary agents. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 962 972, 2018
Yan Zheng, Changjie Fan, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, and Yingfeng Chen. Wuji: Automatic online combat game testing using evolu- tionary deep reinforcement learning. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019, pp. 772-784. IEEE, 2019.
Yan Zheng, Jianye Hao, Zongzhang Zhang, Zhaopeng Meng, Tianpei Yang, Yanran Li, and Changjie Fan. Efficient policy detecting and reusing for non- non-stationarity in markov games. Auton. Agents Multi Agent Syst., 35(1):2, 2021.
14

---

Published as a conference paper at ICLR 2023
A ENVIRONMENT DETAILS
(a) Walker
(b) Quadruped
(c) Jaco
(d) Humanoid
Figure 9: Environment Details. We evaluate our method four manipulation and locomotion
domains. Each domain contains 4 downstream tasks (3 tasks for humanoid).
The details of all environments and corresponding tasks are as follows, as illustrated in Fig. 9:
Walker' (Flip, Run, Stand, Walk): Walker is a biped bound to 2D vertical plane, which learns balancing and locomotion skills.
Quadruped (Jump, Run, Stand, Walk): Quadruped also learns various types of locomotion skills but is harder because of high-dimensional state and action spaces and 3D environ- ment.
Jaco (Reach bottom left, Reach boltom right, Reach top left, Reach top right): Jaco Arm is 6-DOF robotic arm with three-finger gripper, which requires control the robot arm and perform simple manipulation tasks. Prior works (Laskin et al., 2021; Yarats et al., 2021a la) have shown that such reward-spars manipulation tasks are particularly challenging for unsupervised RL.
Humanoid (Run, Stand, Walk): Humanoid a simplified human-like entity with 21 joints, which is extremely challenging to learn locomotion skills due to high-dimensional action spaces and exploration dilemma. We test fine-tuning performance relatively few inter- action steps (150k) whereas prior work typically executes for 30M steps (200x) learning from scratch.
B DETAILS OF UNSUPERVISED RL BASELINES
EUCLID can simply combine with various categories of URL exploration approaches as backbone. We select several typical exploration algorithms as baselines.
Disagreement (Pathak et al., 2019): Disagreement is knowledge-based baseline which trains an ensemble of forward models {g: (Ze+1 1 z,, a:)) predict the feature. Intrinsic rewards are defined as the variance among the ensemble models:
Disagreement
α Var {gr (Re+1 2,, ax)) i == N.
(6)
APT (Liu & Abbeel, 2021b): Active Pre-training (APT) is data-based baseline which utilizes particle-basee estimator (Singh et al., 2003) that uses nearest-neighhors to estimate entropy for given state. Since APT needs the auxiliary representation learning to provide the latent variables for entropy estimation, we implement of APT on the top of EUCLID encoder optimized by latent consistency loss. Intrinsic reward are computed as:
APT r
log I|z, -
humanoid.py
15

---

Published as a conference paper at ICLR 2023
DIAYN (Bysenbach et al., 2019): Diversity is All you need (DIAYN) is competence-based base- line which learn explicit skill w and maximizes the mutual information between states and skills [[w,; Ze). DIAYN decompose the mutual information via [Zz; w.) H(w,) H Hw, 1t). The first term sampled by random prior distribution H(w,) and maximizes the entropy while the
latter term is estimated by the discriminator log ( zz). Intrinsic reward are computed as:
DIAYN r3
log ((w.) 2,) const.
CIC (Laskin et al., 2022): CIC is hybrid data-based and competence-based the previous state- of-the-art method. CIC that uses an intrinsic reward structure based on particle entropy similar to APT and distill behaviors into skills using contrastive learning.
C FULL RESULTS ON THE URLB
The full results of fine-tuning for 100k frames for each task and each method are presented in Table 2 on the state-based URLB. For Disagreement, APT and DIAYN, we utilize the results presented in this Laskin et al. (2021).
Table 2: Full results of pre- tre-training for 2M and fine-tuning for 100k steps on the state- based URLB.
EUCLID
Domain
Tack
Disagremmen
APT
DIAYN
CIC
TDMPCR 100k
(Disagreement wlo MCL
(APT)
(DIAYN)
EUCLID
wio MCL
woo WCL MCL
Flip
491=21
477e16
381=17
631+34
930e28
971=1
967=3
923+34
969
walker
Stund
Run
444=21
344628
860-26
242=11
486=25
940-22
75054
765210
744-9
793+7
770:9
Walk
782+33
661=26
96742
Jump
668:24
462148
578.46
595142
723198
$40113
792-5
807131
858:14
Quadruped
Stand
Run
840m33
161:12
339140
622s57
706:48
415128
505147
761254
465183
765#92
651,35
95326
911m31
589.13
918430
571119
735 16
95825
Walk
721256
434164
406:64
723x43
710277
$741-2
864:21
859433
925
17e5
right
31+4
Reach top leli 117114
112111
1123
15367
17811]
197a23
20127
192110
22515
Reach top right
140
13625
19s4
16384
172124
21927
204-7
19724
Table 2, we find that the best approach to use EUCLID with multi-choice learning and with disagreement as the backbone (EUCLID (Disagreement) with MCL) and we refer to EUCLID for this default setting. EUCLID versions constructed based on each of the explored approaches signif- icantly improve the performance of the corresponding baseline.
D IMPLEMENTATION DETAILS
We provide richer implementation details of BUCLID in this section.
D.1 DEEP DETERMINISTIC POLICY GRADIENT (DDPG)
For continuous action space, we opt for DDPG (Lillicrap et al., 2016) as our base optimization algorithm. DDPG is actor-critic off- off-policy algorithm, which update critics Qo by minimizing the Bellman error:
Lo(
(l [(Oo (st, a.) v(s))
where the Q-target y(s) R(s, a) + 7Qp (Se+1, To (S)+1)), D is a replay buffer and is an slow- moving average of the online critic parameters. At the same time, we learn a policy Te that maxi- mizes Qo by maximizing the objective:
Cx D) = EE,ND (Qo (St, To (s:))
(10)
D.2 HYPER-PANE PARAMETERS
EUCLID studies various categories of exploration algorithms a backbone in the experiments, and we list the individual hyper-parameters of each method in Table 3. As for the world models and policy, most of the parameters remain the same in the original TOLD of TDMPC (Hansen al.,
16

---

Published as a conference paper at ICLR 2023
Table 3: plyper-parameee of the exploration algorithms used in experiments Disagreement hyper-parameter
Value
Ensemble size
5
Forward net
+ - 1024
+ |0| ReLU MLP
APT Hyype-parameter Representation dim
Value
1024
Reward transition
log(r + 1.0)
k in NN
12
Avg top &s in NN
True
DIAYN
Value
Skill dim
16
Skill sampling frequency
Discriminator net
512-+1024
- +1024-11 ReLU MLP
2022). We list the hyper- parameters of EUCLID in Table 4, with particular reference to the fact that some of them change during the pre-training phase (PT) and the fine-tuning phase (FT). Following prior work (Hafner et al., 2020b; Hansen et al., 2022), we use task-specific action repeat hyper- parameter for URLB based on DMControl, which set to 2 by default while 4 for the Quadruped domain. During the pre-training process, we increase the number of policies and prediction heads of the dynamics model one after another based specific snapshot time steps, the detailed parameters are shown the Table 5.
D.3 COMPUTE RESOURCES
We conducted our experiments on an Intel(R) Xeon(R) Platinum 8171M CPU 2.60GHz processor based system. The system consists of processors, each with 26 cores running at 2.60GHz (52 cores in total) with 32KB of LI, 1024 KB of L2, 40MB of unified L3 cache, and 250 GB of memory. Besides, we use a single Nvidia RTX3090 GPU to facilitate the training procedure. The operating system is Ubuntu 16.04. Totally, we conduct experiments on 2500 seeds 15 downstream tasks
E ADDITIONAL ANALYSIS OF ABLATION
110
No Ravers
Normelized Retum (%)
Woln Mode
The results of the complete ablation experiment for each environment are shown in Fig. 10. For the generaliza- tion ability of experiments, the results are based on the average performance of three types of exploration back- bone with 2M steps of pre-training and 100k steps of fine-
80
tuning (A total of 720 runs = algorithms x 12 tasks x 5 seeds x settings).
70
60
Walker
Quadruped
Jac0
Overall
Overall, our default settings, i.c. reusing all components
of the world models (encoder, dynamics model, reward Figure 10: Ablation. Comparison with model) and policies (actor and critic) works best. The agents based on different pre-trained most important modules in EUCLID are dynamics mod- components for fine-tuning. The results els and actor as these two always get a performance gain in all tasks, whereas reusing critic and reward predictor of three types of exploration backbone can improve performance only in some domains.
are based on the average performance
with 2M steps of pre-training.
Using pre-trained policies to initialize downstream policies shows significant performance in Quadruped and Jaco domains, which illustrates the synergistic effect of model based planning and pre-trained policy. Exploration policies trained by intrinsic rewards during the pre-training phase usually exhibit behaviours that beneficial for downstream tasks, e.g. agents in the Quadruped
17

---

Published as a conference paper at ICLR 2023
Table 4: Hyper-parameters of the world model, policy, and planner.
World model
Value
Batch size
1024
Max buffer size
le6
Latent dim
50 (default)
100 (Humanoid)
MLP hidden dim
256 (Encoder)
1024 (otherwise)
MI.P activation
ELU
Optimizer (0)
Adam
Learning rate
1e-4 (PT)
1e-3 (FT)
Reward loss coefficient (c1)
0.5
Consistency loss coefficient (c2)
Value loss coefficient (c3)
0.1
0 update frequency
Policy Seed steps
Value
(PT)
4000 (FT)
Discount factor (r))
0.99
(default)
Action repeat
(Quadruped)
Planning (Only for FT phase)
Value
Iteration
Planning horizon (L)
CEM population size
512
CEM elite fraction
CEM policy fraction (Policy/CEMM)
0.05
CEM Temperature
Table 5: tlyper-parameters of multi-choice learning mechanism
MCL tlyper-paramete
Value
Num of prediction heads (H)
Regularization strength (ax)
0.1
specific interval time steps T
500k
domain trained by the Disagreement can exhibit behaviours such as tumbling and wobbly stand- ing, while randomly initialised agents only fall and shake. Therefore, using such policies to guide the planning process during the fine-tuning phase can help the agents reduce exploration cost and increase sampling efficiency which also demonstrates the benefits of a hybrid strategy.
Since the state transition of the environment are shared during the pre-training and fine-tuning phases, makes sense that pre-trained dynamics model would be more accurate than model initial- ized randomly. Moreover, we empirically find that reusing both the critic and reward predictor at the beginning of FT phase can improve overall performance, although the intrinsic reward which reward predictor predicts in pre-training phase is quite different from the expected extrinsic return given by downstream tasks. Thus, still maintain the reuse of critic and reward predictor by default and leave whether the pre-training of critic and reward predictor can benefit for downstream tasks with theoretical guarantee open question.
PSEUDO CODES OF EUCLID
We show the full process of EUCLID, including both pre-training and fine-tuning phases, in Algo- rithm
18

---

Published as a conference paper at ICLR 2023
Efficient Unsupervised Reinforcement Learning Framework with Multi-choice Dynamies Model (EUCLID)
Algorithm 1:
Input: specific interval time steps T. prediction head size H, regularization strength a, current policy size = O, ensemble policy set S 0.
Require: Initialize all networks: Encoder Ee, Multi-headed dynamics De. Reward predictor
Rg. Critic Qs. Actor Tg. Replay buffer D.
Require: Environment (env), M downstream tasks Tk. k [1....,M, M).
Require: Intrinsic reward ااt extrinsic reward y<xt
Require: pre-train Npr 2M and fine-tune NFT = 100K steps.
# Purt 1 Unsupervised Pre-training
for 1 = 0, N,p.N do
if = T h then
Initialize n with weights of Te. h + h
Extend ensemble policy set S + U n
Update average policy distribution 7 (z.) = Ex= ah (zx) //h
12
Encoder state 2, Ea (8z) and sample action Apply action to the environment 8++1 (. 8;, at) Add transition to replay buffer D + D U (st, a, S(+1)
(zt)
13
14
Sample minibatch from replay buffer DD compute intrinsic reward pit with exploration backbone
15
16
Update encoder, reward predictor, critic and dynamics model parameters with prediction head h
>> see Eq. >> see Eq. 5
17
Update actor o by RL loss with additional diversity encouraging term Purt Supervised Fine-tuning
18
for T' E [Ti...., TM] do
29
Initialize all networks with weights from the pre-training phase and an empty replay buffer
D
21
Select the most appropriate prediction head ht with the highest zero-shot reward
22
Fix head h* and corresponding policy = Initialize fine-tuning actor n.. with weights of mh
24
for 1..Ner do Encoder state 2, Eg (8t) and select action through planning = Plan(z,) guided by
25
Apply action to the environment 1 P-. (1, St, ar)
27
Add transition to replay buffer 2 + D U (st, a,, r2, (s+1) Update encoder, reward predictor, critic and dynamics model parameters with fixed & see Eq. see Eq. 10
28
prediction head h*
Update actor only by RL loss
Evaluate performance of RI. agent on task Tj
G DETAILS OF MULTI-CHOICE LEARNING
To stabilize the training process, fixed the equal number of ensemble policies and prediction heads of the multi-headed dynamics model as hyper-paramedt in advance, and corresponded the policies the prediction heads one by one. Our core intuition is that diverse policies yield diverse data distributions, and we use the different data distributions to train the model primitives separately to obtain set of sub-optimal dynamics models with their specified region. In the beginning of the stage, we initialize an empty set of policies head. We extend ensemble policy set S <- S U a h E 1..H at every specific interval time steps for total 2M pre-training steps where network parameters of Th initialized by the weights of current Tg.
( and dynamics model D, with H prediction
To allow ensemble policies to be as diverse possible, calculate the current average policy distribution The each time we add a member to the ensemble and design policy diversity encour- aging terms DeL. (To (2+) || (2:)) encourage diversity of policy. We balance each policy
19

---

Published as a conference paper at ICLR 2023
of ensemble between the exploration and the diversity objectives that each model primitive models different skill space. After a complete PT phase, we are able to obtain the multi-headed dynamics model and the corresponding set of policies (z
(x)].
In the beginning of FT phase, we should select one head that can benefit the downstream learning the most. We follow the same procedure for competence-base method adaptation in URLB (Laskin et al., 2021). During the first 4k steps, the zero-shot evaluation that we use is to interact with the environment for whole episode by each prediction head and the corresponding pre-trained policy, get the episode extrinsic reward and select the prediction head with the largest reward as the expert. After this, we fix the prediction head h" and finetune world model parameters for the remaining steps.
H DETAILS OF EUCLID MIXTURE PLANNING
Following TDMPC (Hansen et al., 2022), we leverage imaginary trajectories both for planning and policy gradient. As for planning, we perform Model Predictive Control (Bemporad & Morari, 1999) method MPPI (Williams et al., 2015) to update parameters for a family of distributions using an importance weighted average of the estimated top-k sampled trajectories of expected return. Specif- ically, we use the cross entropy method (CEM) (Rubinstein, 1997) to optimize action sequences by iteratively re-sampling action sequences near the best performing sequences from the last iteration. worth noting that we plan at each decision step f and execute only the first action. In addition, long-term model planning computational costly and inaccurate, and model errors accumulate with horizon length (Lai et al., 2020). Therefore, we use short-term reward estimates generated by the learned model and use the Q value function for long-term return estimates via bootstrapping. At the same time, we hope that the pre-trained policy can guide the planning process and improve control performance, For this, we additionally mix the trajectory samples generated by policy with the plan- ning trajectories guarantee the optimization when model not that accurate. The detail pseudo code is given in Algorithm 2.
Algorithm 2: Policy Guided Planning Algorithm
Require: Encoder Eg, Latent dynamics De. Reward predictor Rg. Critic Qs. Actor Tg.
Require: Initial parameters for N (uu,, (a")²).
Require: Planning trajectories num N and policy trajectories num N
Require: Current state st, rollout horizon L, iteration num J, clite trajectories num k. b trajectory starting state
Encoder state 2,- Eg (5:)
# Iterate rounds starting from initial distribution Nie (on),((00) for each iteration j 1,..J do
Sample N trajectories of length L from N' (20-1((0' (a)-1)2)
Sample N₂ trajectories of length L using Da and The Estimating the cumulative discounted rewards for all trajectories, the first L. steps are estimated using the reward predictor Rg and thereafter using Qo
10
11
Select top-k clite trajectories based on cumulative rewards Update μ. a by top-k sampled trajectories using MPPI (Williams et al., 2015)
12
13 return a ~ N (n², (o))(
I ANALYSIS OF MODEL-BASED RL PERSPECTIVE
In the model-based RL community, how to collect data for model fitting is the general challenge. Most of the traditional model-based RL (Zhang et al., 2019; Hafner et al., 2019) approaches use task behavior policies to obtain data and train the world models. Some approaches enhance the breadth of the model by designing additional intrinsic reward-driven goals as auxiliary tasks for learning to ob- tain more diverse samples (Seo et al., 2021: Mazzaglia et al., 2022). However, these approaches still face the problem of poor transferability, which requires complete retraining when faced with similar downstream tasks. And we focus on how to improve the sample efficiency of model-based RL in the perspective of unsupervised RI. and fit general world models for multiple downstream tasks through reasonable reward-free pre-training and fune-nuning paradigms. EUCLID stands for PT and FT
20

---

Published as a conference paper at ICLR 2023
perspective, with multiple choice learning and corresponding policy hoping to train set submodels with separate expert regions and select the most suitable model for the downstream task. In this way, we can reuse the pre-trained model many times, adapting quickly using fewer samples, rather than learning from scratch.
term,
J. ADDITIONAL EXPERIMENTS
J.1 ABLATION OF MULTI-CHOICE LEARNING HUMANOID DOMAIN
SEDE
[gunde Returm
300
Edisode Rutua
Eptoods Roturn
80
200
200
1.00
100
60
30
120
150
%
30
60
90
120
150
30
60
90
120
150
Environment Shops [x101]
Envtronment Steps 1x10
Envirommnnt Stops (x109)
(a) Humanoid-Staan
(b) Humanoid-Wan
(c) Humanoid Run
Figure 11: Learning curves of EUCLID and EUCLID w/o MCL on downstream tasks of the hu- manoid domain. Curves show the mean and 95% confidence intervals of performance across independent seeds. The dashed reference lines are the asymptotic performance of the Disagreement algorithm with 2M PT steps and 2M FT steps.
As shown in Fig. 11, we ablate and evaluate the role of the MCL mechanism in the humanoid domain. Experiments show that the MCL mechanism can achieve stable improvement humanoid domain, which certifies the positive effect of the MCL mechanism in complex reward functions and complex environments.
J.2 STATE PREDICTION ERROR REGRESSION ANALYSIS
To further verify the effectiveness of pre-training and MCL mechanism for dynamics model, we collected ex- pert data for 30 episodes (30000 steps) of quadruped- run tasks. Then, we train a dynamics model on the pre-collected quadruped dataset. In order to ex- clude other factors, we removed the encoder module this experiment, and the prediction loss is defined by
Quadruped-run
12
Pre-toiined
MC,
10
Rundonly
Prediction Erar
[sD, ax) St+ 11 Fig. 12 show the prediction er- ror curves for up to 10000 training iteration, including randomly initialized, pre-trained with and without multi- choice learning using EUCLID.
4
Firstly, find that pre-trained dynamics models have smaller prediction errors than randomly initialization, es- pecially better when using multi-choice learning. Sec- ondly, the pre-trained dynamics models converge more quickly. Thirdly, When the number of training iterations large enough, a single general pre-trained dynamics model (without MCL) and a randomly initialized dynam- ies model eventually converge similar prediction er- ror magnitude, while a multi-headed dynamics model us- ing the MCL mechanism (selecting specific head) signifi- cantly reduces the final prediction error. This indicates that EUCLID is able to predict the dynamics corresponding to the downstream task more accurately.
Training lterations (K10*)
Figure 12: We collect expert data for quadruped-run tasks train dynam- ics model and report the prediction error curves the held-out dataset. Curves show the mean and 95% confidence in- tervals of performance across 5 inde- pendent seeds.
21

---

Published as a conference paper at ICLR 2023
J.3 EXTENSION EXPERIMENTS IN THE PIXEL-BASED URLB
To further demonstrate that EUCLID is a general and effective framework that can significantly accelerate the learning efficiency of downstream tasks through pre-training, we conducted additional pixel-based URLB experiments on four tasks. As for baseline in model-free manner, we opt for DrQ-v2 (Yarats et al., 2021b) (which can be simply viewed as DDPG + Image Augmentation for image inputs) as our base optimization algorithm to learn from images. In the pre-training phase, we consistently use disagreement as the backbone of exploration. The results shown in Table are means over seeds. In the table, TDMPC@ 100k and DrQ-v2@ 100k represent agents with 100k training steps and without pre-training. DrQ-v2 (Disagreement) and EUCLID (Disagreement) represent agents after 500k reward-free pre-training and 100k fine-tuning. The results show that EUCLID can also lead significant performance improvement in visual control tasks.
Table 6: Comparisons on the URLB with image inputs. Results for DrQ-v2@ 100k and DrQ-v2 (Dis- agreement) are obtained from (Laskin al., 2021).
DrQ-v2
EUCLID
Task
DrQ-v2@ 100k
TDMPC@ 100k
(Disagreement)
(Disagreement)
Walker-Flip 81+23
360+11
403+30
6224337
Walker-Run
41+11
131+19
201+15
323+41
Walker-Stand
212+28
398#65
793+40
950=15
Malker-Walk
141#53
348+46
573+43
743+38
22


Reviews:
[{"review_id": "w76ExacO38", "reviewer": "Reviewer_E7J9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces EUCLID, a new paradigm for the unsupervised reinforcement learning task. In the pre-training phase, the paradigm first introduces a multi-choice prediction model to learn the dynamics of the environment. The learned dynamics can adapt the policy to downstream tasks efficiently and effectively. The experimental results show significant improvements over baselines.", "summary_of_the_review": "Overall, I think that the paper is above the accpetance bar considering the idea, difficulty to impletement and the experimental improvements,  but I still have some concerns about the metods details.", "strength_and_weaknesses": "Strength:\n\n1. The motivation is clear and the proposed method make senses.\n2. The idea is novel in this field, although it has been proposed in dynamics generalization task.\n3. The experimentally comparisons are comprehensive and the improvement is impressive.\n\nWeakness:\n\n1.  This paper employs a multi-choice model to learn the environmental dynamics; however, Table 1 demonstrates that the performance difference between with MCL and without MCL is marginal, which raises my concern that MCL is unnecessary for this paradigm. If so, could \"multi-choice\" be removed from the title? Can authors provide further clarification?\n2. According to my knowledge, the scale of rewards in the training phase and the fine-tune phase are vastly different. I am concerned that the scale difference may be greater than the differences between prediction heads in some situations, rendering the MCL ineffective. Can authors provide additional experiment information and observations?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lrxC5ovBuqt", "reviewer": "Reviewer_yoch", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for pretraining an RL agent in a reward-free environment by fitting a dynamics model on the data generated by an exploration policy. The authors position this work in contrast to the body of prior work in unsupervised reinforcement learning (URL) involving model-free policy pretraining, which suffers from what they refer to as the “mismatch problem”: prior URL model-free approaches, like most exploration algorithms, optimize non-stationary reward functions and therefore end up oscillating in state distribution, meaning that transfer performance (which relies on overlap between the state distribution of the pretrained policy and that of the optimal policy for the target task) is highly dependent on the learning update at which the pretraining is stopped. The authors instead propose to pretrain a model-based policy by pretraining a dynamics model on data produced by an exploration policy; to improve the diversity of the pretraining data distribution for the dynamics model, the authors propose to use an ensemble of exploration policies which are additionally incentivized to visit different, and therefore specialize to, subspaces of the latent state space. The authors demonstrate that the pretrained learned dynamics model and policies can be transferred to downstream tasks by using the dynamics model for planning and selecting an expert from the ensemble for guiding the planner. In experiments on the URLB benchmark, they demonstrate that their method outperforms model-free URL baselines, and ablate choices pertaining to the exploration strategy and components of the objective for learning the dynamics model.", "summary_of_the_review": "Overall, this work serves as an interesting case study of a model-based approach on the URLB benchmark. That said, the stated contribution of providing a novel model-fused approach for unsupervised pre-training is mainly positioned with respect to existing model-free approaches in the URL space, even though it ends up being quite related to existing work in model-based RL, and in particular, the challenge of obtaining data for fitting the dynamics model. Moreover, the comparison to transferring e.g. competence based URL methods by learning to control discovered skills is missing, which should serve as a stronger baseline.", "strength_and_weaknesses": "Strengths:\n\nExperiments demonstrate that the resulting pretraining procedure is effective on URLB, compared to baselines. In particular, their approach requires significantly fewer samples under the downstream task.\n\nExperiments demonstrate that unlike the model-free URL pretraining approaches they compare to, the resulting model benefits from more pre-training steps (and data) in a monotonic manner. \n\nThe authors provide a comparison between a significant number of exploration strategies (APT, Disagreement, DIAYN). While this is certainly an added bonus, this also introduces many hyper-parameter tuning challenges. \n\nThe authors provide additional comparisons on tasks based on humanoid, which are typically not considered in such pretraining settings. \nIn general, this work provides a useful case study of the model-based paradigm applied to the URLB benchmark.\n\nWeaknesses:\n\nThe work is mainly positioned with respect to work in URL, but it also relates more broadly to literature in model-based RL, and the general challenge of collecting data for model fitting. It might be more impactful to consider prioritizing the model-based RL perspective and how the proposed method , in which case URLB may serve as one of many benchmarks.\n\nThe authors do not spend much time discussing how the mismatch problem can be circumvented with existing URL approaches, to be effectively transferred to downstream tasks; for example, skill discovery algorithms with discrete skill variables such as DIAYN (Eysenbach et al, 2018) can be interpreted as producing a set of experts which can then be treated as a mixture of experts by a master policy. Such an approach should serve as a baseline. \n\nThe novelty is limited in so far as the approach combines a variant of existing model-based approaches (such as T-CML, Seo et al 2020) with existing exploration strategies.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Uh5CKBTT7E", "reviewer": "Reviewer_486X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a framework for joint pretraining of dynamic model and exploration policy and the adaptation to downstream tasks. The method achieves superior performance compared to prior works with a high sample efficiency. ", "summary_of_the_review": "This work proposes a model-based training framework that leverages dynamic information and skills learnt from the pretraining phase to speed up learning in downstream tasks. The novelty of the framework is limited since the components in this framework are directly taken from the literature, but I think this itself is not a big limitation given the good experimental performance in standard benchmarks.", "strength_and_weaknesses": "Strength\n* The pretraining and finetuning phases, with a specific design of multi-choice dynamics model, is well-motivated. \n* The method achieves SOTA performance on standard benchmarks with high sample efficiency. \n* Rationale behind several design choices, such as using multiple heads for dynamic model training, are empirically supported in experiments. \n\nWeaknesses\n* The method combines multiple components such as MBRL and MCL into a pretraining and finetuning framework. These components are directly taken from the existing literature. While the paper does offer comparisons of which backbones offer the best empirical results, the novelty of the work is limited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U_hqOT_hw7", "reviewer": "Reviewer_wBqh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method called EUCLID for the unsupervised reinforcement learning (URL) setting, that consists of two phases: an unsupervised pretraining phase, and then a downstream task where task-specific rewards are provided. The method uses a model-based pretraining approach, where the model has multiple heads that are trained to be experts on particular subsets of the state space. During finetuning, the model is used in a model-based planning setup to complete downstream tasks. The method is evaluated on state-based tasks from the URLB benchmark. The paper also contributes empirical studies of which unsupervised RL benchmarks improve the performance of EUCLID and ablation studies of the method.", "summary_of_the_review": "In summary, I think this is a nice paradigm for the unsupervised RL setting and includes thorough experimental evaluation and ablations. However, I have a few concerns regarding the novelty and improvements compared to TDMPC, effectiveness of the multi-headed model approach, and clarity of the paper. Therefore I think the paper currently shouldn’t be accepted, but am willing to change my score based on discussion. ", "strength_and_weaknesses": "Strengths:\n- The idea behind the method of the paper is well-motivated, and it conceptually makes sense that a pre-trained dynamics model would be one way to take advantage of an unsupervised pre-training phase.\n- The EUCLID method is agnostic to the choice of intrinsic reward, as the authors point out, and the empirical study on the choice of unsupervised RL method is quite comprehensive.\nWeaknesses:\n- I think that the paper in its current form overclaims the contributions of the multiple-choice dynamics model learning for the final performance. In Table 1, it appears that EUCLID without MCL achieves very similar performance (often within the error bars) as EUCLID with MCL. Although the qualitative results presented show that certain task heads perform better than the other ones, it seems reasonable for that to happen because the heads are specifically trained to be better at specific tasks. But the claim that “This indicates that multi-choice learning reduces the prediction error of the downstream tasks corresponding to local dynamics and obtains better results” doesn’t seem substantiated.\n- I also think that the results for TDMPC @ 100k, which are in the appendix, should be included in Table 1. Because EUCLID uses many of the components of TDMPC, the performance of TDMPC @ 100k is actually quite competitive with EUCLID. \n- The method is only demonstrated on tasks from state, although because of the world model formulated, it seems like it would be readily transferable to learning from pixels. I think it would be helpful to see results on image-based tasks to show the methods’ efficacy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster