COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL
Authors: Xiyao Wang
Ruijie Zheng
Yanchao Sun
Ruonan Jia
Wichayaporn Wongkamjan
Huazhe Xu
Furong Huang
Venue: ICLR 2024 poster (2024)
Keywords: Model-based RL; Model-predictive control; Model rollouts; Model error
Primary area: reinforcement learning

Abstract: Dyna-style model-based reinforcement learning contains two phases: model rollouts to generate sample for policy learning and real environment exploration using current policy for dynamics model learning. However, due to the complex real-world environment, it is inevitable to learn an imperfect dynamics model with model prediction error, which can further mislead policy learning and result in sub-optimal solutions. In this paper, we propose $\texttt{COPlanner}$, a planning-driven framework for model-based methods to address the inaccurately learned dynamics model problem with conservative model rollouts and optimistic environment exploration. $\texttt{COPlanner}$ leverages an uncertainty-aware policy-guided model predictive control (UP-MPC) component to plan for multi-step uncertainty estimation. This estimated uncertainty then serves as a penalty during model rollouts and as a bonus during real environment exploration  respectively, to choose actions. Consequently, $\texttt{COPlanner}$ can avoid model uncertain regions through conservative model rollouts,  thereby alleviating the influence of model error. Simultaneously, it explores high-reward model uncertain regions to reduce model error actively through optimistic real environment exploration. $\texttt{COPlanner}$ is a plug-and-play framework that can be applied to any dyna-style model-based methods. Experimental results on a series of proprioceptive and visual continuous control tasks demonstrate that both sample efficiency and asymptotic performance of strong model-based methods are significantly improved combined with $\texttt{COPlanner}$.

Published as a conference paper at ICLR 2024
COPLANNER: PLAN TO ROLL OUT CONSERVATIVELY BUT TO EXPLORE OPTIMISTICALLY FOR MODEL- BASED RL
Xiyao Wangb Ruijie Zheng' Yanchao Sun
Ruonan Jia Wichayaporn Wongkamjan' Huazhe Xu3.4 Universsity of Maryland, College Park
Furong Huang
3JPMorgan AI Research
"Tsinghua University *Shanghai Qi Zhi Institute (xywang, rzheng12, wwongkam, furongh } @umd. yanchao.sun jpmchase... com jiaruonan9 9 p0 huazhe huazhe_xu@nan xu@mail tsinghua. edu. cn
ABSTRACT
Dyna-style model-based reinforcement learning contains two phases: model roll- outs to generate sample for policy learning and real environment exploration using current policy for dynamics model learning. However, due to the complex real- world environment, it is inevitable to learn an imperfect dynamics model with model prediction error, which can further mislead policy learning and result in sub-optimal solutions. In this paper, we propose COP 1 lanner, a planning saaning--driven framework for model-basee based methods address the inaccurately learned dynam- ics model problem with conservative model rollouts and optimistic environment exploration. COP1 anner leverages uncertainty-aware policy-guided model predictive control (UP-MPC) component to plan for multi-step uncertainty estima- tion. This estimated uncertainty then serves penalty during model rollouts and as a bonus during real environment exploration respectively, to choose actions. Con- sequently, COP lanner can avoid model uncertain regions through conservative model rollouts, thereby alleviating the influence of model error. Simultaneously, it explores high-reward model uncertain regions to reduce model error actively through optimistic real environment exploration. COP lnnlanner is plug-and-play framework that can be applied to any dyna-style model--ased methods. Experi- mental results on series of proprioceptive and visual continuous control tasks demonstrate that both sample efficiency and asymptotic performance of strong model-based methods significantly improved combined with COP lanner. Our code is available at https://giiuub.commmmddd htwp:/huww..oommummmuuuaa-naaaaa -hwagg-lab/COlllaneee
INTRODUCTION
Model-Basse Reinforcement Learning (MBRL. has emerged promising approach improve the sample efficiency of model-free RL methods. Most MBRL methods contain two phases that are alternated during training: 1) the first phase where the agent interacts with the real environment using policy obtain samples for dynamics model learning: 2) the second phase where the learned dynamics model rolls out to generate massive samples for updating the policy. Conse- quently, learning accurate dynamics model is critical the model-ggnerraedd samples with high bias can mislead the policy learning (Deisenroth and Rasmussen 2011 Wu et al. 2022).
Visual Lasks
DreamerV3
COPlanner
Progrio DMC
MBPO
tasks
COPlanner
MujoCo-GYM
MBPO
tasks
COPlanner
Normalized Returns
Figure 1: Mean performance of
COP lanner compared with base- lines across 3 diverse benchmarks.
However, dynamics model errors are inevitable due to the complex real-world environment. Existing methods try to avoid model errors in two main ways. 1) Design different mechanisms such filtering out error-prone samples to mitigate the influence of model errors after model rollouts (Buckman et al.
2018 Yu al. 2020 Pan et al. 2020 Yao et al. 2021). 2) Actively reduce model errors during

---

Published as a conference paper at ICLR 2024
Real sarple buffer
Optimistic
Dynamics Model
Conservat Lve
Env. Interaction
Planner: UP-MPC
Model Rollout
Trojectory y Eutimottonn
Uncersalinty
Uncertsiitt ea Rerard Penelty Inpele
Planning
Sorus
Model
Policy Network (n)
sample buffer
Figure 2: COP1anner anner Framework. The most essential part of COP1 anner is the Uncertainty-aware Policy- Guided MPC (UP-MPC) phase in which we plan trajectories of length H, according to the learned dynamics model and learned policy network to select the action with highest trajectory reward. This UP-MPC phase is implemented differently for the two different purposes: environment exploration v.s. dynamics model rollouts. environment exploration, trajectory reward has an uncertainty bonus term to encourage exploring uncertain regions in the environment. In dynamics model rollouts, trajectory reward, on the contrary. has an uncertainty penalty term to encourage policy learning on confident regions of the learned dynamics model.
real environment interaction through uncertainty- -uuided exploration (Shyam et al. 2019 Ratzlaff al., 2020, Sekar et al., 2020 Mendonca et af.) 202T). While both categories of methods have achieved advancements, each comes with its own set of limitations. For the first category, although these approaches are shown to be empirically effective, they primarily concentrate on estimating uncertainty at the current step, often neglecting the long-term implications that present samples might have on model rollouts. Moreover, post- peoss-rrcessing samples after model rollouts can compromise many model-generated samples discarded or weighted. As for the
rollout efficiency second category, it is intrinsically challenging to achieve low model error and high long-term reward without sacrifieing the sample efficiency by learning exploration policies.
To tackle the aforementioned limitations, we introduce a novel framework, COP 1 lanner, which mitigates the model errors from two aspects: 1) avoid being misled by the existing model errors via conservative model rollouts, and 2) keep reducing the model error via optimistic environment exploration. The two aspects are achieved simultancously by novel uncertainty-aware multi-step planning method, which requires no extra exploration policy training nor additional samples, resulting in stable policy updates and high sample efficiency. COP coplannez is structured around three core components: the Planner, conservative model rollouts, and optimistic environment exploration. In the Planner, we employ an Uncertainty-avare Policy-guided Model Predictive Control (UP-MPC) to forecast future trajectories in terms of selecting actions and to estimate the long-term uncertainty associated with each action. As shown in Figure 2 this long-term uncertainty serves as dual roles. In the model rollouts phase, the uncertainty acts penalty the total planning trajectory, guiding the selection of conservative actions. Conversely, during the model learning phase, it serves as a bonus on the total planning trajectory, steering towards optimistic actions for environment exploration.
Compared to previous methods, COPlanner has the following advantages: (a) COP lanner has higher exploration efficiency, it focuses investigating high-reward uncertain regions to broaden the dynamics model, thereby preventing unnecessary excessive exploration of areas with low rewards. (b) COP 1 Lnnnr has higher model-generated sample utilization rate. Through planning for multi- step model uncertainty estimation, CoPlanner r can prevent model rolled out trajectories from falling into uncertain areas, thereby avoiding model errors before model rollouts and improving the utility of model generated samples.
(c) COPlonnee enjoys an unified policy framework. Unlike previous methods (Sekar et al. 2020 Mendonca et al. 202T) that require training two separate policies for different usage, COP1 anner only requires training single policy and we only change the way model-based planning is utilized, thus improving training efficiency and resolving potential policy distribution mismatches. (d) COP Lnnnr ensures undistracted policy optimization. Notably, COP lanner diverges from existing approaches by not using long-term uncertainty as an intrinsic reward. Instead, the policy's objective remains focused on maximizing environmental rewards, thereby avoiding the introduction of spurious behaviors due to model uncertainty.
Summary of Contributions: (1) We introduce COP coPlanner framework which mitigate the influence of model errors during model rollouts and explore the environment to actively reduce model errors simultaneously by leveraging our proposed uncertainty-awar policy-guided MPC. (2)
2

---

Published as a conference paper at ICLR 2024
COP Lanner is alug-and-play framework that can be applicable to any dyna-style MBRL method. (3) After being integrated with other MBRL baseline methods, COP annnr improves the sample efficiency of these baselines by nearly double. (4) Besides, COPlanner also significantly improves the performance on a suite of proprioceptive and visual control tasks compared with other MBRL. baseline methods (16.9% on proprioceptive DMC, 32.8% on GYM, and 9.6% on visual DMC).
PRELIMINARIES
Model-based reinforcement learning. We consider a Markov Decision Process (MDP) defined by the tuple (S,A, T. Po. r; r,)), where S and A are the state space and action space respectively, Ts''ss, a) is the transition dynamics, Po) the initial state distribution, r(s, a) is the reward function and y is the discount factor. In model-based RL, the transition dynamics T in the real world is unknown, and we aim to construct model (s'ls, a) of transition dynamics and use it to find an optimal policy ı which can maximize the expected sum of discounted rewards,
Σ (r ai) and)
argmax IE -0 ()-a(alog)
Model predictive control. Model predictive control (MPC) has a long history in robotics and control systems (Garcia al. 1989 Qin and Badgwell 2003). MPC find the optimal action through trajectory optimization. Specifically, given the transition dynamics T in the real world, the agent obtains a local solution each step by estimating optimal actions over a finite horizon H (i.e., from t to + H) and executing the first action a, from the computed optimal sequence at time step t:
H Σ Er'r(s,,a)) .8 ~ ((15:-11,0--1
Gt = argmax E
where 7 is typically set to 1. model-based control methods, the transition dynamics T is simulated by the learned dynamics model T' (Chua et al. 2018: Wang and Ba 2019 Hansen et al. (022).
THE COP LANNEN FRAMEWORK
In this section, we will introduce COP1 anner framework. COP lanner consists of three compo- nents: the Planner, conservative model rollouts, and optimistic environment exploration. Within the Planner, we propose using an Uncertainty-aware Policy-guided MPC to predict potential future trajectories when selecting different actions under the current state and estimate the long-tent uncer- tainty associated with each action, which will be introduced in Sec 3.I Depending on the phase, this long-term uncertainty is used further guide the selection of conservative actions for policy learning or optimistic actions for environment exploration which will be introduced in Sec 3.2 and Sec 3.3
3.1 "THE PLANNER": UNCERTAINTY-AWAR POLICY-GUIDED MPC
In this section, we present the core part of our pro- posed framework which is called Uncertainty-avee Policy-guided MPC (UP-MPC). Inspired by MPC, apply the random shooting method (Rao) 2009) to introduce a long-term vision. Specifically, given the current state st, before each interaction with the model or real environment, we first generate an action candidate set containing actions using the (1) dr (2)
Plonnor: UP MPC
Trajectory Reward a Uncerttatnty
Figure 3: The Planner.
...., aj }. Then, for each action candidate, and calculate the reward r, and model uncertainty u for each step. Finally, we select the action according accumulated reward and model uncertainty, (to interact with the learned dynamics for the model rollouts or to interact with the environment for the model learning), as will be discussed in details in Sec 3.2 and See 3.3
policy: a, {a(")
(k)),
perform Hy-step planning
Incorporating model uncertainty is crucial for action selection to compensate for model error. As illustrated in Algorithm we calculate the model uncertainty u through the model disagreement
3

---

Published as a conference paper at ICLR 2024
(Pathak et al. 2019) method. Model disagreement is closely related to model learning and is currently the most common way to estimate model uncertainty in MBRL (Yu et al.) 20200] Kidambi et al.)
2020; Pan et al. 2020 Sekar et al. 2020; Yao et al. 2021: Mendonca et al., 2021D. We train
dynamies model ensemble To {T (T(
T to predict the next state given the current state-action pair (st, dt) input. Utilizing the ensemble, we approximate the model uncertainty by
calculating the variance over predicted states of the different ensemble members. This estimation closely represents the expected information gain (Pathak et al.) 2019h:
u(t:; (z) = - ETT (81, n²)²
A()) (8t; at).
0
See Figure 3 for the illustration of the process. The pseudocode for the Planner, i.e., the UP-MPC process, is summarized in Algorithm
Algorithm 1 The Planner: UP-MPC (T 8, To. K, H, (x)
Require: Policy Tæ. State 8, learned dynamics model To. number of candidates actions K, planning horizon Hp. optimistic/conseraaivee parameter a
Initialize (kk)
for L..., K, so
(k)
for L,.... K
2: for k = 1 to K do
3: for to Hp do
4: Sample at
(-1
5: Rollout dynamics model ri A-11s
(k)
8 (k)
~ To( ,a, (
6: Compute model uncertainty ut (R) according to Eq.
7: R(() (k)) (k) + am,
8: Select k- = arg 11..,=1,..... (k))
(k*) 9: return 0g
Although in Algorithm [ model uncertainty u is implemented through model disagreement, our proposed UP-MPC is generic framework, any method for calculating intrinsic rewards to encourage exploration can be embedded into our framework for computing u. In Appendix D.5 we provide an ablation study of uncertainty estimation methods further illustrate this point.
3.2 CONSERVATIVE MODEL ROLLOUTS
In model-based RL, due to the limited samples available for model learning, model prediction errors inevitable. If a policy trained using model-generated samples with large error, these samples will not provide correct gradient and may mislead the policy update. Previous methods estimate the model uncertainty of each sample after generation and re-weight discarded samples with high uncertainty. However, re- weightitnn samples based on uncertainty still leads to samples with high uncertainty participating in the policy learning process, while filtering requires manually setting an uncertainty threshold, and determining the optimal threshold is difficult. Discarding too many samples can result in inefficient rollouts.
We apply our Planner to plan for maximizing the future reward while minimizing the model uncer- tainty during model rollouts before executing the action. After calculating the reward and model uncertainty for the Hp-step trajectories of K action candidates (line 5 and 6 in Algorithm IT. Mathematically, we select the action according to Eq. 4 to interact with the model for model rollouts:
replace a -a,, for a positive a > 0 at line in Algorithm!] [
a argmax,, ((st, at) > r(Se4s, ($)..)) a Mi u(Ser,, (Sce.)))) Se1s ~ T(- T(11814111, 1,41
(4)
The negative -ae is coefficient that adds the model uncertainty penalty term to the trajectory total reward. By employing this approach, we can prevent model rollout trajectories from falling into model-uncertain regions while obtaining samples with higher rewards.
3.3 OPTIMISTIC ENVIRONMENT EXPLORATION
In addition to model rollouts, another crucial part of MBRL is interacting with the real environment to obtain samples to improve the dynamics model. Since the main purpose of MBRL is improve
4

---

Published as a conference paper at ICLR 2024
sample efficiency, should acquire more meaningful samples for improving the dynamics model within a limited number of interactions. Therefore, unlike previous methods that merely aimed to thoroughly explore the environment to obtain comprehensive model (Shyam et al. 2019 Ratzlaff et al. 2020, Sekar et al., 2020), we do not expect the dynamics model to learn all samples in the state space. This because many low-reward samples do not contribute to policy improvement. Instead, we hope to obtain samples with both high rewards and high model uncertainty to sufficiently expand the model and reduce model uncertainty.
Similar to model rollouts, we also employ our Planner in the process of selecting actions when interacting with the environment. However, the difference lies in that we replace positive a, > 0 at line 7 in Algorithm ! Mathematically, we choose the action with both high cumulative rewards and model uncertainty according to Eq. 5 which is symmetric form of Eq. [ a, is hyperparameter to balance the reward and exploration. Such an action can guide the trajectory towards regions with high rewards and model uncertainty in the real environment, thereby effectively expanding the learned dynamics model.
ao.. for a
argmax (1(8).0)) E r(sc4,, (8i+s)) E u(se4s, (((tt))) .84t ~ T-1
In summary. by simultaneously using conservative model rollouts and optimistic environment explo- ration, COPlannes effectively alleviates the model error problem in MBRL. As we will show in Section 5 this is of great help in improving the sample efficiency and performance. The pseudocode of COP r is shown A and a more detailed figure shown in Appendix A Very importantly, COP COPlaa Lanner achieves both conservative model rollouts and optimistic environment ex- ploration using a single policy. Different from prior exploration methods, the policy that COP 1 lanner learns does not have to be an "exploration" policy which is inevitably suboptimal.
Algorithm 2 Main Algorithm: COPlanner
Require:
Interaction epochs I, rollout horizon Hy, planning horizon Hp. number of candidates actions K, conservative rate ae. optimistic rate a, Initialize policy Tg. dynamics model F, real sample buffer De, model sample buffer D,n
2: for / epochs do
3: while not Done do
4: Select action a = UP-MPC To, K, Hp: (2))
5: Execute in real environment, add (st, at, rt, 8(41) to De
6:
Train dynamics model To with D. for M model rollouts do Sample initial states from real sample buffer D. for = 0 to H, do
7:
8:
9:
10:
Select action d UP-MPC (T, 8h To, K, Hp: Oie)
11:
Rollout learned dynamics model and add to D
Update current policy To with Dm
RELATED WORK
Mitigating model error by improving rollout strategies. Prior methods primarily focus using
dynamics model ensembles (Kurutach et al. (Chua et al.) 2018)) to assess model uncertainty of samples
after they were generated by the model, and then apply weighting techniques (Buckman al. 2018
Yao et al. 202T), penalties (Kidambi et al. 2020 Yu et al. 2020 or filtering (Pan et al. 2020 Wang ct al. (022) y those high uncertainty samples to mitigate the influence of model error. These methods only quantify uncertainty after generating the samples and since their uncertainty metrics are based on the current step and are myopic, these metrics can not evaluate the potential influence of the current sample on future trajectories. Therefore, they fail to prevent the trajectories, which is generated through model rollout on the current policy, from entering high uncertainty regions, eventually leading to failed policy update. Wu et al. (Wu et al. 2022) proposed Plan to Predict (P2P), which reverses model,
the roles of the model and policy during model learning to learn an sesertainty-foreseeeee aiming to avoid model uncertain regions during model rollouts. Combined with MPC, their method achieved promising results. However, their approach lacks effective exploration of the environment. Branched rollout (Janner et al. 2019) and bidirectional rollout (Lai et al., 2020) take advantage of
5

---

Published as a conference paper at ICLR 2024
small model errors in the early stages of rollouts and uses shorter rollout horizons to avoid model errors, but these approaches limit the planning capabilities of the learned dynamics model. Besides, different model learning objectives (Shen et al. 2020 Eysenbach et al., [022 Wang et al. 2203
Zheng et al. 2023b) are designed to solve objective mismatch (Lambert et al. 2020) in model-based
RL and further mitigate model error during model rollouts.
Reducing model error by improving environment exploration. Another approach to mitigate model error is expand the dynamics model by obtaining more diverse samples through exploration during interactions with the environment. However, previous methods mostly focused on pure exploration, i.e., how to make the dynamics model learn more comprehensively (Lowrey et al. 2018
Shyam et al., 2019 Ratzlaff et al. 2020; [Sekar et al. 2020; [Seyde et al., [220;] Ball et al. 2020
Mendonea et al. 2021 Hu et al. 2023). complex environments, thoroughly exploring the entire environment is very samfleninefhiient and not practical in real-world applications. Moreover, using pure exploration to expand the model may lead to the discovery of many low-reward samples, which are not very useful for policy learning. Mendonca et al. (Mendonca et al. 2021) proposed Latent Explorer Achiever (LEXA) which involves a explorer for exploring the environment and one achiever for solving diverse tasks based collected samples, but the explorer and achiever may experience policy distribution shift under specific single-task settings, causing the achiever to potentially not converge to the optimal solution.
Mitigating model error from both sides. One most relevant work is Mosel-Ensemble Exploration and Exploitation (MEEE) (Yao et al. 2021) which simultaneously expands the dynamics model and reduces the impact of model error during model rollouts. During the rollout process, it uses uncertainty to weight the loss calculated for each sample to update the policy and the critic. Before interacting with the environment, they first generate k action candidates and then select the action with the highest sum of Q-value and one-step model uncertainty to execute. However, as we mentioned earlier, weighting samples cannot fundamentally prevent the impact of model errors on policy learning, and may still mislead policy updates. Moreover, since the one-step prediction error of dynamics models is often small (Pan et al. 2020)), relying only on the sum of Q-values and one-step model uncertainty may not effectively differentiate action candidates. As a result, samples collected during interactions with the environment might not efficiently expand the model.
EXPERIMENT
In this section, we conduct experiments to investigate following questions: (a) Can COP 1 lanner be applied to both proprioceptive control MBRL and visual control MBRL methods, to improve their sample efficiency and asymptotic performance? (b) How does each component of COP l laner impact the performance? (c) How does lanner influence model learning and model rollouts?
5.1 EXPERIMENT ON PROPRIOCEPTIVE CONTROL TASKS
Baselines: In this section, conduct experiments to demonstrate the effectiveness of COP lanner on proprioceptive control MBRL methods. We combine COPlanner with MBPO (Janner et al.. 2019), the most classic method in proprioceptive control dyna-style MBRL, and we name the combined method as COPlamer--MBPO. The implementation details can be found in Appendix B Consequently, MBPO naturally becomes one of our baselines. The other three baselines are P2P- MPC (Wu et al. 2022). MEEE (Yao et al. 2021). and M2AC (Pan et al. 2020).. These three methods also aim to mitigate the impact of model errors in model-based RL. We choose one of the most popular model-fre RL mtheod D4PG (Barth-Maron et al. 2018) as another baseline. More details of MPP.MPC, MEEE, and M2AC can found in Section We also provide comparison with more proprioceptive control MBRI methods in Appendix D.1
Environment and hyperparameter settings: We conduct experiments on proprioceptive continu- ous control tasks of DeepMind Control (DMC) and 4 proprioceptive control tasks of MuJoCo-GYa (GYM). MBPO trains an ensemble of networks as the dynamics model while using the Soft Actor-Critic (SAC) as the policy network. In COPlamer-MMBPO, we adopt the setting of MBPO and directly use the dynamics model ensemble to calculate model uncertainty for action selection in Policy- Guided MPC. For hyperparameter setting, we set optimistic rate a to be 1, conservative rate a, to be 2 in all tasks, and set action candidate number K, planning horizon H,, equal to 5 in all tasks. The specific setting shown in the Appendix C.1

---

Published as a conference paper at ICLR 2024
DMC
3005
2-4
28
16:6
24
154
1686
225
1086
2-4
28h
10:00
153
GiTe: Hroppe
264
Papa
1X0
16%6
606
Vielber2d
1204
406
12%
15.0
5/6
2005
15.6
206
4:00
4003
20000
Exoe
1430
1003
-56
605
-23-
1306
15.0c 300
2004
1506
2005
5.56
2000
15.06
F22 MMCC
MBPO
NG2AC DAPG
CDPlansen--RPPP
Figure 4: Experiment results of COPlaneer-MBPO and other five baselines on proprioceptive control environments. The curves in the first eight figures originate from DM Control tasks, while those in the last four are from GYM tasks. The results are averaged over 8 random seeds, and shaded regions correspond to the 95% confidence interval among seeds. During evaluation, for each seed of each method, test for up to 1000 steps in the test environment and perform 10 evaluations obtain an average value. The evaluation interval is every 1000 environment steps.
CoPlanner significantly improves the sample efficiency and performance of MBPO: Through the results in Figure we can find that both sample efficiency and performance of MBPO have a significant improvement after combining COPlannes control DMC, the sample efficiency is improved by 40% on average compared to MBPO. For example, in the Walker-walk task. MBPO requires 100k steps for the performance to reach 700, while COPlaneer-MBPO only needs approximately 60k steps. In more complex GYM tasks, the improvement brought by CCP CCPlannee lanner is even significant. Compared to MBPO, the sample efficiency of COPlanner-MBI MBBPO has almost doubled. (b) Performance: From the performance perspective, shown in Figure the performance of MBPO has improved by 16.9% after combining COP CoPlanner: Moreover, it is worth noting that our method successfully solves the Walker-run task, which MBPO fails to address, further demonstrating the effectiveness of our proposed framework. In GYM tasks, the average performance at convergence has increased by 32.8%. Besides, COPlanner- MBPO also outperforms other baselines.
(a) Sample efficiency: In proprioceptive
5.2 EXPERIMENT ON VISUAL CONTROL TASKS
Baselines: We conduct experiments to demonstrate the effectiveness of our proposed framework on visual control environments. We integrate our algorithm with DreamerV3 (Hafner et al. 2023). the state-of-the-art Dyna-style model-based RL approach recently introduced for visual control. The implementation details can be found in Appendix B We choose LEXA (Mendonca al. 2021 as our another baseline. LEXA uses Plan2Explore (Sckar et al. 2020) as intrinsic reward to explore the environment and learn world model, then using this model train a policy to solve diverse tasks such as goal achieving. Since pure exploration base on Plan2Explore is sample inefficient for model learning when solving specific tasks, we use the real reward provided by environment as extrinsic reward and add it to intrinsic reward provided by Plan2Explore to train the explorer. We adopt this method on DreamerV3 and call it wew.----wwrd-DDeee method with the SOTA model-free visual RL method DrQV2 (Yarats et al. (021). We also provide comparison with more visual control MFRL and MBRL methods in Appendix D.2
Besides, we compare our
Environment and hyperparameter settings: We use 8 visual control tasks of DMC as en- vironment. In COPlanne--Dreamerr3 we learn a latent one-step prediction dynamics model as Plan2Explore (Sekar et al. 2020), the ensemble size is 8. We set action candidate number K and planning horizon H. equal to in all tasks. For optimistic rate a and conservative rate a., we set them to be and 0.5, respectively. Other hyperparameters remain consistent with DreamerV3 paper.
COP calanner significantly improves the sample efficiency and performance of DreamerV3: From the experiment results in Figure 5 observe that COPlanner- OOPlanner-Dreamery. improves the
7

---

Published as a conference paper at ICLR 2024
Hopper-stand
Hopper-hop
Quadrrupes-enk
Quadruped-uu
780
200
2008
2000
1306
T0l
1006
500 Rergenturn-eend
400
Acrobot-ssing swingup
130
Cartpole-seinggp--aa spaare
100
Firgenturn-eer easy
13:0
300
400
2008
2006
75.3
2:00
2000
750
2,00 1508
200
7500
COP annen- Greamerr3
Dreament
DrQ
Figure 5: Experiment results of SuPlanner-Dreamerr and other three baselines on pixel-input DMC. The results are averaged over 8 random seeds, and shaded regions correspond to the 95% confidence interval among seeds. During evaluation, for each seed of cach method, we test for up to 1000 steps in the test environment and perform 10 evaluations to obtain an average value. The evaluation interval every 1000 environment steps.
sample efficiency and performance significantly over D3eamerV3, and it demonstrated significant advantage over DrQv2. The sample efficiency of COPlanner-DreamerV: is more than twice that of Dreamer V3, and the performance is improved by 9.6%. After adding real reward as extrinsic reward for explorer learning, AereAA-dd---raanetee in most environments. It outperforms DreamerV3 in Cartpole-swingup--parss and Hopper-stand. However, its performance and sample efficiency are still worse than COPlanner- Maanner-DreameV3, further indicates the effectiveness of COP lanner..
delivers performance comparable to Dreamer
5.3 ABLATION STUDY
In this section, we aim to investigate the impact of different components within COP lanner on the sample efficiency and performance. We conduct experiments on two proprioceptive control DMC tasks (Walker-stand and Walker-run) using MBPO as baseline and two visual control DMC tasks (Hopper-hop and Cartpole-swinuup-ee demonstrated Figure 6 Due to page limitations, we provide the ablation study on various hyperparameters of COP lanner in Appendix an4 the ablation study of uncertainty estimation methods Appendix D.3
with DreamerV3 as baseline. The results are
From this ablation study, we can see that effectively combining optimistic exploration and conservative rollouts is necessary to achieve the best results. We find that when only using optimistic exploration (COPlanner w. Explore only), the sample efficiency and performance in all tasks are significantly improved, which highlights the importance of expanding the model. When only using conservative rollouts (COPlanner Rollout only), there is some improvement in sample efficiency and performance but to a lesser extent. In more complex visual control tasks, only using conservative rollouts may lead to over-conservatism, resulting in an inability learn effective policy in sparse reward environments (as observed with a broken seed in Sartpol--singuu-spaae decrease in sample efficiency during the early stages of learning (Hopper-hop). This is reasonable because conservative rollouts may avoid high uncertainty and high reward areas to ensure the stability of policy updates. Moreover, without efficiently expanding the model, it challenging to find better solutions using only conservative rollouts in complex visual control tasks. Experimental results show that both optimistic exploration and conservative rollouts are crucial, and using either one individually can lead to an improvement in performance. When combining the two (as in COPlanner), we can achieve the best results, further demonstrating the effectiveness of our method.
or
5.4 MODEL ERROR AND ROLLOUT UNCERTAINTY ANALYSIS
In this section, we will investigate the impact of COP lanner on model learning and model roll- outs. We provide the curves of how model prediction eгтor and rollout uncertainty change as the environment step increases in Figure Z We conduct experiments two proprioceptive control DMC tasks (Cheetah-run and Walker-run) and two visual control DMC tasks (Hopper-hop and Carpple-wwingup-spars))..
8

---

Published as a conference paper at ICLR 2024
Wolker-stand (proprio)
Walker-un (preprio)
Cartpole-swinggeee strpupp-swiiguuppeeeeeeeee (visual)
Hopper-hop (visual)
000
Setume
+03
300
400
234
50%
756
100k
2200%
100k
2008
25.8
200h
500k
Steps
COPlenner
COPtanner w. Explore only
CORanner Rollout only
MBRL Baseline
Figure 6: Ablation studies of optimistic exploration and conservative rollouts on different tasks using different mbrl baselines. In the first two proprioceptive control tasks we use MBPO as baseline. For the last two visual control tasks we employ DreamerV3. The results are averaged over random seeds. We can observe that the best results are achieved when combining optimistic exploration and conservative rollouts. The benefit is more pronounced in more-challenging visual tasks.
Chewtahhou
Walbar-run
Hopper-hop
Cur pop.e--iinguu-eeeee.
Nommen Drssmnn
$4:50
Proua Sace
5:300
4.2%
confunn HEFO
Strpe
180h
18
2003
2.01
19/6
3003 Cortpole swinngu sparse
35.5
1V.
Choctah run
Whlko run
Hoppet hop
2250
1222
23000
28375
20350
032
Steps
2004
150cc
304
J004
25%
3004 10:20
Figure 7: Model learning loss and rollout uncertainty curves for COP 1 anner and two other model-
based RL baselines. The left four proprioceptive control DMC tasks, and the right four are visual control DMC tasks.
(1) In proprioceptive control DMC, we use the MSE loss between model prediction and ground truth next state to evaluate model prediction error during training, while in visual control DMC, we use the KL divergence between the latent dynamics prediction and the next stochastic representa- tion to compute latent model prediction error. We observe that after integrating COP Lanner in proprioceptive control tasks, the model prediction error is significantly reduced. In more complex visual control tasks, due to obtaining more diverse samples through exploration in the early stages of training, the model prediction error of COP anner higher than the baseline (DreamerV3). However, training progresses, the model prediction error rapidly decreases, becoming significantly lower than the model prediction error of DrcamerV3, This allows the model to fully learn from the diverse samples, leading to improvement in policy performance. (2) For the evaluation of rollouts uncertainty, we calculate the model disagreement for each sample in the model -gnerraed replay buffer used for policy training using dynamics model ensemble. We find that COP 1 anner significantly reduces rollout uncertainty due to conservative rollouts, suggesting that the impact of model errors on policy learning minimized. This experiment further demonstrates that the success of COP 1 Lanner is attributed to both optimistic exploration and conservative rollouts.
6 CONCLUSION AND DISCUSSION
We investigate how to effectively address the inaccurate learned dynamics model problem in MBRL. We propose COP 1 anner, a general framework that can be applied to any dyna-style MBRL method. COP Lanner utilizes Uncertainty- tainty of future steps and symmetrically uses this uncertainty as a penalty or bonus to select actions for conservative model rollouts or optimistic environment exploration. In this way, COP CoPlanner can avoid model uncertain areas before model rollouts to minimize the impact of model error, while also exploring high-reward model-uncertain areas in the environment to expand the model and reduce model error. Experiments on a range of continuous control tasks demonstrates the effectiveness of our method. One drawback of COP lanner is that MPC can lead to additional computational time and we provide a detailed computational time consumption in Appendix D.6] We can improve computational efficiency by parallelizing planning, which we leave for future work.
Policy-Guided MPC phase to predict the cumulative uncer-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Wang, Zheng, and Huang are supported by National Science Foundation NSF-11S-2142776 FAI, DOD-ONR-Office of Naval Research under award number N00014-22-1-2335, DOD-AFOSR-Air Force Office of Scientific Research under award number 149550-233110044 ROD-DARPA-Defemt Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD) HRO01 20200077, Adobe, Capital One and JP Morgan faculty fellowships.
REFERENCES
Philip Ball, Jack Ready policy one: World building through active learning. In International Conference on Machine Learning, pages 591-601. PMLR, 2020.
Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts.
Gabriel
Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv: 08617, 2018.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee, Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-11), pages 465-472, 2011.
Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, and Russ R Salakhutdinov. Mismatched no more: Joint model-policy optimization for model- rl. Advances in Neural Information Processing Systems, 35:23230-2333 2022.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and practice-a a survey. Automatica, 25(3):335-348, 1989.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2566 PMLR, 2019b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0411 2023.
Nicklas Hansen, Xiaolong Wang. and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203. 04955, 2022.
Edward S Hu, Richard Chang, Olch Rybkin, and Dinesh Jayaraman. Planning goals for exploration. arXiv preprint 0r3iv:2033.33000 2023.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-basel policy optimization, Advances in neural information processing systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2024
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning, pages 5618-5627. PMLR, 2020.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. arXiv preprint arXivv2002.04523 2020.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint wrXiv:/811.01844 2018.
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Piers Berges, Tingfan Wu, Jay Vakil, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htLlattts///ppeeeeeii net 2 i dd= 6gI.
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 34: 24379-24391, 2021.
Andrew S Morgan, Daljeet Nandha, Georgia Chalvatzaki, Carlo D' D'Eamo, Aaron M Dollar, and Jan Peters. Model predictive actor-critic: Accelerating robot skill acquisition with deep reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6672-6678. IEFE, 2021.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference Robot Learning, 2022. net/
URL
feee/forrue forum Gbpgg
Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the model when it is confident: Masked model-based actor-critic. Advances in neural information processing systems, 33:10537-10546, 2020.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference machine learning, pages 5062-5071. PMLR, 2019.
S Joe Qin and Thomas A Badgwell. survey of industrial model predictive control technology.
Control engineering practice, 11(7):733-764, 2003.
Anil V Rao. A survey of numerical methods for optimal control. Advances in the Astronautical Sciences, 135(1):497-528 2009.
Neale Ratzlaff, Qinxun Bai, Li Fuxin, and Wei Xu. Implicit generative modeling for efficient exploration. In International Conference on Machine Learning, pages 7985-7995. PMLR, 2020.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 8583-8592. PMLR, 2020.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. International Conference on Machine Learning, pages 9443-9454. PMLR, 2021.
Tim Seyde, Wilko Schwarting, Sertac Karaman, and Daniela Rus. Learning plan optimistically: Uncertainty-guider deep exploration via latent model ensembles. arXiv preprint arXiv:2010. 14641, 2020.
11

---

Published as a conference paper at ICLR 2024
Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-basee policy optimization with unsu- pervised model adaptation. Advances in Neural Information Processing Systems, 33:2823-2833 2020.
Pranav Shyam, Wojciech Jaškowski, and Faustino Gomez. Model-based active exploration. In International conference on machine learning, pages 5779-5788. PMLR, 2019.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv:1966.0880 2019.
Xiyao Wang, Wichayapor Wongkamjan, Ruonan Jia, and Furong Huang. Live in the moment: Learning dynamics model adapted to evolving policy. In International Conference Machine Learning. PMLR, 2023.
Zhihai Wang, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Samplc-efficient reinforcement learning via conservative model-based actor-critic. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8612-8620, 2022.
Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, and Hankz Hankui Zhuo Plan to predict: Learning an uncertainty-foreseeing model for masel-based reinforcement learning. Advances in Neural Information Processing Systems, 35:15849-1588 2022.
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Tianying Ji Zhecheng Yuan, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning Representations, 2024. URL https
//operrvvew..o net Ett.
Yao Yao, Xiao, Zhicheng An, Wanpeng Zhang, and Dijun Luo. Sample efficient reinforcement learning via model-ensemble exploration and exploitation. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 4202-4208. IEEE, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint 1rXiv:2107.0940 00945, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129 11114129-1412,, 2020.
Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E Gonzalez, and Stuart Russell. Made: Exploration via maximizing deviation from explored regions. Advances in Neural Information Processing Systems, 34:9663- 0449666-9680,, 2021.
Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. TACO: Temporal latent action-driven contrastive loss for visual reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https://cpenreviie
net forum 2dd
Ruijie Zheng, Xiyao Wang, Huazhe Xu, and Furong Huang. Is model ensemble necessary? model- based rl via a single model with lipschitz regularized value function. In International Conference on Learning Representations, 2023b.
Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Shuang Ma, Hal Daumé III, Huazhe Xu, John Langford, Praveen Palanisamy, Kalyan Shankar Basu, and Furong Huang. Premier-taco: Pretraining multitask representation via temporal action-driven contrastive loss. arXiv preprint 2020:2402..06887, 2024.
12

---

Dynamics Model Rollout Published as a conference paper at ICLR 2024
Appendix
A DETAILED FIGURE OF COPLANNER
We present more detailed figure to illustrate our COPlanne framework. During environment exploration, we first choose action using UP-MPC with multi-step uncertainty bonus, then interact with the real environment to obtain real samples for dynamics model learning. In dynamics model rollouts, at each rollout step, select the actions using UP-MPC with multi-step uncertainty penalty to avoid model uncertain regions and interact with the learned dynamics model to get model-generated samples to update the policy.
Real sample buffer
Real Enviromment
Dynamics Model
UP-MPC phase: a x(s)
Trajectory reward with uncertainty bonds
penalty
%
dg
dg-1
Model sample buffer
Policy Network (rt)
Environment Interaction Dynamics Model Planning
Selected action for Environment Interaction
Selected action for Model Rollout
Figure 8: Figure illustration of COP lanner framework with more details.
B IMPLEMENTATION
COP lanner framework is versatile and applicable to any dyna-style MBRL algorithm. In this section, we are going to introduce implementation of two algorithms we used for experiment in Section 5 COPIanner-MBPO for proprioceptive control and COPlanne--Dreamerk, for visual control.
B. COPLANNER-MO MBPO
MBPO (Janner al. 2019) trains ensemble of probabilistic neural networks (Chua et al. 2018 as dynamics model. utilises negative log-like log-likelihood loss to update each network in the ensemble:
(()) Σi' [ug( (s an) 5n+1
T
3 (ss, an) 1u (sn, a.) 8n+1| log det ( an
(6)
For the policy component, MBPO adopts soft actor-critic (Haarnoja et al. 2018). We combine COPlannes with MBPO, the pseudocode is shown in Algorithm
B.2 COPLANNE--DREAMENT
DreamerV3 (Hafner et al. 2023 is a dyna-style MBRL method that solves long-horizon tasks from visual inputs purely by latent imagination. Its world model consists of image encoder, Recurrent
13

---

Published as a conference paper at ICLR 2024
Algorithm 3 COPlanner-MBPO
Require:
interaction epochs I, rollout horizon Hy, planning horizon Hp. number of candidates actions K, conservative rate a,, optimistic rate 0s
1: Initialize policy Th. dynamics model ensemble Ta = {T) To). real sample buffer De, model sample buffer Pm
2: for epochs do
3: fort= to T do
4: // Optiiissic environment exploration
5:
Select action with optimistic rate a, = UP-MPC 8t, To, K, Hp, 2o)
6: Interact with the real environment with a,, add real sample (se, ae. Tes 8t41) to real sample buffer D.
7:
Train dynamics model To via Equation 6 for M model rollouts do Sample initial rollout states from real sample buffer D.
8:
9:
10:
for = 0 to H, do
11:
// Conservative model rollouts
12:
ân UP-MPC (T&, $/. To, , K, Hp. -ae) (Select action with conservative rate), roll-
out learned dynamics model and add to model sample buffer D,,.
13:
for G gradient updates do
14:
Update current policy Hs using model-ggnerated samples from model sample buffer D.
State-Space Model (RSSM) (Hafner et al. 2019b) to learn the dynamics, and predictors for the image, reward, and discount factor. The world model components are:
Recurrent model:
h4 fo 21-1-111111100 ai-1)
Representation model:
2t ~ To (0(lile x2)
Transition predictor:
2. (2)
Image predictor:
3. ~ por (1)
Reward predictor:
P. p the 2+)
Discount predictor:
îe ~ (
where the recurrent model, the representation model, and the transition predictor components of RSSM. The loss function for the world model learning is:
C(Φ) (m
Impo(xelhua 2x) Inpo(relhn,,)) 21) Impo(Telhu,,)
(7)
+ 3ymazin (1, (Ll
3amar (1, (2c (2/l
where means stop gradient. Besides, DreamerV3 also use actor-critic framework as their policy. In particular, they leverage stochastic actor that chooses actions and deterministic critic. The actor and critic are trained cooperatively. The actor goal is to output actions leading to states that maximize the critic output, while the critic aims to accurately estimate the sum of future rewards that the actor can achieve from each imagined state (or model rollout state), For more training details about DreamerV3, please refer to their original paper (Hafner et al. 2023).
To estimate model uncertainty in COPlanner-Dreameee models 73 { .... Tyj J. each of these models takes latent stochastic state 7; and action a, as input and predicts the next latent deterministic recurrent states h2. The ensemble is trained using MSE loss. During model rollouts, model ensemble to evaluate the uncertainty of sample at each rollout step. Here we provide the pseudocode of COPlanner DreamerV3 COPlanner-Dreamerry: in Algorithm
we train an ensemble of one-step predictive
the world model to generate trajectories, and the step
14

---

Published as a conference paper at ICLR 2024
Algorithm 4 COPlanner--reemmerV3
Require: Rollout horizon Hr, planning horizon Hp. number of candidates actions K, conservative rate Oeey optimistic rate at,
1: Initialize real sample buffer D. with S random seed episodes.
2: Initialize policy Ti, critic vgg one-step model ensemble T. {Tg . ..., T3 world model parame-
ter Φ
3: while not converged do
4: for update step = 1..C do
5:
Ix. Draw data sequences I(, re)l= De 6: Compute latent stochastic states 2t g0(2; hy. 24)
7:
Update world model parameter & via Equation // Conservative model rollouts
8:
9:
Imagine trajectories
{(zr, (,,,,,))==*
from each
with
UP-MPC( TT, 2r, To, K, Hp: -0a)
Update 'gg and Th using imagined trajectories. for time step t 1.IT do Compute 2, ~ q (0(2)(/,,zz)
11:
12:
13:
// Optimistic environment exploration
14:
Select action a UP-MPC pT. 2t, To. K, Hp: an)- Interact with the real environment and obatin (It, aes + 1) Add experience to D. + D. {IT, az, Tt. 2e + 1)]t=1
15:
16:
C HYPERPARAMETERS
In this section, we provide the specific parameters used in each task in our experiments.
Table 1: Hyperparameters of COPIanner-MBPO on proprioceptive control DMC.
Parameter
Value
Conservative rate ae
2
Optimistic rate o
Action candidate number K
5
Planning horizon H, Real ratio
0.5 Reacher-xx
0.8 Finger-spin
0 Others
Rollout horizon H,
[20, 150, 1, 1] Finger-spin
[20, 150, 1, 4] Others
C.I PROPRIOCHPTIVE CONTROL DMC AND McJoCo
We COPlanner-MBPO in all proprioceptive control tasks, For the dynamics model ensemble, adopted the same setup as MBPO (Janner et al.) 2019) original paper, with an ensemble size of 7 and an elite number of which means each time we select the best five of seven neural networks for model rollouts. Fach network in the ensemble is MIP with 4 hidden layers of size 200, using ReLU the activation function. We train the dynamics model every 250 interaction steps with the environment. The actor and critic structures control DMC, the hidden layer size of actor and critic is 512, and updated 10 times each environment step, while in MuJoCo the hidden layer size is 256, and they updated 20 times each environment step. The batch size for model training and policy training is both 256. The learning rate for model training is 1e-3, while the learning rate for policy training 3e-4.
both MLP with 4 hidden layers. In proprioceptive
In MBPO, the authors use samples from both the real sample buffer and the model sample buffer to train the policy, and the ratio of the two referred to the real ratio. In addition, MBPO has unique
15

---

Published as a conference paper at ICLR 2024
mechanism for the rollout horizon H,, which linearly increases with the increase of environment epochs, with each environment epoch including 1000 environment steps. [a, b, 2; yl denotes a thresholded linear function, d.e. epoch e, rollout horizon is h min( max( + _ (v/ z),
The settings for conservative rate des optimistic rate ao. action candidate number K, planning horizon Hp and the above two parameters in different environments are provided in Table and
Table 2: Hyperparameters of COPlanner-MBPO on MuJoCo.
Parameter
Value
Conservative rate ae
2
Optimistic rate a
Action candidate number K
5
Planning horizon Hp Real ratio Rollout horizon Hr,
5
0.05
[20, 100, 1, 4] Hopper
Walker
[20, 150, 1, 15] Ant
[20, 300, 1, 15] Humanoid
C.2 VISUAL CONTROL DMC
In Visual control DMC, we use the COPlanner- Sititnner--DeaeeVV33 method. We keep all parameters consistent with the DreamerV3 original paper (Hafner et al.) 2023), except for newly introduced conservative rate Oge, optimistic rate ag action candidate number K, and planning horizon Hp. In Table 3 we provide the specific settings of conservative rate de. optimistic rate aos action candidate number K, and planning horizon Hp for each task. It's worth noting that, although using a conservative rate of 0.5 can perform well, find that for the tasks in Quadruped, using a conservative rate of 2 yields the best sample efficiency and performance. For other parameters, please refer to the original DreamerV3 paper. For the one-step predictive model ensemble, we use model ensemble with ensemble size of 8, Each network in the ensemble is MLP with 5 hidden layers of size 1024.
Table 3: Hyperparameters of COPlanner- (OPlanner-DreamerV: on visual control DMC. We keep all other hyperparameters consistent with the DreamerV3 original paper.
Parameter
Value
Conservative rate az
0.5
Optimistic rate a
Action candidate number
Planning horizon Hp
D MORE EXPERIMENTS
D.1 COMPARISON WITH MORE PROPRIOCEPTIVE CONTROL MBRL METHODS
In this section, we compared our approach with more proprioceptive control MBRL methods on MuloCo tasks. In addition to the three baseline methods from Section [.1] MBPO (Janner et al.) 2019), P2P-MPC (Wu al. 2022), and MEEE (Yao et al. 2021), we introduced two more baselines: PDML Wang al. 2023) method that dynamically adjusts the weights of each sample in the real sample buffer to enhance the prediction accuracy of the learned dynamics model for the current policy, thereby significantly improving the performance of MBPO. And MoPAC (Morgan et al.] (021). method that also uses policy-guia guidee MPC to reduce model bias. Unlike our approach, MoPAC's quicee MPC is solely used for multi-step prediction during rollout based on total reward to
16

---

Published as a conference paper at ICLR 2024
select actions. It does not incorporate measure of model uncertainty, and therefore, cannot achieve the optimistic exploration and conservative rollouts of COPlanner. The experiment results are shown in Table
As can be seen from Table our method still holds a sigmificant advantage, achieving the best performance in three tasks (Hopper, Walker2d, and Ant). In Humanoid task, it is only surpassed by PDML but is substantially better than the other methods. It's worth mentioning that our approach is orthogonal PDML, and they can be combined. We believe that by integrating COP Pananner with PDML, the performance can be further enhanced.
Table Comparison of different MBRL methods on proprioceptive control MuJoCo tasks. Perfor- mance is averaged over 8 random seeds.
Hopper (70k)
Walker2d (150k)
Ant (150k)
Humanoid (150k)
Ours
3360.3 165.0
4595.3 + 350.2
5268.3 +: 291.4
4833.7 320.1
PDML
3274.2 + 224.1
4378.5 = 248.9
4992.5 + 365.1
5396.7 + 391.3
MBPO
2844.6 158.0
4221.1 281.1
2311.1 = 252.5
1706.0 976.3
P2P-MPC
2316.8 + 459.9
4151.7 516.9
4681.7 + 591.6
3706.1 + 1360.4
MEEE
3076.4 165.3
3873.8 549.6
3932.8 + 352.7
654.2 94.7
MoPAC
3174.2 + 233.8
2893.6 + 472.6
4382.5 + 301.7
1084.6 + 573.2
Besides, we also conduct comparisons with DreamerV3 and D4PG on six medium-difficulty proprio- ceptive control DMC tasks, with the experimental results shown in the Figure We find that after integrating with COP anner, both sample efficiency and performance of DreamerV3 are improved significantly.
serobot-swinggue
Surtool-swinnupp-paee
Cheetah-run
250
800
200
600
600
Retums
Returss
Returns
150
200
400
100
50
0
Steps Finger-turn-eay
Hopper-hop
Walker-run
1000
800
600
200
Retums
600
Reeuum
Returns
400
100
200
200
250k
500k
250k Steps
500k
0
250k Steps
500k
Steps
COPlanner COPlanner-DreamevV DreamerV3
DreamerV3
D4PG
Figure 9: Experiment results of COPlanner-Dreaan V3 DMC tasks. The results are averaged over 8 random seeds, and shaded regions correspond to the 95% confidence interval among seeds.
proprioceptive control
D.2 COMPARISON WITH MORE VISUAL CONTROL METHODS
In this section, we conducted comparisons with more MFRL and MBRL methods for visual control on tasks from visual DMC. addition to DreamerV3 (Hafner et al. 2023) and LEXA-reward DreamerV3 (LEXA-RW), we introduced five more baselines. The first TDMPC (Hansen et al.) 2022). TDMPC learns a task-oriented latent dynamics model and uses this model for planning. During the planning process, TDMPC also learns policy sample a small number of actions, thereby accelerating MPC. The second PlaNet (Hafner et al. 2019b). PlaNet uses the RSSM latent model, which is the same the Dreamer series (Hafner al.
2019a 2020 2023), and directly uses
17

---

Published as a conference paper at ICLR 2024
this model to perform MPC in the latent space to select actions. Additionally, we also include three more model free visual RL baselines: TACO (Zheng et al. 2023a). DrM (Xu al. 2024) and DrQ-v2 (Yarats al. 2021). The experiment results shown in Table From the results,
evident that our method has a significant advantage over all the baselines. future work, as the representation in enner is learned from scratch, will also integrat pretrained visual representations such as Zheng et al. (024)) Majumdar et al. (2023); Nair et al. (2022) into the CCP lanner algorithm framework accelerate the learning process.
Table 5: Performance comparison of different MBRL and MFRL methods on visual DMC tasks at 1
million environment steps.
Hopper-stand
Hopper-hop
Quadruped-wall
Quadruped-run
Ours
937.8 + 9.7
444.2 113.2
803.9 + 59.3
787.9 + 111.9
DreamerV3
888.5 + 60.7
395.3 + 50.1
749.2 1: 65.3
696.0 + 132.2
LEXA-RW
905.7 + 25.1
353.0 + 33.8
591.0 + 110.9
549.5 + 186.2
TDMPC
821.6 770.8
189.2 + 19.7 0.37
427.8 + 50.2
393.8 + 40.9
PlaNet (5m)
5.96
238.90
280.45
TACO
923 + 22
261 + 52
798 + 15
531 + 38
DrM
910 + 15
162 82
750 + 29
452 32
DrQ-v2
918 + 12
192 + 41
702 + 42
407 + 21
Acrobot-swingup
Carpoll-swingup-spars
Finger-turn-easy
Finger-turn-hne
Ours
434.4 + 31.0
787.5 37.5
889.8 +: 28.3
867.1 +: 50.6
DreamerV3
354.9 + 37.9
814.7 22.3
853.9 + 37.5
751.3 93.1
LEXA-RW
337.7 + 24.9
683.5 + 145.6
825.5 + 55.9
740.7 + 162.0
TDMPC
227.5 + 16.9
668.3 49.1
703.8 65.2
402.7 112.6
PlaNet (5m)
3.21
0.64
451.22
312.55
TACO
241 +
802 21
752 +
618 17
DrM
228 + 15
799 + 7
692 + 39
459 + 36
DrQ-v2
210 + 12
794 14
576 42
248 52
D.3 EXPERIMENTS COMBINED WITH DREAMERV2
We also combine COP 1 anner with DreamerV2 (Hafner et al. (020) for experimentation, with the results shown Figure IO, After integrating with DreamerV2, our method also achieves a significant improvement in both sample efficiency and performance.
Hopper-stand
Hopper-hop
Quadruped-walk
Quadruped-run
Retumtt
500
125
500
200
100
5234
Stops
Stops
Stops
Stops
Walker-stand
Walker-wall
Walker-run
1000
1000
7003
10.0
COPlanner-Dreamette DreamerV2
S00
000
200
200
2525
7.06
25.05
3000
753%
2506
3036 1524
S3egps
Figure 10: Experiment results of COPlameer-DreamerV2 on visual control DMC tasks. The results are averaged over 4 random seeds, and shaded regions correspond to the 95% confidence interval among seeds. During evaluation, for each seed of each method, we test for up to 1000 steps in the test environment and perform 10 evaluations to obtain average value. The evaluation interval is every 1000 environment steps.
18

---

Published as a conference paper at ICLR 2024
D.4 HYPERPARAMETER STUDY
Optimaai
Conservaine rate
Action cand date number
Fanning ncrion
COPlanner-MBPS on proprioceptive Walker-run
140
743
140
Optimas 1010
Corsorvtiie rale
Action cand number
Flaniing noriznn
(b) COPlanner-MBPO proprioceptive Cheetah-run
300
103
Optimeeie
number
(c) DuPllaner-Dreammrr proprioceptive Hopper-hop
Figure 11: Ablation studies of COP lanne''s different hyperparameters. Experiments are conducted using COPIanner-MBPO on proprioceptive Walker-run and proprioceptive Cheetah-run, and using COPlanner- 3
proprioceptive Hopper-hop. The results averaged over 4 random seeds. From left to right, the results are for different parameters of optimistic rate a., conservative rate dee. action candidate number K, and planning horizon Hp.
In this section, we conduct hyperparameter studies to investigate the impact of different hyperpa- rameters COP anner. We perform experiments the Walker-run and Cheetah-run task of proprioceptive control DMC using COPIamer-MBBPO, and on the Hopper-hop using COPlanner- DreamerV3. The original hyperparameter settings are: optimistic rate a, is conservative rate a, is 2, action candidate number K is 5, and planning horizon Hp is 5. When conducting ablation experiments for each hyperparameter, other parameters remain unchanged. The results are shown together in Figure
Optimistic rate ag: observe that the best a, lies between 0.5 1. When the a is large, COP lanner tends to excessively explore high uncertainty areas while neglecting rewards, leading decrease in sample efficiency and performance. On the other hand, when the a, is too small, COP lanner fails to achieve the desired exploration effect.
Conservative rate ae: the optimal range for the ae is between and A too large ae may lead to overly conservative selection of low-reward actions, while too small would be unable to make model rollouts avoid model uncertain areas.
Action candidate number K; we find that K has significant impact on sample efficiency and set to 2, the improvement of COPlanner over MBPO terms of
performance. When performance and sample efficiency is relatively limited. This is reasonable because if there are only a few action candidates, our selection space very limited, and even with the use uncertainty bonus and penalty to select actions, there may not be much difference. When K increases more than 5, the effect of COP lanner becomes very stable, and more candidates do not bring noticeable improvements in performance and sample efficiency.
19

---

Published as a conference paper at ICLR 2024
Planning horizon Hp: when Hp is 1, we find that COPlanner's improvement performance and sample efficiency relatively limited. This also confirms what we mentioned in Section [] only considering the current step while ignoring the long-term uncertainty impact cannot completely avoid model errors, as samples with low current model uncertainty might still lead to future rollout trajectories falling into model uncertain regions. As the planning horizon gradually increases, performance and sample efficiency also rise. When the planning horizon is too long (Hp equals to 7 9), it possible that due the bottleneck of the model planning capability, most action candidates' corresponding trajectories fall into model uncertain areas, leading to slight decline in performance and sample efficiency.
D.5 ABLATION STUDY MODEL UNCERTAINTY ESTIMATION METHOOSS
We conduct an ablation study on the Hopper-hop task in visual control DMC to evaluate different uncertainty estimation methods. We adopt two methods, RE3 (Sco et al. 2021) and MADE (Zhang ctt al. 202I), which are used to estimate intrinsic rewards in pixel input, to replace the disagreement in calculating u(s, a) in Equation 4 and 5 The results performance achieved using these two methods is similar to that of disagreement. This demonstrate that using disagreement to calculate uncertainty not the primary reason for the observed performance improvement.
shown in Figure 12 We find that the
Hopper-hop
Disagreement
400
Returns
200
100
0k
250k
500k
750k
1m
Steps
Figure 12: Ablation study of different uncertainty estimation methods.
D.6 COMPUTATIONAL. TIME CONSUMPTION COP NNNER
We provide comparison of the computational time consumption between the baseline methods and COP lanner across different domains in Table 6 All timings are reported using single NVIDIA 2080ti GPU.
Table 6: Average time consumption (h).
MuJoCo
Proprioceptive DMC
Visual DMC
COPlanner-MBPO MBPO
41.2
11.3
N/A
33.78
10.6
N/A
JOPlanner-Dreamce
N/A
N/A
17.9
DreamerV3
N/A
N/A
13.1
20

---

Published as a conference paper at ICLR 2024
D.7 DIVERSITY EVALUATION REAL SAMPLE BUFFER
In this section, to further demonstrate that our method achieves better exploration of the environment, we evaluate the average state entropy of the real sample buffer obtained using our method and DreamerV3 at I million environment steps. A higher average state entropy implies that the real sample buffer covers more states the real environment, indicating that the samples in the real sample buffer are more diverse and thus suggesting more thorough exploration of the environment (Seo et al., 2021). We conduct evaluations on four visual DMC tasks include Hopper-hop. Quadruped- walk, Acrobot-swingup. and Finger-turn-hard. Following the work of RE3 (Seo et al.) 202 1). in order to estimate state entropy in environments with high- gigh-dimensional observations, we utilize a k-nearest neighbor entropy estimator in the low-dimessional representation space of randomly initialized encoder. Our encoder consists of three convolutional layers with 3x3 kernels, a stride of 2, and padding of 1, followed by flattening layer. The activation function between each layer is ReLU. After passing through the encoder, each image from the replay buffer is compressed into a 12-dimensional latent state, and the k-nearest neighbor state entropy is estimated as follows:
e(xt) = log(lyy V.NNN 1),
where where y: = (s; is fixed representation from a random encoder and y - the k-nearest neighbor of y/ within set of N representations (y1, 1/2, .... 1/2,.... }n We set N = 1024. Then we average
the k-nearest neighbor state entropy of each sample in real sample buffer. The final results are shown in Figure 13
CSP Pnnne Breamer'2
Dreamerys
0.2
0.
0.4
0.3
Hlpper hop
Quedruped walk
Acrobet sningue
finger Bumn
taoks
Figure 13: K-nearest neighbor state entropy estimation on different visual DMC tasks compared with DreamerV3.
The figure clearly shows that the state entropy of the real sample buffer significantly increases after integrating our method. This indicates that the real samples obtained by our method are more diverse, achieving better exploration of the environment.
D.8 VISUALIZATION EXPERIMENT RESULTS
Figure 14: Visualization of policy learned by DreamerV3 on Hopper-hop.
Figure 15: Visualization of policy learned by COPlanner- 3 on Hopper-hop.
21

---

Published as a conference paper at ICLR 2024
Figure 16: Visualization of policy learned by DreamerV3 on Quadruped-wand
Figure 17: Visualization of policy learned by COPlaner-DreamerVV on Quadruped -walk. walk.
In this section, to better demonstrate the improvements brought by our method, we visualize the trajectories obtained from evaluations the real environment after convergence using Dreamer and We visualize trajectories in Hopper-hop and Quadruped-wand
our method COPlanner-DreamerrV
tasks, as shown Figure 14
15 16 and
From the comparison between Figure and Figure 1. we can observe that in the Hopper hop task, DreamerV3 is only able learn to jump using the knee, whereas our method can learn to jump using the feet and perform somersaults during the jump. Through the comparison of Figure [6] task, our method able to learn more stable
Figure we can see that in the Quadruped-vee behavior of walking using all four legs, opposed to DreamerV3, which learns to walk using only three legs.
22


Reviews:
[{"review_id": "QIMVLrY9td", "reviewer": "Reviewer_EHTZ", "summary": "This paper proposes a model based reinforcement learning method that target for solving the challenge of inaccurately learned dynamic model problem through a combination of conservative model rollouts offline and optimistic exploration with the environment online. To estimate the dynamic model uncertainty, the authors utilize the model disagreement method which learns an ensemble of dynamic models.  This estimate uncertainty are utilized in two folds, it can serve as a penalty term during rollouts and as an incentive when interating with the environment online. The authors conduct experiments in environments like MuJoCo and DeepMind Control and demonstrate the proposed method achieves better sample efficiency and performance than other model-based RL baselines.", "questions": "Where do the authors see the main factor for the extra computation cost?  Is it the uncertainty calculation for all action candidates through the ensemble of dynamic models?\n\nWhat options is available for trading off computation cost for uncertainty estimation performance?\n\nIn Figure 7, while the proposed method reached lower loss, this does not imply that the method achieved better exploration, which the authors claim in related work section, how can this be demonstrated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is easy to follow and the proposed approach is also simple and flexible to use on existing model based RL methods. The authors have done extensive experiments and ablations to show their strengths in sample efficiency and model performance. I think the part of the success of this paper attributes to the good uncertainty estimation.", "weaknesses": "The good uncertainty estimation comes with the cost of extra computation, in Appendix D.6, there is an above 20\\% increase for MBPO variant and around 40\\% increase for the DreamerV3 variant. The test time comparison should also be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PzCzd6Rk99", "reviewer": "Reviewer_NmVv", "summary": "This paper proposes a model-based reinforcement learning framework COPlanner to mitigate the model errors from the model rollouts and environment exploration. This framework includes three components, and the most crucial component is the Planner to predict future trajectories based on selected actions and their corresponding uncertainties, and the uncertainties work as penalties during model rollouts and bonuses during environment exploration.", "questions": "1. It's still not quite clearly to me how Conservative rate and Optimistic rate are selected. What's the intuition here about why they are always different than each other in every experiment settings? Are they selected according to Action candidate number, and Planning horizon as well?\n2. Following the weakness, experiments for comparison and more visualizations are necessary.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to understand.\n2. The idea of using the variance over predicted states of the ensemble members to approximate the model uncertainty either penalty or reward seems interesting to me.\n3. The proposed framework outperforms baselines in almost all experiments.", "weaknesses": "1. Lack of comparisons to some framework.  In section 3.2, the paper mentions some previous methods to estimate uncertainty samples after generations or to decrease model error by re-weighting or discarding samples with high uncertainty. This should require a comparison to demonstrate using variance through model ensemble.\n2. Lack of visualization of experiments. The results are all basically tables, line plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QcRRImB7Dn", "reviewer": "Reviewer_D4cs", "summary": "This work presents COPlanner, a model-based reinforcement learning algorithm that combines optimistic exploration of high-reward model uncertain regions with pessimistic model learning to avoid biased updates in information-sparse regions. The approach leverages an MPC-style action selection and is evaluated on both proprioceptive as well as visual control tasks.", "questions": "-\tAction selection by optimizing over a 5-step rollout without terminal value function is surprising. Do you have intuition for why this short-term reasoning is sufficient? \n-\tThe original Dreamer-v3 paper also evaluated on proprioceptive DMC environments - why not use Dreamer-v3 as a proprioceptive baseline as well?\n-\tThere are several task-specific parameter choices for proprioceptive control. How impactful are these? Ideally a single set of hyperparameters would be used throughout.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tIncreasing learning efficiency of model-based reinforcement learning agents is an important research direction, particularly in light of hardware deployment under costly data generation\n-\tCombining optimistic exploration of uncertain high-reward behaviors with conservative model rollouts to improve information quality of samples is a very promising research direction\n-\tMPC-style action selection provides several benefits over direct policy evaluation\n- The approach is evaluated against several baseline agents on OpenAI Gym as well as DeepMind Control Suite environments, with both proprioceptive and visual tasks for the latter", "weaknesses": "-\tGenerally, the paper would benefit from a stronger selection of established baselines. E.g., for proprioceptive DMC model-free MPO/DMPO/D4PG or model-based DreamerV3, for visual DMC DrQ-v2 as a model-free baseline\n-\tThe visual control experiments should be run for more than 1M steps as agents have not converged on the Acrobot/Finger/Quadruped tasks. Comparing with Dreamer-v1 Figure 10, agents would be expected to converge and solve the tasks at approximately 2M steps.\n-\tThe Dreamer-v2 scores in Figure 9 appear to be significantly lower than the ones provided by the original authors on GitHub – which Dreamer implementation was used to obtain these scores?\n-\tThe task selection for proprioceptive DMC consists of relatively easy tasks and some of the “MuJoCo”/Gym tasks have not been run to convergence\n-\tThe hyperparameter study in Appendix D.4 is interesting, but should be extended to more environments to observe clearer trends\n-\tSome highly related work is missing. E.g., the optimistic exploration of uncertain future returns for visual control has previously been investigated in [1], also building on the Dreamer-v2 agent, while optimistic finite-horizon exploration under nominal reward functions was studied in [2]. Additionally, work such as POLO [3] would be relevant.\n\n[1] T. Seyde, et al. \"Learning to plan optimistically: Uncertainty-guided deep exploration via latent model ensembles,\" CoRL 2021.\n\n[2] P. Ball, et al. \"Ready policy one: World building through active learning,\" ICML 2020.\n\n[3] K Lowrey, et al.  \"Plan online, learn offline: Efficient learning and exploration via model-based control,\" ICLR 2019.\n\n\nMinor:\n\n-\tThe DMC tasks are MuJoCo-based, so “MuJoCo” task description should be replaced by “Gym”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)