On Effective Scheduling of Model-based Reinforcement Learning
Authors: Hang Lai
Jian Shen
Weinan Zhang
Yimin Huang
Xing Zhang
Ruiming Tang
Yong Yu
Zhenguo Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep reinforcement learning
model-based reinforcement learning
automatic hyperparameter scheduling

Abstract: Model-based reinforcement learning has attracted wide attention due to its superior sample efficiency. Despite its impressive success so far, it is still unclear how to appropriately schedule the important hyperparameters to achieve adequate performance, such as the real data ratio for policy optimization in Dyna-style model-based algorithms. In this paper, we first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance. Inspired by the analysis, we propose a framework named AutoMBPO to automatically schedule the real data ratio as well as other hyperparameters in training model-based policy optimization (MBPO) algorithm, a representative running case of model-based methods. On several continuous control tasks, the MBPO instance trained with hyperparameters scheduled by AutoMBPO can significantly surpass the original one, and the real data ratio schedule found by AutoMBPO shows consistency with our theoretical analysis.

On Effective Scheduling of Model-based Reinforcement Learning
Hang Lai Jian Shen'!, Weinan Zhang?/, Yimin Huang', Xing Zhang", Ruiming Tang". Yong Yu Zhenguo Li Shanghai Jiao Tong University, HHuawei Noah's Ark Lab (laihang, wnzhang) (aaegx.)aappepaaaa. ajtu.eduu edu.cn
Abstract
Model-based reinforcement learning has attracted wide attention due to its perior sample efficiency. Despite its impressive success so far, is still unclear how appropriately schedule the important hyperparameters to achieve adequate performance, such as the real data ratio for policy optimization in Dyna-style model-based algorithms. In this paper, first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance. Inspired by the analysis, we propose framework named AutoMBPO to automatically schedule the real data ratio as well as other hyperparameters in training model-based policy optimization (MBPO) algorithm, a representative running case of model-based methods. On several continuous control tasks, the MBPO instance trained with hyperparameters scheduled by AutoMBPO can significantly surpass the original one, and the real data ratio schedule found by AutoMBPO shows consistency with our theoretical analysis.
1 Introduction
Deep model-free reinforcement learning (MFRL) has achieved great successes in complex decision- making problems such Go [26] and robotic control [10], to name a few. Although MFRL can achieve high asymptotic performance, a tremendous number of samples collected in interactions with the environment required. In contrast, model--ased reinforcement learning (MBRL). which alternately learns a model the environment and derives an agent with the help of current model estimation, considered be more sample efficient than MFRL 27
MBRL methods generally fall into different categories according the specific usage of the learned model 22 33). Among them, Dyna-style algorithms (28) adopt some off-the-shelf MFRL methods to train a policy with both real data from environment and imaginary data generated by the model. Since Dyna-style algorithms can seamlessly take advantage of innovations in MFRL literature and have recently shown impressive performance [12 algorithms. Although some theoretical analysis [17 12 22 and thorough empirical evaluation [30] have been conducted in the previous literature, it remains unclear how to appropriately schedule the hyperparameters to achieve optimum performance when training Dyna-style MBRL algorithm. In practice, many important hyperparameters may primarily affect performance.
30 this paper mainly focuses on Dyna-style
Firstly, since Dyna-style MBRL algorithms consist of model learning and policy optimization, how to balance the alternate optimization of these two parts is crucial problem (22]. Intuitively, insufficient model training may lead to inaccurate estimation while excessive model training will cause overfitting, and small amount of policy training may not utilize the model adequately while exhaustive policy training will exploit the model's deficiencies. Moreover, the rollout length of the imaginary trajectory
equal contribution. Weinan Zhang is the corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

is also critical [12 since too short rollout length fails to sufficiently leverage the model to plan forward, while too long rollout length may bring disastrous compounding error [.. Janner et al. [[2] manually design schedule that linearly increases the rollout length across epochs. Finally, when using both real samples and imaginary samples to train the policy, how to control the ratio of the two datasets remains unclear. Janner et al. [122 fix the ratio of real data as 5%, while Kalweit and Boedecker [13] use the uncertainty to adaptively choose the ratio, which tends to use more imaginary samples initially and gradually use more real samples afterward. According to these existing works, the optimal hyperparameters in model-based methods may be dynamic during the whole training process, which further strengthens the burden of manually scheduling the hyperparameters.
Based on these considerations, in this work, we aim to investigate how to appropriately schedule these hyperparameters, i.c.. real data ratio, model training frequency, policy training iteration, and rollout length, to achieve optimal performance of Dyna-style MBRL algorithms. Although real data ratio is an essential factor empirically [33 12). it has not yet been studied thoroughly in theory. To bridge this gap, we first derive return discrepancy upper bound for model-based value iteration, which reveals that gradually increasing the ratio of real data yields a tighter bound than choosing a fixed value. Inspired by the analysis, considering the complex interplay between real data ratio and other hyperparameters in practice, we develop AutoMBPO to automatically determine the joint policy optimization (MBPO), a representative
schedule of the above hyperparameters in running case of synna-ttle MBRL algorithms. Specifically, AutoMBPO introduces parametric hyper-controller to sequentially choose the value of hyperparameters in the whole optimization process to maximize the performance of MBPO. We apply AutoMBPO to several continuous control tasks. On all these tasks, the MBPO instance trained with hyperparameters scheduled by AutoMBPO can significantly surpass the one with original configuration [12]. Furthermore, the hyperparameter schedule found by AutoMBPO is consistent with our theoretical analysis, which yields a better understanding of MBPO and provides insights for the design of other MBRL methods.
2 Related Work
Model-based reinforcement learning (MBRL) has been widely studied due to its high sample effi- ciency. MBRL methods can be roughly categorized into four types according to different model usage: (i) Dyna-style algorithms (28 17
12 [5] leverage the model to generate imaginary data and adopt some off-the-shelf MFRL algorithms to train policy using both real data and imaginary data; (ii) shooting algorithms 19 use model predictive control (MPC) to plan directly without explicit policy; (iii) salltic-gradien algorithms 7 [6] 5 search policies with back- proppaapaantionn through time by exploiting the model derivatives; (iv) model-augmented value expansion algorithms 9 2) utilize model rollouts to improve the target value for temporal difference (TD) updates. This
paper mainly focuses on the first category.
Dyna-style algorithms (28) typically alternate between sampling real data, learning dynamics models, generating imaginary data, and optimizing policy. Many efforts have been made to improve one or more of these steps. For model learning, the deep ensemble technique [21 3 14] has been leveraged to resist overfitting, As for imaginary data generation, it is suggested to generate short rollouts to reduce compounding model error 12] and the model's uncertainty is further incorporated choose reliable imaginary data dynamically (20). When optimizing the policy, we can use merely imaginary data [17 [4] or a mixture of real data and imaginary one in a fixed ratio [12]. Moreover, the ratio can also be dynamically adjusted according to the estimated uncertainty of the Q-function [I.. Besides, Rajeswaran et al. (22) present game-theoretic framework for MBRL by formulating the optimization of model and policy as two-player game.
As for automatic hyperparameter optimization for MBRL, Zhang et al. 32 utilize population based training (PBT) to tune the hyperparameters of a shooting algorithm dynamically. The reinforcement on reinforcement (RoR) framework proposed by Dong et al. [8 applies an RL algorithm to control the sampling and training process of an MBRL method. Our work differs from theirs in four major aspects: (1) We provide theoretical analysis for the hyperparameter being scheduled, which is not included in their work. (2) In terms of problem settings, RoR assumes one can directly access an initial state without further interaction from that state, which impractical for most RL problems. (3) As for the method design, our formulated hDp in Section 5.1 is totally different from RoR's, including state, action and reward. In detail, our state definition contains more training information, and the reward design is based on the return rather than a single-step reward in the target-MDP. (4)
2

---

Finally, in the experiments in Section 6 our method achieves better performance and generalization all environments, and the empirical hiddins also show differences.
3
Preliminaries
We first briefly introduce the RL problem with notations used throughout this paper and the previous MBRL method which our proposed framework is based.
3.1 Reinforcement Learning
In reinforcement learning (RL), there originally exists a Markov decision process (MDP), which we will call target-MDP throughout the paper. The target-MDP is defined by the tuple (S, A, I, 7). S and A are the state and action spaces, respectively, and T'ss' ss, a) is the transition density of state s' given action a taken under state s. The reward function is denoted as r(s, a), and (0, 1) is the discount factor. The goal of RL is find the optimal policy x that maximizes the expected return, denoted by 77:
== arg max T/(T] arg max E Σ E()n(a)a)))
(1)
where S1+1 ~ T(s st, at) and a, ~ T(a st). Generally, the ground truth transition T' is unknown, and MBRL methods aim to construct a model T of the transition dynamics to help improve the policy,
To evaluate reinforcement learning method (no matter whether it is model-free model-based), mainly care about two metrics: (1) asymptotic performance: the expected return of the converged policy in real environments; (2) sample efficiency: the number of samples (s, a, s', r) collected in target-MDP to achieve some performance. Therefore, an effective RL. method means achieves higher expected returns with fewer real samples.
3.2 Model-Based Policy Optimization
Model-based policy optimization (MBPO) [2] is state-of-the-art MBRL method which will be used as the running case in our framework. MBPO learns bootstrapped ensemble of probabilistic dynamics models (T)) ..... f. where E is the ensemble size. Each individual dynamics model in the ensemble is probabilistic neural network that outputs Gaussian distribution with diagonal covariance Th a)) = A/ (u6 (s,a), Ey (8, a)). These models are trained independently via maximum likelihood with different initializations and training batches. The corresponding loss function is
N [Ho (Sn, an) - Sn+1] Eg (sn: an) [M0 (Sn, an) - 11 + log det Eg (Sn an)
(0) =
==1
In practical implementation, an early stopping trick adopted in training the model ensemble. To be more specific, when training each individual model, hold-out dataset will be created, and the training will early stop the loss evaluated on the hold-out data does not decrease.
Every time the ensemble models are trained, they are then used to generate k-length imaginary rollouts, which begin from states sampled from real data and follow the current policy, Each time the policy interacts with the target-MDD, it will be trained for G' gradient updates using both real data and imaginary data certain ratio. To make it clear, in each batch for policy training, we denote the proportion of real data to the whole batch as B and the proportion of imaginary data as 1 - B. We call / the real ratio and call G the policy training iteration throughout the paper. The policy optimization algorithm is SAC (Soft Actor-Critte [1],, and the loss functions for the actor mu and the critic Qo are
(.(w) ENND (aarsBBBr-ar BRI logg(om - Qo(se,
(2)
(O(Ø) B((nee)~ []
2 (Qo at) (r(s),
(3)
+ + 1Q (441) log xw tw ((444)))))))
where Q3 is the target function..
3

---

4 Analysis of real ratio schedule
Previous theoretical findings in MBRL mainly focus on the optimization of model and policy [7 22
or the rollout length [12. 15]. while the usage of real data in policy optimization remains unclear. Kalweit and Boedecker [13 only use real data near convergence but accept noisier imaginary data initially, which seems confusing since the model may not be accurate enough at the beginning. In this section, we provide a theoretical analysis of the real ratio schedule by deriving a return discrepancy upper bound of sampling-base -based Fitted Value Iteration (FVI) in model- The analysis mainly follows Munos and Szepesvári [8], where all the training data are sampled from the underlying dynamics. We extend the framework sampled from the underlying dynamics w.p. (with probability) / or from a learned model w.p. 1 - B, which call 3-mixture sampling-based FVI. Besides, we consider deterministic environments here, ie., given (s, a), the next state = T(s, a) is unique, just as in all the environments in Section 6
settings.
more general case: the data can either be
For the sake of specificity, we first present detailed description of the 3-mixture sampling-base FVI algorithm. Let F be the value function space, and Vo E F the initial value function. The FVI algorithm produces a sequence of functions {Vi K F iteratively. Given V&2 and N sampled states {51, SN ), the value function of the next iteration Vk+1 is computed as
(*:) max (ri 7Ve (6/)) 1,2, aEA
(4)
N
Σ f(s)) ( V
V argmin
(5)
feF
where 8; is uniformly sampled from a certain state distribution μt, and for each possible action the next state sf is sampled from the underlying dynamics w.p. 3 or from the learned model w.p. B. As final preparatory steps, we define the Bellman operator for deterministic MDPs B F - F as
(s))) max (r(s, +VV('') aEA
where s' = T(s, a); and for function g over X, ||g|lp. is defined as | p. Following Munos and Szepesvári [18], we begin with the error bound in a single iteration.
J (((=))
Lemma 4.1. (Single Iteration Error Bound) Let Va and V be the value functions of iteration k and k+1, and Vmaz Tmmoz/ / "maz/(1-7)) For 2 1 and certain state distribution let the inherent Bellman ervor of the value function space FF be defined by dp.((Ba (BF,F)
supv e inf (EF fl BV Assume the value functions are Ky-Lipschitz continuous. iee, for any k and states pair (st, 83), holds that (v+)))) - Vi(s;)]= I Kvlls; -8yll2- And assume the I?2norm model error between the real next state s' and the predicted ones s¹ obeys half-normal distribution, i.e., the probubility density function f([1'' 8 8'112)
2 exp
11 1122=1113
Then for any co, do >> 0, the inequality
V2n0
20
I1V++1 BVellpn (pF.(BFFF)) +.0
holds w.p. least 8, provided that
8Vmax
2p (Iog(L/6)) log (32.V)
N 128
(6)
ED
and
Eo
4yK (1 10:00--1110000111 66N 33))
Proof. See Appendix A Lemma A.2
According to the model error assumption (more details in Appendix B) the model will give precise predictions if a is small enough. Therefore, Lemma 4. states that with an accurate model and sufficient large N, the errors between BV; and V in each iteration can be bounded with high probability. Now us turn to bound the return discrepancy between the greedy policy w.r.t. Vx and the optimal policy.

---

Theorem 4.1. (3-mixture campling-based FVI bound) Under the concentrability assumption and the same assumptions of Lemma 4.1 let p be an arbitvary state distribution, Cppy be the discounted- average concentrability coefficient and πT K be the greedy policy w.r.t Vg. Let Vak and V be the expected return of executing k and the optimal policy x² in real environment, respectively. Define Nreal- N |A| 3 as the expected number of real samples. Then the following bound holds w.p. at least 1 6:
27
(0
((3/)) (log( Nreal
(/114)
Nreal +log( (o8(6)))
||V VTx 11,0 S
Cl/ed p. (BF,F)
(1 7)2
3|A
(8)
0 x (1
8K 33 )0) + 0 +0(100 (T///Vmux) V
38
Proof. See Appendix
Theorem
A.7
The definition of the concentrability assumption is provided in Appendix Theorem 4.1 gives upper bound on the return discrepancy. To achieve small return discrepancy, we hope to minimize the upper bound, For the first term in the bound, the approximation error be made small by selecting F to be large enough, and the last term arises due to the finite number of iterations. For the remaining two terms, as 3 3/Nreal increases, the second term increases while the third term decreases. Therefore, there exists an optimal value for 3/Nreal to trade off these two terms. In online MBRL, Neea increases as real samples are continuously collected from the environment throughout the training process, so that gradually increasing the real ratio 3 is promising to achieve good performance according to the upper bound.
Remark.
F) can
From another perspective, the second term corresponds to the estimation error of using limited data to learn the value functions. By utilizing a learned model, we can generate as much data as we want, which, however, tends to suffer from the model error represented by the third term. The hyperparameter 3 serves as a trade-off between these two kinds of errors. Intuitively, with more real data available, the pressure of limited data is alleviated, which means we can increase 3 accordingly to reduce the reliance on inaccurate models.
The AutoMBPO Framework
The analysis in the previous section shows that the optimal real ratio during training should gradually increase according to the current optimization states, such as the number of real samples and model error. These states may also be affected by other hyperparameters, e.g., model training frequency and rollout length, which makes the hyperparameter scheduling non-trivial in practice. In order investigate the optimal joint scheduling of these hyperparameters, we propose practical framework named AutoMBPO to schedule these hyperparameters automatically when training an MBRL algorithm. The high-level idea of AutoMBPO to formulate hyperparameter scheduling as a sequential decision-making problem and then adopt some RI. algorithm to solve it. Here we choose MBPO 112 as a representative running case since MBPO has achieved remarkable success in continuous control benchmark tasks and can naturally take all the hyperparameters we care about into consideration (a comparison of the hyperparameters that can be scheduled in different MBRL algorithms is provided in Appendix C). We illustrate the overview framework of AutoMBPO in Figure I and detail the algorithm in Algorithm !
5.1 Hyper-MDP Formulation
We formulate the problem of determining the hyperparameter schedule as new sequential decision- making problem, which we will call hyper-MDP. In the hyper-MDP, the hyper-controller needs to decide the hyperparameter values throughout one complete run of the MBPO learning process. Below we elaborate on several critical aspects of the hyper-MDP formulation used in this paper.
Episode. An episode of the hyper-MDP consists of the whole process of training an MBPO instance from scratch, which contains multiple episodes of the tDr. We will terminate an episode of the hyper-MDP if N samples have been collected in the target-MDP. To be more specific, if the episode length of the target-MDD is H, and we only train MBPO for m episodes, then we have N = H m.
5

---

State. The state should be de- fined include all the information of MBPO training to help hyper- controller make decisions accord- ingly. We use concatenation of several features to represent the state. As suggested by Theorem 4.1] the state should contain Noaal, the num- ber of real samples so far, and the model error, which can estimated by the model loss Cp. on valida- tion set. Besides, we also add other features to better capture the cur- rent optimization situation: (1) critic loss: Lq. which some extent repre- sents how well the value function fits the training data; (2) policy change: 6x ((t_))~=D (a(s)) D (a]s)) s)]], which evaluates the distance of cur- rent policy π and the data-collecting policy TD on sampled data. In prac- tice, we calculate 6 using recently collected data to reflect the updating speed of policy roughly; (3) validation metrics: we use the average return on the target-MDP to represent the performance of the policy; (4) current hyperparameters: the hyperparameter values used currently by MBPO. All these features are normalized into [0, to make the learning process stable.
State, Reward
Hyper-MDP
Policy
Target-MDP
Interact
Train
Store
Cyper-Controller
Rollout
Train
Imaginary
Model
Action
Figure 1: Illustration of AutoMBPO framework. The hyper- controller (left) treats the whole learning process of MBPO (right) as its hyper-MDP and takes action according the state containing all the information of MBPO training. The action to adjust the hyperparameters of MBPO, e.g., real ratio, model training frequency, policy training iteration, and rollout length. Afterward, reward is given to the hyper-controller based on the performance of MBPO policy in target-MDP.
Action. The action set in the hyper-MDP should contain all the hyperparameters we care about. Firstly, we would like adjust the ratio of real data when optimizing the policy. Secondly, also consider investigating the model training and policy learning frequency to balance their alternate optimization [22 Lastly, the rollout length when generating imaginary rollouts is crucial in MBPO, but it remains unclear how to choose its value [2]. Besides, since there exists an optimal value for 3/Nreal according the theoretical analysis and Nraal gradually increases during training in practice, for the action space, adjusting the real ratio by multiplying constant is enough to get an effective schedule. To sum up, the action set and the corresponding space of each action are: (1) adjusting the real ratio B: { ixc- x1, xc), where is constant larger than 1; (2) deciding whether to train the model: {0. 1.. Note that in MBPO, the model training iterations are implicitly decided by an early stopping trick, so we only decide whether to train the model; (3) adjusting the policy training iteration G: (-1,0, +1}; (4) adjusting the model rollout length k: {-1, O, +1). In practice, the actions are taken every real samples are collected where 7 is the hyperparameter of the hyper-controller.
Reward. One direct way is to define a reward at the end of an episode in hyper-MDP, which, however, will lead to the sparse reward problem. We define our reward as follows: every H real samples, i.c., one episode in MDPP we evaluate the MBPO policy on target-MDP and define the reward as the average evaluated return.
5.2 Hyper-Controller Learning
After formulating the hyper-MDP, fact, we can use any off-the-shelf reinforcement learning algorithm, and we choose Proximal Policy Optimization (PPÓ) [24] the hyper-controller algorithm since PPO has achieved considerable To be more specific, we maximize the following
ready to learn hyper-controller in the hyper-MDP. In
success in the optimization schedule problem 31. clipped PPO objective:
J( R2 min (rx(1) A., clip (r:(),, 1 - -e,,-,,1 1 +))
where (2) = (Cn(a.)))) MCau (as)st) and MCs is the hyper-controller policy parameterized by Ω.
There are multiple choices of the advantage function and we use the baseline version of the Monte-Carle returns to reduce the variance:
A D
H/T (R,- R.).
(9)

---

Algorithm 1: AutoMBPO
Initialize hyper-controller MC repeat
Initialize MBPO instance; initialize the hyper-controller sample count i for target-MDP episodes do
for h = 0: - real timesteps (an episode in target-MDP) do
Interact with the target-MDP using the MBPO policy; add the real samples to Den h mod f 0 then
Calculate the current state S, for hyper-MDP take action based Sy; A4 + MC(S,)
Decide whether to train the model, and adjust the hyperparameters k G, according to A;
10
for F modde rollouts do
11
Sample & uniformly from Denvi perform k-step rollout from 8s; add the imaginary data to model
for G gradient updates do
13
Train the MBPO policy using SAC on Denv and D model in the ratio |
14
h H then
Evaluate MBPO and construct the reward as the average return: R, Aug(n); update i+]
16
else if h mod and h + T then
17
Construct the reward R₃ = 0; update = -i++
Train the hyper-controller MC on the data {(S,, A;, R2) H/T until acceptable performance is achieved
18
19
Here, R/ is the average return of MBPO policy trained in advance with the same amount of real data using the original hyperparameters, and we treat E E/--H// R² as the baseline. The advantage funetion can also be regarded as the performance improvement compared with the original MBPO configuration.
Experiment
In this section, conduct experiments verify the effectiveness of AutoMBPO and provide comprehensive empirical analysis. Our experiments aim to answer the following three questions: i) How does the MBPO instance learned by AutoMBPO perform compared the original configuration? ii) Do the learned hyper- cyper-controllers in different target-MDPss MDDs share similar schedules, and are the schedules consistent with previous theoretical analysis? iii) Which hyperparameter in MBPO is more important/sensitive to the final performance? More experimental results are provided in Appendix D
6.1 Comparative Evaluation
We mainly compare the MBPO instance found by AutoMBPO the MBPO with original hyperpa- rameters 12] since AutoMBPO is directly based on MBPO, and there is no need to compare to other model-based methods. Besides, we also compare Soft Actor-Critie (SAC) (IT), the state-of-the-art model-free algorithm in terms of sample efficiency and asymptotic performance. Actually, SAC can be viewed as an extreme variant of MBPO where the real ratio training policy is 1. To make fair comparison, we also provide the result of SAC where the policy training iteration is enforced be the same in MBPO. To be more specific, the original SAC, denoted as SAC(1), trains the policy one time step in the real environment, while in MBPO, the policy is trained 20 times per real step, so we add the baseline SAC(20). We also compare our method the PBT (32 and RoR [ algorithm for comprehensive comparison. Note that the hyper-controllers in AutoMBPO and RoR are trained advance and are only evaluated here. The whole training process of AutoMBPO provided in Appendix D.3

---

Hopper
Ant
Humanoid
1.000
330
4200
5000
Sverage retum
3.000
Sverage retum
avorage retum
-.290
0.00
30000
3000
2000
.00
0
22%
42%
50%
50%
136
30%
50%
50%
128K
150K
0.
00%
123K
1B8%
242
302
slaps
stops
steps
HopperBullet
Walker2dBBllet
HaltChoetahBullett
4000
2000
3200
2000
3000
2.00
000rago re5um
2200
15:00
1000
1000
500
300
MBPO
AutaMBPO
PBT
SAC(1)
SAC/20)
Figure 2: Learning curves of AutoMBPO and baselines on different continuous control environments. The solid lines indicate the mean, and shaded areas indicate the standard deviation of 10 trials over different random seeds. Each trial is evaluated every 1000 environment steps, where each evaluation reports the average return over 10 episodes.
The compared methods are evaluated on three MuJoCo 29] (Hopper, Ant, Humanoid), and three PyBullet [6] (HopperBullet, Walker2dBullet, HalcheetahBullet) continuous control tasks as our target-MDPs. In practice, the three hyperparameters (real ratio, model training frequency, and policy training iteration) are scheduled on all tasks, while the rollout length is not scheduled on the PyBullet tasks since preliminary experiments showed that even 2-step rollouts degrade the performance on the PyBullet environments. Moreover, due to the high computational cost, the hyper-controllet is only trained in previous m episodes of MBPO and will be used to continuously control MBPO instances running for M episodes, which is usually 2 - 3 times larger than m (e.g.. Furthermore, we give a penalty for the model training to speed up the training process, and we find does not influence the results much. More experimental details and hyperparameter configuration for hyper-controllk can be found in Appendix and
50, M = 150 on Ant).
The comparison results are shown in Figure We can observe that: i) the MBPO instances learned by AutoMBPO significantly outperform MBPO with original configuration in all tasks, highlighting the strength of automatic scheduling. ii) Compared to other hyperparameter optimization baselines (i.e., PBT and RoR), our approach AutoMBPO achieves better performance and generalization in different environments, verifying the effectiveness of our proposed framework. iii) MFRL methods with exhaustive policy training may overfit to small amount of real data since SAC(20) achieves worse performance than SAC(1) some complex tasks like Ant and Walker2dBullet. The overfitting problem is much less severe in MBPO since we can generate plentiful data using the model. iv) Though only trained in previous episodes, the hyper-controller can still find reasonable hyperparameter schedules and can be extended to the longer training process of MBPO.
6.2 Hyperparameter Schedule Visualization
We visualize the schedules of the hyperparameters found by AutoMBPO in Figure 3 From the results, the hyperparameter schedules on different tasks show both similarities and differences: i) A similar increasing schedule of the real ratio can be observed in all six environments, which is consistent with our derivation in Section 4 and the empirical results in Kalweit and Boedecker [13] ii) The hyper-controller tends to train the model more frequently complex environments, such as Ant and Humanoid, iii) To certain extent, increasing the policy training iteration can better exploit the model. However, it may also increase the risk of instability (22). and we observe that the hyper-controller decreases the policy training iteration in the subsequent training phase of Hopper when the policy
8

---

is close to convergence. iv) As for the rollout length, the hyper-controller adopts a nearly linear increasing schedule, which is close the manually designed one used in the original MBPÒ [2].
Real Ratio
Model Training Frequency
Policy Training lteraion
Rollout Length
modd tse ing Eoquency
polcyy rma teatos
rolluu longgh a
00 64
08
06
205
bolting ponoentage
trakning percomaage
training percentage
boining porcentage
Ant
Humanoid
HopporBulist
Waler2dBullee
HallChootahBulitt
MBPO
Figure 3: Hyperparameter schedule found by AutoMBPO on different tasks. The dashed lines indicate the original hyperparameter configuration in MBPO. For better visualization, model training frequency is scaled by the frequency of the original MBPO, and the training process (x-axis) is scaled to [0,1] since the max episodes vary on different tasks.
6.3 Importance of Hyperparameters
In this section, we investigate the importance of different hyperparameters. Specifically, we utilize the hyper-controller trained previously schedule certain hyperparameter respectively, i.e., real ratio (AutoMBPO-R) model training frequency (AutoMBPO-M) policy training iteration (AutoMBPO-P) and rollout length (AutoMBPO-L), while fixing other hyperparameters to the same in the original MBPO. As shown in Figure the importance of various hyperparameters changes across different tasks: using the hyper-controller to schedule the policy training iteration is helpful on Hopper but fails on Ant, while the situation reverses for model training frequency. However, using the hyper-controller to schedule the real ratio retains much of the benefit of AutoMBPO on almost all tasks, which further verifies the importance of real ratio.
Happar
Walker2dBullet
1500
$000
3300
2000
2000
3verage rdtum
3300
overage cot.m
(0:0
Sverage
1500
3200
1500
1000
1300
1000
300
AutsMBPO
AutoMBPO-R
AutsMBPO-M
AutoMBPOPP
AutoMBPO-L
Figure 4: Performance of the instances with only one specific hyperparameter scheduled by hyper- controller while other hyperparameters fixed to the same as in MBPO. The complete figures of all six environments are provided in Appendix D.I.
7 Conclusion
In this paper, we conduct a theoretical analysis to justify real data usage for policy optimization in MBRL algorithms. Inspired by the analysis, we present framework called AutoMBPO to automatically schedule the hyperparameters in the MBPO algorithm. Extensive experiments on complex continuous control tasks demonstrate that the derived MBPO instance with hyperparameters scheduled by AutoMBPO can achieve significantly better performance than the MBPO instance of the original configuration. Moreover, the real ratio schedule discovered by AutoMBPO is consistent with our theoretical analysis. We hope this insight can help design more effective MBRL algorithms. For future work, we plan apply hyper-controller training scheme to other MBRL methods.
Acknowledgments
The SJTU team supported by "New Generation of AI 2030" Major Project (2018AAA01009))) Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and National

---

Natural Science Foundation of China (62076161, 61772333, 61632017). The work is also sponsored by Huawei Innovation Research Program.
References
[1] Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning, pages 264-273. PMLR, 2018.
[2] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pages 8224 8234, 2018.
[3] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4754 4765, 2018.
[4] Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning, pages 617-629, 2018.
[5] Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-augmentte actor-critic: Backpropagating through paths. arXiv preprint arXiv:2005.0800 2020.
[6] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
[7] Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465 4422, 2011.
[8] Linsen Dong, Yuanlong Li, Xin Zhou, Yonggang Wen, and Kyle Guan. Intelligent trainer for dyna-style model- deep reinforcement learning. IEEE Transactions on Neural Nenworks and Learning Systems, 2020.
[9] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint 1rXiv:1803.001100 2018.
[10] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. 2017 IEEE international conference robotics and automation (ICRA), pages
IEEE, 2017.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 1rXivv1801.01290, 2018.
[12] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, pages 2211-125330 2019.
[13] Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning, pages 195-206, 2017.
[14] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference Learning Representations, 2018.
[15] Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy op- timization. In International Conference on Machine Learning, pages 5618-5627. PMLR. 2020.
[16] Sergey Levine and Vladlen Koltun, Guided policy search. In International Conference on Machine Learning, pages 1-9, 2013.
10

---

[17] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmii framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations, 2018.
[18] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, (MMa))8855885) 2008
[19] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566. IEEE, 2018.
[20] Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the model when it is confident: Masked model-base based actor-critic. arXiv preprint arXiv:2010.0483 04893, 2020.
[21] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learn- ing robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2016.
[22] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. arXiv preprini arXiv:2004.07804 2020.
[23] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0244 02438, 2015.
[24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707. 063477 2017.
[25] Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-based policy optimization with unsupervised model adaptation. Advances in Neural Information Processing Systems, 33, 2020.
[26] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che. Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
[27] Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based in contextual decision processes: Pac bounds and exponential improvements over model- free approaches. In Proceedings of the Thirty-Second Conference on Learning Theory, pages 2898-2933, 2019.
[28] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
[29] Emanuel Todorov. Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-500 5033. IEEE, 2012.
[30] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.00 19000020577 2019.
[31] Haowen Xu, Hao Zhang, Zhiting Hu, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Autoloss: Learning discrete schedule for alternate optimization. In International Conference on Learning Representations, 2018
[32] Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Staristics, pages 4015 4023. PMLR, 2021.
[33] Guangxiang Zhu, Minghao Zhang, Honglak Lee, and Chongjie Zhang. Bridging imagination and reality for model-based deep reinforcement learning. Advances Neural Information Processing Systems, 33, 2020.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] We described the high computa-
tional cost and propose several tricks to mitigate this problem Section 6.1
(c) Did you discuss any potential negative societal impacts of your work? [No] No potential negative societal impacts have been found yet.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section and Appendix A
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material supplemental material.
URL)? [Yes] In the
(b) Did you specify all the training details (e.g.. data splits. hyperparameters, how they were chosen)? [Yes] See Section 6.1| Appendix Flandddd(aanndd H
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We reported the standard deviation of 10 trails over different random seeds in the experimental results, indicated by the shaded areas. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the authors of the MBPO algorithm.
(b) Did you mention the license of the assets? [No] MIT license.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We included our code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Our work does not use existing data.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] Our work does not use existing data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] Our work does not use crowdsourcing or conduct research with human subjects.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] Our work does not use crowdsourcing conduct research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Our work does not crowdsourcing conduct research with human subjects.
12


Reviews:
[{"review_id": "ef4d_VUgf1z", "reviewer": "Reviewer_yNuW", "summary": "The paper is about learning a hyper-parameter setter for some of the hyper-parameters used to train a model-based reinforcement learning algorithm. Theoretical results are provided to show that there exists an optimal ratio of real-data to model-generated data during training. Experiments show that setting hyper-parameters with a parametric policy improves performance over manual hyper-parameter selection or hand-crafted schedules.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is good at conveying its ideas. The introduction section clearly states what we are going to be dealing with. The related work section does a good job of summarizing the different ways in which models have been used in MBRL.\n\nI am not sure if the theoretical analysis should be placed in such a prominent place, given that it is somewhat dense with notation. Perhaps the results of the analysis can be stated earlier on, or be highlighted more from the rest of this section, such that the first-time reader does not immediately have to confront the math right at the stage when they are most eager to find out about the implementation of the ideas presented in the introduction.\n\nThe methods section does a good job of setting up the hyper-MDP. However I have some questions about the execution:\n\n* Algorithm 1 starts with \"initialize MBPO\". Does that mean the model and policy parameters are re-initialized periodically during the lifetime of the whole algorithm?\n* The advantage calculation uses the average return of MBPO trained with the same amount of real data from original parameters. Does this mean using AutoMBPO requires logs from a previous MBPO training run (or a separate process training plain MBPO in parallel and providing such logs)?\n\nThe experiments section is well-designed. The comparative results are in favor of the method. The hyper-parameter schedules and the ablation study over the hyper-parameters are both welcome inclusions.\n\nThat being said, some points are unclear to me:\n\n* A little confused by the experiments. Does AutoMBPO here mean an experiment where both MBPO and its hyper-controller are being trained in tandem, or has AutoMBPO been trained in advance, and only being applied here? If the two are being trained jointly, shouldn't we see periodic drops in performance, since MBPO gets re-initialized every time a new hyper-episode is started, according to Algorithm 1?\n* In case we are indeed training MBPO and its hyper-controller in parallel, I am a little surprised that the scheduling curves in fig. 3 can make sense throughout. I'd expect the hyper-controller to make errors in the beginning of training. Can you comment on this?\n* How were the hyper-parameters set for plain MBPO? Is it the result of a hyper-parameter search, or carried over from previous work? If they are carried over, I would be curious about the improvement of AutoMBPO over a near-optimal fixed hyper-parameter setup found by a hyper-parameter search. This is especially true of the bullet-based environments which MBPO was not originally applied to.\n\nAll in all, this is a good paper and it would benefit the research community to accept it.\n\n**Other comments**\n\n* Do the theoretical results extend to stochastic transition functions?\n* Is it right to refer to the hyper-parameter setting problem as an MDP, given that the state does not fully describe all of the information in the system? I think POMDP would be more accurate here.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is alright.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "c4Gk6yhl3M", "reviewer": "Reviewer_AyeP", "summary": "This work seeks to bring some clarity the hyperparameter management of Dyna-style deep RL (i.e. MBPO). \nThe main parameter of interest being the real data to synthetic data ratio.\nIt does this through theoretical analysis which is followed up by an automl framework in the style of 'RL for RL' (RoR). \nTraining an outer policy to schedule policy performs better than standard MBPO across several tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The MBPO algorithm and DYNA-style RL algorithms are very popular at the moment, so I can see this work being of interest to the deep RL community. \nThe outer policy does seem to have a widespread improvement on standard MBPO in the mean and the theoretical analysis looks correct. \n\nI see this paper more as an analysis into hyperparameters rather than a practical MBRL algorithm.\nThe need to train a PPO outer loop to me throws away all the sample efficiency that MBRL is supposed to provide, and judging form the hyperparameters in the appendix the full algorithm requires ~1e6 samples that model-free RL usually requires. \nI would be useful to see the 'full' training curves (including the outer loop) to see who effective the outer RL is.\n\nThe paper often refers to this method as 'simple yet effective'. I would disagree that a bilevel deep RL problem is 'simple'.\nThe phrase 'simple yet effective' also a suggests that its a surprise that this method works, whereas for me it is more a question of whether the extensive computational requirements are worthwhile for the task in hand. \n\nIt is also discussed that the empirical results verify that the theory is correct and that it is optimal to slowly increase the data percentage from low to high. However, in the experiements it looks like the initial value of the ratio is set to 0.05 (Table 4), so the policy appears biased to start low.  I think a more uniform initial exploration is required before that claim can be made. \n\nA rather specific observation I would like some clarification on: on HopperBullet the ratio quickly goes to 1.0, which suggests that MBPO should behave the same as SAC(20) within 10% of training (Fig 3), yet in the performance (Fig 2) AutoMBPO is worse that SAC(20) for the first ~10% but then is better for the remaining 90%. Shouldn't the performance be the same?   \n\nGiven the amount of computation required to run this bi-level optimizaiton, it seems the performance increase is not really statistically significant apart from on WalkerBullet. I think some discussion of the statistical significance of the results (since requires so much more computation) would benefit the reader. \n\nRegarding the baselines, I would appreciate more of a discussion in the appendix regarding how PBT and RoR are implemented as baselines. \n\nBayesian optimization is a popular method for HPO. I wonder if, rather that optimizing a neural network policy, you could define a 'schedule spline' (or some similar parameterization) that is only a few parameters but covers a sensible space of possible schedules. \n\nMinor\nline 4 of algorithm 1, typo on target", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KvnNha4Oe-E", "reviewer": "Reviewer_SPME", "summary": "The paper deals with the problem of hyper-parameters optimization in model-based reinforcement learning (MBRL) Specifically, the paper tries to answer the question of how to schedule hyperparameters such as real data ratio, model training frequency, policy training iteration, and rollout length to achieve better performance. Supplementals materials are provided. \n\nThere are 2 contributions in the paper:\n- The paper provides some analysis to show that real data ratios affect the performance of MBRL. Specifically, they show that increasing the real data ratio improves the performance of MBRL.\n- They propose a new MBRL algorithm that learns to schedule hyper-parameters on the fly. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main Strengths: \n- The paper makes an effort to analyze the effect of real data ratio on the performance of MBRL. The analysis is based on the one by Munos and Szepesvári but the author did a good job in modifying the proof in the mixed data setting. \n- The author proposes a hyper-parameters tuning method based on Reinforcement Learning. Although the idea is not new, the author did a good job by choosing the appropriate MDP setup. \n- The experiments show good results of their proposed method. \n- Overall, the paper is well written. The way the authors present the paper is easy to follow. \n\nMain Weaknesses: \n- There is one problem with the comment that the author made about their bound. Since Nreal = N · |A| · β, the ratio β/Nreal  is a constant. Thus it is not clear how gradually increasing the real ratio β is promising to achieve good performance according to the upper bound as what the authors claim.\n- The proposed algorithm and the theoretical analysis do not have any relation. The analysis only shows that the real data ratio plays an important role in MBRL. Besides, the theory does not provide any hints in the design of the proposed algorithm. However, the authors claim that their algorithm is inspired by the theory, which is not totally correct in my humble opinion.\n- The idea of using RL to tune hyper-parameters is not new and so not a novel contribution. Intuitively, the paper might get trapped in a no-ending route. In detail, the purpose of the paper is to improve MBRL by dynamically tuning hyper-parameters. However, the authors again propose another RL algorithm to achieve this purpose. How are you going to efficiently tune the hyper-parameters of this new RL algorithm? Please let me know if I miss something here. \n\nQuestion: Have you tried your model on Humanoid and if so what was the result? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not provide limitations and societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that the paper is well-motivated, well-written and the the proposed method, motivated by theoretical analysis, shows convincing experimental results on hyperparameter tuning of model-based RL, a hard yet important problem to address. On the other hand, minor concerns remain mainly about whether the theory is connected well with the practice, whether another outside RL loop over model-based RL is "too heavy" in practice and requires high sample complexity.  Authors have address many of the concerns raised by the reviewers and I am happy to accept this paper. 

Decision: Accept (Poster)