On Effective Scheduling of Model-based Reinforcement Learning
Year: 2021

Abstract:
Model-based reinforcement learning has attracted wide attention due to its superior sample efficiency. Despite its impressive success so far, it is still unclear how to appropriately schedule the important hyperparameters to achieve adequate performance, such as the real data ratio for policy optimization in Dyna-style model-based algorithms. In this paper, we first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance. Inspired by the analysis, we propose a framework named AutoMBPO to automatically schedule the real data ratio as well as other hyperparameters in training model-based policy optimization (MBPO) algorithm, a representative running case of model-based methods. On several continuous control tasks, the MBPO instance trained with hyperparameters scheduled by AutoMBPO can significantly surpass the original one, and the real data ratio schedule found by AutoMBPO shows consistency with our theoretical analysis.

heading: 1 Introduction
text: Deep model-free reinforcement learning (MFRL) has achieved great successes in complex decisionmaking problems such as Go [26] and robotic control [10], to name a few. Although MFRL can achieve high asymptotic performance, a tremendous number of samples collected in interactions with the environment are required. In contrast, model-based reinforcement learning (MBRL), which alternately learns a model of the environment and derives an agent with the help of current model estimation, is considered to be more sample efficient than MFRL [27].
MBRL methods generally fall into different categories according to the specific usage of the learned model [25, 33]. Among them, Dyna-style algorithms [28] adopt some off-the-shelf MFRL methods to train a policy with both real data from environment and imaginary data generated by the model. Since Dyna-style algorithms can seamlessly take advantage of innovations in MFRL literature and have recently shown impressive performance [12, 30], this paper mainly focuses on Dyna-style algorithms. Although some theoretical analysis [17, 12, 22] and thorough empirical evaluation [30] have been conducted in the previous literature, it remains unclear how to appropriately schedule the hyperparameters to achieve optimum performance when training a Dyna-style MBRL algorithm. In practice, many important hyperparameters may primarily affect performance.
Firstly, since Dyna-style MBRL algorithms consist of model learning and policy optimization, how to balance the alternate optimization of these two parts is a crucial problem [22]. Intuitively, insufficient model training may lead to inaccurate estimation while excessive model training will cause overfitting, and a small amount of policy training may not utilize the model adequately while exhaustive policy training will exploit the model’s deficiencies. Moreover, the rollout length of the imaginary trajectory
∗equal contribution. †Weinan Zhang is the corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
is also critical [12] since too short rollout length fails to sufficiently leverage the model to plan forward, while too long rollout length may bring disastrous compounding error [1]. Janner et al. [12] manually design a schedule that linearly increases the rollout length across epochs. Finally, when using both real samples and imaginary samples to train the policy, how to control the ratio of the two datasets remains unclear. Janner et al. [12] fix the ratio of real data as 5%, while Kalweit and Boedecker [13] use the uncertainty to adaptively choose the ratio, which tends to use more imaginary samples initially and gradually use more real samples afterward. According to these existing works, the optimal hyperparameters in model-based methods may be dynamic during the whole training process, which further strengthens the burden of manually scheduling the hyperparameters.
Based on these considerations, in this work, we aim to investigate how to appropriately schedule these hyperparameters, i.e., real data ratio, model training frequency, policy training iteration, and rollout length, to achieve optimal performance of Dyna-style MBRL algorithms. Although real data ratio is an essential factor empirically [13, 12], it has not yet been studied thoroughly in theory. To bridge this gap, we first derive a return discrepancy upper bound for model-based value iteration, which reveals that gradually increasing the ratio of real data yields a tighter bound than choosing a fixed value. Inspired by the analysis, considering the complex interplay between real data ratio and other hyperparameters in practice, we develop AutoMBPO to automatically determine the joint schedule of the above hyperparameters in model-based policy optimization (MBPO), a representative running case of Dyna-style MBRL algorithms. Specifically, AutoMBPO introduces a parametric hyper-controller to sequentially choose the value of hyperparameters in the whole optimization process to maximize the performance of MBPO. We apply AutoMBPO to several continuous control tasks. On all these tasks, the MBPO instance trained with hyperparameters scheduled by AutoMBPO can significantly surpass the one with original configuration [12]. Furthermore, the hyperparameter schedule found by AutoMBPO is consistent with our theoretical analysis, which yields a better understanding of MBPO and provides insights for the design of other MBRL methods.

heading: 2 Related Work
text: Model-based reinforcement learning (MBRL) has been widely studied due to its high sample efficiency. MBRL methods can be roughly categorized into four types according to different model usage: (i) Dyna-style algorithms [28, 17, 4, 14, 12, 15] leverage the model to generate imaginary data and adopt some off-the-shelf MFRL algorithms to train a policy using both real data and imaginary data; (ii) shooting algorithms [19, 3] use model predictive control (MPC) to plan directly without explicit policy; (iii) analytic-gradient algorithms [7, 16, 5] search policies with back-propagation through time by exploiting the model derivatives; (iv) model-augmented value expansion algorithms [9, 2] utilize model rollouts to improve the target value for temporal difference (TD) updates. This paper mainly focuses on the first category.
Dyna-style algorithms [28] typically alternate between sampling real data, learning dynamics models, generating imaginary data, and optimizing policy. Many efforts have been made to improve one or more of these steps. For model learning, the deep ensemble technique [21, 3, 14] has been leveraged to resist overfitting. As for imaginary data generation, it is suggested to generate short rollouts to reduce compounding model error [12], and the model’s uncertainty is further incorporated to choose reliable imaginary data dynamically [20]. When optimizing the policy, we can use merely imaginary data [17, 14] or a mixture of real data and imaginary one in a fixed ratio [12]. Moreover, the ratio can also be dynamically adjusted according to the estimated uncertainty of the Q-function [13]. Besides, Rajeswaran et al. [22] present a game-theoretic framework for MBRL by formulating the optimization of model and policy as a two-player game.
As for automatic hyperparameter optimization for MBRL, Zhang et al. [32] utilize population based training (PBT) to tune the hyperparameters of a shooting algorithm dynamically. The reinforcement on reinforcement (RoR) framework proposed by Dong et al. [8] applies an RL algorithm to control the sampling and training process of an MBRL method. Our work differs from theirs in four major aspects: (1) We provide a theoretical analysis for the hyperparameter being scheduled, which is not included in their work. (2) In terms of problem settings, RoR assumes one can directly access an initial state without further interaction from that state, which is impractical for most RL problems. (3) As for the method design, our formulated hyper-MDP in Section 5.1 is totally different from RoR’s, including state, action and reward. In detail, our state definition contains more training information, and the reward design is based on the return rather than a single-step reward in the target-MDP. (4)
Finally, in the experiments in Section 6, our method achieves better performance and generalization in all environments, and the empirical findings also show differences.

heading: 3 Preliminaries
text: We first briefly introduce the RL problem with notations used throughout this paper and the previous MBRL method on which our proposed framework is based.

heading: 3.1 Reinforcement Learning
text: In reinforcement learning (RL), there originally exists a Markov decision process (MDP), which we will call target-MDP throughout the paper. The target-MDP is defined by the tuple (S,A, T, r, γ). S and A are the state and action spaces, respectively, and T (s′ | s, a) is the transition density of state s′ given action a taken under state s. The reward function is denoted as r(s, a), and γ ∈ (0, 1) is the discount factor. The goal of RL is to find the optimal policy π∗ that maximizes the expected return, denoted by η:
π∗ := argmax π η[π] = argmax π Eπ [ ∞∑ t=0 γtr(st, at) ] , (1)
where st+1 ∼ T (s | st, at) and at ∼ π(a | st). Generally, the ground truth transition T is unknown, and MBRL methods aim to construct a model T̂ of the transition dynamics to help improve the policy.
To evaluate a reinforcement learning method (no matter whether it is model-free or model-based), we mainly care about two metrics: (1) asymptotic performance: the expected return of the converged policy in real environments; (2) sample efficiency: the number of samples (s, a, s′, r) collected in target-MDP to achieve some performance. Therefore, an effective RL method means it achieves higher expected returns with fewer real samples.

heading: 3.2 Model-Based Policy Optimization
text: Model-based policy optimization (MBPO) [12] is a state-of-the-art MBRL method which will be used as the running case in our framework. MBPO learns a bootstrapped ensemble of probabilistic dynamics models {T̂ 1θ , ..., T̂Eθ } where E is the ensemble size. Each individual dynamics model in the ensemble is a probabilistic neural network that outputs a Gaussian distribution with diagonal covariance T̂ iθ (s ′|s, a) = N ( µiθ (s, a) ,Σ i θ (s, a) ) . These models are trained independently via maximum likelihood with different initializations and training batches. The corresponding loss function is
LT̂ (θ) = N∑ n=1 [µθ (sn, an)− sn+1]> Σ−1θ (sn, an) [µθ (sn, an)− sn+1] + log det Σθ (sn, an) .
In practical implementation, an early stopping trick is adopted in training the model ensemble. To be more specific, when training each individual model, a hold-out dataset will be created, and the training will early stop if the loss evaluated on the hold-out data does not decrease.
Every time the ensemble models are trained, they are then used to generate k-length imaginary rollouts, which begin from states sampled from real data and follow the current policy. Each time the policy interacts with the target-MDP, it will be trained for G gradient updates using both real data and imaginary data in a certain ratio. To make it clear, in each batch for policy training, we denote the proportion of real data to the whole batch as β and the proportion of imaginary data as 1 − β. We call β the real ratio and call G the policy training iteration throughout the paper. The policy optimization algorithm is SAC (Soft Actor-Critic) [11], and the loss functions for the actor πω and the critic Qφ are
Lπ(ω) = Est∼D [ Eat∼πω [α log(πω(at|st))−Qφ(st, at)] ] , (2)
LQ(φ) = E(st,at)∼D [1
2
( Qφ(st, at)− (r(st, at) (3)
+ γEst+1,at+1 [Qφ̂(st+1, at+1)− α log πω(at+1|st+1)]) )2] ,
where Qφ̂ is the target Q-function.

heading: 4 Analysis of real ratio schedule
text: Previous theoretical findings in MBRL mainly focus on the optimization of model and policy [17, 22] or the rollout length [12, 15], while the usage of real data in policy optimization remains unclear. Kalweit and Boedecker [13] only use real data near convergence but accept noisier imaginary data initially, which seems confusing since the model may not be accurate enough at the beginning. In this section, we provide a theoretical analysis of the real ratio schedule by deriving a return discrepancy upper bound of sampling-based Fitted Value Iteration (FVI) in model-based settings. The analysis mainly follows Munos and Szepesvári [18], where all the training data are sampled from the underlying dynamics. We extend the framework to a more general case: the data can either be sampled from the underlying dynamics w.p. (with probability) β or from a learned model w.p. 1− β, which we call β-mixture sampling-based FVI. Besides, we consider deterministic environments here, i.e., given (s, a), the next state s′ = T (s, a) is unique, just as in all the environments in Section 6.
For the sake of specificity, we first present a detailed description of the β-mixture sampling-based FVI algorithm. Let F be the value function space, and V0 ∈ F be the initial value function. The FVI algorithm produces a sequence of functions {Vk}0≤k≤K ⊂ F iteratively. Given Vk and N sampled states {s1, ..., sN}, the value function of the next iteration Vk+1 is computed as
V̂k(si) = max a∈A
( ri + γVk (s ′ i) ) , i = 1, 2, . . . , N, (4)
Vk+1 = argmin f∈F N∑ i=1 ∣∣∣f (si)− V̂k(si)∣∣∣p , (5) where si is uniformly sampled from a certain state distribution µ, and for each possible action a, the next state s′i is sampled from the underlying dynamics w.p. β or from the learned model w.p. 1− β. As final preparatory steps, we define the Bellman operator for deterministic MDPs B : F → F as
BV (s) = max a∈A {r(s, a) + γV (s′)} ,
where s′ = T (s, a); and for function g over X , ‖g‖p,µ is defined as ‖g‖pp,µ = ∫ |g(x)|pµ(dx). Following Munos and Szepesvári [18], we begin with the error bound in a single iteration.
Lemma 4.1. (Single Iteration Error Bound) Let Vk and Vk+1 be the value functions of iteration k and k+1, and Vmax = rmax/(1−γ). For p ≥ 1 and a certain state distribution µ, let the inherent Bellman error of the value function space F be defined by dp,µ(BF ,F) = supV ∈F inff∈F ‖f − BV ‖p,µ. Assume the value functions are KV -Lipschitz continuous, i.e., for any k and states pair (si, sj), it holds that |Vk(si) − Vk(sj)| ≤ KV ‖si − sj‖2. And assume the L2-norm model error between the real next state s′ and the predicted ones ŝ′ obeys a half-normal distribution, i.e., the probability density function f(‖ŝ′ − s′‖2) = 2√2πσ exp ( −‖ŝ ′−s′‖22 2σ2 ) . Then for any 0, δ0 > 0, the inequality
‖Vk+1 −BVk‖p,µ ≤ dp,µ(BF ,F) + 0
holds w.p. at least 1− δ0 provided that
N > 128 ( 8Vmax 0 )2p ( log(1/δ0) + log (32N) ) , (6)
and σ <
0 4γKV Φ−1 ( 1− δ0/ ( 8N |A|(1− β) )) . (7) Proof. See Appendix A, Lemma A.2.
According to the model error assumption (more details in Appendix B), the model will give precise predictions if σ is small enough. Therefore, Lemma 4.1 states that with an accurate model and sufficient large N , the errors between BVk and Vk+1 in each iteration can be bounded with high probability. Now let us turn to bound the return discrepancy between the greedy policy w.r.t. VK and the optimal policy.
Theorem 4.1. (β-mixture sampling-based FVI bound) Under the concentrability assumption and the same assumptions of Lemma 4.1, let ρ be an arbitrary state distribution, Cρ,µ be the discountedaverage concentrability coefficient and πK be the greedy policy w.r.t. VK . Let V πK and V ∗ be the expected return of executing πK and the optimal policy π∗ in real environment, respectively. Define Nreal = N · |A| · β as the expected number of real samples. Then the following bound holds w.p. at least 1− δ:
‖V ∗ − V πK‖p,ρ ≤ 2γ
(1− γ)2 C1/pρ,µ dp,µ(BF ,F) +O (( β|A| Nreal ( log( Nreal β|A| ) + log( K δ ) )) 12p)
+O ( Φ−1 ( 1− βδ 8KNreal(1− β) ) σ ) +O ( γK/pVmax ) .
(8)
Proof. See Appendix A, Theorem A.1.
Remark. The definition of the concentrability assumption is provided in Appendix A. Theorem 4.1 gives an upper bound on the return discrepancy. To achieve small return discrepancy, we hope to minimize the upper bound. For the first term in the bound, the approximation error dp,µ(BF ,F) can be made small by selecting F to be large enough, and the last term arises due to the finite number of iterations. For the remaining two terms, as β/Nreal increases, the second term increases while the third term decreases. Therefore, there exists an optimal value for β/Nreal to trade off these two terms. In online MBRL, Nreal increases as real samples are continuously collected from the environment throughout the training process, so that gradually increasing the real ratio β is promising to achieve good performance according to the upper bound.
From another perspective, the second term corresponds to the estimation error of using limited data to learn the value functions. By utilizing a learned model, we can generate as much data as we want, which, however, tends to suffer from the model error represented by the third term. The hyperparameter β serves as a trade-off between these two kinds of errors. Intuitively, with more real data available, the pressure of limited data is alleviated, which means we can increase β accordingly to reduce the reliance on inaccurate models.

heading: 5 The AutoMBPO Framework
text: The analysis in the previous section shows that the optimal real ratio β during training should gradually increase according to the current optimization states, such as the number of real samples and model error. These states may also be affected by other hyperparameters, e.g., model training frequency and rollout length, which makes the hyperparameter scheduling non-trivial in practice. In order to investigate the optimal joint scheduling of these hyperparameters, we propose a practical framework named AutoMBPO to schedule these hyperparameters automatically when training an MBRL algorithm. The high-level idea of AutoMBPO is to formulate hyperparameter scheduling as a sequential decision-making problem and then adopt some RL algorithm to solve it. Here we choose MBPO [12] as a representative running case since MBPO has achieved remarkable success in continuous control benchmark tasks and can naturally take all the hyperparameters we care about into consideration (a comparison of the hyperparameters that can be scheduled in different MBRL algorithms is provided in Appendix C). We illustrate the overview framework of AutoMBPO in Figure 1 and detail the algorithm in Algorithm 1.

heading: 5.1 Hyper-MDP Formulation
text: We formulate the problem of determining the hyperparameter schedule as a new sequential decisionmaking problem, which we will call hyper-MDP. In the hyper-MDP, the hyper-controller needs to decide the hyperparameter values throughout one complete run of the MBPO learning process. Below we elaborate on several critical aspects of the hyper-MDP formulation used in this paper.
Episode. An episode of the hyper-MDP consists of the whole process of training an MBPO instance from scratch, which contains multiple episodes of the target-MDP. We will terminate an episode of the hyper-MDP if N samples have been collected in the target-MDP. To be more specific, if the episode length of the target-MDP is H , and we only train MBPO for m episodes, then we have N = H ·m.
State. The state should be defined to include all the information of MBPO training to help hypercontroller make decisions accordingly. We use a concatenation of several features to represent the state. As suggested by Theorem 4.1, the state should contain Nreal, the number of real samples so far, and the model error, which can be estimated by the model loss LT̂ on a validation set. Besides, we also add other features to better capture the current optimization situation: (1) critic loss: LQ, which to some extent represents how well the value function fits the training data; (2) policy change: π = E(s,a)∼πD [|π(a|s)−πD(a|s)|], which evaluates the distance of current policy π and the data-collecting policy πD on sampled data. In prac-
tice, we calculate π using recently collected data to reflect the updating speed of policy roughly; (3) validation metrics: we use the average return on the target-MDP to represent the performance of the policy; (4) current hyperparameters: the hyperparameter values used currently by MBPO. All these features are normalized into [0, 1] to make the learning process stable.
Action. The action set in the hyper-MDP should contain all the hyperparameters we care about. Firstly, we would like to adjust the ratio of real data when optimizing the policy. Secondly, we also consider investigating the model training and policy learning frequency to balance their alternate optimization [22]. Lastly, the rollout length when generating imaginary rollouts is crucial in MBPO, but it remains unclear how to choose its value [12]. Besides, since there exists an optimal value for β/Nreal according to the theoretical analysis and Nreal gradually increases during training in practice, for the action space, adjusting the real ratio by multiplying a constant is enough to get an effective schedule. To sum up, the action set and the corresponding space of each action are: (1) adjusting the real ratio β: {×c−1,×1,×c}, where c is a constant larger than 1; (2) deciding whether to train the model: {0, 1}. Note that in MBPO, the model training iterations are implicitly decided by an early stopping trick, so we only decide whether to train the model; (3) adjusting the policy training iteration G: {−1, 0,+1}; (4) adjusting the model rollout length k: {−1, 0,+1}. In practice, the actions are taken every τ real samples are collected where τ is the hyperparameter of the hyper-controller.
Reward. One direct way is to define a reward at the end of an episode in hyper-MDP, which, however, will lead to the sparse reward problem. We define our reward as follows: every H real samples, i.e., one episode in target-MDP, we evaluate the MBPO policy on target-MDP and define the reward as the average evaluated return.

heading: 5.2 Hyper-Controller Learning
text: After formulating the hyper-MDP, we are now ready to learn a hyper-controller in the hyper-MDP. In fact, we can use any off-the-shelf reinforcement learning algorithm, and we choose Proximal Policy Optimization (PPO) [24] as the hyper-controller algorithm since PPO has achieved considerable success in the optimization schedule problem [31]. To be more specific, we maximize the following clipped PPO objective:
J (Ω) = Êt [ min ( rt(Ω)Ât, clip (rt(Ω), 1− , 1 + ) Ât )] ,
where rt(Ω) = MCΩ(at|st)/MCΩold(at|st) and MCΩ is the hyper-controller policy parameterized by Ω.
There are multiple choices of the advantage function [23], and we use the baseline version of the Monte-Carlo returns to reduce the variance:
Ât = Σ m·H/τ i=t (Ri −R ′ i). (9)
Algorithm 1: AutoMBPO 1 Initialize a hyper-controller MC 2 repeat 3 Initialize a MBPO instance; initialize the hyper-controller sample count i = 0 4 for m target-MDP episodes do 5 for h = 0 : H − 1 real timesteps (an episode in target-MDP) do 6 Interact with the target-MDP using the MBPO policy; add the real samples to Denv 7 if h mod τ = 0 then 8 Calculate the current state Si for hyper-MDP; take an action based on Si: Ai ←MC(Si) 9 Decide whether to train the model, and adjust the hyperparameters k,G, β
according to Ai 10 for F model rollouts do 11 Sample st uniformly from Denv; perform k-step rollout from st; add the imaginary data to Dmodel 12 for G gradient updates do 13 Train the MBPO policy using SAC on Denv and Dmodel in the ratio β 14 if h = H − 1 then 15 Evaluate MBPO and construct the reward as the average return: Ri = Avg(η); update i = i+ 1 16 else if h mod τ = 0 and h 6= H − τ then 17 Construct the reward Ri = 0; update i = i+ 1
18 Train the hyper-controller MC on the data {(Si,Ai,Ri)}m·H/τi=0 19 until acceptable performance is achieved
Here, R′i is the average return of MBPO policy trained in advance with the same amount of real data using the original hyperparameters, and we treat Σm·H/τi=t R ′ i as the baseline. The advantage function can also be regarded as the performance improvement compared with the original MBPO configuration.

heading: 6 Experiment
text: In this section, we conduct experiments to verify the effectiveness of AutoMBPO and provide comprehensive empirical analysis. Our experiments aim to answer the following three questions: i) How does the MBPO instance learned by AutoMBPO perform compared to the original configuration? ii) Do the learned hyper-controllers in different target-MDPs share similar schedules, and are the schedules consistent with previous theoretical analysis? iii) Which hyperparameter in MBPO is more important/sensitive to the final performance? More experimental results are provided in Appendix D.

heading: 6.1 Comparative Evaluation
text: We mainly compare the MBPO instance found by AutoMBPO to the MBPO with original hyperparameters [12] since AutoMBPO is directly based on MBPO, and there is no need to compare to other model-based methods. Besides, we also compare to Soft Actor-Critic (SAC) [11], the state-of-the-art model-free algorithm in terms of sample efficiency and asymptotic performance. Actually, SAC can be viewed as an extreme variant of MBPO where the real ratio in training policy is 1. To make a fair comparison, we also provide the result of SAC where the policy training iteration is enforced to be the same as in MBPO. To be more specific, the original SAC, denoted as SAC(1), trains the policy one time per step in the real environment, while in MBPO, the policy is trained 20 times per real step, so we add the baseline SAC(20). We also compare our method to the PBT [32] and RoR [8] algorithm for a comprehensive comparison. Note that the hyper-controllers in AutoMBPO and RoR are trained in advance and are only evaluated here. The whole training process of AutoMBPO is provided in Appendix D.3.
The compared methods are evaluated on three MuJoCo [29] (Hopper, Ant, Humanoid), and three PyBullet [6] (HopperBullet, Walker2dBullet, HalcheetahBullet) continuous control tasks as our target-MDPs. In practice, the three hyperparameters (real ratio, model training frequency, and policy training iteration) are scheduled on all tasks, while the rollout length is not scheduled on the PyBullet tasks since preliminary experiments showed that even 2-step rollouts degrade the performance on the PyBullet environments. Moreover, due to the high computational cost, the hyper-controller is only trained in previous m episodes of MBPO and will be used to continuously control MBPO instances running for M episodes, which is usually 2− 3 times larger than m (e.g., m = 50,M = 150 on Ant). Furthermore, we give a penalty for the model training to speed up the training process, and we find it does not influence the results much. More experimental details and hyperparameter configuration for hyper-controller can be found in Appendix F and H.
The comparison results are shown in Figure 2. We can observe that: i) the MBPO instances learned by AutoMBPO significantly outperform MBPO with original configuration in all tasks, highlighting the strength of automatic scheduling. ii) Compared to other hyperparameter optimization baselines (i.e., PBT and RoR), our approach AutoMBPO achieves better performance and generalization in different environments, verifying the effectiveness of our proposed framework. iii) MFRL methods with exhaustive policy training may overfit to a small amount of real data since SAC(20) achieves worse performance than SAC(1) on some complex tasks like Ant and Walker2dBullet. The overfitting problem is much less severe in MBPO since we can generate plentiful data using the model. iv) Though only trained in previous episodes, the hyper-controller can still find reasonable hyperparameter schedules and can be extended to the longer training process of MBPO.

heading: 6.2 Hyperparameter Schedule Visualization
text: We visualize the schedules of the hyperparameters found by AutoMBPO in Figure 3. From the results, the hyperparameter schedules on different tasks show both similarities and differences: i) A similar increasing schedule of the real ratio can be observed in all six environments, which is consistent with our derivation in Section 4 and the empirical results in Kalweit and Boedecker [13]. ii) The hyper-controller tends to train the model more frequently in complex environments, such as Ant and Humanoid. iii) To a certain extent, increasing the policy training iteration can better exploit the model. However, it may also increase the risk of instability [22], and we observe that the hyper-controller decreases the policy training iteration in the subsequent training phase of Hopper when the policy
is close to convergence. iv) As for the rollout length, the hyper-controller adopts a nearly linear increasing schedule, which is close to the manually designed one used in the original MBPO [12].

heading: 6.3 Importance of Hyperparameters
text: In this section, we investigate the importance of different hyperparameters. Specifically, we utilize the hyper-controller trained previously to schedule one certain hyperparameter respectively, i.e., real ratio (AutoMBPO-R), model training frequency (AutoMBPO-M), policy training iteration (AutoMBPO-P), and rollout length (AutoMBPO-L), while fixing other hyperparameters to the same as in the original MBPO. As shown in Figure 4, the importance of various hyperparameters changes across different tasks: using the hyper-controller to schedule the policy training iteration is helpful on Hopper but fails on Ant, while the situation reverses for model training frequency. However, using the hyper-controller to schedule the real ratio retains much of the benefit of AutoMBPO on almost all tasks, which further verifies the importance of real ratio.

heading: 7 Conclusion
text: In this paper, we conduct a theoretical analysis to justify real data usage for policy optimization in MBRL algorithms. Inspired by the analysis, we present a framework called AutoMBPO to automatically schedule the hyperparameters in the MBPO algorithm. Extensive experiments on complex continuous control tasks demonstrate that the derived MBPO instance with hyperparameters scheduled by AutoMBPO can achieve significantly better performance than the MBPO instance of the original configuration. Moreover, the real ratio schedule discovered by AutoMBPO is consistent with our theoretical analysis. We hope this insight can help design more effective MBRL algorithms. For future work, we plan to apply our hyper-controller training scheme to other MBRL methods.

heading: Acknowledgments
text: The SJTU team is supported by “New Generation of AI 2030” Major Project (2018AAA0100900), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and National
Natural Science Foundation of China (62076161, 61772333, 61632017). The work is also sponsored by Huawei Innovation Research Program.

Review summaries:
The paper deals with the problem of hyper-parameters optimization in model-based reinforcement learning (MBRL) Specifically, the paper tries to answer the question of how to schedule hyperparameters such as real data ratio, model training frequency, policy training iteration, and rollout length to achieve better performance. Supplementals materials are provided.   There are 2 contributions in the paper: - The paper provides some analysis to show that real data ratios affect the performance of MBRL. Specifically, they show that increasing the real data ratio improves the performance of MBRL. - They propose a new MBRL algorithm that learns to schedule hyper-parameters on the fly.  

This work seeks to bring some clarity the hyperparameter management of Dyna-style deep RL (i.e. MBPO).  The main parameter of interest being the real data to synthetic data ratio. It does this through theoretical analysis which is followed up by an automl framework in the style of 'RL for RL' (RoR).  Training an outer policy to schedule policy performs better than standard MBPO across several tasks.

The paper is about learning a hyper-parameter setter for some of the hyper-parameters used to train a model-based reinforcement learning algorithm. Theoretical results are provided to show that there exists an optimal ratio of real-data to model-generated data during training. Experiments show that setting hyper-parameters with a parametric policy improves performance over manual hyper-parameter selection or hand-crafted schedules.