Plan To Predict: Learning an Uncertainty-Foreseeing Model For Model-Based Reinforcement Learning
Authors: Zifan Wu
Chao Yu
Chen Chen
Jianye HAO
Hankz Hankui Zhuo
Venue: NeurIPS 2022 Accept (2022)
Keywords: model-based reinforcement learning

Abstract:  In Model-based Reinforcement Learning (MBRL), model learning is critical since an inaccurate model can bias policy learning via generating misleading samples. However, learning an accurate model can be difficult since the policy is continually updated and the induced distribution over visited states used for model learning shifts accordingly. Prior methods alleviate this issue by quantifying the uncertainty of model-generated samples. However, these methods only quantify the uncertainty passively after the samples were generated, rather than foreseeing the uncertainty before model trajectories fall into those highly uncertain regions. The resulting low-quality samples can induce unstable learning targets and hinder the optimization of the policy. Moreover, while being learned to minimize one-step prediction errors, the model is generally used to predict for multiple steps, leading to a mismatch between the objectives of model learning and model usage. To this end, we propose Plan To Predict (P2P), an MBRL framework that treats the model rollout process as a sequential decision making problem by reversely considering the model as a decision maker and the current policy as the dynamics. In this way, the model can quickly adapt to the current policy and foresee the multi-step future uncertainty when generating trajectories. Theoretically, we show that the performance of P2P can be guaranteed by approximately optimizing a lower bound of the true environment return. Empirical results demonstrate that P2P achieves state-of-the-art performance on several challenging benchmark tasks. 

Plan To Predict: Learning an Uncertainty-Foreeeeing Model for Model-Based Reinforcement Learning
Zifan Wu
Chao Yu*
School of Computer Science and Engineering Sun Yat-sen University
School of Computer Science and Engineering Sun Yat-sen University
Guangzhou, China wuzf50mai12 sysu. wyw..dd..u.ooo.o. edu. cn
Guangzhou, China yuchao3@mail, sysu. edu.cn
Chen Chen Noah's 's Ark Lab Huawei Beijing, China cc1vr0163. com
Jianye Hao Noah's Ark Lab Huawei Beijing, China haojianye@huavei..o
Hankz Hankui Zhuo
School of Computer Science and Engineering
Sun Yat-sen University
Guangzhou, China
zhuohank@mail. wtt.:/iaaaaass..see.e.ee. edu. cn
Abstract
In Model-based Reinforcement Learning (MBRL), model learning is critical since inaccurate model can bias policy learning via generating misleading samples. However, learning an accurate model can be difficult since the policy is continually updated and the induced distribution over visited states used for model learning shifts accordingly. Prior methods alleviate this issue by quantifying the uncertainty of model-generated samples. However, these methods only quantify the uncertainty passively after the samples were generated, rather than foreseeing the uncertainty before model trajectories fall into those highly uncertain regions. The resulting low- quality samples the policy. Morcover, while being learned to minimize one-step prediction errors, the model is generally used to predict for multiple steps, leading to mismatch between the objectives model learning and model usage. To this end, we propose Plan To Predict (P2P), MBRL framework that treats the model rollout process as a sequential decision making problem by reversely considering the model decision maker and the current policy the dynamies. In this way, the model can quickly adapt to the current policy and foresee the multi-step future uncertainty when generating trajectories. Theoretically, performance of P2P can be guaranteed by approximately optimizing lower bound of the true environment return. Empirical results demonstrate that P2P achieves state-of-the- performance on several challenging benchmark tasks.
induce unstable learning targets and hinder the optimization
show that the
Introduction
Through learning an approximate dynamics of the environment and then using it to assist policy learning, Model-based Reinforcement Learning (MBRL) methods [Moerland et al., 2020] have been shown to be much more sample-efficient than model-free methods both theoretically and empirically (Nagabandi et al., 2018, Luo al., 2019, Janner al., 2019]. The learning of the model critical for MBRL, since an inaccurate model can bias policy learning through generating misleading pseudo samples, which is also referred as the model bias issue [Deisenroth and Rasmussen, 2011] 1.
"correpponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Gool
#
(a)
(b)
Low Uncertainty
Movement Induced by Fiss Movement Induced by fT
Low Uncertainty/ High Value
Low Uncertainty Low Value
Model Prediction that Foresees the Uncertainty
High Uncertainty/ Spurious High Value
Model Prodiction Leads to Lange
Figure I: (a) Intuition for an setertaaintt-freseeeiin model; (b) A specific maze task modified from the D4RL benchmark [Fu et al., 2020], where an ant falls from the sky and randomly lands the two sides of a wall. The ant aims to reach the goal using shortest time (assuming that the goal cannot be seen from the sky).
However, learning an accurate model can be difficult, since the update of the policy continually shifts the distribution over visited states, and due to lack of enough training data for the modified distribution, the model can suffer from overfitting and become inaccurate in regions where the updated policy is likely visit. This issue is further aggravated when using expressive models like deep neural networks, which are inherently prone to overfitting.
Prior methods mitigate this issue by applying uncertainty quantification techniques, typically by using a bootstrapped model ensemble [Kurutach et al., 2018, Chua et al., 2018 88, Janner et al., 2019]. However, due to the lack of explicit considerations of the aforementioned policy shift issue, these methods can only quantify the uncertainty passively after the samples were generated, and thus fail to prevent the trajectories induced by the current policy from rumning into regions with high uncertainty. This incapability to avoid uncertain regions can result low quality of the generated samples, which provides unstable learning targets and thus may hinder the learning of the policy. Moreover, the model the existing methods is usually used to predict for multiple steps, while the model learning objective is commonly minimize the one-step prediction errors. In other words, there exists an objective mismatch [Lambert et al., 2020] between model learning and model usage, which means that the policy optimization generally expects accurate multi-step trajectories (i.e., model usage), yet the model is only estimated to consider the accuracy of immediate predictions (i.e., model learning).
To gain conceptual understanding of the above two motivations, consider the scenario in Figure I(a):
1) The approximate model P of the true environment dynamics P predicts P(s (aro) 0, P(s (mo) > O and P(s rr) Vs & { sf} at the state-action pair æo: and 2) The policy updated from Told to Tnew to exploit the regions with high values predicted by the value function in the model, and the resulting distribution shift exposes the model's inaccuracy at $1 the current policy, leading to large expected prediction error the next step, i.e., Σar new (a [s] (P((-ss) a'), P(-)s), >> Ev Tnew "noo(aa (d'ss{)) ) (P (P(-ss a'), P(-1s/, a'),, where ((.,-) is some distance metrics and the symbol >> means significantly greater than. Now starting from Fu, under the updated policy Tnews predicting is likely to direct the trajectory into highly uncertain regions with spurious high values (e.g.. the grey region Figure 1(b)), and cause considerably larger error in the subsequent trajectory generation than that of predicting si- In contrast, if the model can foresee this uncertainty through explicit considerations of the current policy, then it can learn to plan for future predictions and thus jump to sf more often to prevent the trajectory from falling into uncertain regions (c.g., the ant falls into the right hand side of the wall in Figure 1(b)). Therefore, to generate trajectories that have small accumulation errors, the model should be aware of the update of the policy and change its predicting "strategy" by taking into account the long-term effect of immediate predictions.
Inspired by the above intuition, we propose new framework for MBRL, named Plan To Predict (P2P). Our core idea is to treat the model rollout process as a sequential decision making problem and learn the model by the corresponding algorithms, such as reinforcement learning and model predictive control. During the learning of the model, the current policy fixed and serves as the environment dynamics, while at the same time the model becomes a decision maker and is penalized via the

---

prediction errors at each step. Through actively interacting with the current policy and minimizing the accumulative prediction errors on the resuiting state distribution, the model can quickly adapt to the state distribution shift by considering the long-term effect of the immediate predictions, thus foreseeing the uncertainty of regions that model trajectories are likely to fall into. Such foreiight of uncertainty enables the model to plan to predict for multiple steps, which improves the quality of the pseudo samples generated by the model and eventually benefits policy learning. Note that the typical treatment of minimizing one-step prediction errors regardless of the current policy is in fact a greedy optimization process, and thus can be viewed as a special case of P2P. Some prior works, such [Nagabandi et al., 2018, Luo et al., 2019], adopt similar multi-step prediction loss to learn the model, yet these works simply compute the loss on trajectories collected by previous policies and can still suffer from severe distribution shift due to the lack of considerations of the policy update.
Theoretically, we prove that P2P approximately optimizes lower bound of the true environment return. Different from prior works [Luo et al., 2019, Janner et al., 2019], our theoretical analysis does not scale up the step-wise prediction errors to their maximum over timesteps when considering the impact of model error the return approximation, thus resulting in tighter bound of the true return and also justifying the objective mismatch issue. Optimizing this bound not only provides a stronger guarantee for policy improvement, but also enables the model to quickly adapt to the current policy, Empirical results demonstrate that P2P outperforms the existing State-of-the-ar methods on several challenging benchmark tasks. It also deserves to note that, due to the capability to avoid model error, P2P may be particularly suited for offline learning tasks, where incremental data collection is not allowed and thus the model error is harder to be corrected than in online settings, inducing an irreversible bias on policy learning. Therefore, we also provide an evaluation of P2P in offiin learning settings in the experiments, and present an interpretative visualization on how the learning objective of P2P affects the model predictions.
2 Preliminaries and Notations
A discrete time Markov desision process (MDP) M is defined by tuple (S, R, P. po. 7). where is the state space. A the action space, R(s, a) scalar reward function of s 8 and E A, P(s") (s, a) e [o, 1] the transition dynamics which is assumed to be unknown, 7 E [0, 1) the discount factor and Pu the initial state distribution. A stochastic policy (als) maps states to probability distribution over the action space. The expected return of π defined by ((x) == E,.p iEi= R(s,, at) is the expectation of the sum of discounted rewards induced by Following the general setting of reinforcement learning, the goal is to search for an optimal policy that maximizes the expected return from the environment, i.e., arg max, [TT.
MBRL methods learn model P to approximate the unknown dynamics P. and then use this model to aid policy learning. While the model can be utilized in various ways, this paper focuses on the usage of generating pseudo samples to enrich the dataset, which is one of the most common usages of the model.
The expected return of predicted by the model P is denoted as J (T) == E,, IE R(s, at)]. In our analysis, the reward function is assumed to be available, 2 and the policy obtained in the last iteration is denoted as the data-collecting policy TD.
3 The Plan To Predict (P2P) Framework
In this section, we present detailed description of our proposed algorithmic framework for MBRL, i.c., Plan To Predict (P2P). In Section 3.1. the model rollout process is reformulated as a sequential decision making problem using the notion of MDP and then a meta-algorithh of the P2P framework is provided as generic solution to this reformulated problem. In Section 3.2, theoretical results are provided to guarantee the performance of P2P. In Section 3.3, two practical algorithms are developed under the P2P framework by handling some practical issues.
Note that this is a commonly used assumption since the sample complexity of learning the reward function with supervised learning is a lower order term compared to the one of learning the transition model [Ghesh- laghi Azar et al., 2013].
3

---

Algorithm 1 Meta-Algorithm of the P2P Framework
1: Initialize the policy и and the approximate model P:
2: Initialize empty dataset D;
3: for each epoch do
4: Collect data with in real environment: D <- DU f(s, di ri, 8/)
5: Optimize P for multi-step prediction under the current policy æ:: P t- arg max J (PT)
6: Optimize under P: < arg max. j (x')
7: end for
3.1
The Overall Framework
We now define the model MDP to reformulate the model rollout problem and outline our overall algorithm design.
Definition 1. The model MDP Ma is defined by tuple (STh AT, Rm, pm :7, ;7,Pp") where at
timestep t, 2:" ;= (st,at) E 5tm att == S1+1 E A (s. oz := -(P(st+1) a.) P(s++1|8e; at)) pm(s7"11877 "ir_1s8",,,a aa.') ==
P((e))))))((())))))) 0z).7 is the discount factor. p :- - Po. and po is the initial state distribution of the true environment.
In the model learning process of P2P, the roles played by the policy л and the model P are reversed: ı is fixed to serve as the environment dynamics and P. now as the decision maker, interacts with and is updated to minimize the accumulative prediction error along the generated trajectories. Similar to the optimization of the policy, the objective of this learning process can be written as arg max,, J J=(P) (P) where J J=(P)= == EI R (s"", a.")), The overall algorithmic framework of P2P is depicted in Algorithm 1, where the main difference compared most existing MBRL methods lies in Line 5, i.e., the learning of the model. Specifically, this process can be achieved in a way similar to policy learning, i.e., generating samples by actively interacting with the policy (now viewed as the background environment), and optimizing the expected return J (P) accordingly. This optimization can in principle be done by any approaches solving sequential decision making problems. As two representatives of these approaches, Model Predictive Control (MPC) [Camacho and Alba, 2013] and Reinforcement Learning (RL) [Sutton and Barto, 2018] are both applied in our practical version of P2P, which are detailed in Section 3.3. In addition, one of the biggest gaps between this theoretical construction of the model MDP and the practical implementation may lie in the derivation of RT According to Definition 1, the true environment dynamies is required to compute this reward function, while in practice the true dynamics is not accessible in the model learning phase. Therefore, depending on which underlying approach (i.e., MPC or RL) is chosen for model optimization, we propose two different alternative plans to approximate 1 in Section 3.3.
3.2 Theoretical Results
In general, MBRL methods learn approximate model of the real world dynamics and learn the policy by maximizing the expected return predicted by the model. The better this pseudo return approximates the true environment return, the stronger guarantee the model can provide for policy improvement. Thus, a critical problem in theory is how to bound the gap between the expected return of the model and the true environment, i... ()((z) bounded by C, and the policy is updated from TD to æ with the expected model return improved by 2C, then the policy improvement in terms of the true environment can be guaranteed:
[.r]]. Suppose that this gap can be upper
J(m) - J(TD) 2 ( (T) c) - (1P (TD) + +C)
2 J 20 O.
Our main theoretical result is the following theorem: Theorem 1. The gap between the expected return of the model and the environment is bounded as:
[((T) - fp(m)) S 2Rmax
(2 (2-7)ex + (1 - 7e
(1)
(1 >)²
t=1

---

denotes the policy distribution shift, 4 (e,0x))) Drv (P a) [I,(-)s aj)] denotes the upper bound of one-step model prediction ervor at timestep t of the model rollout trajectory, Drv(pl q) refers the total variation between dis- tribution p and q, Rmaz == max,,a R(s, a), and Pt-1(8, (8, a; π) denotes the state-action distribution at t under P and π.
where Cx == max,
Proof. Please refer to Appendix A.
Transforming the inequality in Eq. (1) results in (TT) 2 - T), where C(P, x) denotes the right hand side in Eq. (1). Thus by iteratively applying the update rule in Eq. (2), the above lower bound of the policy performance can be monotonically improved:
T P
arg max a,P
J 2C(P,
This update rule is ideal and usually impractical since it involves an exhaustive search in the state and action space compute C, and requires full-horizon rollouts in the model for estimating the accumulative model errors. Thus, similar to how algorithms like TRPO [Schulman et al., 2015] approximate their theoretically monotonic version, P2P approximates this update rule by maximizing the expected model return while keeping the accumulative model error small. Specifically, via replacing the computationally expensive Drv with expectation over P in the definition of e"", it be easily derived that maximizing J (P) defined in Section 3.1 is equivalent minimizing E. E.E As for the policy shift term 6,, though the bound suggests that this term may also be constrained, we found empirically that it is sufficient only control the model error. This may be explained by the relatively small scale of policy shift with respect to model error, as observed in [Janner et al., 2019].
is worth noting that the accumulative error here is not simply summing up the multi-step errors on trajectories collected by any other policies in [Nagabandi et al., 2018, Luo et al., 2019], because the definition of the step-wise model error (i.e.. <f") in Theorem involves a policy-dependent expectation over the state space, indicating that this error is also affected by the update of the policy. To enable quick adaptation to this policy update, P2P minimizes the accumulative error on the state distribution induced by active interaction between the model and the current policy, and thus better approximates the accumulative eгror defined in Theorem 1.
Different from most of prior work [Luo al., 2019, Janner et al., 2019], in Theorem I the step- wise model prediction errors are not scaled up to their maximum over timesteps (i.e., max, e,"), leading tighter error bound. As result, this bound not only provides stronger guarantee for policy improvement, but also justifies the objective mismatch issue between the model learning and model usage from theoretical perspective: Scaling up these eπrоr terms by such a potentially large magnitude yields a loose bound which leads to algorithms aiming at minimizing the upper bound of individual one-step prediction losses. However, to obtain better policy improvement, the tighter bound in Eq. (1) implies that a more fine-nrained model learning process is desired, which considers consecutive rollout steps and minimizes the accumulative prediction error, i.e., E. 7ee"" Using the notion of model MDP defined in Section 3.1, the model induced by minimizing one-step prediction errors can be intuitively interpreted as a greedy decision maker, which is often considered shortsighted and may easily lead to severe suboptimality.
3.3 Practical Implementation
In this subsection, we instantiate Algorithm 1 by specifying explicit approaches to learn the model and providing solutions to some practical issues. Detailed pseudo codes can be found in Appendix E.
P2P-MPC: Since in our problem formulation the "dynamics" of the model rollout process, i..., the current policy π, is accessible, one of the simplest yet effective ways to learn the model can be the model predictive control (MPC) [Camacho and Alba, 2013], which utilizes the dynamics to plan and optimize for a sequence of actions. Given the state x at step f, the MPC controller first optimizes the sequence of actions a over finite horizon H, and then employs the first action of
the optimal action sequence, i.c., aHi f == arg max mm 6448
Rd a. ). Computing the Ep Et=l
5

---

InverteaDoubiieenndllum
Hopper
HalfCheetah
10000
4000
10000
Aver Nerrage R.c0m
Anal sapa Natumn
2000
5000
0
5k
10k
23k
40k
0
35k Steps
70k
Steps
Steps
Walker2d
Ant
Humanoid
5000
5000
6000
Avcrage Retuu Retum
Avdrage Retur Retum
2500
2500
150k
P2P-MPC
P2P-RL
MBPO
STEVE
SAC
SLBO
Comparisons against baselines on MuJoCo. Solid curves represent the mean of runs over 5 different random sceds, and shaded regions correspond to standard deviation among these runs.
Figure 2:
exact arg max is computationally challenging due the dimension of the state and action spaces, so we adopt the random-sampinn shooting method [Rao, 2009] which generates random action sequences, executes them respectively, and chooses the one with highest return predicted by the dynamics. Besides, the reward RT which cannot be directly computed by definition, is approximated here using a neural network and trained on the environment dataset. Specifically, during each training iteration, P2P-MPC first trains the model via traditional one-step prediction loss, and then trains the R network by taking transitions sampled from the environment dataset as inputs, and the minus prediction errors on these transitions as labels. The prediction error of an environment transition (8, a, r, 8') is computed via ||8 - 8'1l + || -rl. where 8', f are sampled from P(s (l,
P2P-RL: It is well-known that though effective, RL methods often suffer from high sample complexity. Hence, the model trained the environment dataset instead of the samples generated by the interaction of the model and policy. However, there are two issues to be addressed: 1) As the "environment dynamics", the policy is highly non-stationary due its continuing update; and 2) The decision maker which generates the dataset is the true environment rather than our model. The first issue can be addressed by updating the next state each sampled transitions by applying the current policy, i.e., $1+1 + (St+1, π 1)). As for the second issue, since the model-generated samples not used in model learning. it can be seen approximately as offline learning problem and thus can be addressed by well well-developed offline RL methods [Fujimoto et al., 2019, Levine et al., 2020]. Due to the effectiveness and the implementational simplicity, we adopt SAC [Haarnoja et al., 2018] with behavior cloning as the underlying learning algorithm, which bears similarities to the TD3+BC approach proposed by [Fujimoto and Gu, 2021]. Besides, an approach called DualDICE [Nachum et al., 2019] is also applied to correct the estimation of the state distribution. Finally, since the samples come from the true environment, the reward Rm can be simply approximated by re-predicting the dynamics in the sampled transitions and computing the prediction errors for the current model. Specifically, P2P-RL trains the model on the environment dataset and treats the model learning process offline RL problem, where the true dynamics becomes the "decision maker" of the environment dataset in our problem formulation. Thus, regarding a transition (8, a, r, 8'), R can be directly approximated by computing -||3' - s'll || - rl] where 8', " ~ (1,5)88)))
Experiments
In this section, we first evaluate the empirical performance P2P on set of MuJoCo continuous control tasks [TTodorvv et al., 2012] in Section 4.1, and then present an quantitative analysis for the model error in Section 4.2. The ablation study provided in Section 4.3 verifies the necessity of an active model interaction when computing the multi-step prediction loss. Besides, key

---

Hopper
Ant
Humanoid
Acamuutive
2
7
9
Stops
Stass
Steps
P2P-MPC
P3P-RIL
Figure 3: Quantitative analysis of the accumulative model errors. While the true environment dynamics is not accessible during learning, the errors approximated via supervised learning on the environment dataset.
parameter of P2P-MPC is also studied in Section 4.3. Finally, to gain a deeper understanding of how P2P work in offline settings, a visualization of the model's rollout trajectories is shown in Section 4.4.
4.1 Comparisons with Baselines
We compare P2P against some shate-of-thh-art model-free and model-based methods. The model-free baseline is SAC [Haarnoja et al., 2018], which is the model-free counterpart of P2P. The model- based baselines include MBPO [Janner et al., 2019], which uses short-horizon rollouts branched from the state distribution of previous policies, SLBO [Luo et al., 2019], which enjoys theoretical performance guarantee and uses model rollouts from the initial state distribution, and STEVE [Buckman et al., 2018), which also generates short rollouts but uses model data for estimating target values instead of policy learning. Note that framework only focuses model learning and can be employed as a flexible plug-in component by most MBRL algorithms that generate pscudo samples to assist policy learning. In our experiments, we plug the model learning process of P2P into MBPO since is a widely accepted strong baseline in MBRL. The implementational details and the hyperparameter settings of P2P, as well as the environment settings, are all described in Appendix B. As shown in Figure 2, P2P-MPC significantly outperforms the baselines on several tasks, especially on Hopper, Humanoid and Ant, which are generally considered to be the most challenging MuJoCo tasks. In Walker2d, the best rollout length is one step in our implementation and MBPO, thus resulting in similar performance between these two methods. Compared to P2P-MPC, the performance of P2P-RI is less stable. Through an investigation on the model learning process (see Appendix C). we find that P2P-RL sometimes struggles balance the loss of behavior cloning and RL, leading to the difficulty in hyperparameter tuning and the instability of learning network parameters, while the optimization of P2P-MPC is non-parametric and thus does not suffer from this issue.
4.2 Model Error Analysis
The theoretical result presented in Section 3.2 implies that to guarantee better policy improvement, the model should learn to minimize the accumulative errors along the trajectories induced by the current policy and the model. Since P2P is designed based on this principle, the empirical results in Figure 2 verify the theoretical result in terms of the final performance. To further verify the effectiveness of this algorithmic design, we study the accumulative prediction error on three tasks that require rollouts with relative long horizons, i.e., Hopper, Ant and Humanoid. The results shown in Figure 3 together with Figure 2 strongly support our theoretical claims by demonstrating that the algorithm inducing less accumulative error achieves better performance. Besides, we can also observe that in Hopper the model errors of P2P-MPC in the first few steps are slightly larger than the other two methods. Nevertheless, as the rollout trajectory goes longer, the accumulative error induced by P2P-MPC grows significantly slower than the other methods. This further agrees with the intuition mentioned in Figure 1 by showing that P2P is able to trade the one-step model error for accumulative model error by considering the long-term effect of the immediate prediction.
4.3 Ablation Study
Multi-Step Prediction Loss In Section 3.2, we emphasize that the loss of P2P is essentially different from the multi-step prediction loss used in prior work, since P2P computes the multi-step loss on the state distribution induced from active interaction between the model and the current
Code available https: //tp:::///ttbb../// wtt.ttttb..co//ifaaa////// ttredd...
7

---

Hopper
Humandid
Ant
6000
Nerrge Setum
Neeron Seturn
Anurage Retum
2000
3000
2500
0
20k
40k
40k
Bick
120k
50c
1.0kk
1.5k
Steps
Steps
Staps
Computed by Interaction
Computed by Previous Trajectories
Figure 4: Results comparing performances induced by two ways of computing the multi-step prediction loss. The yellow curves represent the performance of P2P-MPC, which minimizes the multi-step loss the trajectories generated by active interactions between the model and the current policy. The blue curves show the results of an ablation version MBPO where the original one-step loss is replaced by multi-step loss computed over the trajectories sampled from the environment dataset. The lengths of these trajectories are set to the same in this comparison.
Hopper
Humanoid
Ant
4000
8000
5000
Avertee Roturn
Average
2000
3000
10ck
Horizon-O
Horizon-2
Horizon-4
Horizon-6
Figure 5: Results comparing performances induced by different planning horizons of the model in P2P-MPC.
policy, and the contrary, prior work computes this loss on the trajectories collected by previous policies. The empirical results in Figure 4 compare the performances induced by these two kind of losses and highlight the importance of the interactive model learning process in P2P.
Planning Horizon Different from most RL methods, MPC methods assign an explicit horizon for planning, hence P2P-MPC is able to set the planning horizon of the model to be exactly the same with the rollout horizon. This difference may partially explain the superior performance of P2P-MPC against P2P-RL. Nevertheless, longer planning horizon incurs higher computational cost, especially in tasks with high-dimensional state spaces. To trade-off the performance and the computational cost, our solution is to simply limit the planning horizon, and the empirical results shown Figure suggests that horizon of 6 is satisfactory for the tasks experimented.
4.4 An Interpretative Experiment
To gain an intuitive understanding of how the learning objective of P2P affects the model predictions, here we present a visualization of the rollout trajectories in maze2d-media medium, an offline learning task in the D4RL benchmark [Fu al., 2020]. As shown in Figure 6, the envi- ronment is slightly modified by lengthening the wall at the middle of the maze and fixing the goal to the top right corner. Besides, the number of offline samples that involve the grey region are partially discarded to increase this region's 's uncer- tainty. Figure 6 shows the top four trajectories that most frequently generated from fixed starting point using our methods and MOPO [Yu et al., 2020], a well-known offline model-based RL method that uses one-step error for model learning. The visualization clearly demonstrates the different rollout preferences of the two kinds of models: the model P2P-MPC tends generate trajectories heading to the regions with low uncertainty, while the model of MOPO fails to prevent the trajectories from being absorbed into the highly uncertain region. The resulting accumulative model errors are quantified in Figure 7, and as shown in Table 1, the final performance the induced policy of P2P-MPC is significantly better than MOPO. Note that in online settings, the model error is possible to be corrected by the newly collected data from the environment, while in offline
Figure 6: Visualiza- tion of models' rollout trajectories. The blue trajectories are generated MOPO, while the yel- low ones are generated by PPP-MPC.
settings this incremental data-collecting process is not allowed. Thus, the model error can have

---

Prediction Emors
P2PMFPC
MOPO
Environment
P2P-MPC
MOPO
maze2d-medium (modified)
230.2 + 39.5
171.2 + 32.4
maze2d-large-dieess
8.1 + 1.1
0.8 + 0.0
Figure 7: Accumulative prediction error along the rollout trajectories on the modified maze2d- medium task.
Table 1: Comparison with MOPO on two maze tasks.
more detrimental effect on policy learning, which may severely degrade the asymptotic performance as evidenced in Table 1. Additional results on much harder task (i.e., maze2d-large-vl are also provided in Table to further verifies the effectiveness of P2P-MPC.
5 Related Work
MBRL aims at improving the sample efficiency of model-free methods while maintaining high asymptotic performance. The research of MBRL can be roughly divided into two lines: the model usage and the model learning. The most common usage of the model is generating pseudo samples to enrich the data buffer, as reduce the interaction with the environment and thus accelerate policy learning [Sutton, 1990, 1991, Deisenroth et al., 2013, Kalweit and Boedecker, 2017, Luo et al., 2019, Janner et al., 2019, Pan et al., 2020]. The main challenge of this kind of usage is the compounding model error of long-term predictions, which is also known as the model bias issue [Deisenroth and Rasmussen, 2011] 1]. A number of approaches are proposed address this issue, varying from using Dyna-like hooit- rollouts [Sutton, 1991, Janner et al., 2019], interpolating between real and pseudo samples [Kalweit and Boedecker, 2017], learning an uncertainty-aware probabilistic model ensemble [Chua et al., 2018], to masking the rollouts according to the uncertainty estimation [Pan et at 2020]. Falling into this category of model usage, the focus of our work is to improve the quality of model-geeneratd trajectories by reducing the accumulative model error along these trajectories. Through an active interaction between the model and the current policy, P2P reinforces quick model adaptation to the update of policy, so as to foresee and thus avoid the uncertainty in the future trajectory during rollout. Hence, one of the major differences between P2P and prior work can be interpreted as: not just to passively estimate the uncertainty after encountering it, but to actively avoid it before this encounter.
It is worth noting that there are also methods based on the optimism in face of uncertainty, which try exploit the uncertainty of the model for sufficient exploration in the real environment, such [Curi et al., 2021]. If the model's promise of large return turns out to be false, executing the policy trained by such a model will hopefully collect real-world counterfactual data to the previously erroneous model. However, this correcting process itself requires additional samples, which may run counter to the principle of developing model-based methods. Furthermore, exploring the erroncous part of the model may not only provides the agent with false information about the real environment, but also be prone to the model exploitation issue which severely hurts the asymptotic performance of the policy. Therefore, how to carefully exploit the model uncertainty remains an open question.
In the line of model learning, the most commonly used objective is to simply minimize each one-step prediction error for transitions available in the environment dataset [Kurutach et al., 2018, Chua et al., 2018, Janner et al., 2019]. However, since the data distribution of the environment dataset varies with the update of policy, it is unlikely to learn an accurate model for all the transitions in complex environments. Thus, when the model is used to predict for multiple steps, there exists an objective mismatch [Lambert et al., 2020] between model usage and model learning. To address this issue, we propose to minimize the accumulative prediction error under the state distribution of the current policy. The most relevant work to ours includes [Nagabandi et al., 2018, Luo et al., 2019], where a similar multi-step step prediction loss employed for model learning. The essential difference between their multi-step loss and ours is clarified in Section 3.1 and 3.2. Generative models [Kingma and Welling, 2014] have also been utilized to perform multi-step prediction [Mishra et al., 2017, Ha and Schmidhuber, 2018]. However, the need for a highly exploratory dataset limits their application to relatively simple and low-dimensional tasks. Other learning objectives improve traditional model learning from angles that are less similar to ours, including goal-aware model

---

learning that predicts how far the current state is from the goal state [Nair et al., 2020] and decision- aware model learning that aims to better aids policy learning by re-weighting the model error on different samples [Farahmand et al., 2017, Farahmand, 2018, 8, D'Oro et al., 2020].
In recent years, a series of work has attempted learn the model by treating the model rollout process as a sequential periiionnmaking problem. While these work shares some similarity with our work, the targeted problems and the proposed solutions are not exactly the same as P2P. Shang et al. [2019] propose an environment reconstruction method which models the influence of the hidden confounder on the environment by treating the platform, the user and the confounder as three agents interacting with each other. They focus on the offline setting (i.e., RL-based recommendation) and simultaneously train the model and the policy using a multi multi-agent imitation learning method. Xu et al. [2020] treat the model as dual agent and analyze the error bounds of the model. They propose to train the model using imitation learning methods. Chen et al. [2022] also consider multi-step model error, yet they mainly focus on handling counterfactual data queried by adversarial policies. Unlike the above two work [Xu et al., 2020, Chen et al., 2022] that both focus solely on model learning. P2P aims at proposing a general MBRL framework and attempts to adapt the model to the continuously
updating policy quickly.
Conclusion and Future Work
In this paper, we propose an MBRL framework named Plan To Predict (P2P), which novelly treats the model rollout process sequential decision making problem. In this problem reformulation, the roles played by the model and the policy are reversed, where the model becomes a decision maker and the policy instead serves as the outer environment. Through an active interaction between the model and the current policy, the model can minimize the multi-step accumulative errors on the induced trajectories and thus quickly adapt the state distribution of the current policy. We provide theoretical guarantee for P2P, which also clarifies the difference of our multi-step loss from prior methods and justifies the mismatch of objectives between model learning and model usage. Empirical results on several challenging benchmark tasks verify the effectiveness of P2P. Moreover, observing that the capability to avoid model error makes P2P promising for offline settings, we also provide preliminary results on some offline learning tasks. While this is not the major focus of this paper, studying how P2P can be applied in offline tasks more properly is left as an important future work.
Acknowledgments and Disclosure of Funding
The work is supported by the National Natural Science Foundation of China under Grant 62076259. The authors would like thank Qian Lin, Zongkai Liu and Xuejing Zheng for the useful discussions, and thank Hanlin Yang for the help in drawing Figure (b). Finally, the reviewers and metareviewer highly appreciated for their constructive feedback the paper.
References
Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. Model-based reinforcement learning: A survey. arXiv preprini urXiv:2006. 16712, 2020.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ics for bbsel deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference Robotics and Automation, pages 1559-7566. IEEE, 2018
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorith- mic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: model-based policy optimization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages
2019.
Marc Deisenroth and Carl E Rasmussen. Pilco: A model- and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning, pages 465-472. Citeseer, 2011.
10

---

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference Learning Representations, 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 4759-4770, 2018.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. arXiv preprint arXiv:2002.04523. 2020.
Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning, 91)3): 325-349, 2013.
Eduardo F Camacho and Carlos Bordons Alba, Model predictive control. Springer science & business media, 2013.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
Anil Rao. survey of numerical methods for optimal control. Advances the Astronautical Sciences, 135(1):497-228, 2009.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164) 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning. pages 1861-1870. PMLR, 2018.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Proceedings of the 35rd International Conference on Neural Information Processing Systems, pages 20132-20145, 2021.
Ofir Nachum, Yınlam Chow, Bo Dai, and Lihong Li. Dualdice: behavior-agnostic estimation of discounted stationary distribution corrections. Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 2318-2328, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEFE/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8234 8244, 2018.
Tianhe Yu. Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: model-based offline policy optimization. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 14129-14142, 2020.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990. pages 216-224. Elsevier, 1990.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 264):160-163, 1991.

---

Mare Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and trends in Robotics, 2(1-2):388-405 2013.
Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference Robot Learning, pages 195-206. PMLR, 2017.
Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the model when it is confident: masked model- based actor-critic. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 10537-10546, 2020.
Sebastian Curi, lija Bogunovic, and Andreas Krause. Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning. In International Conference on Machine Learning, pages 2254-2264. PMLR, 2021.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Nikhil Mishra, Pieter Abbeel, and Igor Mordatch. Prediction and control with temporal segment models. In International Conference on Machine Learning, pages 2459-2468, PMLR, 2017.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 2455-2467, 2018.
Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-aware prediction: Learning to model what matters. International Conference on Machine Learning, pages 7207-7219. PMLR, 2020.
Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski. Value-aware loss function for model- based reinforcement learning. In Artificial Intelligence and Statistics, pages 1486-1494.
PMLR, 2017.
mmssoud Farahmand. Iterative value-aware model learning. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 9090-9101, 2018
Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, and Marcello Restelli. Gradient-aware model-based policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3801-3808, 2020.
Wenjie Shang, Yang Yu, Qingyang Li, Zhiwei Qin, Yiping Meng, and Jieping Ye. Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 566-576. 2019.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 15737- 15749, 2020.
Xiong-Hui Hui Chen, Yang Yu, Zheng-Mao Zhu, Zhihua Yu, Zhenjun Chen, Chenghe Wang, Yinan Wu, Hongqiu Wu, Rong-Jun Qin, Ruijin Ding, et al. Adversarial counterfactual environment model learning, arXiv preprint arXiv:2206. 04890, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See the supplementary material and Appendix B.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the supplementary material and Appendix B.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "Rtzaz04-dOz", "reviewer": "Reviewer_bd9p", "summary": "The authors propose an MBRL framework named Plan to Predict (P2P), which treats the model rollout process as a sequential decision-making problem. The model in P2P can minimize the multi-step accumulative errors on the induced trajectories and thus quickly adapt to the state distribution of the current policy, the authors give a theoretical guarantee for P2P. Empirical results on several MuJoCo benchmark tasks verify the effectiveness of P2P.", "questions": "\n1. The description in Section 3.3 is a bit confusing. For P2P-MPC, Line 191 mentions \"Besides, the reward R^m, which cannot be directly computed by definition, is approximated here using a neural network and trained on the environment dataset\". How do we estimate R^m using a neural network, do you mean directly predicting the model error (sounds like a difficult job)? In P2P-RL, the authors mention (Line 205), \"simply approximated by re-predicting the dynamics in the sampled transitions and computing the prediction errors for the current model-policy\". It seems that R^m is obtained based on the data P^*, calculated by the \\hat P prediction error. But if we use the model for multi-step rollout, in the successor time steps, we cannot get the ground-truthP^*(s’|\\hat s,a), where \\hat s is predicted by \\hat P, how to calculate R^m? Also, for P2P-RL, a lot of technical details are mentioned here, such as: “updating the next state in each sampled transition by applying the current policy-dynamics“; ”adopting SAC with behavior cloning as the underlying learning algorithm“; ”DualDICE is also applied to correct the estimation of the state distribution“. I suggest that the authors can add pseudocode or a more complete description in the appendix to improve the readability of this part.\n\n\n2. The author's explanation for the poor performance of the P2P-RL method (in Line 230) is that \"we find that P2P-RL sometimes struggles to balance the loss of behavior cloning and RL, leading to the difficulty in hyperparameter tuning and the instability of learning network parameters”. But we can find that in the Hopper environment, the cumulative error of P2P-RL and P2P-MPC algorithms is similar (Figure 3), while the policy performance of P2P-RL is twice as bad. This seems to be contrary to the explanation proposed by the authors.\n\n3. I think some related work is missing in this article. In recent years, a series of practical work and theoretical analyses have attempted to learn the model using the \"treats the model rollout process as a sequential decision-making problem\" framework. Its solution is similar but not exactly the same as P2P-RL. I recommend the author to read the following articles and add to the related work, which may inspire the author's follow-up research and improvement work [1,2,3].\n\n\n[1] Wenjie Shang et al. Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation.  2019.\n\n[2] Tian Xu et al. Error Bounds of Imitating Policies and Environments. 2020.\n\n[3] Xiong-Hui Chen et al. Adversarial Counterfactual Environment Model Learning. 2022.\n\n\n\n\n", "limitations": "\nOverall, the article is sound and has no major drawbacks to me. If the authors clarify the proposed questions and improve the writing of Section 3.3, I will consider increasing the score of the article.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The article is overall easy to follow except Section 3.3 (see Questions below);\n2. The motivation to fix the mismatch between model learning and model usage is reasonable and valuable. \n3. The experiment is well-designed to demonstrate the mechanism of P2P.\n\nWeaknesses\n\n\nThe article is sound and has no major drawbacks to me. But there are several questions for the authors to further clarify (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QE7DLdPRSaz", "reviewer": "Reviewer_xhVo", "summary": "The paper proposes an improved method for learning models for model-based reinforcement learning, by recasting the supervised regression approach commonly used as a control problem.", "questions": "- Figure 1: while nicely designed, I do not think I fully understand what the figure represents. It has a lot of details and an unfamiliar setup. Please simplify and clarify.\n- Line 42: Since the term \"objective mismatch\" is used, I would encourage the authors to cite previous work by Lambert et al. and others.\n- Justification: It is unclear to me why exactly the model should prevent the policy from going into high uncertainty regions. First, in most model based methods, several trajectories would be computed (explicitly with random shooting, or implicitly, by repeatedly querying a probabilistic model). These should to some extend counteract the overestimation, as the model would be less controllable in this region. Furthermore, executing the next policy on the real environment will hopefully provide counterfactual data to the previously erroneous model.\n- How does predicting $R^m$ actually differ between this approach and MBPO? How is the expectation with regard to the current policy realized?\n- Line 162: I do not fully understand what is meant by this sentence? Is the difference in the data used for the update?\n- Line 190: The $R^m$ is trained as a neural network? How exactly? This seems crucial for making the method work, but is treated as a side comment.\n\nI think a lot of my confusion stems from the last question: how is the reward/loss for the mode actually computed. I think a minimal change that would alleviate a lot of my concerns would be to add a more clear, step by step explanation of the algorithm to the top of Section 3, instead of relegating implementation details to later? Since the setup is fairly different from other model learning approaches, a more \"tutorial style\" of writing would immensely benefit my (and hopefully others) understanding.", "limitations": "The method discusses an optimization method and therefore does not clearly require a discussion of societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The following will be very brief and most of my comments will be under \"Questions\", because I have serious doubts that I actually understood the method the authors are proposing. This is not due to lack of familiarity with the field, I have done research and published in model-based reinforcement learning. I will point out my problems in the next section.\n\nGiven the very nice results and the overall introduction, I am very favorable to recommend acceptance. This is not currently reflected in the score however, due to the questions I have about the paper. Therefore I would ask the authors to go through them and hopefully improve the presentation (or convince me that I overlooked crucial details of course :) ), and to not be discouraged by the low score! I think the ideas warrant a presentation at NeurIPS, but I think that the paper can be heavily improved to make it accessible and give it a chance to shine in the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NlaBpnCwz5g", "reviewer": "Reviewer_N1PQ", "summary": "Model based reinforcement learning promises to reduce sample complexity in reinforcement learning. MBRL is responsible for some of the major breakthroughs in AI, zuch as AlphaGo Zero, and AlphaFold. MBRL may suffer from model inaccuracy, especially when the learning of the model, which is typically doen with 1 step lookahead, is separate from model usage, which is typically done with many step lookahead. The paper suggests a solution to this problem, using a meta-approach, to approach the model learning as an MDP problem itself. \nThe paper offers both a theoretical analysis, and convincing experimental evidence of strongly better performance than some of the best model based, and some of the best model free approaches. This a convincing contribution.", "questions": "- can you compare to more other MBRL approaches\n- can you run on larger problems?", "limitations": "few limitations are mentioned in the paper. The authors are invited to provide more\nThere are no ethical issues to address", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- clear writing, clear problem statement\n- clear contribution\n- good theoretical analysis\n- convincing experimental evidence\n- important new idea, that performs substantially better. important contribution\n\nWeaknesses\n- few", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A_YB_J1pfmD", "reviewer": "Reviewer_q2Vv", "summary": "The paper proposes a model-based reinforcement learning algorithm that alternates between model-learning with the policy fixed, and policy improvement with the model fixed. The key insight is that the model-learning phase is treated as a Markov decision process (MDP) with the estimated model as the \"policy\" to be learned (with the actual policy fixed), where the reward is the negative mismatch between the estimated model and the actual system transition dynamics. The paper claims that encapsulating model-learning with a fixed policy as a \"reversed\" MDP encourages model-learning to more quickly adapt to policy updates and produce a model that is more accurate over multiple transition steps.", "questions": "Based on the discussion above in \"Strengths and Weaknesses\", I have the following questions and requests for clarification:\n\n- Is the reward function R(s,a) assumed to be known? If not, where and how do you learn R(s,a) in Algorithm 1?\n\n- For P2P-MPC, on lines 190-191 the authors state \"the reward Rm, which cannot be directly computed by definition, is approximated here using a neural network and trained on the environment dataset.\" For P2P-RL, on the lines 205-207 the authors state \"since the samples come from the true environment, the reward Rm can be simply approximated by re-predicting the dynamics in the sampled transitions and computing the prediction errors for the current model-policy\". Both statements lack significant detail and yet seem to be critical to model learning since they quantify the difference between the model P_hat and the true transition dynamics P. Please expand on these points.\n\n- Please expand on your evaluations of the model error induced by P2P and multi-step loss function baselines, particularly in Figure 4. As it stands, the lack of details severely hampers reproducibility.\n\n- The authors promote that their proposed method P2P learns by \"actively interacting with the current policy\" (line 66) and has the \"capability to avoid model error\" (line 84). Distinctly, the authors' ant-in-a-maze example scenario has the goal in a region of low model uncertainty. However, it is certainly possibly to encounter a scenario where the goal is in a region of high model uncertainty, thereby requiring further exploration in the online setting. Could P2P underperform or fail in such scenarios? Further experiments would be welcome here.\n", "limitations": "There is no substantial discussion of limitations in this work. There is some superficial comparison of P2P-MPC and P2P-RL at the end of Section 4.1 and in Appendix C, but little insight or discussion is provided. I have raised an issue regarding exploration-exploitation in the review sections above that may be useful for the authors to consider.\n\nThe authors have adequately addressed the societal impact of their work.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*** ORIGINALITY ***\n\nThe key contribution of the paper is the treatment of the unknown transition model as the \"decision-maker\" when paired with a fixed policy during model learning. The novelty of this idea is clearly established by the authors in their comparisons to prior work. The paper proposes the \"meta-algorithm\" Algorithm 1, which requires the user to choose from existing algorithms to specify the model and policy learning phases. Overall, the proposed novelty can be viewed as a specification on the objective for the model learning phase. However, the empirical improvements offered by this key insight are clearly demonstrated in Section 4, so there is strong value in it as a contribution.\n\n*** QUALITY ***\n\nThe paper is technically sound for the most part, and the methods used seem appropriate. However, it is unclear if the authors learn the reward function in Algorithm 1 at all -- there does not seem to be a description anywhere in the main paper if R(s,a) is known or must be learned. In addition, in Figure 4 of Section 4, the results cannot be interpreted because the authors have not provided a clear description of what these plots show. Indeed, the caption only vaguely states that these results compare \"performances induced by two ways of computing the multi-step prediction loss\". Since the authors repeatedly tout the benefit of their approach over prior approaches using multi-step objectives, this part of Section 4.3 should be expanded and improved.\n\n*** CLARITY ***\n\nThe paper is clearly written for the most part. Some notation in Theorem 1 is undefined in the main body of the paper and in the appendix (e.g., R_max, D_TV). The use of the term \"value\" throughout the paper should be clarified (e.g., does this refer to a value function?). The use of the terms \"policy-dynamics\" for the policy pi and \"model-policy\" for the dynamics model P_hat beginning on line 180 is confusing. It would probably be clearer to just refer to them as the policy and model, respectively, and ensure it is clear to the reader when either the policy-learning or the model-learning phase is being discussed.\n\n*** SIGNIFICANCE ***\n\nThe key idea in treating dynamics learning with a fixed policy as an MDP with the transition model as the \"policy\" is interesting and could be useful to other researchers. The empirical results, particularly in Figures 3 and 7, seem to show that P2P can potentially reduce model error over multi-step rollouts.\n\nThe primary concern I have with this paper is its treatment of regions in state-action space with high model uncertainty. The authors seem to consider such regions as ones that should be avoided during reinforcement learning. On line 272, the authors state that their method instead \"tends to generate trajectories heading to the regions with low uncertainty\", while the baseline method MOPO \"fails to prevent the trajectories from being absorbed into the highly uncertain region\". This line of reasoning runs counter to the exploration-exploitation principle in reinforcement learning. Indeed, while the scenario in Figures 1 and 6 seems to have the goal in a region of low model uncertainty, it is certainly possible to encounter scenarios where the goal is instead in a region of high model uncertainty. Thus, exploration into regions with high model uncertainty in the online setting perhaps should be encouraged rather than avoided (subject to, e.g., safety constraints). This is a potentially critical limitation of the proposed work that is not discussed, and perhaps should be investigated with an additional experiment where the goal is located in a region of high model uncertainty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept