Objective Mismatch in Model-based Reinforcement Learning
Authors: Nathan Lambert
Brandon Amos
Omry Yadan
Roberto Calandra
Venue:  (2020)
Keywords: Model-based Reinforcement learning
dynamics model
reinforcement learning

Abstract: Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks. Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, leaving the general framework virtually unchanged since its conception. In this paper, we identify a fundamental issue of the standard MBRL framework -- what we call the objective mismatch issue. Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized. In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t. the likelihood of the one-step ahead prediction, and the overall goal of improving performance on a downstream control task. For example, this issue can emerge with the realization that dynamics models effective for a specific task do not necessarily need to be globally accurate, and vice versa globally accurate models might not be sufficiently accurate locally to obtain good control performance on a specific task. In our experiments, we study this objective mismatch issue and demonstrate that the likelihood of the one-step ahead prediction is not always correlated with downstream control performance. This observation highlights a critical flaw in the current MBRL framework which will require further research to be fully understood and addressed. We propose an initial method to mitigate the mismatch issue by re-weighting dynamics model training. Building on it, we conclude with a discussion about other potential directions of future research for addressing this issue.

Under review as a conference paper at ICLR 2020
OBJECTIVE MISMATCH IN MODEL-BASED REINEORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Model-based reinforcement learning (MBRL) has been shown be powerful framework for data-efficiently learning control of continuous tasks. Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, leaving the general framework virtually unchanged since its conception, this paper, we identify a fundamental issue of the standard MBRL framework what we call the objective mismatch issue. Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, met- ric will also be optimized. In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t. the likelihood of the one-step ahead prediction, and the overall goal of improving performance a downstream control task. For example, this issue can emerge with the realization that dynamics models effective for specific task do not necessarily need be globally accurate, and vice versa globally accurate models might not be sufficiently accurate locally to obtain good control performance on a specific task. In our experiments, we study this objective mismatch issue and demonstrate that the like- lihood of the one-step ahead prediction is not always correlated with downstream control performance. This observation highlights a critical flaw in the current MBRL framework which will require further research to be fully understood and addressed. We propose initial method to mitigate the mismatch issue by re- weighting dynamics model training. Building on it, we conclude with a discussion about other potential directions of future research for addressing this issue.
INTRODUCTION
Model-basee reinforcement learning (MBRL) is popular approach for learning to control nonlinear systems that cannot be expressed analytically (Bertsekas) 1995| SSttton and Barto 2018 Deisen roth and Rasmussen 2011 Williams et al. 2017). MBRL techniques achieve the state of the art performance for
cocuuuuuu-coorror problems with access to a limited number of trials (e.g. CCua al. (0018); Wang and Ba ( and in controlling systems given only visual observations with no observations of the original system's 's state (e.g. HHfnnr et al. (018); Zhang et al. (018)). MBRL approaches typically learn a forward dynamics model that predicts how the dynamical system will evolve when a set of control signals are applied. This model is classically fit with respect to the maximum likelihood of set of trajectories collected on the real system, and then used as part of control algorithm to be executed on the system (e.g., model-predictive control).
In this paper, we highlight fundamental problem in the standard MBRL learning scheme: the objective mismatch issue. The learning of the forward dynamics model is decoupled from the subsequent controller and policy that it induces through the optimization of two different objective functions negative log- likeiiidd (or its equivalent for deterministic models, e.g., the RMSE) of the single- or multi-step look-ahead prediction for the dynamics model, and task performance (i.e. reward) for the policy optimization. While the use of negative log-likelihood (NLL) for system identification is an historically accepted objective, it results in optimizing an objective that does not necessarily correlate to the actual performance. The contributions of this paper are to: 1) identify and formalize the problem of objective mismatch in MBRL; 2) examine the signs of and the effects of objective mismatch simulated control tasks; 3) propose initial mechanism to mitigate objective mismatch; 4) discuss the impact of objective mismatch on existing MBRL and outline future directions to address this problem.

---

Under review as a conference paper at ICLR 2020
Control
Interacis
Dynamics fa
Policy Ry (x)
Environment
Training: Maximum Lizellbood
Objocsive Mismutch
Responses
Trajectories
State Transitions
Reward
Figure 1: Objective mismatch MBRL arises when dynamics model trained to maximize the likelihood but then used for the policy maximize reward signal that is not used during training.
MODEL-BASED REINFORCEMENT LEARNING
We now outline the MBRL formulation used in the paper. At time I, we denote the state 8, E IR the actions Og Rd,, and the reward r(*,; (t). We say that the MBRL agent acts in an environment gov- erned by a state transition distribution p($ ar). We denote a parametric model to approximate this distribution with ( (8, MBRL follows the general approach of agent acting in its environ- ment, learning a model said environment, and then leveraging the model to act more effectively, summarized in Alg. While iterating over para- metric control policies, the agent collects measurements of state, action, next-state and forms a dataset D ((,,, an an. s,,) where N is the sum of the length of all episodes. With the dy- namics data D, the agent learns the environment in the form of a neural network forward dynamics model, learning approximate dynamics po- This dynamics model is leveraged by controller that takes in the current state 9t and returns an action sequence d maximizing the expected reward (eo t-- r(s;, where T is the predictive horizon and (*())) ) is the set of state transitions induced by the model pa- In our paper, we primarily use probabilistic networks designed to minimize the NLL of the predicted parametric distribution pg. denoted as P. ensembles of probabilistic networks denoted PE, and compare to deterministic networks minimizing the mean squared er- the models in PETS (Chua et al.,
Algorithm 1: Model-based RL
Data: Initialize D from random actions while Improving do
Train model p (sc, az) on
Collect data D' with controller using real environment Aggregate dataset D = DUD'
(MSE), denoted D or DE. Unless otherwise stated 2018) with an expectation- based trajectory planner and a cross-entropy meees-eerrppy-etthod method (CEM) optimizer.
3 OBJECTIVE MISMATCH AND ITS CONSEQUENCES
The Origin of Objective Mismatch: The Sub- tle Differences between MBRL and System Identification Many ideas and concepts in model-based RI. are rooted in the field of opti- mal control and system identification (Bertsckas)
Goal
1995: Zhou al.] 1996; Kirk 2012 Bryson 2018 Sutton and Barto 2018). In system identifi cation, the main idea is to two-step process where we first generate on the robot (optimal) elicitation trajectories
Start
fit dynamics model (typically analytical), and subsequently we ap- ply this model to a specific task. This particular scheme has multiple assumptions: 1) the data col- lected cover the entire state-action space (done by appropriate elicitation trajectories); 2) the pres- ence of large (virtually infinite) amount of data; 3) the global nature of the model resulting from the system identification process (which should be generalizable to large range of tasks). With these assumptions, the general idea of system identification effectively collect large amount of data covering the whole state-space create
Figure 2: Sketch of the state-action space for system identification and MBRL. (Lefi) In sys- tem identification, the elicitation trajectories are designed off-line to cover the entire state-action space, without considering a specific task. (Right) In MBRL instead, the data collected during the learning often concentrated wards the goal, with other parts of the state-action space being completely unexplored (grey area).
trajectories to-
2

---

Under review as a conference paper at ICLR 2020
a global model that is sufficiently accurate that we can at deployment time specify any desired task, and still obtain good performance.
When adopting the idea of learning the dynamics model used in optimal control for MBRL, it is important to consider if these assumptions still hold. The assumption of virtually infinite data is visibly in tension with the explicit goal of MBRL which is to reduce the number of interactions with the environment by being "smart" about the sampling of new trajectories. In fact, in MBRL the offline data collection performed via elicitation trajectories is largely replaced by on-policy sampling (with the exception of some initial motor babbling (Chua el al. 2018) in order to explicitly reduce the need to collect large amount of data. Moreover in practice in the MBRL setting the data will not usually cover the entire state-action space, since they are generated by policy trying to optimize one specific task. conjunction with the use of non-parametrie models (i.e., not analytical model), this results in learned models that are strongly biased towards capturing the distribution of the data, and that are only locally accurate. Nonetheless, this is at first sight not an issue since the common MBRL setting consider the learning aimed at solving only one specific task, and rarely test for generalization capabilities of the learned dynamics.
In practice, we can now see how the assumptions and goals of system identification are in contrast with the ones of MBRL. Understanding these differences and the downstream effects on algorithmic approach is crucial to design new families of MBRL algorithms.
Objective Mismatch During the MBRL process of iteratively learning controller, the reward signal from the environment diluted by the training of forward dynamics model with independent metric, as showing in Fig. [. In our experiments, we highlight that the minimization of some network training cost does not hold a strong correlation to maximization of episode reward. As dynamic environments becoming increasingly complex in dimensionality, the assumptions of collected data distributions become weaker and over-fitting to different data poses an increased risk.
Formally, the problem of objective mismatch appears two de-coupled optimization problems repeated over many cycles of Alg. []] shown in Eq. (Ia.b). which could be at the cost of minimizing the final reward. This loop becomes increasingly difficult analyze as the dataset used for model training changes with each experimental trial a step that is needed to include new data from previously unexplored states. In this paper we characterize the problems introduced by the interaction of these two optimization problems, but avoid to consider the interactions added by the changes in data distribution during the learning process, as this would significantly increase the complexity of the analysis. In addition, we discuss potential solutions, but do not make claims about the best way to do so, which is left for future work.
N
I+T
Training: arg min
log po po(s((s;,,a)
Control: arg max (x)
Σ r(s;,a))
(la.b)
IDENTIFYING OBJECTIVE MISMATCH
In this section, we experimentally study the issue of objective mismatch in MBRL to answer the following questions: 1) Does the distribution of models obtained from running a MBRL algorithm show a strong correlation between NI.I and reward? 2) Are there signs of sub-optimality in the dynamics models training process that could be limiting performance? 3) What model differences are reflected in reward but not in NLL?
Experimental Setting In our experiments, we use two popular RL benchmark tasks: the cartpole and half cheetah. For more details on these tasks see Chua et all (201)). For our cartpole experiments, we aggregate dynamics models at each episode from 50 runs of PETS (20 trials per run) paired with the on-policy episode reward achieved, giving the number of models Mcp 1000. Our half cheetah experiments consist runs (300 trials per run) of PETS, tabulating the reward with the dynamics models again, yielding Mhe 2400. All dynamics models are of depth 3, hidden width 500, and use the tanh activation function. With a large set of dynamics models in the feasible space of on- on-policy models, then used series of control datasets to evaluate the NLL of the dynamics models versus performance. The datasets we use are designed to investigate how different assumptions made in MBRL trickle down into final performance. We start with expert datasets to
3

---

Under review as a conference paper at ICLR 2020
Eplicd
Au
Libelioodd
Log Likelhhood
Litehhodd
(a) Cartpole Expert (p- 0.59)
(b) Cartpole On-Policy (p 0.34)
(c) Cartpole Grid (p 0.06)
Eqteod Boveed
-10
Log Likeltood
Log Likdibood
Log Likellbood
(d) Half Cheetah Expert (p 10=00 0.07)
(e) Half Cheetah On-Policy
(l) Half Cheetah Sampled (p 0.19)
(p 0.46)
Figure 3: The distribution of dynamics models from our PETS experiments plotting in the Le-Reward space on three datasets, with correlation coefficients p. (Notes: we are using log likelihood (LL) here rather than NLL easier interpretability and the reward at each point is the mean of 10 trials with the CEM optimizer, to disentangle the stochasticity of MPC.) The numbers of models evaluated are Map 1000 and Mr.- = 2400. For the datasets shown in (c,d.f) many of the LL's are extremely high and outside of the range of the figure, but the trend lines and correlation coefficients are calculated on all points. There is trend of high reward to 'good' LL that breaks down the datasets contain more of the state-space than only expert trajectories. Trend lines are regressed from total least squares, and included when the sum of squares solution implemented converges.
test if on-policy performance is only linked to having adequately explored the environment. As baseline, we compare the expert data to datasets collected by standard data aggregation procedures when learning. Finally, dataset representing the entire state space is generated as a grid over the state-space or by sampling from reasonable values and measuring the transitions in order to test how complete dynamics models relates to model accuracy and final performance. Specifically, the cartpole datasets are as follows: an expert selection of episodes data with r > 179 (2, 400 points), a standard on-policy dataset aggregated from the end of PETS run (3, 780 points), and a grid of datapoints to approximate the full system dynamics (16, 807 points). For half cheetah we compared three similar datasets: an expert dataset from a PETS run of half cheetah using the true dynamics for planning (3, 000 points), an on-policy dataset aggregated from PETS (90. 900 points), and an uniform sampled dataset attempting to approximate the full system dynamics (200, 000 points).
4.1 EXPLORATION OF MODEL LOSS VS EPISODE REWARD SPACE
In the standard MBRL framework, it is assumed that there is a clear proportional relationship between model testing loss and on-policy performance. We here show that this is a strong assumption even in simple domains, and on specific on-policy data. The relationships between model accuracy on data representing the full environment space and reward show no concrete trend in Fig. 3... The simplicity of the cartpole environment results in quick learning and a concentration of networks around peak reward. The distribution of rewards versus log-likelihood (L.L. shown in Fig. 3 where there is a trend of increased reward as LL decreases, but there is substantial variance and points of disagreement. This bi-model distribution on the half cheetah expert data set, shown in most clearly Fig. 31, relates to a failure mode in early half cheetah trials where the agent enters unrecoverable state. The contrast between Fig. 3 and Fig. Bi. shows substantial difference in the transitions represented within the datasets. The multiple modes by which half cheetah runs during different stages of learning warrants more investigation into how new data should be incorporated into a training dataset while maintaining the knowledge of older data.
4

---

Under review as a conference paper at ICLR 2020
Validation Error
Episode Reward (x)
(a) D model.
(b) DE model (E = 5).
(c) model.
(d) PE model (E 5).
The reward versus epoch when re-evaluating the controller leveraging a dynamics model at each training epoch for different types of dynamics models. Even for the simple cartpole environment, networks of width 500 and depth 3 cannot learn the entire grid dataset: simple models (D, DE) fail to achieve full performance, while more advanced models reach higher performance but eventually over-fit to available data (P, PE). The over-fitting of the P model is further evaluated in Fig. 5a
Figure 4:
These results show that objective mismatch is not preventing MBRL from functioning, which would appear as model loss being fully dissociated with performance and the plots shown lacking any coherent shape. Rather, the mismatch we show likely is ceiling on the performance of current MBRL algorithms by reducing the correlation between model accuracy and evaluation reward. The distributions in this section of on-policy data in Fig. B3e show noisy trend of higher reward with better model loss and not the guaraniee of improvement that is expected when training a better model.
4.2 MODEL LOss VS EPISODE REWARD DURING TRAINING
This section explores the relationship between how model training impacts performance at the per-epoch
Grid Data (A)
Policy Data (O)
Expert Data (+)
Episode Reward
level. These experiments also shed light onto the im-
pact the strong dataset assumptions outlined Sec. [3] Figures F and are generated by re-evaluating the con- troller leveraging a dynamics model at each training epoch. As dynamics model is trained, there two key inflection points the first is the training epoch where episode reward is maximized, and the second is when error on the validation set is minimized. When training data from the entire state state-space, our experi- ments show that the reward is maximized at a drastically different time than when validation loss is minimized. These experiments are focused on showing the discon- nect between three standard practices in MBRL a) the assumption that the dynamics data can express large portions of the state-space when collected on policy, b) the idea that simple neural networks can satisfacto- rily capture complex dynamics, c) and the practice that model training is a simple optimization problem discon- nected from reward. For the cartpole environment with the grid dataset, Fig. Fig.44sho shows that the performance of the controller degrades for P, PE models and fails to adequately learn with the D, DE models. For cartpole, training on on-policy data replicates results of the tri- als (> 175 reward). shown in Fig. [5b] while the expert dataset fails to achieve any learned performance (< 20 reward) because it lacks robust enough dataset for stable prediction generation during stochastic action sampling. Fig. Fig.5Jhightt highhligs how the trained models are able to represent other datasets that they are trained which points to the need for fine-tuming fine-tuning of that assumption on datasets made at the outset between the state space and that collected on-policy. There is no information indicating the on-policy dataset will lead to a complete dynamics understanding because on- oolioiic and grid show little correlation in evaluation loss. When training the gird cartpole, the fact that the on-policy data diverges before
Trming Upeck
(a) Trained: grid
Tested: expert, on-policy
190
Traising Speck
(b) Trained:
on-policy Tested: expert, grid
Figure 5: The effect of the dataset choice on model (P) training and accuracy in differ- ent regions of the state-space. (Top) when training the complete dataset, the model begin over-fitting to the on- on-policy data even before the performance drops in the con- troller. (Boltom) A model trained only on policy data does not accurately model the entire state-space.
5

---

Under review as a conference paper at ICLR 2020
Suit Vinu
87
Solt
14.2
182
-04
Planned Trajectorios
-04
Planned Trajectomes
88
Expert Traeectoyy
Expert Trajectory
Time (emvrrommnn
Time (frvitroment
(a) Initial Model: NLL:- 4.83, Reward:176
(b) Adversarially Generated Model:
NLL:-4.85, Reward:98
Figure 6: Planned trajectories along the expert trajectory for the initial model and the adversarially generated model trained to lower the reward. It can be seen how the planned trajectories are qualitatively similar except for the peak at 25. There, the adversarially generated model learned that applying a small nudge to the dynamics model at the right place/moment yield to significantly influencing the control outcome with minimal change terms of NLL..
the reward decreasing is encouraging objective mismatch may be preventable in simple tasks, but the problem becomes increasingly complex as the dynamics data distribution becomes large and multi-modal, as in half cheetah. Similar experiments on half cheetah data failed to achieve matching reward to the full trials because the training process in PETS involves incremental network with dataset aggregation. When training on single set of dynamics data for many epochs the controller did not achieve comparable reward to networks incrementally trained on new data after each episode.
4.3 DECOUPLING MODEL LOSS FROM CONTROLLER PERFORMANCE
When two dynamics models evaluated on a control dataset have similar validation loss, there can be a large variance in the reflected control policy. In this section, we explore how differences in dynamics models are qualitatively reflected in control policies to show the reader that a accurate dynamics model does not guarantee performance.
Adversarial attack on model performance We performed an adversarial attack (Goodfellow et al. 2015;; (Szegedy et al.] 2013) on neural network dynamics model so that it attains good likelihood but poor reward to continue illustrating objec- tive mismatch, start with a dynamics model that achieves high likelihood and high reward and tweak the parameters so that it continues achieving high likelihood but has a low reward. We fine-tune the network's ' last layer to minimize the episode reward, with a large penalty if the model validation likelihood drops below the original value. We use zeroth- order optimizer (CMA-ES) because the reward is generally non-differentiable. As starting point for this experiment sampled P dynamics model from the last trial of a PETS run on cartpole. This model achieves reward of 176 and has NLL of -48227 on it's on-policy training dataset. Using CMA-ES, we reduced the on-policy reward of the model to 98 45% decrease in performance compared to the original model while slightly improving the NL.L to -4.845; the CMA-ES convergence over population iterations is shown in Fig. and the difference between the two models visualized in Fig. 6 Fine tuning of all model parameters would be even more likely to find sub-optimal performing controllers with low model loss because the output layer consists of about 1 1% of the total model parameters. This experiment shows that the model parameters that achieve low model loss inhabit broader space than the subset that also achieves high reward.
180
Nacationn
tea
Reward
120
100
80
4.0
-52
-5.4
NL.1
Figure 7: Convergence of the CMA- ES population's is best member over iterations to the minimum reward achieved with comparable NLL.
Sampling models with similar NLLs, different rewards To better understand the objective mis- match, we also compared how a difference of model loss can impact a control policy. We sampled models with similar NLL's and extremely different rewards from Fig. Bile and visualized the chosen optimal action sequences along an expert trajectory. The control policies and dynamics models appear to be converging to different regions of state spaces. In these visualizations, there not a emphatic

---

Under review as a conference paper at ICLR 2020
3crert
6.9
633
170
341
1.7
war
0.2
Standaed Duraee Stze
Welghted Dosei Siee
Figure 8: Mean reward of PETS trials, with and without model re-weighting, on a log-grid of dynamics model training sets with number of points S E [10, 10000) and sampling expert-distance bounds < E [.28, 15.66]. The re-weighting shows an ability to learn moderate performance at substantially lower number of datapoints, but suffers from increased variance in larger set sizes. The performance of PETS declines when the dynamics model is trained on points too near to the expert dataset because the model lacks robustness when running online with the stochastic MPC.
reason why the models achieved different reward, so further study is needed to quantify the impact of model differences. The interpretability the difference between models and controllers will be issue.
important to solving the
5 ADDRESSING OBJECTIVE MISMATCH DURING MODEL TRAINING
Tweaking dynamics model training can partially mitigate the problem of objective mismatch. While keeping the NLL minimization standard in MBRL, trainer can prioritize state transitions associated with an expert trajectory over tuples further from expert, and dynamics model will use its capacity to learn relevant tasks more quickly. We show improved sample efficiency on Cartpole via re-weighting network training by measure of Euclidean distance in the state-action space.
Given a element a state space (s,, o.), we can quantify the distance of any two tuples {(s;, Oi, &f), 1,2] d Il[s1; 01] - [82, az]]l- With this distance, I(y), of points further from an expert policy to be lower, so that points in the expert trajectory get weight w(y)
re-weight the loss,
and points at the edge of the grid dataset used in Sec. 4 get a weight w(y) (in our experiments, multiple monotonic functions from 0 to had similar results). With this notion of distance and weighting, we used the expert dataset discussed in Sec. as a distance baseline. We generated a base dataset consisting of 25, 000, 000 tuples of (s, s') uniformly sampling across the state and action space of cartpole. We taxonomized this dataset by taking the minimum orthogonal distance, d", from each of the points to the 200 element dataset from one expert trajectory that achieved reward of 180. To create different datasets that range from near-expert to nearly uniform across the state space, vary the distance bound, and number of points, S, trained on. For each sampling bound, e, we sample 5 different datasets such that dž < from our distance-tabulated random dataset, and for each dataset (S, c) trained 5 P models, giving 25 episodes from PETS algorithm on the cartpole task for each point in the heatmap shown Fig. g This simple form of re- weighting the neural network loss, shown in Eq. (2a.bc). demonstrated
improvement in sample efficiency to learn the cartpole task, as seen in the left halves of each plot in Fig. g Developing an iterative method to re- seieighht samples in online training method has the potential further improve the sample efficiency of MBRL baselines.
(d"(s)
Weighting w(y)
Standard I(0.v)
Re-weight I(I. v) w(v)
(2a.b.c)
DISCUSSION, RELATED WORK, AND FUTURE WORK
Objective mismatch seriously impacts the performance of MBRL. Our experiments have gone deeper into this fragility in the context of the state-of-the-ar MBRL algorithms from an analysis perspective rather than from solution perspective. Beyond the re-weighting of the NLL presented in Sec. [5] here we summarize and discuss the key pieces in the community starting to address this issue.
7

---

Under review as a conference paper at ICLR 2020
Learning the dynamics model to optimize the task performance Most relevant to work are research directions on controllers that directly connect the reward signal back to terms of the controller. In theory, this exactly solves the model mismatch problem we discuss in this paper but in practice the current approaches are borderline intractable and have proven difficult to scale to complex systems. One popular way to do this is by designing systems that are fully differentiable so that the task reward can be backpropagated all the way to the dynamics. This has been investigated with differentiable MPC (Amos et al. 2018), Path Integral control (Okada et al. 2017 Pereira et al. 2018), and stochastic optimization (Donti et al. (017). Universal Planning Networks (Srinivas et al., 2018) propose differentiable planner that unrolls gradient descent steps over the action space of a planning network. An alternative explored in Bansal et al. (011 is to use a zero-order optimizer instead (Bayesian optimization specifically) which can used to learn locally linear dynamics to maximize the controller's performance without having to compute gradients explicitly.
Shaping the cost or reward Closely related are methods that, instead of learning a dynamics model optimizing task performance, fix the dynamics model and fine-tune the cost function of controller to optimize, e.g,, imitation loss Trimpe et al. (2014); Talvitie (2018): (Tamar et al. (2017).
Add heuristics to the dynamics model structure or training process to make control easier If infeasible or intractable to shape the cost or dynamics components of a controller, adding heuristics to the structure or training process of the dynamics model is reasonable and can give state-of-the results in many settings. One challenge in these heuristics is that they may be unstable and difficult to fix improve when they do not work in new environments. These heuristics can manifest in the form of learning a latent space that is locally linear, e.g., in Embed to Control and related methods (Watter et al.] 2015 Banijamali et al. 2017), by enforcing that the model makes long-horizon predictions (e et al. (019), ignoring parts of the state space that actions can not control (Ghssh et al., 2018). detecting and correcting when a predictive model steps off the manifold of reasonable states (Talvitie) 2017), adding reward signal prediction on top of the latent space Gelada al. (2019), adding noise when training transitions Mankowitz et al. (2019). among other approaches Jonschkowski et al. (2018): Ke et al. (2019); Miladinović et al. (2019); lchter and Pavone (2019);
Singh et al.. (2019).
Using incorrect models Instead of fixing the issues directly in the model or cost parameters, one can assume that the approximations are always incorrect but to use them anyway and potentially try to correct them Abbeel et al. 2006); /Sorg et al. (2010); Joseph et al. (2013); Nagabandi et al. (201 8).
Add inductive biases to the controller Prior knowledge and inductive biases can be added the controller in the form of saper-parameters such as the horizon length, or by penalizing unreasonable control sequences by using, e.g., a slew rate penalty. These heuristics can significantly improve the performance if done correctly but can be difficult to tune. Jiang et al. (2015) use complexity theory to justify using a short planning horizon with an approximate model to reduce the the class of induced policies. WWhiney and Fergus (2018) empirically show that long horizons are better for some tasks.
Continuing Experiments Our experiments represent an initial exploration into the challenges of objective mismatch in MBRL, but more analysis is needed. Our exploration in Sec. 4.2 is limited to cartpole due to computational challenges of training with large dynamics datasets and Sec. 443 could be strengthened by defining quantitative comparisons in controller performance other than episode reward. Additionally, the effects of model mismatch should be quantified in other state of the art algorithms in MBRL such as MBPO (Janner et al. 2019) and POPLIN (Wang and Ba 2019
7 CONCLUSION
This paper identifies, formalizes and analyzes the issue of objective mismatch in MBRL. This fundamental disconnect in objectives between the training of the model w.r.t. the likelihood, and the overall task reward emerges from the subtle differences at the origins of MBRL with assumptions that have not been fully adapted. Experimental results highlighted the negative effects that objective mismatch have on the performance of a current state of the art MBRL algorithm. Our results are an exposition of current limitations and an encouragement to investigate nuance in current practices. In providing a first insight on the issue of objective mismatch MBRL, we hope future work will more deeply examine these underlying assumptions. With our re- reiieigttnng method to mitigate mismatch, which also shows improvements in sample efficiency, we share our more thoughts about promising directions of future research to address this issue. We believe that fundamentally understanding and addressing the objective mismatch issue will contribute to improving the performance of MBRL.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Abbeel, M. Quigley, and A. Y. Ng. Using inaccurate models in reinforcement learning. In International Conference on Machine Learning (ICML), pages 1-8, 2006. ISBN 1-59593-383-2. doi: 10. 143844. 143845.
B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter. Differentiable MPC for end-to-end planning and control. Neural Information Processing Systems, pages 8289-8300, 2018.
Banijamali, R. Shu, M. Ghavamzadeh, H. Bui, and Ghodsi. Robust locally- controllable embedding. arXiv preprint arXiv: urXiv:/710.05377 2017.
S. Bansal, R. Calandra, T. Xiao, S. Levine, and C. J. Tomlin. Goal-driven dynamics learning via
Bayesian optimization. In IEEE Conference on Decision and Control (CDC), pages 5168-5173, 2017. doi:
D. P. Bertsekas. Dynamic programming and optimal control, volume 1. Athena scientific Belmont, MA, 1995.
A. E. Bryson. Applied optimal control: optimization, estimation and control. Routledge, 2018.
K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4754-4765, 2018.
M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465-472, 2011.
P. Donti, B. Amos, and J. Z. Kolter. Task-based end-to-end model learning in stochastic optimization. In Advances Neural Information Processing Systems, pages 5484-5494 2017.
C. Gelada, Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. arXiv preprint arXiv: 1906::/90.0027550 02736, 2019.
D. Ghosh, A. Gupta, and S. Levine. Learning actionable representations with goal-conditioned policies. arXiv preprint arXiv: 1811.07819, 2018
I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference Learning Representations (ICLR), 2015.
D. Hafner, T. Lillicrap. I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 1.04551,, 2018.
B. Ichter and M. Pavone. Robot motion planning in learned latent spaces. JEEE Robotics and Automation Letters, 2019.
M. Janner, Fu, M. Zhang, and S. Levine. When trust your model: Model-based policy optimiza- tion. arXiv preprini arXiv: /001:/990000255555 2019.
N. Jiang, A. Kulesza, S. Singh, and R. Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference Autonomous Agents and Multiagent Systems, pages 1 1181-1189, International Foundation for Autonomous Agents and Multiagent Systems, 2015.
R. Jonschkowski, D. Rastogi, and O. Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. arXiv preprint arXiv: 1 122, 2018.
J. Joseph, A. Geramifard, J. W. Roberts, J. P. How, and N. Roy. Reinforcement learning with misspecified model classes. In 2013 IEEE International Conference on Robotics and Automation, pages 939-946. IEEE, 2013.
N. R. Ke, A. Singh, A. Touati, A. Goyal, Y. Bengio, D. Parikh, and D. Batra. Learning dynamics model in reinforcement learning by incorporating the long term future. arXiv preprint 1rXiv:1903.01550 2019.

---

Under review as a conference paper at ICLR 2020
D. E. Kirk. Optimal control theory: an introduction. Courier Corporation, 2012.
D. J. Mankowitz, N. Levine, R. Jeong, A. Abdolmaleki, J. T. Springenberg, T. Mann, T. Hester, and M. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:1906.077
2019.
D. Miladinović, M. W. Gondal, B. Schölkopf, J. M. Buhmann, and S. Bauer. Disentangled state space representations. arXiv preprint 1906vv//00..::00 2019.
A. Nagabandi, Clavera, S. Liu, R. S. Fearing, Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv: 11.4111377 2018.
M. Okada, L. Rigazio, and T. Aoshima. Path integral networks: End-to-end differentiable optimal control. arXiv preprint arXiv: urXiv:1706.09597 2017.
M. Pereira, D. D. Fan, G. N. An, and E. Theodorou. Mpe-inspired neural network policies for sequential decision making. arXiv preprint arXiv:/80..0888 180200580, 2018.
S. Singh, S. M. Richards, V. Sindhwani, J.-J. E. Slotine, and M. Pavone. Learning stabilizable nonlinear dynamics with contraction based regularization, 2019.
Sorg, R. L. Lewis, and S. P. Singh. Reward design via online gradient ascent. In Advances in Neural Information Processing Systems, pages 2190-2198, 2010.
A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn. Universal planning networks. arXiv preprint 1804 2018.
R. Sutton and A. Barto. Reinforcement learning: An introduction. MIT press, 2018
C. Szegedy, W. Zaremba, I. Sutskever. Bruna, D. Erhan, I. Goodfellow, and Fergus. Intriguing properties of neural networks. arXiv preprint 1312.699,, 2013.
E. Talvitie. Self-correcting models for model-based reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
E. Talvitie. Learning the reward function for misspecified model. arXiv preprint arXiv:1801.00622 2018.
A. Tamar, G. Thomas, T. Zhang, S. Levine, and P. Abbeel. Learning from the hindsight plan episodic MPC improvement. In IEEE International Conference on Robotics and Automation (ICRA), pages 336-343, May 2017. doi: 10. (00./CCRAA201...998888
Trimpe, A. Millane, S. Doessegger, and R. D'Andrea. A self-tuning lqr approach demonstrated on an inverted pendulum. IFAC Proceedings Volumes, 47(3): 1281-1128811 2014.
T. Wang and J. Ba. Exploring model-based planning with policy networks. arXiv preprint 1rXiv:1906.086 2019.
M. Watter, Springenberg. Boedecker, and M. Riedmiller. Embed control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pages 2746-2755 22544, 2015.
W. Whitney and Fergus. Understanding the asymptotic performance of model-based methods. 2018.
G. Williams, N. Wagener. B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic MPC for model-based reinforcement learning. In International Conference on Robotics and Automation (ICRA), pages 1714-1721, 2017.
M. Zhang, S. Vikram, I. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: Deep structured latent representations for model-based reinforcement learning. arXiv preprint 1808 09105, 2018.
K. Zhou, J. C. Doyle, K. Glover, et al. Robust and optimal control, volume 40. Prentice hall New Jersey, 1996.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
A EFFECT OF DATASET DISTRIBUTION WHEN LEARNING
Learning speed can be slowed by many fac-
Random Transitions: 200
Random Transitions: 2000
tors in dataset distribution, such as adding additional irrelevant transitions. When ex- tra transitions from a specific area the state space are included in the training set, the dynamics model will spend increased expression on these transitions. NLL of the model will be biased down as it learns this data, but will reduce the learning speed as new, more relevant transitions are added to the training set.
Random Transitions: 4000
Random Transitions: 20000
Cunulitive Reward
150
100
S0
Running cartpole random data collection with short horizon 10 steps (while fore-
Trial Num
ing initial babbling state to always be O),
Figure 9: Cartpole (Mujoco simulations) learning effi- ciency is suppressed when additional data not relevant to the task is added to the dynamics model training set. This effect related to the issue objective mismatch because model training needs to account for potential off-task data.
for 20, 200,400 and 2000 babbling roll-
outs (that sums up to 200, 2000, 4000 and 20000 transitions in the dataset fi- nally shows some regression in the learning speed for runs with more useless data in the motor babbling. This data highlights the importance of careful exploration vs exploitation tradeoff's, or changing how models are trained to be selective with data.
B TASK GENERALIZATION IN SIMPLE ENVIRONMENTS
In this section, we compare the perfor- mance model trained on data for the standard cartpole task (x position goal at 0) to policies attempting to move the cart different positions in the x-axis. Fig. 10 is a learning curve of PETS with a PE model using the CEM optimizer. Even though performance levels out, the NLL contin- decrease the dynamics models accrue more data. With more complicated systems, such as halfcheetah, the reward of different tasks verses global likelihood of the model would likely be more inter- esting (especially with incremental model training) we will investigate this in future work. Below, we show that the dynamics model generalizes well to tasks close zero (both positive in (Fig. [Tb) and nega- tive positions (Fig. Ia), but performance
Validation Error
Episode Reward
140
90
60
12
14
Teadl Nammber
Figure 10: Learning curve for the standard Cartpole task used in this paper (X (X 0). The median reward from trials is plotted with the mean NLL of the dynamics models at each iteration. The reward reaches maximum (180) well before the NLL is at it' minimum. drops off in areas the training set does not cover well.
Below the learning curves in Fig. 12 we include snapshots of the distributions of training data used for these models at different trials, showing how coverage relates reward in cartpole. It is worth investigating how many points can be removed from the training set while maintaining peak performance each task.
11

---

Under review as a conference paper at ICLR 2020
good .1
gyoal .2
godd
.5
X 1.0
to
120
100
20
12 14 16
18
18
(a) X goo!
(b) X gont
Figure 11: MPC control with different reward functions with the same dynamics models loaded from trials shown in Fig. I0 The cartpole solves tasks further from 0 proportional to the state space coverage (Goal further from zero causes reduced performance). The distribution of æ data encountered is shown in Fig.
(a) distribution after trial 1.
(b) x distribution after trial 5.
(c) r distribution after trial 20.
Figure 12: Distribution of æ position encountered during the trials shown in Fig. 10 The distribution converges to high concentration around O, making it difficult for MPC to control outside of the area close to O.
C WAYS MODEL MISMATCH CAN HARM THE PERFORMANCE OF A CONTROLLER
Model mismatch between fitting the likelihood and optimizing the task's 's reward manifests itself in many ways. Here we highlight two of them and Sec. we discuss how related work connects in with these issues.
Long-horizon rollouts of the model may be unstable and inaccurate. Time-series or dynamics models that are unrolled for long periods of time easily diverge from the true prediction and can easily step into predicting future states that are not on the manifold of reasonable trajectories. Taking these faulty dynamics models and using them as a smaller part of controller that optimizes some cost function under poor approximation to the dynamics. Issues can especially manifest if, e.g., the approximate dynamics do not properly capture stationarity properties necessary the optimality of the true physical system being modeled.
Non-convex and non-smooth models may make the control optimization problem challenging The approximate dynamics might have bad properties that make the control optimization problem much more difficult than on the true system, even when the true optimal action sequence is optimal under the approximate model. This is especially non-line con-lincarities and montsmootsssss that make many classical control approaches difficult.
when using neural network they introduce
12


Reviews:
[{"review_id": "Ske8EvwptS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper claims that it identifies a fundamental issue in model-based reinforcement learning methods. The issue is called objective mismatch, which arises when one objective is optimized (for example, model learning objective) without taking into consideration of another objective (for example policy optimization). The author shows several experiments to illustrate the issue and proposes a method to mitigate it by assigning priorities to samples when training the model. \n\nThe issue of objective mismatch is not being noticed the first time. The paper \"Reinforcement learning with misspecified model classes\" has mentioned similar phenomenon. And other work such as https://arxiv.org/abs/1710.08005 also discussed similar issue. \n\nI disliked the way how the paper is motivated. The paper says “in standard MBRL framework”, what is the standard MBRL framework? I think there is no such standard so far. The claim saying that the mismatch is a crucial flaw in current MBRL framework is too strong. The paper at least completely ignored two broad classes of MBRL methods. The first is value-aware MBRL (which attempts to take into account decision making when learning a model), there are actually many MBRL methods are in this category. Some examples: value prediction network, Predictron, Value-Aware Model Learning. The second class of MBRL approach is Dyna. Several works (continuous deep q-learning with model based acceleration, Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains, Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation, Recall Traces: Backtracking Models for Efficient Reinforcement Learning, Hill Climbing on Value Estimates for Search-control in Dyna) show that even with the same model (hence the same model error), using different simulated experiences play a significant role of improving sample efficiency. It is unclear whether model-error plays a decisive role in improving sample efficiency. \n\nThe proposed method in section 5 lacks of justification, even a regular ER buffer is asymptotically on-policy, and hence it is indirectly linked with the control performance. It is unclear why introducing the weights based on expert trajectory can be helpful — it can be worse because it should be far away from on-policy distribution. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eiQv52tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"OBJECTIVE MISMATCH IN MODEL-BASED REINFORCEMENT LEARNING\" explores the relationships between model optimization and control improvement in model-based reinforcement learning. While it is an interesting problem, the paper fails at demonstrating really useful effects, and the writting needs to be greatly improved to help reader to focus on salient points. \n\nFrom my point of view, the main problem of this paper is that it is too messy and it is very difficult to understand what authors want to show, as i) there is a very important lack of experimental details (e.g., main aspects of models and controllers should be clearly stated) and ii) analysis is to wordy, authors should emphasize the message in each part. From the experiments in 4.1, the only thing that I got is from the last sentence \"noisy trend of higher reward with better model loss\". are these results from LL computed on a validation set ? If not, this is not reallly meaningfull since high LL may only indicate overfitting. If yes, how was the validation data collected ? If the collection is not inline with training it is difficult to understand what we observe since we only need LL to be good on the path from the current to the opitmal policy, not everywhere. Even if the validation data is inline with training, there remains the difficulty of over-fitting in the policy area (for the on-policy experiments at least). Is there something else ?  From 4.2 we observe that it is unsurprisingly better to learn the model from the policy trajectories. From 4.3, we observe that an adversarial is able to reduce rewards without losing in LL. Ok, the adversarial is able to lock the controler in a sub-optimal area while still being good to model the dynamics elsewhere, but what does it show ?  Finally, proposal to cope with the identified mismatch are not clearly explained and not very convincing. Is re-weighting helping in collecting higher rewards ? \n\nFrom my point of view, this work is in a too preliminary state to be published at ICLR ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "B1xf9mFAtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the old problem of mismatch between the ultimate reward obtained after optimizing a  decision (planning or control) over a probabilistic model (of dynamics) and  the training  objective for the model (log-likelihood). Experiments highlight that the NLL and reward can be very poorly correlated, that improvements in NLL initially improve reward but can later degrade it, and that models with similar NLLs can lead to very different rewards. A  reweighting trick is proposed and summarily evaluated.\n\nI like the topic of this paper but there are several aspects which I see as making it weaker than my acceptance threshold.\n\nFirst, the paper overclaims in originality. This mismatch problem is not new, it is an instance of a more general issue that end-to-end training and meta-learning try to address, and has been already studied in the context of MBRL by many authors, who actually proposed more substantial solutions. When I read the abstract I had the impression that the paper actually had a theoretical analysis showing the correlation problem, but there is no such thing, only experiments. Section 3 does not actually provide a new insight. Still, the experiments are interesting in that they reveal that the magnitude of the mismatch is probably more serious than most RL researchers believed.\n\nSecond, the 'fix' proposed is not well justified nor well tested (e.g. no quantiative comparisons, no comparisons against existing alternative methods to address the same problem, etc). This seriously weakens conclusions like \"shows improvements in sample efficiency\".\n\nOne concern I have about the experiments of fig 3 is that NLL can be really bad, thus distorting rho, which is not a robust measure. So I would only look at NLLs of models with good NLLs, to obtain a more interesting analysis.\n\nAnother concern about experiments is that I am not convinced that they were performed with SOTA MBRL methods and hyper-parameters (as demonstrated by SOTA performance on known benchmarks). Otherwise I could easily imagine how the mismatch could be much more severe than in the actual scenarios of interest.\n\nMinor points:\n\n\nBottom of page 6 refers to visualizations but I did not see if or where they were shown.\n\nWhy the e in the numerator of eq 2e? Seems useless to put any constant there.\n\nThe section on 'Shaping the cost or reward' was not clear enough to me (please expand).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the reviewers point out, this paper has potentially interesting ideas but it is in too preliminary state for publication at ICLR.  
Decision: Reject