Objective Mismatch in Model-based Reinforcement Learning

1 INTRODUCTION . Model-based reinforcement learning ( MBRL ) is a popular approach for learning to control nonlinear systems that can not be expressed analytically ( Bertsekas , 1995 ; Sutton and Barto , 2018 ; Deisenroth and Rasmussen , 2011 ; Williams et al. , 2017 ) . MBRL techniques achieve the state of the art performance for continuous-control problems with access to a limited number of trials ( e.g . Chua et al . ( 2018 ) ; Wang and Ba ( 2019 ) ) and in controlling systems given only visual observations with no observations of the original system ’ s state ( e.g . Hafner et al . ( 2018 ) ; Zhang et al . ( 2018 ) ) . MBRL approaches typically learn a forward dynamics model that predicts how the dynamical system will evolve when a set of control signals are applied . This model is classically fit with respect to the maximum likelihood of a set of trajectories collected on the real system , and then used as part of a control algorithm to be executed on the system ( e.g. , model-predictive control ) . In this paper , we highlight a fundamental problem in the standard MBRL learning scheme : the objective mismatch issue . The learning of the forward dynamics model is decoupled from the subsequent controller and policy that it induces through the optimization of two different objective functions – negative log-likelihood ( or its equivalent for deterministic models , e.g. , the RMSE ) of the single- or multi-step look-ahead prediction for the dynamics model , and task performance ( i.e. , reward ) for the policy optimization . While the use of negative log-likelihood ( NLL ) for system identification is an historically accepted objective , it results in optimizing an objective that does not necessarily correlate to the actual performance . The contributions of this paper are to : 1 ) identify and formalize the problem of objective mismatch in MBRL ; 2 ) examine the signs of and the effects of objective mismatch on simulated control tasks ; 3 ) propose a initial mechanism to mitigate objective mismatch ; 4 ) discuss the impact of objective mismatch on existing MBRL and outline future directions to address this problem . 2 MODEL-BASED REINFORCEMENT LEARNING . Algorithm 1 : Model-based RL Data : Initialize D from random actions while Improving do Train model pθ ( st+1|st , at ) on D Collect data D′ with controller using pθ in real environment Aggregate dataset D = D ∪D′ We now outline the MBRL formulation used in the paper . At time t , we denote the state st ∈ Rds , the actions at ∈ Rda , and the reward r ( st , at ) . We say that the MBRL agent acts in an environment governed by a state transition distribution p ( st+1|st , at ) . We denote a parametric model to approximate this distribution with pθ ( st+1|st , at ) . MBRL follows the general approach of an agent acting in its environment , learning a model of said environment , and then leveraging the model to act more effectively , summarized in Alg . 1 . While iterating over parametric control policies , the agent collects measurements of state , action , next-state and forms a dataset D = { ( sn , an , s′n ) } Nn=1 , where N is the sum of the length of all episodes . With the dynamics data D , the agent learns the environment in the form of a neural network forward dynamics model , learning an approximate dynamics pθ . This dynamics model is leveraged by a controller that takes in the current state st and returns an action sequence at : t+T maximizing the expected reward Eπθ ( st ) ∑t+T i=t r ( si , ai ) , where T is the predictive horizon and πθ ( st ) is the set of state transitions induced by the model pθ . In our paper , we primarily use probabilistic networks designed to minimize the NLL of the predicted parametric distribution pθ , denoted as P , or ensembles of probabilistic networks denoted PE , and compare to deterministic networks minimizing the mean squared error ( MSE ) , denoted D or DE . Unless otherwise stated we use the models as in PETS ( Chua et al. , 2018 ) with an expectation-based trajectory planner and a cross-entropy-method ( CEM ) optimizer . 3 OBJECTIVE MISMATCH AND ITS CONSEQUENCES . The Origin of Objective Mismatch : The Subtle Differences between MBRL and System Identification Many ideas and concepts in model-based RL are rooted in the field of optimal control and system identification ( Bertsekas , 1995 ; Zhou et al. , 1996 ; Kirk , 2012 ; Bryson , 2018 ; Sutton and Barto , 2018 ) . In system identification , the main idea is to use a two-step process where we first generate on the robot ( optimal ) elicitation trajectories τ to fit a dynamics model ( typically analytical ) , and subsequently we apply this model to a specific task . This particular scheme has multiple assumptions : 1 ) the data collected cover the entire state-action space ( done by appropriate elicitation trajectories ) ; 2 ) the presence of large ( virtually infinite ) amount of data ; 3 ) the global nature of the model resulting from the system identification process ( which should be generalizable to a large range of tasks ) . With these assumptions , the general idea of system identification is effectively to collect large amount of data covering the whole state-space to create once a global model that is sufficiently accurate that we can at deployment time specify any desired task , and still obtain good performance . When adopting the idea of learning the dynamics model used in optimal control for MBRL , it is important to consider if these assumptions still hold . The assumption of virtually infinite data is visibly in tension with the explicit goal of MBRL which is to reduce the number of interactions with the environment by being “ smart ” about the sampling of new trajectories . In fact , in MBRL the offline data collection performed via elicitation trajectories is largely replaced by on-policy sampling ( with the exception of some initial motor babbling ( Chua et al. , 2018 ) ) in order to explicitly reduce the need to collect large amount of data . Moreover – in practice – in the MBRL setting the data will not usually cover the entire state-action space , since they are generated by a policy trying to optimize one specific task . In conjunction with the use of non-parametric models ( i.e. , not analytical model ) , this results in learned models that are strongly biased towards capturing the distribution of the data , and that are only locally accurate . Nonetheless , this is at first sight not an issue since the common MBRL setting consider the learning aimed at solving only one specific task , and rarely test for generalization capabilities of the learned dynamics . In practice , we can now see how the assumptions and goals of system identification are in contrast with the ones of MBRL . Understanding these differences and the downstream effects on algorithmic approach is crucial to design new families of MBRL algorithms . Objective Mismatch During the MBRL process of iteratively learning a controller , the reward signal from the environment is diluted by the training of a forward dynamics model with a independent metric , as showing in Fig . 1 . In our experiments , we highlight that the minimization of some network training cost does not hold a strong correlation to maximization of episode reward . As dynamic environments becoming increasingly complex in dimensionality , the assumptions of collected data distributions become weaker and over-fitting to different data poses an increased risk . Formally , the problem of objective mismatch appears as two de-coupled optimization problems repeated over many cycles of Alg . 1 , shown in Eq . ( 1a , b ) , which could be at the cost of minimizing the final reward . This loop becomes increasingly difficult to analyze as the dataset used for model training changes with each experimental trial – a step that is needed to include new data from previously unexplored states . In this paper we characterize the problems introduced by the interaction of these two optimization problems , but avoid to consider the interactions added by the changes in data distribution during the learning process , as this would significantly increase the complexity of the analysis . In addition , we discuss potential solutions , but do not make claims about the best way to do so , which is left for future work . Training : argmin θ N∑ i=1 log pθ ( s ′ i|si , ai ) , Control : argmax at : t+T Eπθ ( st ) t+T∑ i=t r ( si , ai ) ( 1a , b ) 4 IDENTIFYING OBJECTIVE MISMATCH . In this section , we experimentally study the issue of objective mismatch in MBRL to answer the following questions : 1 ) Does the distribution of models obtained from running a MBRL algorithm show a strong correlation between NLL and reward ? 2 ) Are there signs of sub-optimality in the dynamics models training process that could be limiting performance ? 3 ) What model differences are reflected in reward but not in NLL ? Experimental Setting In our experiments , we use two popular RL benchmark tasks : the cartpole and half cheetah . For more details on these tasks see Chua et al . ( 2018 ) . For our cartpole experiments , we aggregate dynamics models at each episode from 50 runs of PETS ( 20 trials per run ) paired with the on-policy episode reward achieved , giving the number of models Mcp = 1000 . Our half cheetah experiments consist of 8 runs ( 300 trials per run ) of PETS , tabulating the reward with the dynamics models again , yielding Mhc = 2400 . All dynamics models are of depth 3 , hidden width 500 , and use the tanh activation function . With a large set of dynamics models in the feasible space of on-policy models , we then used a series of control datasets to evaluate the NLL of the dynamics models versus performance . The datasets we use are designed to investigate how different assumptions made in MBRL trickle down into final performance . We start with expert datasets to test if on-policy performance is only linked to having adequately explored the environment . As a baseline , we compare the expert data to datasets collected by standard data aggregation procedures when learning . Finally , a dataset representing the entire state space is generated as a grid over the state-space or by sampling from reasonable values and measuring the transitions in order to test how complete dynamics models relates to model accuracy and final performance . Specifically , the cartpole datasets are as follows : an expert selection of episodes data with r > 179 ( 2 , 400 points ) , a standard on-policy dataset aggregated from the end of a PETS run ( 3 , 780 points ) , and a grid of datapoints to approximate the full system dynamics ( 16 , 807 points ) . For half cheetah we compared three similar datasets : an expert dataset from a PETS run of half cheetah using the true dynamics for planning ( 3 , 000 points ) , an on-policy dataset aggregated from PETS ( 90 , 900 points ) , and an uniform sampled dataset attempting to approximate the full system dynamics ( 200 , 000 points ) .

Summary: This paper discusses the old problem of mismatch between the ultimate reward obtained after optimizing a  decision (planning or control) over a probabilistic model (of dynamics) and  the training  objective for the model (log-likelihood). Experiments highlight that the NLL and reward can be very poorly correlated, that improvements in NLL initially improve reward but can later degrade it, and that models with similar NLLs can lead to very different rewards. A  reweighting trick is proposed and summarily evaluated.