On-Policy Model Errors in Reinforcement Learning

1 INTRODUCTION . Model-free reinforcement learning ( RL ) has made great advancements in diverse domains such as single- and multi-agent game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Vinyals et al. , 2019 ) , robotics ( Kalashnikov et al. , 2018 ) and neural architecture search ( Zoph & Le , 2016 ) . All of these model-free approaches rely on large numbers of interactions with the environment to ensure successful learning . While this issue is less severe for environments that can easily be simulated , it limits the applicability of model-free RL to ( real-world ) domains where data is scarce . Model-based RL ( MBRL ) aims to reduce the amount of data required for policy optimization by learning a model that approximates the true environment , which we can use to generate cheap , simulated state transitions ( Sutton , 1990 ; Racanière et al. , 2017 ; Moerland et al. , 2020 ) . While early approaches on low-dimensional tasks by Schneider ( 1997 ) ; Deisenroth & Rasmussen ( 2011 ) used probabilistic models with closed-form posteriors , recent methods rely on neural networks to scale to more complex tasks on discrete ( Kaiser et al. , 2019 ) and continuous ( Chua et al. , 2018 ; Kurutach et al. , 2018 ) action spaces . However , the learned representation of the true environment always remains imperfect and as such introduces an error source to the RL problem ( Atkeson & Santamaria , 1997 ; Abbeel et al. , 2006 ) . Hence , a key challenge in MBRL is model-bias ; small errors in the learned models that can compound over multi-step predictions and lead to lower asymptotic performance in contrast to model-free approaches . To address these challenges with both model-free and model-based RL , Levine & Koltun ( 2013 ) ; Chebotar et al . ( 2017 ) propose to combine the merits of both . While there are multiple possibilities to combine the two methodologies , in this work we focus on improving the model ’ s predictive state distribution such that it more closely resembles the data distribution of the true environment . Contributions The main contribution of this paper is on-policy corrections ( OPC ) , a novel hyperparameter-free methodology that uses on-policy transition data on top of a separately learned model to enable accurate long-term predictions for MBRL . A key strength of our approach is that it does not introduce any new parameters that need to be hand-tuned for specific tasks . We theoretically motivate our approach by means of a policy improvement bound and show that we can recover the true state distribution when generating trajectories on-policy with the model . We illustrate how OPC improves the quality of policy gradient estimates in a simple toy example and evaluate it on various continuous control tasks from the MuJoCo control suite and their PyBullet variants . There , we demonstrate that OPC improves current state-of-the-art MBRL algorithms in terms of data efficiency , especially for the more difficult PyBullet environments . Related Work To counteract model-bias , several approaches combine ideas from model-free and model-based RL . For example , Levine & Koltun ( 2013 ) guide a model-free algorithm via modelbased planning towards promising regions in the state space , Kalweit & Boedecker ( 2017 ) augment the training data by an adaptive ratio of simulated transitions , Talvitie ( 2017 ) use ‘ hallucinated ’ transition tuples from simulated to observed states to self-correct the model , and Feinberg et al . ( 2018 ) ; Buckman et al . ( 2018 ) use a learned model to improve the value function estimate . Janner et al . ( 2019 ) mitigate the issue of compounding errors for long-term predictions by simulating short trajectores start from real states . Cheng et al . ( 2019 ) extend first-order model-free algorithms via adversarial online learning to leverage prediction models in a regret-optimal manner . Clavera et al . ( 2020 ) employ a model to augment an actor-critic objective , which they optimize via back-propagation through time , adapting the planning horizon interpolates between a purely model-based and a modelfree approach . Morgan et al . ( 2021 ) combine actor-critic methods with model-predictive rollouts to guarantee near-optimal simulated data and retain exploration on the real environment . A downside of most existing approaches is that they introduce additional hyperparameters that are critical to the learning performance ( Zhang et al. , 2021 ) . In addition to empirical performance , recent work builds on the theoretical guarantees for model-free approaches by Kakade & Langford ( 2002 ) ; Schulman et al . ( 2015 ) to provide guarantees for MBRL . Luo et al . ( 2018 ) provide a general framework to show monotonic improvement towards a local optimum of the value function , while Janner et al . ( 2019 ) present a lower-bound on performance for different rollout schemes and horizon lengths . Yu et al . ( 2020 ) show guaranteed improvement in the offline MBRL setting by augmenting the reward with an uncertainty penalty , while Clavera et al . ( 2020 ) present improvement guarantees in terms of the model ’ s and value function ’ s gradient errors . A related field to MBRL that also combines models with data is iterative learning control ( ILC ) ( Bristow et al. , 2006 ) . The differences between the two methods are the following : RL typically focuses on finding parametric feedback policies for general reward functions , whereas ILC seeks an open-loop sequence of actions with fixed length to improve state tracking performance . Further , the model in ILC is often derived from first principles and then kept fixed , whereas in MBRL the model is continuously improved upon observing new data . The method most closely related to RL and our approach is optimization-based ILC ( Owens & Hätönen , 2005 ; Schöllig & D ’ Andrea , 2009 ) , in which a linear dynamics model is used to guide the search for optimal actions . Recently , Baumgärtner & Diehl ( 2020 ) extended the ILC setting to nonlinear dynamics and more general reward signals . Little work is available that draws connections between RL and ILC ( Zhang et al. , 2019 ) with one notable exception : Abbeel et al . ( 2006 ) use the observed data from the last rollout to account for a mismatch in the dynamics model . The limitations of this approach are that deterministic dynamics are assumed , the policy optimization itself requires a line search procedure with rollouts on the true environment and that it was not combined with model learning . We build on this idea and extend it to the stochastic setting of MBRL by making use of recent advances in RL and model learning . Lastly , Harutyunyan et al . ( 2016 ) propose a similar correction term as the one introduced in this paper in the context of off-policy policy evaluation and correct the state-action value function instead of the transition dynamics . Similarly , Fonteneau et al . ( 2013 ) consider the problem of off-policy policy evaluation but in the batch RL setting and propose to generate ‘ artificial ’ trajectories from observed transitions instead of using an explicit model for the dynamics . 2 PROBLEM STATEMENT AND BACKGROUND . We consider the Markov decision process ( MDP ) ( S , A , p , r , γ , ρ ) where S ⊆ RdS and A ⊆ RdA are the continuous state and action spaces , respectively . The unknown environment dynamics are described by the transition probability p ( st+1 | st , at ) , an initial state distribution ρ ( s0 ) and the reward signal r ( s , a ) . The goal in RL is to find a policy πθ ( at | st ) parameterized by θ that maximizes the expected return , discounted by γ ∈ [ 0 , 1 ] over episodes of length T , η = E s0 : T , a0 : T [ ∑T t=0 γtr ( st , at ) ] , st+1 ∼ p ( st+1 | st , at ) , s0 ∼ ρ , at ∼ πθ ( at | st ) . ( 1 ) Algorithm 1 General Model-based Reinforcement Learning 1 : for n = 1 , . . . do 2 : for b = 1 , . . . , B do 3 : Rollout policy πθn on environment and store transitions Dbn = { ( ŝn , bt , ân , bt , ŝn , bt+1 ) } T−1t=0 4 : p̃n ( st+1 | st , at ) : Learn a global dynamics model given all data D1 : n = ⋃n i=1 ⋃B b=1Dbi 5 : θn+1 = θn + α∇η̃n : Optimize the policy based on the model p̃ with any RL algorithm The expectation is taken with respect to the trajectory under the stochastic policy πθ starting from some stochastic initial state . Direct maximization of Eq . ( 1 ) is challenging , since we do not know the environment ’ s transition probabilities p. In MBRL , we learn a model for the transitions and reward function from data , p̃ ( st+1 | st , at ) ≈ p ( st+1 | st , at ) and r̃ respectively , and subsequently maximize η̃ = E s0 : T , a0 : T [ ∑T t=0 γtr̃ ( st , at ) ] , st+1 ∼ p̃ ( st+1 | st , at ) , s0 ∼ ρ̃ , at ∼ πθ ( at | st ) . ( 2 ) as a surrogate problem for Eq . ( 1 ) . The choice of initial state distribution ρ̃ ( s0 ) depends on the optimization method : For model predictive control , Chua et al . ( 2018 ) use the current state to optimize the next action based on a rollout , while Janner et al . ( 2019 ) use the empirical state distribution observed from the true environment . In the following , we focus on the latter and for ease of exposition assume a known reward function r̃ = r , even though we learn it jointly with p̃ in our experiments . We let ηn denote the policy ’ s performance under the policy πn = πθn at iteration n and use ŝ and â for states and actions that are observed on the true environment . Algorithm 1 summarizes the overall procedure for MBRL : At each iteration n we store B on-policy trajectories Dbn = { ( ŝn , bt , ân , bt , ŝn , bt+1 ) } T−1t=0 obtained by rolling out the current policy πn on the real environment in Line 3 . Lastly , we approximate the environment with a learned model p̃n ( st+1 | st , at ) based on the data in Line 4 , and optimize the policy based on the proxy objective in Eq . ( 2 ) in Line 5 . Note that the policy optimization algorithm can be off-policy and employ its own , separate replay buffer . Model Choices The choice of model p̃ plays a key role , since it defines the surrogate problem Eq . ( 2 ) . Inside the optimization loop , the model is typically used to predict sequences of transitions τ with length H . One option is to use a purely data-based model that replays past transitions from a recent on-policy trajectory b starting at time step t , p̃datan ( τt : t+H | t , b ) = t+H∏ t′=t p̃datan ( st′ | t′ , b ) with p̃datan ( st | t , b ) = δ ( st − ŝn , bt ) , ( 3 ) where δ ( · ) denotes a Dirac-delta distribution and 1 ≤ b ≤ B . This model choice effectively reduces model-based algorithms back to the model-free setting . An alternative is to summarize all the past dataD1 : n = ∪ni=1∪Bb=1Dbi in a parametric model coming from a distribution family P by maximizing the marginal log-likelihood L , p̃modeln ( st+1 | st , at ) = arg max p̃∈P ∑ ( ŝt , ât , ŝt+1 ) ∈D1 : n L ( ŝt+1 ; ŝt , ât ) , ( 4 ) which for each state-action pair ( st , at ) models a distribution over the next state st+1 , such that p̃modeln ( τt : t+H | t , b ) = δ ( st−ŝn , bt ) ∏t+H−1 t′=t p̃ model n ( st′+1 | st′ , at′ ) with at′ ∼ πθ . For both models , we can sample H-step predictions by first sampling a trajectory index b ∼ Categorical ( { 0 , . . . , B } ) and then rolling out following the respective state sequence distributions p̃ ( τt : t+H | t , b ) .

Summary: This paper addresses on-policy errors in model-based Reinforcement Learning. The authors decompose the policy improvement bound in terms that depend on the off-policy and on-policy model errors. They first present two major techniques, one purely based on the replay buffer, which asymptotically obtains zero on-policy error, and one based on a learned model, which leverages off-policy data. Then they propose On-Policy Corrections (OPC), a combination of the two techniques which can use a model and still maintain low on-policy error using the replay buffer. This method has asymptotically (with infinite data) zero on-policy error when using deterministic policies. If the policy is stochastic, the error grows exponentially with the trajectory length T. The authors implement their algorithm on top of MBPO and show experimentally that OPC-MBPO has superior performance in some environments.