Blending MPC & Value Function Approximation for Efficient Reinforcement Learning

1 INTRODUCTION . Model-free Reinforcement Learning ( RL ) is increasingly used in challenging sequential decision-making problems including high-dimensional robotics control tasks ( Haarnoja et al. , 2018 ; Schulman et al. , 2017 ) as well as video and board games ( Silver et al. , 2016 ; 2017 ) . While these approaches are extremely general , and can theoretically solve complex problems with little prior knowledge , they also typically require a large quantity of training data to succeed . In robotics and engineering domains , data may be collected from real-world interaction , a process that can be dangerous , time consuming , and expensive . Model-Predictive Control ( MPC ) offers a simpler , more practical alternative . While RL typically uses data to learn a global model offline , which is then deployed at test time , MPC solves for a policy online by optimizing an approximate model for a finite horizon at a given state . This policy is then executed for a single timestep and the process repeats . MPC is one of the most popular approaches for control of complex , safetycritical systems such as autonomous helicopters ( Abbeel et al. , 2010 ) , aggressive off-road vehicles ( Williams et al. , 2016 ) and humanoid robots ( Erez et al. , 2013 ) , owing to its ability to use approximate models to optimize complex cost functions with nonlinear constraints ( Mayne et al. , 2000 ; 2011 ) . However , approximations in the model used by MPC can significantly limit performance . Specifically , model bias may result in persistent errors that eventually compound and become catastrophic . For example , in non-prehensile manipulation , practitioners often use a simple quasi-static model that assumes an object does not roll or slide away when pushed . For more dynamic objects , this can lead to aggressive pushing policies that perpetually over-correct , eventually driving the object off the surface . Recently , there have been several attempts to combine MPC with model free RL , showing that the combination can improve over the individual approaches alone . Many of these approaches involve using RL to learn a terminal cost function , thereby increasing the effective horizon of MPC ( Zhong et al. , 2013 ; Lowrey et al. , 2018 ; Bhardwaj et al. , 2020 ) . However , the learned value function is only applied at the end of the MPC horizon . Model errors would still persist in horizon , leading to sub-optimal policies . Similar approaches have also been applied to great effect in discrete games with known models ( Silver et al. , 2016 ; 2017 ; Anthony et al. , 2017 ) , where value functions and policies learned via model-free RL are used to guide Monte-Carlo Tree Search . In this paper , we focus on a somewhat broader question : can machine learning be used to both increase the effective horizon of MPC , while also correcting for model bias ? One straightforward approach is to try to learn ( or correct ) the MPC model from real data encountered during execution ; however there are some practical barriers to this strategy . Hand-constructed models are often crude-approximations of reality and lack the expressivity to represent encountered dynamics . Moreover , increasing the complexity of such models leads to computationally expensive updates that can harm MPC ’ s online performance . Model-based RL approaches such as Chua et al . ( 2018 ) ; Nagabandi et al . ( 2018 ) ; Shyam et al . ( 2019 ) aim to learn general neural network models directly from data . However , learning globally consistent models is an exceptionally hard task due to issues such as covariate shift ( Ross & Bagnell , 2012 ) . We propose a framework , MPQ ( λ ) , for weaving together MPC with learned value estimates to trade-off errors in the MPC model and approximation error in a learned value function . Our key insight is to view MPC as tracing out a series of local Q-function approximations . We can then blend each of these Q-functions with value estimates from reinforcement learning . We show that by using a blending parameter λ , similar to the trace decay parameter in TD ( λ ) , we can systematically trade-off errors between these two sources . Moreover , by smoothly decaying λ over learning episodes we can achieve the best of both worlds - a policy can depend on a prior model before its has encountered any data and then gradually become more reliant on learned value estimates as it gains experience . To summarize , our key contributions are : 1 . A framework that unifies MPC and Model-free RL through value function approximation . 2 . Theoretical analysis of finite horizon planning with approximate models and value functions . 3 . Empirical evaluation on challenging manipulation problems with varying degrees of model-bias . 2 PRELIMINARIES . 2.1 REINFORCEMENT LEARNING . We consider an agent acting in an infinite-horizon discounted Markov Decision Process ( MDP ) . An MDP is defined by a tupleM= ( S , A , c , P , γ , µ ) where S is the state space , A is the action space , c ( s , a ) is the per-step cost function , st+1∼P ( ·|st , at ) is the stochastic transition dynamics and γ is the discount factor and µ ( s0 ) is a distribution over initial states . A closed-loop policy π ( ·|s ) outputs a distribution over actions given a state . Let µπM be the distribution over state-action trajectories obtained by running policy π onM . The value function for a given policy π , is defined as V πM ( s ) =EµπM [ ∑∞ t=0γ tc ( st , at ) |s0 =s ] and the action-value function as QπM ( s , a ) = EµπM [ ∑∞ t=0γ tc ( st , at ) |s0 =s , a0 =a ] . The objective is to find an optimal policy π∗ = argmin π Es0∼µ [ V πM ( s0 ) ] . We can also define the ( dis ) -advantage function AπM ( s , a ) =Q π M ( s , a ) −V π ( s ) , which measures how good an action is compared to the action taken by the policy in expectation . It can be equivalently expressed in terms of the Bellman error as AπM ( s , a ) =c ( s , a ) +γEs′∼P , a′∼π [ QπM ( s′ , a′ ) ] −Ea∼π [ QπM ( s , a ) ] . 2.2 MODEL-PREDICTIVE CONTROL . MPC is a widely used technique for synthesizing closed-loop policies for MDPs . Instead of trying to solve for a single , globally optimal policy , MPC follows a more pragmatic approach of optimizing simple , local policies online . At every timestep on the system , MPC uses an approximate model of the environment to search for a parameterized policy that minimizes cost over a finite horizon . An action is sampled from the policy and executed on the system . The process is then repeated from the next state , often by warm-starting the optimization from the previous solution . We formalize this process as solving a simpler surrogate MDP M̂= ( S , A , ĉ , P̂ , γ , µ̂ , H ) online , which differs fromM by using an approximate cost function ĉ , transition dynamics P̂ and limiting horizon to H. Since it plans to a finite horizon , it ’ s also common to use a terminal state-action value function Q̂ that estimates the cost-to-go . The start state distribution µ̂ is a dirac-delta function centered on the current state s0 =st . MPC can be viewed as iteratively constructing an estimate of the Q-function of the original MDPM , given policy πφ at state s : QφH ( s , a ) =EµπφM [ H−1∑ i=0 γiĉ ( si , ai ) +γ HQ̂ ( sH , aH ) |s0 =s , a0 =a ] ( 1 ) MPC then iteratively optimizes this estimate ( at current system state st ) to update the policy parameters φ∗t =argmin φ QφH ( st , πφ ( st ) ) ( 2 ) Alternatively , we can also view the above procedure from the perspective of disadvantage minimization . Let us define an estimator for the 1-step disadvantage with respect to the potential function Q̂ as A ( si , ai ) =c ( si , ai ) +γQ̂ ( si+1 , ai+1 ) −Q̂ ( si , ai ) . We can then equivalently write the above optimization as minimizing the discounted sum of disadvantages over time via the telescoping sum trick argmin π∈Π E µ πφ M [ Q̂ ( s0 , a0 ) + H−1∑ i=0 γiA ( si , ai ) |s0 =st ] ( 3 ) Although the above formulation queries the Q̂ at every timestep , it is still exactly equivalent to the original problem and hence , does not mitigate the effects of model-bias . In the next section , we build a concrete method to address this issue by formulating a novel way to blend Q-estimates from MPC and a learned value function that can balance their respective errors . 3 MITIGATING BIAS IN MPC VIA REINFORCEMENT LEARNING . In this section , we develop our approach to systematically deal with model bias in MPC by blending-in learned value estimates . First , we take a closer look at the different sources of error in the estimate in ( 1 ) and then propose an easy-to-implement , yet effective strategy for trading them off . 3.1 SOURCES OF ERROR IN MPC . The performance of MPC algorithms critically depends on the quality of the Q-function estimatorQφH ( s , a ) in ( 1 ) . There are three major sources of approximation error . First , model bias can cause compounding errors in predicted state trajectories , which biases the estimates of the costs of different action sequences . The effect of model error becomes more severe asH−→∞ . Second , the error in the terminal value function gets propagated back to the estimate of the Q-function at the start state . With discounting , the effect of error due to inaccurate terminal value function diminishes asH increases . Third , using a smallH with an inaccurate terminal value function can make the MPC algorithm greedy and myopic to rewards further out in the future . We can formally bound the performance of the policy with approximate models and approximate learned value functions . In Theorem 3.1 , we show the loss in performance of the resulting policy as a function of the model error , value function error and the planning horizon . Theorem 3.1 ( Proof Appendix A.1.2 ) . Let MDP M̂ be an α-approximation ofM such that ∀ ( s , a ) , we have ∣∣∣∣∣∣P̂ ( s′|s , a ) −P ( s′|s , a ) ∣∣∣∣∣∣ 1 ≤α and |̂c ( s , a ) −c ( s , a ) |≤α . Let the learned value function Q̂ ( s , a ) be an -approximation of the true value function ∣∣∣∣∣∣Q̂ ( s , a ) −Qπ∗M ( s , a ) ∣∣∣∣∣∣∞≤ . The performance of the MPC policy is bounded w.r.t the optimal policy as ∣∣∣∣V π∗M ( s ) −V π̂M ( s ) ∣∣∣∣∞ ≤2 ( γ ( 1−γH−1 ) ( 1−γH ) ( 1−γ ) αH ( cmax−cmin 2 ) + γHαH 1−γH ( Vmax−Vmin 2 ) + α 1−γ + γH 1−γH ) ( 4 ) This theorem generalizes over various established results . SettingH=1 , =0 gives us the 1-step simulation lemma in Kearns & Singh ( 2002 ) ( Appendix A.1.1 ) . Settingα=0 , i.e . true model , recovers the cost-shaping result in Sun et al . ( 2018 ) . Further inspecting terms in ( 4 ) , we see that the model error increases with horizon H ( the first two terms ) while the learned value error decreases withH which matches our intuitions . In practice , the errors in model and value function are usually unknown and hard to estimate making it impossible to set the MPC horizon to the optimal value . Instead , we next propose a strategy to blend the Q-estimates from MPC and the learned value function at every timestep along the horizon , instead of just the terminal step such that we can properly balance the different sources of error .

Summary: The paper proposes to combine MPC and model-free RL to overcome the possible modelling errors. Thereby the approach achieves the sample-efficiency of MPC and the control quality of model-free RL. The resulting MPQ(\lambda) algorithm uses MPPI to obtain the actions by optimizing the blended MPC objective. The Q-targets for fitting the q function also use the blended Q-estimate.