BLENDING MPC & VALUE FUNCTION APPROXIMATION FOR EFFICIENT REINFORCEMENT LEARNING
Year: 2021

Abstract:
Model-Predictive Control (MPC) is a powerful tool for controlling complex, real-world systems that uses a model to make predictions about future behavior. For each state encountered, MPC solves an online optimization problem to choose a control action that will minimize future cost. This is a surprisingly effective strategy, but real-time performance requirements warrant the use of simple models. If the model is not sufficiently accurate, then the resulting controller can be biased, limiting performance. We present a framework for improving on MPC with model-free reinforcement learning (RL). The key insight is to view MPC as constructing a series of local Q-function approximations. We show that by using a parameter λ, similar to the trace decay parameter in TD(λ), we can systematically trade-off learned value estimates against the local Q-function approximations. We present a theoretical analysis that shows how error from inaccurate models in MPC and value function estimation in RL can be balanced. We further propose an algorithm that changes λ over time to reduce the dependence on MPC as our estimates of the value function improve, and test the efficacy our approach on challenging high-dimensional manipulation tasks with biased models in simulation. We demonstrate that our approach can obtain performance comparable with MPC with access to true dynamics even under severe model bias and is more sample efficient as compared to model-free RL.

heading: 1 INTRODUCTION
text: Model-free Reinforcement Learning (RL) is increasingly used in challenging sequential decision-making problems including high-dimensional robotics control tasks (Haarnoja et al., 2018; Schulman et al., 2017) as well as video and board games (Silver et al., 2016; 2017). While these approaches are extremely general, and can theoretically solve complex problems with little prior knowledge, they also typically require a large quantity of training data to succeed. In robotics and engineering domains, data may be collected from real-world interaction, a process that can be dangerous, time consuming, and expensive.
Model-Predictive Control (MPC) offers a simpler, more practical alternative. While RL typically uses data to learn a global model offline, which is then deployed at test time, MPC solves for a policy online by optimizing an approximate model for a finite horizon at a given state. This policy is then executed for a single timestep and the process repeats. MPC is one of the most popular approaches for control of complex, safetycritical systems such as autonomous helicopters (Abbeel et al., 2010), aggressive off-road vehicles (Williams et al., 2016) and humanoid robots (Erez et al., 2013), owing to its ability to use approximate models to optimize complex cost functions with nonlinear constraints (Mayne et al., 2000; 2011).
However, approximations in the model used by MPC can significantly limit performance. Specifically, model bias may result in persistent errors that eventually compound and become catastrophic. For example, in non-prehensile manipulation, practitioners often use a simple quasi-static model that assumes an object does not roll or slide away when pushed. For more dynamic objects, this can lead to aggressive pushing policies that perpetually over-correct, eventually driving the object off the surface.
Recently, there have been several attempts to combine MPC with model free RL, showing that the combination can improve over the individual approaches alone. Many of these approaches involve using RL to learn a terminal cost function, thereby increasing the effective horizon of MPC (Zhong et al., 2013; Lowrey et al., 2018; Bhardwaj et al., 2020). However, the learned value function is only applied at the end of the MPC horizon. Model errors would still persist in horizon, leading to sub-optimal policies. Similar approaches have also been applied to great effect in discrete games with known models (Silver et al., 2016; 2017; Anthony et al., 2017), where value functions and policies learned via model-free RL are used to
guide Monte-Carlo Tree Search. In this paper, we focus on a somewhat broader question: can machine learning be used to both increase the effective horizon of MPC, while also correcting for model bias?
One straightforward approach is to try to learn (or correct) the MPC model from real data encountered during execution; however there are some practical barriers to this strategy. Hand-constructed models are often crude-approximations of reality and lack the expressivity to represent encountered dynamics. Moreover, increasing the complexity of such models leads to computationally expensive updates that can harm MPC’s online performance. Model-based RL approaches such as Chua et al. (2018); Nagabandi et al. (2018); Shyam et al. (2019) aim to learn general neural network models directly from data. However, learning globally consistent models is an exceptionally hard task due to issues such as covariate shift (Ross & Bagnell, 2012).
We propose a framework, MPQ(λ), for weaving together MPC with learned value estimates to trade-off errors in the MPC model and approximation error in a learned value function. Our key insight is to view MPC as tracing out a series of local Q-function approximations. We can then blend each of these Q-functions with value estimates from reinforcement learning. We show that by using a blending parameter λ, similar to the trace decay parameter in TD(λ), we can systematically trade-off errors between these two sources. Moreover, by smoothly decaying λ over learning episodes we can achieve the best of both worlds - a policy can depend on a prior model before its has encountered any data and then gradually become more reliant on learned value estimates as it gains experience.
To summarize, our key contributions are: 1. A framework that unifies MPC and Model-free RL through value function approximation. 2. Theoretical analysis of finite horizon planning with approximate models and value functions. 3. Empirical evaluation on challenging manipulation problems with varying degrees of model-bias.

heading: 2 PRELIMINARIES

heading: 2.1 REINFORCEMENT LEARNING
text: We consider an agent acting in an infinite-horizon discounted Markov Decision Process (MDP). An MDP is defined by a tupleM= (S,A,c,P,γ,µ) where S is the state space, A is the action space, c(s,a) is the per-step cost function, st+1∼P (·|st,at) is the stochastic transition dynamics and γ is the discount factor and µ(s0) is a distribution over initial states. A closed-loop policy π(·|s) outputs a distribution over actions given a state. Let µπM be the distribution over state-action trajectories obtained by running policy π onM. The value function for a given policy π, is defined as V πM(s)=EµπM [ ∑∞ t=0γ
tc(st,at) |s0 =s] and the action-value function as QπM(s, a) = EµπM [ ∑∞ t=0γ
tc(st,at) |s0 =s,a0 =a]. The objective is to find an optimal policy π∗ = argmin
π Es0∼µ [V πM(s0)]. We can also define the (dis)-advantage
function AπM(s,a)=Q π M(s,a)−V π(s), which measures how good an action is compared to the action taken by the policy in expectation. It can be equivalently expressed in terms of the Bellman error as AπM(s,a)=c(s,a)+γEs′∼P,a′∼π[QπM(s′,a′)]−Ea∼π[QπM(s,a)].

heading: 2.2 MODEL-PREDICTIVE CONTROL
text: MPC is a widely used technique for synthesizing closed-loop policies for MDPs. Instead of trying to solve for a single, globally optimal policy, MPC follows a more pragmatic approach of optimizing simple, local policies online. At every timestep on the system, MPC uses an approximate model of the environment to search for a parameterized policy that minimizes cost over a finite horizon. An action is sampled from the policy and executed on the system. The process is then repeated from the next state, often by warm-starting the optimization from the previous solution.
We formalize this process as solving a simpler surrogate MDP M̂= (S,A,ĉ,P̂ ,γ,µ̂,H) online, which differs fromM by using an approximate cost function ĉ, transition dynamics P̂ and limiting horizon to H. Since it plans to a finite horizon, it’s also common to use a terminal state-action value function Q̂ that estimates the cost-to-go. The start state distribution µ̂ is a dirac-delta function centered on the current state s0 =st. MPC can be viewed as iteratively constructing an estimate of the Q-function of the original MDPM, given policy πφ at state s:
QφH(s,a)=EµπφM [ H−1∑ i=0 γiĉ(si,ai)+γ HQ̂(sH,aH) |s0 =s,a0 =a ] (1)
MPC then iteratively optimizes this estimate (at current system state st) to update the policy parameters
φ∗t =argmin φ QφH(st,πφ(st)) (2)
Alternatively, we can also view the above procedure from the perspective of disadvantage minimization. Let us define an estimator for the 1-step disadvantage with respect to the potential function Q̂ as A(si,ai)=c(si,ai)+γQ̂(si+1,ai+1)−Q̂(si,ai). We can then equivalently write the above optimization as minimizing the discounted sum of disadvantages over time via the telescoping sum trick
argmin π∈Π E µ πφ M
[ Q̂(s0,a0)+
H−1∑ i=0 γiA(si,ai) |s0 =st
] (3)
Although the above formulation queries the Q̂ at every timestep, it is still exactly equivalent to the original problem and hence, does not mitigate the effects of model-bias. In the next section, we build a concrete method to address this issue by formulating a novel way to blend Q-estimates from MPC and a learned value function that can balance their respective errors.

heading: 3 MITIGATING BIAS IN MPC VIA REINFORCEMENT LEARNING
text: In this section, we develop our approach to systematically deal with model bias in MPC by blending-in learned value estimates. First, we take a closer look at the different sources of error in the estimate in (1) and then propose an easy-to-implement, yet effective strategy for trading them off.

heading: 3.1 SOURCES OF ERROR IN MPC
text: The performance of MPC algorithms critically depends on the quality of the Q-function estimatorQφH(s,a) in (1). There are three major sources of approximation error. First, model bias can cause compounding errors in predicted state trajectories, which biases the estimates of the costs of different action sequences. The effect of model error becomes more severe asH−→∞. Second, the error in the terminal value function gets propagated back to the estimate of the Q-function at the start state. With discounting, the effect of error due to inaccurate terminal value function diminishes asH increases. Third, using a smallH with an inaccurate terminal value function can make the MPC algorithm greedy and myopic to rewards further out in the future.
We can formally bound the performance of the policy with approximate models and approximate learned value functions. In Theorem 3.1, we show the loss in performance of the resulting policy as a function of the model error, value function error and the planning horizon.
Theorem 3.1 (Proof Appendix A.1.2). Let MDP M̂ be an α-approximation ofM such that ∀(s,a), we have ∣∣∣∣∣∣P̂(s′|s,a)−P(s′|s,a)∣∣∣∣∣∣ 1 ≤α and |̂c(s,a)−c(s,a)|≤α. Let the learned value function Q̂(s,a) be
an -approximation of the true value function ∣∣∣∣∣∣Q̂(s,a)−Qπ∗M(s,a)∣∣∣∣∣∣∞≤ . The performance of the MPC
policy is bounded w.r.t the optimal policy as ∣∣∣∣V π∗M (s)−V π̂M(s)∣∣∣∣∞
≤2 (
γ(1−γH−1) (1−γH)(1−γ) αH
( cmax−cmin
2
) + γHαH
1−γH
( Vmax−Vmin
2
) + α
1−γ +
γH
1−γH
) (4)
This theorem generalizes over various established results. SettingH=1, =0 gives us the 1-step simulation lemma in Kearns & Singh (2002) (Appendix A.1.1). Settingα=0, i.e. true model, recovers the cost-shaping result in Sun et al. (2018). Further inspecting terms in (4), we see that the model error increases with horizon H (the first two terms) while the learned value error decreases withH which matches our intuitions.
In practice, the errors in model and value function are usually unknown and hard to estimate making it impossible to set the MPC horizon to the optimal value. Instead, we next propose a strategy to blend the Q-estimates from MPC and the learned value function at every timestep along the horizon, instead of just the terminal step such that we can properly balance the different sources of error.

heading: 3.2 BLENDING MODEL PREDICTIVE CONTROL AND VALUE FUNCTIONS
text: A naive way to blend Q-estimates from MPC with Q-estimates from the value function would be to consider a convex combination of the two
(1−λ)Q̂(s,a)︸ ︷︷ ︸ model-free +λQφH(s,a)︸ ︷︷ ︸ model-based
(5)
where λ∈ [0,1]. Here, the value function is contributing to a residual that is added to the MPC output, an approach commonly used to combine model-based and model-free methods (Lee et al., 2020). However, this is solution is rather ad hoc. If we have at our disposal a value function, why invoke it at only at the first and last timestep? As the value function gets better, it should be useful to invoke it at all timesteps.
Instead, consider the following recursive formulation for the Q-estimate. Given (si,ai), the state-action encountered at horizon i, the blended estimateQλ(si,ai) is expressed as
Qλ(si,ai)︸ ︷︷ ︸ current blended estimate =(1−λ)Q̂(si,ai)︸ ︷︷ ︸ model-free +λ( ĉ(si,ai)︸ ︷︷ ︸ model-based +γ Qλ(si+1,ai+1)︸ ︷︷ ︸ future blended estimate ) (6)
where λ∈ [0,1]. The recursion ends at Qλ(sH,aH) = Q̂(sH,aH). In other words, the current blended estimate is a convex combination of the model-free value function and the one-step model-based return. The return in turn uses the future blended estimate. Note unlike (5), the model-free estimate is invoked at every timestep.
We can unroll (6) in time to showQλH(s,a), the blendedH−horizon estimate, is simply an exponentially weighted average of all horizon estimates
QλH(s,a)=(1−λ) H−1∑ i=0 λiQφi (s,a)+λ HQφH(s,a) (7)
where Qφk(s,a) = EµπφM
[∑k−1 i=0 γ iĉ(si,ai)+γ kQ̂(sk,ak) |s0 =s,a0 =a ] is a k-horizon estimate. When
λ=0, the estimator reduces to the just using Q̂ and whenλ=1 we recover the original MPC estimateQH in (1). For intermediate values of λ, we interpolate smoothly between the two by interpolating allH estimates.
Implementing (7) naively would require running H versions of MPC and then combining their outputs. This is far too expensive. However, we can switch to the disadvantage formulation by applying a similar telescoping trick
QλH(s,a)=EµπφM
[ Q̂(s0,a0)+
H−1∑ i=0 (γλ)iA(si,ai)
] (8)
This estimator has a similar form as the TD(λ) estimator for the value function. However, while TD(λ) uses the λ parameter for bias-variance trade-off, our blended estimator aims trade-off bias in dynamics model with bias in learned value function.
Why use blending λ when one can simply tune the horizonH? First,H limits the resolution we can tune since it’s an integer – as H gets smaller the resolution becomes worse. Second, the blended estimator QλH(s,a) uses far more samples. Say we have access to optimal horizonH ∗. Even if bothQλH andQ φ H∗ had the same bias, the latter uses a strict subset of samples as the former. Hence the variance of the blended estimator will be lower, with high probability.
4 THE MPQ(λ)ALGORITHM
We develop a simple variant of Q-Learning, called Model-Predictive Q-Learning with λWeights (MPQ(λ)) that learns a parameterized Q-function estimate Q̂θ. Our algorithm, presented in Alg. 1, modifies Q-learning to use blended Q-estimates as described in the (8) for both action selection and generating value targets. The parameter λ is used to trade-off the errors due to model-bias and learned Q-function, Q̂θ. This can be viewed as an extension of the MPQ algorithm from Bhardwaj et al. (2020) to explicitly deal with model bias by incorporating the learned Q-function at all timesteps. Unlike MPQ, we do not explicitly consider the entropy-regularized formulation, although our framework can be modified to incorporate soft-Q targets.
Algorithm 1: MPQ(λ)
Input: Initial Q-function weights θ, Approximate dynamics P̂ and cost function ĉ Parameter: MPC horizonH, λ schedule [λ1,λ2,...],
discount factor γ, minibatch sizeK, num mini-batchesN , update frequency tupdate 1 D←∅ 2 for t=1...∞ do
// Update λ 3 λ=λt
// Blended MPC action selection
4 φt←argmin φ E µ πφ M
[ Q̂θ(s0,a0)+ ∑H−1 i=0 (γλ) iA(si,ai) |s0 =st ]
5 at∼πφt 6 Execute at on the system and observe (ct,st+1) 7 D←(st,at,ct,st+1) 8 if t%tupdate==0 then
9 Sample N minibatches ({
sk,n,ak,n,ck,n,s ′ k,n }K k=1 )N n=1 fromD
// Generate Blended MPC value targets
10 ŷk,n=ck,n+γminEµπφM
[ Q̂θ(s0,a0)+ ∑H−1 i=0 (γλ) iA(si,ai) |s0 =s′k,n ]
11 Update θ with SGD on loss L= 1N 1 K ∑N n=1 ∑K k=1 ( ŷk,n−Q̂θ(sk,n,ak,n) )2
At every timestep t, MPQ(λ) proceeds by using H-horizon MPC from the current state st to optimize a policy πφ with parameters φ. We modify the MPC algorithm to optimize for the greedy policy with respect to the blended Q-estimator in (8), that is
φ∗t =argmin φ E µ πφ M
[ Q̂θ(s0,a0)+
H−1∑ i=0 (γλ)iA(si,ai) |s0 =st
] (9)
An action sampled from the resulting policy is then executed on the system. A commonly used heuristic is to warm start the above optimization by shifting forward the solution from the previous timestep, which serves as a good initialization if the noise in the dynamics in small (Wagener et al., 2019). This can significantly cut computational cost by reducing the number of iterations required to optimize ( 9) at every timestep.
Periodically, the parameters θ are updated via stochastic gradient descent to minimize the following loss function withN mini-batches of experience tuples of sizeK sampled from the replay buffer
L(θ)= 1 N 1 K N∑ n=1 K∑ k=1 ( ŷk,n−Q̂θ(sk,n,ak,n) )2 (10)
TheH-horizon MPC with blended Q-estimator is again invoked to calculate the targets
ŷj=c(sj,aj)+γmin φ E µ πφ M
[ Q̂θ(s0,a0)+
H−1∑ i=0 (γλ)iA(si,ai) |s0 =s′k,n
] (11)
Using MPC to reduce error in Q-targets has been previously explored in literature (Lowrey et al., 2018; Bhardwaj et al., 2020), where the model is either assumed to be perfect or model-error is not explicitly accounted for. MPC with the blended Q-estimator and an appropriate λ allows us to generate more stable Q-targets than using Qθ or model-based rollouts with a terminal Q-function alone. However, running H-horizon optimization for all samples in a mini-batch can be time-consuming, forcing the use of smaller batch sizes and sparse updates. In our experiments, we employ a practical modification where during
the action selection step, MPC is also queried for value targets which are then stored in the replay buffer, thus allowing us to use larger batch sizes and updates at every timestep.
Finally, we also allow λ to vary over time. In practice, λ is decayed as more data is collected on the system. Intuitively, in the early stages of learning, the bias in Q̂θ dominates and hence we want to rely more on the model. A larger value of λ is appropriate as it up-weights longer horizon estimates in the blended-Q estimator. As Q̂θ estimates improve over time, a smaller λ is favorable to reduce the reliance on the approximate model.

heading: 5 EXPERIMENTS
text: Task Details: We evaluate MPQ(λ) on simulated robot control tasks, including a complex manipulation task with a 7DOF arm and in-hand manipulation with a 24DOF anthropomorphic hand (Rajeswaran* et al., 2018) as shown in Fig. 1. For each task, we provide the agent with a biased version of simulation that is used as the dynamics model for MPC. We use Model Predictive Path Integral Control (MPPI) (Williams et al., 2017), a state-of-the-art sampling-based algorithm as our MPC algorithm throughout.
1. CARTPOLESWINGUP: A classic control task where the agent slides a cart along a rail to swingup the pole attached via an unactuated hinge joint. Model bias is simulated by providing the agent incorrect masses of the cart and pole. The masses are set lower than the true values to make the problem harder for MPC as the algorithm will always input smaller controls than desired as also noted in Ramos et al. (2019). Initial position of cart and pole are randomized at every episode.
2. SAWYERPEGINSERTION: The agent controls a 7DOF Sawyer arm to insert a peg attached to the end-effector into a hole at different locations on a table in front of the robot. We test the effects of inaccurate perception by simulating a sensor at the target location that provides noisy position measurements at every timestep. MPC uses a deterministic model that does not take sensor noise into account as commonly done in controls. This biases the cost of simulated trajectories, causing MPC to fail to reach the target.
3. INHANDMANIPULATION: A challenging in-hand manipulation task with a 24DOF dexterous hand from Rajeswaran* et al. (2018). The agent must align the pen with target orientation within certain tolerance for succcess. The initial orientation of the pen is randomized at every episode. Here, we simulate bias by providing larger estimates of the mass and inertia of the pen as well as friction coefficients, which causes the MPC algorithm to optimize overly aggressive policies and drop the pen.
Please refer to the Appendix A.2 for more details of the tasks, success criterion and biased simulations.
Baselines: We compare MPQ(λ) against both model-based and model-free baselines - MPPI with true dynamics and no value function, MPPI with biased dynamics and no value function and Proximal Policy Optimization (PPO) Schulman et al. (2017).
Learning Details: We represent the Q-function with a feed-forward neural network. We bias simulation parameters like mass or friction coefficients using the formulam=(1+b)mtrue, where b is a bias-factor. We also employ a practical modification to Alg. 1 in order to speed up training times as discussed in Section 4. Instead of maintaining a large replay-buffer and re-calculating targets for every experience tuple in a mini-batch, as done by approaches such as Bhardwaj et al. (2020); Lowrey et al. (2018), we simply query MPC for the value targets online and store them in a smaller buffer, which allows us to perform updates at every timestep. We use publicly the available implementation at https://bit.ly/38RcDrj for PPO. Refer to the Appendix A.2 for more details.
0 5 10 15 20 25 Validation Iteration
900
800
700
600
500
400
300
200
A ve
ra ge
R ew
ar d
Validation rewards
MPPIH64(true), MPPIH64(biased) = 0.5 = 0.55 = 0.6 = 0.65 = 0.7 = 0.75 = 0.8 = 0.85 = 0.9 = 0.95 = 1.0 PPO PPO (asymptotic)
(a) Fixed λ
0 5 10 15 20 25 Validation Iteration
900
800
700
600
500
400
300
200
A ve
ra ge
R ew
ar d
Validation rewards
MPPIH64(true) MPPIH64(biased) = 0.6 = 0.65 = 0.7 = 0.75 F = 0.6 F = 0.65 F = 0.7 F = 0.75
(b) Fixed v/s Decaying λ
0 5 10 15 20 25 Validation Iteration
800
600
400
200
A ve
ra ge
R ew
ar d
Validation rewards
MPPIH64(true), b = 0 b = 0.1 b = 0.2 b = 0.3 b = 0.4 b = 0.5 b = 0.6 b = 0.7 b = 0.8 b = 0.9 PPO PPO (asymptotic)
(c) Varying Model Bias
0 5 10 15 20 25 Validation Iteration
900
800
700
600
500
400
300
200
A ve
ra ge
R ew
ar d
Validation rewards
MPPIH64(true), MPPIH64(biased) F = 0.5 F = 0.55 F = 0.6 F = 0.65 F = 0.7 F = 0.75 F = 0.8 F = 0.85 F = 0.9 F = 0.95 PPO PPO (asymptotic)
(d) λ decay withH=64
0 5 10 15 20 25 Validation Iteration
1000
800
600
400
200
A ve
ra ge
R ew
ar d
Validation rewards
MPPIH32(true), MPPIH32(biased) F = 0.5 F = 0.55 F = 0.6 F = 0.65 F = 0.7 F = 0.75 F = 0.8 F = 0.85 F = 0.9 F = 0.95 PPO PPO (asymptotic)
(e) λ decay withH=32
0 5 10 15 20 25 Validation Iteration
1000
800
600
400
200
A ve
ra ge
R ew
ar d
Validation rewards
H = 1 H = 2 H = 4 H = 8 H = 16 H = 32 H = 64 H = 64, F = 0.8
(f) Varying Horizon v/s λ
1 2 4 8 16 32 64
800
600
400
200
M ax
V al
id at
io n
R ew
ar d
100 particles
1 2 4 8 16 32 64
800
600
400
200
50 particles
1 2 4 8 16 32 64
800
600
400
200
20 particles
1 2 4 8 16 32 64
800
700
600
500
400
300
10 particles
Horizon
= 1.0 = 0.8
(g) Bias-Variance Trade-off
Figure 2: CARTPOLESWINGUP experiments. Solid lines show average rewards over 30 validation episodes (fixed start states) with length of 100 steps and 3 runs with different seeds. The dashed lines are average reward of MPPI for the same validation episodes. Shaded region depicts the standard error of the mean that denotes the confidence on the average reward estimated from finite samples. Training is performed for 100k steps with validation after every 4k steps. When decaying λ as per a schedule, it is fixed to the current value during validation. In (b),(d),(e), (f) λF denotes the λ value at the end of training. PPO asymptotic performance is reported as average reward of last 10 validation iterations. (g) shows the best validation reward at the end of training for different horizon values and MPPI trajectory samples (particles) using λ=1.0 and λ=0.8

heading: 5.1 ANALYSIS OF OVERALL PERFORMANCE
text: O 1. MPQ(λ) is able to overcome model-bias in MPC for a wide range of λ values. Fig. 2(a) shows a comparison of MPQ(λ) with MPPI using true and biased dynamics with b=−0.5 and H=64 for various settings of λ. There exists a wide range of λ values for which MPQ(λ) can efficiently trade-off model-bias with the bias in the learned Q-function and out-perform MPPI with biased dynamics. However, setting λ to a high value of 1.0 or 0.95, which weighs longer horizons heavily leads to poor performance as compounding effects of model-bias are not compensated byQθ. Performance also begins to drop as λ decreases below 0.6. MPQ(λ) outperforms MPPI with access to the true dynamics and reaches close to asymptotic performance of PPO. This is not surprising as the learned Q-function adds global information to the optimization and λ corrects for errors in optimizing longer horizons. O 2. Faster convergence can be achieved by decaying λ over time. As more data is collected on the system, we expect the bias inQθ to decrease, whereas model bias remains constant. A larger value of λ that favors longer horizons is better during initial steps of training as the effect of a randomly initialized Qθ is diminished due to discounting and better exploration is achieved by forward lookahead. Conversely, as Qθ gets more accurate, model-bias begins to hurt performance
and a smaller λ is favorable. We test this by decaying λ in [1.0,λF ] using a fixed schedule and observe that indeed faster convergence is obtained by reducing the dependence on the model over training steps as shown in 2(b). Figures 2(d) and 2(e) present ablations that show that MPQ(λ) is robust to a wide range of decay rates with H=64 and 32 respectively. When provided with true dynamics, MPPI with H = 32 performs better than H = 64 due to optimization issues with long horizons. MPQ(λ) reaches performance comparable with MPPI H=32 and asymptotic performance of PPO in both cases showing robustness to horizon values which is important since in practice we wish to set the horizon as large as our computation budget permits. However, decaying λ too fast or too slow can have adverse effects on performance. An interesting question for future work is whether λ can be adapted in a state-dependent manner. Refer to Appendix A.2 for details on the decay schedule. O 3. MPQ(λ) is much more sample efficient compared to model-free RL on high-dimensional continuous control tasks, while maintaining asymptotic performance. Figures 2 and 3 show a comparison of MPQ(λ) with the model-free PPO baseline. In all cases, we observe that MPQ(λ), through its use of approximate models, learned value functions, and a dynamically-varying λ parameter to trade-off different sources of error, rapidly improves its performance and achieves average reward and success rate comparable to MPPI with access to ground truth dynamics and model-free RL in the limit. In INHANDMANIPULATION, PPO performance does not improve at all over 150k training steps. In SAWYERPEGINSERTION, the small magnitude of reward difference between MPPI with true and biased models is due to the fact that despite model bias, MPC is able to get the peg close to the table, but sensor noise inhibits precise control to consistently insert it in the hole. Here, the value function learned by MPQ(λ) can adapt to sensor noise and allow for fine-grained control near the table. O 4. MPQ(λ) is robust to large degree of model misspecification. Fig. 2(c) shows the effects of different values of the bias factor b used to vary the mass of the cart and pole for MPQ(λ) with a fixedλ decay rate of [1.0,0.75]. MPQ(λ) achieves performance better than MPPI (H=64) with true dynamics and comparable to model-free RL in the limit for a wide range of bias factors b, and convergence rate is generally faster for smaller bias. For large values of b, MPQ(λ) either fails to improve or
diverges as the compounding effects of model-bias hurt learning, making model-free RL the more favorable alternative. A similar trend is observed in Figures 3(a) and 3(b) where MPQ(λ) outperforms MPPI with corresponding bias in the mass, inertia and friction coefficients of the pen with atleast a margin of over 30% in terms of success rate. It also achieves performance comparable to MPPI with true dynamics and modelfree RL in the limit, but is unable to do so for b=1.0. We conclude that while MPQ(λ) is robust to large amount of model bias, if the model is extremely un-informative, relying on MPC can degrade performance.
O 5. MPQ(λ) is robust to planning horizon and number of trajectory samples in sampling-based MPC. TD(λ) based approaches are used for bias-variance trade-off for value function estimation in model-free RL. In our framework, λ plays a similar role, but it trades-off bias due to the dynamics model and learned value function against variance due to long-horizon rollouts. We empirically quantify this on the CARTPOLESWINGUP task by training MPQ(λ) with different values of horizon and number of particles for λ=1.0 and λ=0.8 respectively. Results in Fig. 2(g) show that - (1) using λ can overcome effects of model-bias irrespective of the planning horizon (except for very small values ofH=1 or 2) and (2) using λ can overcome variance due to limited number of particles with long horizon rollouts. The ablative study in Fig. 2(f) lends evidence to the fact that is preferable to simply decay λ over time than trying to tune the discrete horizon value to balance model bias. Not only does decaying λ achieve a better convergence rate and asymptotic performance than tuning horizon, the performance is more robust to different decay rates (as evidenced from Fig. 2(d)), whereas the same does not hold for varying horizon.

heading: 6 CONCLUSION
text: In this paper, we presented a general framework to mitigate model-bias in MPC by blending model-free value estimates using a parameter λ, to systemativally trade-off different sources of error. Our practical algorithm achieves performance close to MPC with access to the true dynamics and asymptotic performance of model-free methods, while being sample efficient. An interesting avenue for future research is to vary λ in a state-adaptive fashion. In particular, reasoning about the model and value function uncertainty may allow us to vary λ to rely more or less on our model in certain parts of the state space. Another promising direction is to extend the framework to explicitly incorporate constraints by leveraging different constrained MPC formulations.

heading: ACKNOWLEDGMENTS
text: This work was supported in part by ARL SARA CRA W911NF-20-2-0095. The authors would like to thank Aravind Rajeswaran for help with code for the peg insertion task.

heading: A APPENDIX

heading: A.1 PROOFS
text: We present upper-bounds on performance of a greedy policy when using approximate value functions and models. We also analyze the case of finite horizon planning with an approximate dynamics model and terminal value function which can be seen as a generalization of (Sun et al., 2018). For simplicity, we switch to using V̂ (s) to the learnt model-free value function (instead of Q̂(s))
Let V̂ (s) be an -approximation ∣∣∣∣∣∣V̂ (s)−V π∗M (s)∣∣∣∣∣∣∞≤ . Let MDP M̂ be an α-approximation ofM
such that ∀(s,a), we have ∣∣∣∣∣∣P̂(s′|s,a)−P(s′|s,a)∣∣∣∣∣∣
1 ≤α and |̂c(s,a)−c(s,a)|≤α.

heading: A.1.1 A GENTLE START: BOUND ON PERFORMANCE OF 1-STEP GREEDY POLICY
text: Theorem A.1. Let the one-step greedy policy be
π̂(s)=argmin a∈A
ĉ(s,a)+γΣs′P̂(s ′|s,a)V̂ (s′) (12)
The performance loss of π̂(s) w.r.t optimal policy π∗ on MDPM is bounded by
∣∣∣∣∣∣V π̂M(s)−V π∗M (s)∣∣∣∣∣∣∞≤ 2 ( γ +α+γα ( Vmax−Vmin 2 )) 1−γ
(13)
Proof. From (12) we have ∀s∈S
ĉ(s,π̂(s))+γ ∑ s′ P̂(s′|s,π̂(s))V̂ (s′)≤ ĉ(s,π∗(s))+γ ∑ s′ P̂(s′|s,π∗(s))V̂ (s′)
ĉ(s,π̂(s))−ĉ(s,π∗(s))≤γ (∑ s′ P̂(s′|s,π∗(s))V̂ (s′)− ∑ s′ P̂(s′|s,π̂(s))V̂ (s′) ) (using
∣∣∣∣∣∣V̂ (s)−V π∗M (s)∣∣∣∣∣∣∞≤ ) ĉ(s,π̂(s))−ĉ(s,π∗(s))≤γ
(∑ s′ P̂(s′|s,π∗(s))V π ∗ M (s ′)− ∑ s′ P̂(s′|s,π̂(s))V π ∗ M (s ′) ) +2γ
(using |̂c(s,a)−c(s,a)|≤α)
c(s,π̂(s))−c(s,π∗(s))≤2γ +2α+γ (∑ s′ P̂(s′|s,π∗(s))V π ∗ M (s ′)− ∑ s′ P̂(s′|s,π̂(s))V π ∗ M (s ′) ) (14)
Now, let s be the state with the max loss V π̂M(s)−V π ∗ M (s),
V π̂M(s)−V π ∗ M (s)=c(s,π̂)−c(s,π∗)+γ ∑ s′ ( P(s′|s,π̂)V π̂M(s′)−P(s′|s,π∗)V π ∗ M (s ′) )
Substituting from (14)
V π ∗ M (s)−V π̂M(s)≤2γ +2α+γ ∑ s′ P̂(s′|s,π∗(s))V π ∗ M (s ′)−γ ∑ s′ P̂(s′|s,π̂(s))V π ∗ M (s ′)
−γ ∑ s′ P(s′|s,π∗)V π ∗ M (s ′)+γ ∑ s′ P(s′|s,π̂)V π̂M(s′)
Add and subtract γ ∑ s′P(s ′|s,π̂)V π∗M (s′) and re-arrange
V π ∗ M (s)−V π̂M(s)≤2γ +2α+γ ∑ s′ ( P̂(s′|s,π∗)−P(s′|s,π∗) ) V π ∗ M (s ′)
−γ ∑ s′ ( P̂(s′|s,π̂)−P(s′|s,π̂) ) V π ∗ M (s ′)
+γ ∑ s′ P(s′|s,π̂) ( V π̂M(s ′)−V π ∗ M (s ′) )
≤2γ +2α+2γα ( Vmax−Vmin
2
) +γ ∑ s′ P(s′|s,π̂) ( V π̂M(s ′)−V π ∗ M (s ′) )
Since s is the state with largest loss∣∣∣∣∣∣V π∗M (s)−V π̂M(s)∣∣∣∣∣∣∞≤2γ +2α+2γα ( Vmax−Vmin 2 ) +γ ∑ s′ P(s′|s,π̂) ∣∣∣∣∣∣V π∗M (s)−V π̂M(s)∣∣∣∣∣∣∞
≤2γ +2α+2γα ( Vmax−Vmin
2
) +γ ∣∣∣∣∣∣V π∗M (s)−V π̂M(s)∣∣∣∣∣∣∞
Re-arranging terms we get∣∣∣∣∣∣V π∗M (s)−V π̂M(s)∣∣∣∣∣∣∞≤ 2 ( γ +α+γα ( Vmax−Vmin 2 )) 1−γ
(15)
which concludes the proof.

heading: A.1.2 BOUND ON PERFORMANCE OF H-STEP GREEDY POLICY
text: Notation: For brevity let us define the following macro,
〈V,π,M〉H=EµπM [ H−1∑ i=0 γic(si,ai)+γ HV (sH) ] (16)
which represents the expected cost achieved when executing policy π onM using V as the terminal cost. We can substitute different policies, terminal costs and MDPs. For example, 〈 V̂ ,π̂,M̂ 〉 H is the expected cost obtained by running policy π̂ on simulator M̂ forH steps with approximate learned terminal value function V̂ . Lemma A.1. For a given policy π, the optimal value function V π ∗
M and MDPsM,M̂ the following performance difference holds
∣∣∣∣∣∣〈V π∗M ,π,M〉 H − 〈 V π ∗ M ,π,M̂ 〉 H ∣∣∣∣∣∣ ∞ ≤γ ( 1−γH−1 1−γ ) αH ( cmax−cmin 2 ) +γHαH ( Vmax−Vmin 2 ) + 1−γH 1−γ α
Proof. We temporarily introduce a new MDPM′ that has the same cost function as aM, but transition function of M̂ 〈
V π ∗ M ,π,M 〉 H − 〈 V π ∗ M ,π,M̂ 〉 H = 〈 V π ∗ M ,π,M 〉 H − 〈 V π ∗ M ,π,M′ 〉 H
+ 〈 V π ∗ M ,π,M′ 〉 H − 〈 V π ∗ M ,π,M̂ 〉 H
(17)
Let ∆P(s0...sH)=P(s0...sH)−P̂(s0...sH) represent the difference in distribution of states encountered by executing π onM and M̂ respectively starting from state s0. Expanding the RHS of (17)
= ∑
s0,...,sH
∆P(s0...sH) ( H−1∑ i=0 γic(si,ai)+γ HV π ∗ M (sH) ) +Eµπ M̂ [ H−1∑ i=0 γi(c(si,ai)−ĉ(si,ai)) ] (18)
Since the first state s1 is the same = ∑
s1,...,sH
∆P(s1...sH) ( H−1∑ i=1 γic(si,ai)+γ HV π ∗ M (sH) ) +Eµπ M̂ [ H−1∑ i=0 γi(c(si,ai)−ĉ(si,ai)) ]
≤ ∣∣∣∣∣ ∣∣∣∣∣ ∑ s1,...,sH ∆P(s1...sH) ( H−1∑ i=1 γic(si,ai)+γ HV π ∗ M (sH) )∣∣∣∣∣ ∣∣∣∣∣ ∞ + ∣∣∣∣∣ ∣∣∣∣∣EµπM̂ [ H−1∑ i=0 γi(c(si,ai)−ĉ(si,ai)) ]∣∣∣∣∣ ∣∣∣∣∣ ∞
≤ ∣∣∣∣∣ ∣∣∣∣∣ ∑ s1,...,sH ∆P(s1...sH) ( H−1∑ i=1 γic(si,ai)+γ HV π ∗ M (sH) )∣∣∣∣∣ ∣∣∣∣∣ ∞ + 1−γH 1−γ α (19) where the first inequality is obtained by applying the triangle inequality and the second one is obtained by applying triangle inequality followed by the upper bound on the error in cost-function.
≤ ∣∣∣∣∣∣ ∣∣∣∣∣∣ ∑
s2,...,sH+1
∆P(s2...sH+1) ∣∣∣∣∣∣ ∣∣∣∣∣∣ ∞ sup( H−1∑ i=1 γic(si,ai)+γ HV π ∗ M (sH)−K)+ 1−γH 1−γ α (20)
By choosing K = ∑H−1 i=1 γ i(cmax +cmin)/2+γ H(Vmax +Vmin)/2 we can ensure that the term inside sup is upper-bounded by γ(1−γH−1)/(1−γ)((cmax−cmin)/2)+γH(Vmax−Vmin)/2
≤γ ( 1−γH−1
1−γ
) αH ( cmax−cmin
2
) +γHαH ( Vmax−Vmin
2
) + 1−γH
1−γ α (21)
The above lemma builds on similar results in (Kakade et al., 2003; Abbeel & Ng, 2005; Ross & Bagnell, 2012).
We are now ready to prove our main theorem, i.e. the performance bound of an MPC policy that uses an approximate model and approximate value function.
Proof of Theorem 3.1
Proof. Since, π̂ is the greedy policy when using M̂ and V̂ ,〈 V̂ ,π̂,M̂ 〉 H ≤ 〈 V̂ ,π∗,M̂ 〉 H〈
V π ∗ M ,π̂,M̂ 〉 H ≤ 〈 V π ∗ M ,π ∗,M̂ 〉 H +2γH (using ∣∣∣∣∣∣V̂ −V π∗M ∣∣∣∣∣∣ 1 ≤ )
(22)
Also, we have 〈 V π ∗ M ,π̂,M 〉 H − 〈 V π ∗ M ,π ∗,M 〉 H = (〈 V π ∗ M ,π̂,M 〉 H − 〈 V π ∗ M ,π̂,M̂ 〉 H ) − (〈 V π ∗
M ,π ∗,M 〉 H − 〈 V π ∗ M ,π ∗,M̂ 〉 H ) + (〈 V π ∗ M ,π̂,M̂ 〉 H − 〈 V π ∗ M ,π ∗,M̂ 〉 H
) (23) The first two terms can be bounded using Lemma A.1 and the third term using Eq. (22) to get
〈 V π ∗ M ,π̂,M 〉 H − 〈 V π ∗ M ,π ∗,M 〉 H
≤2 ( γ 1−γH−1
1−γ αH
( cmax−cmin
2
) +γHαH ( Vmax−Vmin
2
) + 1−γH
1−γ α+γH ) (24) Now, let s be the state with max loss V π̂M(s)−V π ∗ M (s)
V π̂M(s)−V π ∗ M (s)= 〈 V π̂M,π̂,M 〉 H − 〈 V π ∗ M ,π ∗,M 〉 H
= (〈 V π̂M,π̂,M 〉 H − 〈 V π ∗ M ,π̂,M 〉 H ) + (〈 V π ∗ M ,π̂,M 〉 H − 〈 V π ∗ M ,π ∗,M 〉 H ) =γH ( V π̂M(sH+1)−V π ∗ M (sH+1) ) + (〈 V π ∗ M ,π̂,M 〉 H − 〈 V π ∗ M ,π ∗,M 〉 H
) ≤γH ( V π̂M(s)−V π ∗ M (s) )
+2
( γ(1−γH−1)
1−γ αH
( cmax−cmin
2
) +γHαH ( Vmax−Vmin
2
) + 1−γH
1−γ α+γH ) (25)
where last inequality comes from applying Eq. (24) and the fact that s is the state with max loss. The final expression follows from simple algebraic manipulation.

heading: A.2 EXPERIMENT DETAILS

heading: A.2.1 TASK DETAILS

heading: CARTPOLESWINGUP
text: • Reward function: x2cart+θ2pole+0.01vcart+0.01ωpole+0.01a2
• Observation: [xcart,θpole,vcart,ωpole] (4 dim)
SAWYERPEGINSERTION We simulate sensor noise by placing a simulated position sensor at the target location in the MuJoCo physics engine that adds Gaussian noise with σ = 10cm to the observed 3D position vector. MPPI uses a deterministic model that does not take sensor noise into account for planning. Every episode lasts for 75 steps with a timestep of 0.02 seconds between steps
• Reward function: −1.0 ∗ ||xee − xtarget||1 − 5.0 ∗ ||xee − xtarget||2 + 5 ∗ 1(||xee−xtarget||2<0.06)
• Observation: [ qpos,qvel,xee,xtarget,xee−xtarget,||xee−xtarget||1,||xee−xtarget||2 ] (25 dim)
An episode is considered successful if the peg stays within the hole for atleast 5 steps.
INHANDMANIPULATION This environment was used without modification from the accompanying codebase for Rajeswaran* et al. (2018) and is available at https://bit.ly/3f6MNBP
• Reward function: −||xobj−xdes||2 +zTobjzdes+ Bonus for proximity to desired pos + orien, zTobjzdes represents dot product between object axis and target axis to measure orientation similarity.
• Observation: [qpos,xobj,vobj,zobj,zdes,xobj−xdes,zobj−zdes] (45 dim)
Every episode lasts 75 steps with a timestep of 0.01 seconds between steps. An episode is considered successful if the orientation of the pen stays within a specified range of desired orientation for atleast 20 steps. The orientation similarity is measured by the dot product between the pen’s current longitudinal axis and desired with a threshold of 0.95.

heading: SAWYERPEGINSERTION

heading: A.2.2 LEARNING DETAILS
text: Validation: Validation is performed after every N training episodes during training for Neval episodes using a fixed set of start states that the environment is reset to. We ensure that the same start states are sampled at every validation iteration by setting the seed value to a pre-defined validation seed, which is kept constant across different runs of the algorithm with different training seeds. This helps ensure consistency in evaluating different runs of the algorithm. For all our experiments we setN=40 andNeval =30.
MPQ(λ): For all tasks, we represent Q function using 2 layered fully-connected neural network with 100 units in each layer and ReLU activation. We use ADAM (Kingma & Ba, 2014) for optimization with a learning rate of 0.001 and discount factor γ = 0.99. Further, the buffer size is 1500 for CARTPOLESWINGUP and 3000 for the others, with batch size of 64 for all. We smoothly decay λ according to the following sublinear decay rate
λt= λ0
1+κ √ t
(26)
where the decay rate κ is calculate based on the desired final value of λ. For batch size we did a search from [16, 64] with a step size of 16 and buffer size was chosen from 1500, 3000, 5000. While batch size was tuned for cartpole and then fixed for the remaining two environments, the buffer size was chosen independently for all three.
Proximal Policy Optimization (PPO): Both policy and value functions are represented by feed forward networks with 2 layers each with 64 and 128 units for policy and value respectively. All other parameters are left to the default values. The number of trajectories collected per iteration is modified to correspond with the same number of samples collected between validation iterations for MPQ(λ). We collect 40 trajectories per iteration. Asymptotic performance is reported as average of last 10 validation iterations after 500 training iters in SAWYERPEGINSERTION and 2k in INHANDMANIPULATION.
MPPI parameters Table 1 shows the MPPI parameters used for different experiments. In addition to the standard MPPI parameters, in certain cases we also use a step size parameter as introduced by Wagener et al. (2019). For INHANDMANIPULATION and SAWYERPEGINSERTION we also apply autoregressive filtering on the sampled MPPI trajectories to induce smoothness in the sampled actions, with tuned filter coefficients. This has been found to be useful in prior works (Summers et al., 2020; Lowrey et al., 2018) for getting MPQ(λ) to work on high dimensional control tasks. The temperature, initial covariance and step size parameters for MPPI were tuned using a grid search with true dynamics. Temperature and initial covariance were set within the range of [0.0,1.0] and step size from [0.5,1.0] with a discretization of 0.05. The number of particles were searched from [40,120] with a step size of 10 and the horizon was chosen from 4 different values [16,20,32,64]. The best performing parameters then chosen based on average reward over 30 episodes with a fixed seed value to ensure reproducibility. The same parameters were then used in the case of biased dynamics and MPQ(λ), to clearly demonstrate that MPQ(λ) can overcome sources of error in the base MPC implementation.

Review summaries:
The paper proposes to combine MPC and model-free RL to overcome the possible modelling errors. Thereby the approach achieves the sample-efficiency of MPC and the control quality of model-free RL. The resulting MPQ(\lambda) algorithm uses MPPI to obtain the actions by optimizing the blended MPC objective. The Q-targets for fitting the q function also use the blended Q-estimate.