Blending MPC & Value Function Approximation for Efficient Reinforcement Learning
Authors: Mohak Bhardwaj
Sanjiban Choudhury
Byron Boots
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
model-predictive control

Abstract: Model-Predictive Control (MPC) is a powerful tool for controlling complex, real-world systems that uses a model to make predictions about future behavior. For each state encountered, MPC solves an online optimization problem to choose a control action that will minimize future cost. This is a surprisingly effective strategy, but real-time performance requirements warrant the use of simple models. If the model is not sufficiently accurate, then the resulting controller can be biased, limiting performance. We present a framework for improving on MPC with model-free reinforcement learning (RL). The key insight is to view MPC as constructing a series of local Q-function approximations. We show that by using a parameter $\lambda$, similar to the trace decay parameter in TD($\lambda$), we can systematically trade-off learned value estimates against the local Q-function approximations. We present a theoretical analysis that shows how error from inaccurate models in MPC and value function estimation in RL can be balanced. We further propose an algorithm that changes $\lambda$ over time to reduce the dependence on MPC as our estimates of the value function improve, and test the efficacy our approach on challenging high-dimensional manipulation tasks with biased models in simulation. We demonstrate that our approach can obtain performance comparable with MPC with access to true dynamics even under severe model bias and is more sample efficient as compared to model-free RL.

Published as a conference paper at ICLR 2021
BLENDING MPC & VALUE FUNCTION APPROXIMATION FOR EFFICIENT REINFORCEMENT LEARNING
Mobak Bhardwaj'
Sanjiban Choudhury?
Byron Boots
University of Washington 2 Aurora Innovation Inc.
ABSTRACT
Model-Predictive Control (MPC) is a powerful tool for controlling complex, real-world systems that uses model make predictions about future behavior. For each state encountered, MPC solves an online optimization problem to choose a control action that will minimize future cost. This a surprisingly effective strategy, but real-time performance requirements warrant the use of simple models. If the model is not sufficiently accurate, then the resulting controller can be biased, limiting performance. We present framework for improving (RL). The key insight is to view MPC as constructing a series of local Q-function approximations. We show that by using parameter X, similar to the trace decay parameter in TD(A), we can systematically trade-off learned value estimates against the local Q-uuctioon approximations. We present a theoretical analysis that shows how error from inaccurate models in MPC and value function estimation in RI. can be balanced. We further propose algorithm that changes À over time reduce the dependence MPC as our estimates of the value function improve, and test the efficacy our approach on challenging high-ddmensiooal manipulation tasks with biased models in simulation. We demonstrate that our approach can obtain performance comparable with MPC with access to true dynamics even under severe model bias and more sample efficient as compared to model-free RL.
MPC with model-free reinforcement learning
INTRODUCTION
Model-free Reinforcement Learning (RL) is increasingly used in challenging sequential decision-making problems including high-dimensional robotics control tasks (Haamojj et al., 2018; Schulman et al., 2017) as well as video and board games (Silver et al., 2016; 2017). While these approaches are extremely general, and can theoretically solve complex problems with little prior knowledge. they also typically require large quantity of training data to succeed. In robotics and engineering domains, data may be collected from real-world interaction, process that can be dangerous, time consuming, and expensive.
Preei-Prrddettve Control (MPC) offers a simpler, more practical alternative. While RL typically uses data to learn global model offline, which is then deployed at test time, MPC solves for policy online by optimizing an approximate model for a finite horizon at a given state. This policy is then executed for a single timestep and the process repeats. MPC is one of the most popular approaches for control of complex, safety- critical systems such as autonomous helicopters (Abbeel et al., 2010), aggressive off-road vehicles (Williams al., 2016) and humanoid robots (Erez et al., 2013), owing to its ability use approximate models to optimize complex cost functions with nonlinear constraints (Mayne et al., 2000; 201
However, approximations the model used by MPC can significantly limit performance. Specifically, model bias may result in persistent emorr that eventually compound and become catastrophic. For example, non- prehensie manipulation, practitioners often use a simple quasi-static model that assumes an object does not roll or slide away when pushed. For more dynamic objects, this can lead to aggressive pushing policies that perpetually over-correct, eventually driving the object off the surface.
Recently, there have been several attempts to combine MPC with model free RL, showing that the combination can improve over the individual approaches alone. Many of these approaches involve using RL to learn a terminal cost function, thereby increasing the effective horizon of MPC (Zhong et al., 2013; Lowrey et al., 2018; Bhardwaj et al., 2020). However, the leamndd value function is only applied at the end of the MPC horizon. Model errors would still persist in horizon, leading to sub-optimal policies. Similar approaches have also been applied to great effect in discrete games with known models (Silver et al., 2016; 2017; Anthony et al., 2017), where value functions and policies leamed via model-free RL used to

---

Published as a conference paper at ICLR 2021
guide Monte-Cate Carlo Tree Search. In this paper, focus somewhat broader question: can machine learning be used to both increase the effective horizon of MPC, while also correcting for model bias?
One straightforward approach is to try to learn (or correct) the MPC model from real data encountered dur- ing exccution; however there are some practical barriers to this strategy. Hand-constructed models are often suude-apprxximttoon of reality and lack the expressivity represent encountered dynamics. Moreover, in- creasing the complexity of such models leads to computationally expensive updates that can harm MPC's on- line performance. Model-based RL approaches such as Chua et al. (2018); Nagabandi et al. (2018) 8); Shyam et al. (2019) aim to learn general neural network models directly from data. However, learning globally consistent models an exceptionally hard task due to issues such as covariate shift (Ross & Bagnell, 2012).
We propose framework, (A) for weaving together MPC with learned value estimates to trade-off errors in the MPC model and approximation error learned value function. Our key insight is to view MPC as tracing out a series of local Q-function approximations. We can then blend each of these Q-functions with value estimates from reinforcement leaming. We show that by using a blending parameter X, similar to the trace decay parameter in TD(A), we can systematically trade-off errors between these two sources. Moreover, by smoothly decaying over learning episodes we can achieve the best of both worlds- policy can depend on prior model before its has encountered any data and then gradually become more reliant on learned value estimates as it gains experience.
To summarize, our key contributions are:
1. A framework that unifies MPC and Model-free RL through value function approximation.
2. Theoretical analysis of finite horizon planning with approximate models and value functions.
3. Empirical evaluation on challenging manipulation problems with varying degrees of model-bias.
2 PRELIMINARIES
2.1
REINFORCEMENT LEARNING
We consider agent acting in aefinite-horizon discounted Markov Decision Process (MDP). An MDP is defined by a tuple M = (S,,Ac.P.,..)) where S is the state space, AA is the action space, c(s,a) is the per-step cost function, St+1 ~ P(-(st,at) is the stochastic transition dynamics and is the discount factor and (a(sp) is a distribution over initial states. A closed-loop policy π (-|ss) outputs a distribution over actions given a state. Let u be the distribution over state-action trajectories obtained by running policy t on M. The value function for a given policy ग,, is defined as (M) = Epy, x 7 c()s,a))) | = and the action-value function as Q7 a) = Ey - 80 8,00 = a). The objective is to find an optimal policy πe argmin Engay VM (so)]. We can also define the (dis)-advantage
" function (s,a) - (T (s). which measures how good an action is compared to the action
taken by the policy in expectation. It can be equivalently expressed in terms of the Bellman error as AJ (s.2)= =)) Paa [ [Q (s' (s's,'')== ,d)] (s,aa)
2.2 MODEL-PREDICTIVVE CONTROL.
MPC is a widely used technique for symthesizing closed-loop policies for MDPs. Instead of trying to solve for single, globally optimal policy, MPC follows more pragmatic approach of optimizing simple, local policies online. At every timestep on the system, MPC uses an approximate model of the environment to search for a parameterized policy that minimizes cost over finite horizon. An action is sampled from the policy and executed on the system. The process is then repeated from the next state, often by warm-starting the optimization from the previous solution.
We formalize this process as solving a simpler surrogate MDP (M...A....);..... online, which differs from M by using an approximate cost function è, transition dynamics P and limiting horizon to H. Since it plans to a finite horizon, it's also common to use a terminal state-action value function Q that estimates the toot-to-go. The start state distribution fa is dirac-delts function centered on the current state 80- = St. MPC can be viewed as iteratively constructing an estimate of the Q-function of the original MDP M, given policy ाे. at state s:
#--1 Σ
(7 EE
m PM
(1)
2

---

Published as a conference paper at ICLR 2021
MPC then iteratively optimizes this estimate (at current system state s,) update the policy parameters
i = Frggmnn Q2 (8)))
(2)
Alternatively, also view the above procedure from the perspective of disadvantage minimization. Let define estimator for the 1-step disadvantage with respect the potential function Q as A( A(s,,a,) = ++ -(0++11((((,00)) (s,,a)) We can then equivalently write the above optimization as minimizing the discounted sum of disadvantages over time via the telescoping sum trick
H
((00,00) + Σ r (8.,0)))8000)
argmin E well
AM
Although the above formulation queries the Q at every timestep, it is still exactly equivalent to the original problem and hence, does not mitigate the effects model-bias. the next section, we build concrete method to address this issue by formulating novel way to blend Q-estimates from MPC and learned value function that can balance their respective errors.
MITIGATING BIAS IN MPC VIA REINFORCEMENT LEARNING
In this section, develop approach systematically deal with model bias in MPC by blending-in learned value estimates. First, we take a closer look at the different sources of error in the estimate in (1) and then propose easy-to-implement, yet effective strategy for trading them off.
3.1 SOURCES OF ERROR IN MPC
The performance of MPC algorithms critically depends on the quality of the Q-function estimator Qir(s (s,a) in (1). There are three major sources of approximation error. First, model bias can cause compounding errors predicted state trajectories, which biases the estimates of the costs different action sequences. The effect of model error becomes more severe H 0o. Second, the eтror in the terminal value function gets propagated back to the estimate of the Q-function the start state. With discounting, the effect of error due to inaccurate terminal value function diminishes as H increases. Third, using small H with an inaccurate terminal value function can make the MPC algorithm greedy and myopic to rewards further out in the future.
We can formally bound the performance of the policy with approximate models and approximate learned value functions. Theorem 3.1, we show the loss in performance the resulting policy as function of the model error, value function error and the planning horizon.
Theorem 3.1 (Proof Appendix A.1.2). Let MDP M be a-approximation of M such that V(s,a), have P(s'ss,a)- - P(s' (s,a) S and
Sa. a. Let the learned value function ((s.a) be [(s.a) (x, <e. The performance the MPC (
an c-approximation the true value function policy is bounded wart the optimal policy as
<2 7(1 (1-2"111 all (1-7()(1>))
Cmax
Cmin
M MaH 1-
Vmaa Vmnn
a 1-y
(4)
2
2
This theorem generalizes over various established results. Setting =11,-=0 gives us the I-step simulation lemma in Keams & Singh (2002) (Appendix A.1. Setting =0, ie. true model, recovers the cost- cost-shaping result in Sun et al. (2018). Further inspecting terms in (4), we see that the model error increases with horizon H (the first terms) while the learned value emrr decreases with H which matches intuitions.
In practice, the emrors in model and value function are usually unknown and hard to estimate making impossible the MPC horizon to the optimal value. Instead, we next propose a strategy to blend the Q-estimates from MPC and the learned value function at every timestep along the horizon, instead of just the terminal step such that we can properly balance the different sources of error.
3

---

Published as a conference paper at ICLR 2021
3.2 BLENDING MODEL PREDICTIVE CONTROL AND VALUE FUNCTIONS
A naive way to blend Qeestimates from MPC with Qeestimates from the value function would be to consider a convex combination of the two
(1 (1-)) +XQ (s,a)
(5)
model-fice
model-based
where A [0,1]. Here, the value function is contributing a residual that is added the MPC output, an approach commonly used to combine model-based and model-free methods (Lee et al., 2020). However, this is solution is rather ad hoc. If we have at our disposal a value function, why invoke at only the first and last timestep? As the value function gets better, it should be useful to invoke it at all timesteps.
Instead, consider the following recursive formulation for the Q-estimate. Given (s,,a;), the state-action encountered at horizon i, the blended estimate Q* (s;,a:) is expressed as
(staa))
(1 (1,a)) ((8,,a) ty (+++1,
current blended estimate
modee free
model bused
future blended evtimate
where Àe E [0, The recursion ends at (2HH.ar)) (s)) In other words, the current blended estimate a convex combination of the model-free value function and the one-step model-based return. The return in turn uses the future blended estimate. Note unlike (5), the model-free estimate is invoked every timestep.
We can unroll (6) in time to show Q2 (8,0), the blended H -horion estimate, is simply an exponentially weighted average of all horizon estimates
H-1 ((1
XO) X (sa)
1=0
where (s.a)) = E "s E= + $00 is k-horizon estimate. When
A=0, the estimator reduces to the just using Q and when x= we recover the original MPC estimate QH in (1). For intermediate values of X, interpolate smoothly between the two by interpolating all H estimates.
Implementing (7) naively would require running H versions of MPC and then combining their outputs. This is far too expensive. However, can switch to the disadvantage formulation by applying similar telescoping trick
H-1
(n,a (1ssa)-E,,, 555 Q(50 Σ (TA)AA(s,,a)) 1=0
(8)
This estimator has similar form as the (A) estimator for the value function. However, while TA) uses the > parameter for bias-variance trade-off, our blended estimator aims trade-off bias in dynamics model with bias in leamed value function.
Why use blending À when one can simply tune the horizon H? First, H limits the resolution we can tune since it's an integer (s,a) uses far more samples. Say we have access to optimal horizon H", Even both Q7 and Q had the same bias, the latter uses a strict subset of samples as the former. Hence the variance of the blended estimator will be lower, with high probability.
H gets smaller the resolution becomes worse. Second, the blended estimator
4 THE MPQ(A) ALGORITHM
We develop simple variant of Q-Leaming, called Model-Predictive Q-Learning with À Weights (MPQ(A)) that learns a parameterized Q- Q-function estimate Qo. Our algorithm, presented in Alg. 1, modifies Q-learning to use blended Q-estimates as described in the (8) for both action selection and generating value targets. The parameter À is used to trade-off the errors due to model-bias and learned Q-function, Qo- This can be viewed an extension of the MPQ algorithm from Bhardwaj et al. (2020) to explicitly deal with model bias by incorporating the learned Q-function at all timesteps. Unlike MPQ, we do not explicitly consider the
formulation, although our framework be modified to incorporate soft-Q Q targets.
4

---

Published as a conference paper at ICLR 2021
Algorithm 1: MPQ(A)
Input: Initial Q-function weights 0, Approximate dynamics P and cost function Parameter: MPC horizon H, À schedule .2....... discount factor minibatch size K, num mini-batches N, update frequency tupdate
D+ for
t= 1...x do // Update A
A= Ae
Blended MPC action selection
Φ + argmin E no MM
Execute a, the system and observe (c,,8441) D+ (1,,,,1,,, if 17ouupaaee then
(f (f k=1
Sample N minibatches
from D
// Generate Blended MPC value targets
Ih. Ch,n +yminE (0(50.00)+ -1-1 (TA) A(s;,a)) Skmn
10
MM
1 Update with SGD on loss -=
N K &
(ik.. Q0(sem
11
At every timestep t, MPQ(A) proceeds by using H H-horizon MPC from the current state s, optimize a policy Ty with parameters d. We modify the MPC algorithm to optimize for the greedy policy with respect to the blended Q-estimator in (8), that is
H-1 Qo(so.a0)+ + Σ A(s,))(s
6 =argmin E PM
An action sampled from the resulting policy is then executed on the system. A commonly used heuristic is to warm start the above optimization by shifting forward the solution from the previous timestep, which serves as good initialization the noise in the dynamics in small (Wagener et al., 2019). This can significantly cut computational cost by reducing the number of iterations required to optimize (9) every timestep.
Periodically, the parameters 0 are updated via stochastic gradient descent to minimize the following loss function with N mini-batches of experience tuples of size K sampled from the replay buffer
N K
((0) - N
Un.n (o((s.......
(10)
K
The H-horizon MPC with blended Q-estimator is again invoked to calculate the targets
ú ))((())0g)) +ymin
(0(50. + Σ A(s,a:))=s
(11)
Using MPC to reduce error Q-targets has been previously explored in literature (Lowrey et al., 2018; Bhardwaj al., 2020), where the model is either assumed to be perfect model-error is not explicitly accounted for. MPC with the blended Q-estimator and an appropriate allows us to generate more stable Q-targets than using Qo model- based rollouts with terminal Q-function alone. However, running H-horizon optimization for samples a mini-batch can be time-consuming, forcing the use smaller batch sizes and sparse updates. In experiments, employ practical modification where during
5

---

Published as a conference paper at ICLR 2021
Figure 1: Tasks for evaluating MPQ(A). Left to right cartpole, peg insertion with 7DOF arm, and in-hand manipulation orient align pen(blue) with target(green) with 24DOF dexterous hand.
the action selection step, MPC is also queried for value targets which are then stored in the replay buffer, thus allowing us to use larger batch sizes and updates at every timestep.
Finally, we also allow À to vary over time. In practice, À is decayed more data is collected on the system. Intuitively, in the early stages of learning, the bias in Qo dominates and hence we want rely more on the model. A larger value of À is appropriate as it up-weights longer horizon estimates in the blended-Q estimator. As Qo estimates improve over time, a smaller À is favorable to reduce the reliance on the approximate model.
5 EXPERIMENTS
Task Details: We evaluate MPQ(A) simulated robot control tasks, including complex manipulation task with a 7DOF arm and in-hand manipulation with a 24DOF anthropomorphic hand (Rajeswaran" et al., 2018) as shown in Fig. For each task, we provide the agent with a biased version of simulation that used as the dynamics model for MPC. We use Model Predictive Path Integral Control (MPPI) (Williams al., 2017), a of-tc-of-thh-ar sampling algorithm as our MPC algorithm throughout.
A classic control task where the agent slides a cart along a rail to swingup the pole attached via an unactuated hinge joint. Model bias is simulated by providing the agent incorrect masses of the cart and pole. The masses are set lower than the true values to make the problem harder for MPC as the algorithm will always input smaller controls than desired as also noted in Ramos et al. (2019). Initial position of cart and pole are randomized at every episode.
I. CARTPOLESWINGUS
2. PEGINSRRTION The agent controls a 7DOF Sawyer arm to insert a peg attached to the end-effeccor into hole at different locations on a table in front of the robot. We test the effects of inaccurate perception by simulating a sensor at the target location that provides noisy position measurements at every timestep. MPC uses deterministic model that does not take sensor noise into account as commonly done in controls. This biases the cost of simulated trajectories, causing MPC to fail to reach the target.
3. INHANDMANIPULATIION A challenging in-hand manipulation task with 24DOF dexterous hand from Rajeswaran* et al. (2018). The agent must align the pen with target orientation within certain tolerance for succcess. The initial orientation of the pen is randomized at every episode. Here, we simulate bias by providing larger estimates of the mass and inertia of the pen as well as friction coefficients, which causes the MPC algorithm to optimize overly aggressive policies and drop the pen.
Please refer to the Appendix A.2 for more details the tasks, success criterion and biased simulations.
Baselines: We compare MPQ(X) against both model-based and model-free baselines MPPI with true dynamics and no value function, MPPI with biased dynamics and no value function and Proximal Policy Optimization (PPO) Schulman et al. (2017).
We represent the Q-function with foed-forward neural network. We bias simulation parameters like mass friction coefficients using the formula m= (1+ (1+6))m bymrree where b is a bias-factor. We also employ a practical modification to Alg. 1 in order to speed up training times as discussed in Section 4. Instead of maintaining large replay-buffer and re-calculating targets for every experience tuple in a mini-batch, as done by approaches such as Bhardwaj et al. (2020); Lowrey et al. (2018), we simply query MPC for the value targets online and store them smaller buffer, which allows us to perform updates at every timestep. We use publicly the available implementation at https://bit.l//8RDDDD for PPO. Refer the Appendix 4.2 for more details.
Learning Details:

---

Published as a conference paper at ICLR 2021
(a) Fixed A
(b) Fixed w/s Decaying À
(c) Varying Model Bias
(d) A decay with H
(e) À decay with 32
(l) Varying Horizon v/s
2 1.0
A- 0.8
100 particles
50 partieles
20 particles
10 partieles
Max Validation Reward
-200
200
-300
4400
500
-600
770
-000
-800
-000
16
Horizon
Slas--Vriance Trade-off
Figure KARTPOLESWINGUS experiments. Solid lines show average rewards over 30 validation episodes (fixed start states) with length of 100 steps and 3 runs with different scods. The dashod lines are average reward of MPPI for the same validation episodes. Shaded region depicts the standard error of the mcan that denotes the confidence the average reward estimated from finite samples. Training performed When decaying as per schedule, is fixed to the current value during validation. (b),d))()), (f) Ap denotes the value at the end of training. PPO asymptotic performance is reported as average reward of last 10 validation iterations. (g) shows the best validation reward at the end of training for different horizon values and MPPI trajectory samples (particles) using A= 1.0 and À 0.8
100k steps with validation after every 4k steps.
5.1 ANALYSIS OF OVERALL PERFORMANCE
1. MPQ(A) is able to overcome model-bias in MPC for wide range of values.
Fig. 2(a) shows a comparison of MPQ(A) with MPPI using true and biased dynamics with b=-0.5 and = for various settings of X. There exists a wide range of À values for which MPQ(A) can efficiently trade-off model-bias with the bias in the learned Q-function and out-perform MPPI with biased dynamics. However, setting À to a high value of 1.0 or 0.95, which weighs longer horizons heavily leads to poor performance as compounding effects model-bias are not compensated by Qp. Performance also begins to drop À decreases below 0.6. MPQ(A) outperforms MPPI with access the true dynamics and reaches close to asymptotic performance PPO. This is not surprising as the learned Q-function adds global information to the optimization and À corrects for errors in optimizing longer horizons.
O 2.
Faster convergence can be achieved by decaying À over time.
As more data is collected on the system, we expect the bias in Q9 to decrease, whereas model bias remains constant. A larger value of À that favors longer horizons is better during initial steps of training the effect of randomly initialized Qa is diminished due to discounting and better exploration achieved by forward lookahead. Conversely, Qo gets more accurate, model-bias begins hurt performance

---

Published as a conference paper at ICLR 2021
1003
2003
Avrrage Strcees
MPPILL3 -0.
1883
MPRRIDS
MPRRIOS v0.6
D-
Valiaaion Meration
Validatinn Iterntion
(a) INHANDMANIPULATIIN Reward
(b) INHANDMANIPULLTIONN Success Rate
100
Avenge Succes Rate
Averuss Rooard
-300
-400
VEPARTTTeS sertian PROlecmpii
20
Validation Eteration
V Validation Iterndia
(c) SAWYER PAGNNERPGINNSRRTION Reward
(d) PEGINETTNIN Success Rate
Figure 3: Robustness and sample efficiency of MPQ(A). (a)((b) Varying bias factor over mass, inertia and friction of pen (d) Peg insertion with noisy perception. Total episode length is 75 steps for both. Same bias factor b is used for all altered properties per task. Curves depict average reward over 30 validation episodes with multiple seeds and shaded areas are the standard error of the mean. Validation done after every steps and is decayed 0.85 end of 75k training steps in both. Asymptotic performance of PPO is average last validation itcrations. Refer to Appendix A.2 for details on tasks and success metrics.
and a smaller À favorable. We test this by decaying À in [1.0,Ar] using a fixed schedule and observe that indeed faster convergence is obtained by reducing the dependence on the model over training steps as shown in 2(b). Figures 2(d) and 2(e) present ablations that show that MPQ(A) is robust to a wide range of decay rates with H = 64 and 32 respectively. When provided with true dynamics, MPPI with H = 32 performs better than H = 64 due to optimization issues with long horizons. MPQ(A) reaches performance comparable with MPPI H 32 and asymptotic performance of PPO in both cases showing robustness to horizon values which is important since in practice we wish to set the horizon as large as our computation budget permits. However, decaying À fast or too slow can have adverse effects on performance. An interesting question for future work is whether À can be adapted in state-dependent manner. Refer to Appendix A.2 for details on the decay schedule.
03. (PQ(A) is much control tasks, while maintaining asymptotic performance.
sample efficient compared to model-free RI. on high-dimensional continuous
Figures 2 and 3 show a comparison of MPQ(X) with the model-free PPO baseline. all cases, we observe that MPQ(A), through its of approximate models, learned value functions, and dynamically-varyyin parameter to trade-off different sources of error, rapidly improves its performance and achieves average reward and success comparable to MPPI with access to ground truth dynamics and model-free RL in the limit. In INHANDMANIULL ATION, PPO performance does not improve at all over 150k training steps. In PEIEEEERSEETRIOONN the small magnitude of reward difference between MPPI with true and biased models is due to the fact that despite model bias, MPC is able to get the peg close to the table, but sensor noise inhibits precise control consistently insert in the hole. Here, the value function learned by MPQ(A) can adapt to sensor noise and allow for fine tme-grained control near the table.
O 4. MPQ(A) is robust to large degree of model misspecification.
Fig. 2(c) shows the effects of different values of the bias factor ò used to vary the mass of the cart and pole for MPQ(A) with fixed X decay rate of ..0.0.75). MPQ(A) (X) achieves performance better than MPPI (H = =44) with true dynamics and comparable to model-free RL in the limit for a wide range of bias factors b, and convergence rate is generally faster for smaller bias. For large values of b, MPQ(A) either fails improve
8

---

Published as a conference paper at ICLR 2021
divenges the compounding effects of model-bias hurt learning, making model-free RL the more favorabe alternatie.. A similar trend is observed in Figures 3(a) and 3(b) where (PQ(A) outperforms MPPI with corresponding bias in the mass, inertia and friction coefficients of the pen with atleast margin of over 30% in terms of success rate. It also achieves performance comparable to MPPI with true dynamics and model- free RL in the limit, but is unable to do so for b= 1.0. We conclude that while MPQ(A) is robust large amount of model bias, if the model is extremely un-informative, relying on MPC can degrade performance.
05. ()Q(A) is robust to planning horizon and number of trajectory samples in sampling-haser MPC. TD(A) based approaches are used for bias-variance trade-off for value function estimation in model-free RL. In our framework, À plays similar role, but trades-off bias due the dynamics model and learned value function against variance due to long-horizon rollouts. We empirically quantify this on the CARTPOLESWING WIIGUUP task by training MPQ(A) with different values of horizon and number of particles for >= 1.0 and x= respectively, Results in Fig. 2(g) show that - (1) using À can overeome effects of model-bias irrespective of the planning horizon (except for very small values of 1 or 2) and (2) using A can overcome variance due to limited number of particles with long horizon rollouts. The ablative study in Fig. 2(f) lends evidence to the fact that is preferable to simply decay À over time than trying to tune the discrete horizon value to balance model bias. Not only does decaying À achieve a better convergence rate and asymptotic performance than tuning horizon, the performance is more robust to different decay rates (as evidenced from Fig. 2(d)), whereas the same does not hold for varying horizon.
CONCLUSION
In this paper, we presented a general framework to mitigate model-bias in MPC by blending model-free value estimates using parameter X, to systemativally trade-off different sources of error. Our practical algorithm achieves performance close to MPC with access to the true dynamics and asymptotic performance of model-free methods, while being sample efficient. An interesting avenue for future research is to vary À in a state-adaptive fashion. In particular, reasoning about the model and value function uncertainty may allow to vary À to rely more less on our model in certain parts of the state space. Another promising direction is to extend the framework to explicitly incorporate constraints by leveraging different constrained MPC formulations.
ACKNOW
This work was supported in part by ARL SARA CRA 191-22200220005 The authors would like to thank Aravind Rajeswaran for help with code for the peg insertion task.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Exploration and apprenticeship learning in reinforcement learning. In Proceedings of the 22nd international conference on Machine learning, pp. 1-8, 2005.
Pieter Abbeel, Adam Coates, and Andrew Y Ng. Autonomous helicopter aerohatics through apprenticeship learning. The International Journal of Robotics Research, 29(13): 1008))96081163, 2010.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems, pp. 5360-5370, 2017.
Mohak Bhardwaj. Ankur Handa, Dieter Fox, and Byron Boots. Information thcoretic model predictive q-learning. In Learning for Dynamics and Control, pp. 840 850-550,22 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement leariing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754 4765, 2018
Tom Erez, Kendall Lowrey, Yuval Tassa, Vikash Kumar, Svetoslav Kolev, and Emanuel Todorov. An integrated system for real-time model predictive control of humanoid robots. In 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 292-299, IEEE, 2013.
Tuomas Haamoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 2,55000 2018.

---

Published as a conference paper at ICLR 2021
Sham Kakade, Michael Kearns, and John Langford. Exploration in metric state spaces. In Proceedings of the 20th Interaational Conference on Machine Learning (ICML-03), pp. 306 306-312, 2003.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2-3):209-22 2002.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Gilwoo Lee, Brian Hou, Sanjiban Choudhury, and Siddhartha S Srinivasa. Bayesian residual policy optimization: Scalable bayesian reinforcement learning with clairvoyant experts. arXiv preprint arXiv:2002.03042, 2020.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via modell-ased control. arXiv preprint arXiv:1811.01888, 2018.
David Q Mayne, James B Rawlings, Christopher Rao, and Pierre OM Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789 814, 2000.
David Q Mayne, Erric C Kerrigan, EJ Van Wyk, and Paola Falugi. Tube- based robust nonlinear model predictive control. International Journal of Robust and Nonlinear Control, 21(1 11:1311-1353 201
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, In 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 7559-7566. IEEE, 2018
Aravind Rajeswaran", Vikash Kumar*, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint /rXiv:/906.011 2019.
Stephane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning. arXiv preprint 1203. 1007, 2012.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Pranav Shyam, Wojciech Jaškowski, and Faustino Gomez. Model-basee active exploration. In International Conference on Machine Learning, pp. 5779-5788, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre,, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver. Thomas Hubert, Julian Schritwwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 171200115,, 2017.
Colin Summers, Kendall Lowrey, Aravind Rajeswaran, Siddhartha Srinivasa, and Emanuel Todorov. Lyceum: An efficient and scalable ecosystem for robot learning. arXiv preprint arXiv:2011.03443, 2020.
Wen Sun, J Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning & imitation learning. arXiv preprint arXiv: 105. /240, 2018.
Nolan Wagener, Ching-An Cheng, Jacob Sacks, and Byron Boots. An online learning approach to model predictive control. arXiv preprint arXiv: 2019.
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA). pp. 1433-1440. IEEE, 2016.
10

---

Published as a conference paper at ICLR 2021
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpe for model-bused reinforcement learning. In 2017 IEEE Interationall Conference on Robotics and Automation (ICRA), pp. 1714-1721. IEEE, 2017.
Mingyuan Zhong, Mikala Johnson, Yuval Tassa, Tom Frez, and Emanuel Todorov. Value function approximation and model predictive control. In 2013 IEEE symposium adaptive dynamic programming and reinforcement learning (ADPRL), pp. 100-107. IEEE, 2013.
A
APPENDIX
A.1 PROOFS
We present upper-bounds on performance of a greedy policy when using approximate value functions and models. We also analyze the case of finite horizon planning with approximate dynamics model and terminal value function which can be seen as generalization of (Sun et al., 2018). For simplicity, we switch to using (s) to the learnt model-free value function (instead of (()))
Let (s) be an e-approximation such that V(sa), we have P(s((sa))P((((l..)
[[ (s)
S c. Let MDP M be an a-approximation M
0
<a and [(s,,,)
A.1.1 A GENTLE START: BOUND ON PERFORMANCE OF 1-STEP GREEDY POLICY
Theorem A.1. Let the one-step greedy policy be
(8) =
P(s)s,))
(12)
aeA
The performance loss of R(s)) war optimal policy nt MDP M is bounded by
VV(()) (xt)) - yau Viuax - Viniu
(13)
Proof. From (12) we have YsES
((s,t(s))
((s,f(s)) (b)) (0))() (6)) (((
(using ([)) Vz l== 5e)
(of((s))-((s.))) (s),
- EPMNSAE)N)))) +270
(using ((s,,)) <a)
P (s)) EP(61 (
((,.((s)) (s)) s2y
(14)
Now, let a be the state with the max loss (x)

---

Published as a conference paper at ICLR 2021
Substituting from (14)
(s) 22ye+2a-22 2 - (5)
P(s E P(
Add and subtract " (5") and re-arrange
( (2) - (s') (5')
(P(6))ast) P( V (8)
Σ P(sl)s.tt) (V (V)
Viaxx Vinin
P()((s,5t) ((6105)(VV(()))VV +7
2 20x +
2
Since is the state with largest loss
Vmax= Vinin
(z) 22
+7
Σ
P(s (s)
2
Vmax Vmax-Vmin
1 (8) -
20
2
Re-arranging terms we get
||))()) 2
Vinx Venin Ventn
(15)
which concludes the proof.
A.1.2 BOUND ON PERFORMANCE OF H-STEP GREEDY POLICY
Notation: For brevity let us define the following macro,
H -
(V.T,M) =Ep Σ
(16)
which represents the expected cost achieved when executing policy π on M using V as the terminal cost.
We can substitute different policies, terminal costs and MDPs. For example, (V.t.M is the expected cost obtained by running policy á on simulator M for steps with approximate learned terminal value function V.
Lemma A.1. For a given policy n, the optimal value function V and MDPs M.M the following performance difference holds
(VZI (V /1 S (0('TT))
11 1-y
aH
Cinax
Cmin
+ aH
Vinax
Vnnnn
+
1 1
2
2
1 1-7
Proof. We temporarily introduce new MDP M' that has the same cost function as M, but transition function of M
(V (MM) (V = (VRimTmM) 1
(vzi
(17)
12

---

Published as a conference paper at ICLR 2021
Let AP(s P(so.sss)))) ) (at..sss)-PPss....... (S) by executing on M and M respectively starting from state 80-
represent the difference in distribution of states encountered
Expanding the RHS of (17)
Σ S
Σ 1=0
7 +o +
7 (((s)a)) (18)
1=
Since the first state $1 is the same
H-1
Σ AP(S)..88H)
Y (((s,a)++ (sn)
+ +R
(c(s,ai)
Σ
AP(S1...HH)
Σ
y td(s),aa) + + (SH)
+
Bug
Σ 1=0
((((s.a))
M
H-1
1-7
Σ AP(S)..8H)
Σ
('(s). +7 V (sm)
+
1
1-y
AL
(19)
where the first inequality is obtained by applying the triangle inequality and the second one is obtained by applying triangle inequality followed the upper bound on the error in cost-function.
H-1
1-""
Σ $2 #1
AP(22...8H4)
sup Σ
x fdss,a;) + VV (8t) K)+
(20)
1-7
By choosing K = E
H-1 72 (Crmux + (2 (Vina + Vinin/22 we can ensure that the term inside
sup is upper-bounder boupded by (1-1111 //1 (1->) (1->))(Cmmx Cmnin) 2 "Vmmax - (Vni)))
1- 11 <
Cmax - Cmin ) + + aH
Vmax Vinin
1-
aH
+
(21)
1-y
2
2
1-7
The above lemma builds on similar results in (Kakade al., 2003; Abbeel & Ng, 2005; Ross & Bagnell, 2012).
We are now ready to prove our main theorem, i.e. the performance bound of an MPC policy that uses an approximate model and approximate value function.
Proof of Theorem 3.1
Proof. Since, t is the greedy policy when using M and V.
(V.R.M) H <(VVT' (Vi < VVi -M) +2 He (using V-V Se)
(22)
MM)
Also, we have
(VM - -(Viio= =
((VT a.
H1 ( ) ( " .M)
H
- -(-ViVi (Vii wnl
(23)
11
- -(Xi m -M)
The first two terms be bounded using Lemma A.1 and the third term using Eq.. (22) get
13

---

Published as a conference paper at ICLR 2021
(VMiauM -(VZi (MM MM)
1 1-1
aHl
Crax
Crii
+ +1" aH
Vinxx
-Viin
Vnnu
1-
1-yath"s
(24)
1-7
2
2
1-7
Now, let 8 be the state with max loss (M(s)-Vii
- (VMDAM) # ( (M)
-(VI, - AM
H (VV - -(Vi or
H
(V -V (SH+1)
V3t sirM) - -V H
5T V
+2
(1- all Cmax Cmin
Cmin
+ all
Vmnxx Sltax-Vine Vimin
1- a
1-7
1-7
(25)
where last inequality comes from applying Eq. (24) and the fact that the state with max loss. final expression follows from simple algebraic manipulation.
The
A.2
EXPERIMENT DETAILS
A.2.1
TASK DETAILS
CARTPOLESWINGUP
Reward function: 2oort + 0 +0.01 +0.00e000000......0 +.00 +
pole
Observation: [Neartt pote: (4 dim)
SOWYERPEGINSERTION We simulate sensor noise by placing simulated position sensor at the target location the MuloCo physics engine that adds Gaussian noise with a = 10cm to the observed 31) position vector. MPPI uses deterministic model that does not take sensor noise into account for planning. Every episode lasts for 75 steps with timestep of 0.02 seconds between steps
Reward function: 11. * ||
Ftarget ||:
5.0 * ||Fee
Ttargetlla
1 1[| ([Iaeee-Ftargelle <0.06)
Observation: Neet Ptarrttt Ftarget,ll
Starget || aleee Tsargetlla (25 dim)
An episode is considered successful if the peg stays within the hole for atleast steps.
INHANDMANPUULATTION This environment was used without modification from the accompanying codebase for Rajeswaran* al. (2018) and is available htpps//hitlly336MNNBBP
Reward function: -I|may - 2 + 20by Ades Bonus for proximity to desired pos + orien,
-T' Zdes represents dot product between object axis and target axis to measure orientation similarity.
Observation: [gpos Vobi (obbj,Poo Fides Foby - -Ides - Zdee] (45 dim)
Every episode lasts 75 steps with a timestep of 0.01 seconds between steps. An episode is considered successful if the orientation of the pen stays within specified range of desired orientation for atleast 20 steps. The orientation similarity is measured by the dot product between the pen's current longitudinal axis and desired with threshold of 0.95.
14

---

Published as a conference paper at ICLR 2021
Table 1: MPPI Parameters
CARTPOLESWINGUP
SAWYERPEGINSSRRIIN
INHANDMANIPLLATION
Parameter
Value
Parameter
Value
Parameter
Value
Horizon
32
Horizon
20
Horizon
32
Num particles
60
Num particles
100
Num particles
100
Covariance (E)
0,45
Covariance (2)
0.25
Covariance (2)
0.3
Temperature( 3)
0.1
Temperature( 5)
0.1
Temperature( 3)
0.15
Filter coeff'
[1.0, 0.0, 0.0]
Filter coeffs
[0.25, 0.8, 0.0]
Filter coeff'
[0.25, 0.8, 0.0]
Step size
1.0
Step size
0.9
Step size
1.0
7
0.99
7
0.99
7
0.99
A.2.2 LEARNING DETAILS
Validation: Validation is performed after every N training episodes during training for Neval episodes using a fixed set of start states that the environment reset to. We ensure that the same start states are sampled at every validation iteration by setting the seed value to pre-defined validation seed, which is kept constant acmoss different runs of the algorithm with different training seeds. This helps ensure consistency in evaluating different runs of the algorithm. For all our experiments set N and Neval =30.
(PQ(A): For all tasks, we represent Q function using 2 layered fully-connected neural network with 100 units in each layer and ReLU activation. We use ADAM (Kingma & Ba, 2014) for optimization with a learning rate of 0.001 and discount factor 7 = 0.99. Further, the buffer size is 1500 for CARTPOLESWINGUS and 3000 for the others, with batch size of 64 for all. We smoothly decay according to the following sublinear decay rate
Ao 1+NVE
A
(26)
where the decay rate k. is calculate based on the desired final value of X. For batch size we did a search from [16, 64] with step size of 16 and buffer size was chosen from 1500, 3000, 5000. While batch size was tuned for cartpole and then fixed for the remaining two environments, the buffer size was chosen independently for all three.
Proximal Policy Optimization (PPO): Both policy and value functions are represented by feed forward networks with 2 layers each with 64 and 128 units for policy and value respectively. All other parameters are left to the default values. The number of trajectories collected per iteration is modified to correspond with the same number of samples collected between validation iterations for MPQ(A). We collect 40 trajectories per iteration. Asymptotic performance is reported as average of last 10 validation iterations after 500 training iters in SAWYER
and 2k in INHANDMANIPULATOON
MPPI parameters Table shows the MPPI parameters used for different experiments. In addition to the standard MPPI parameters, in certain cases we also use step size parameter as introduced by Wagener al. (2019). For INNHADDANIPULAAATIO ATION and SAWYER PERYERREIINNT INGERNSTTION we also apply autoregressive filtering on the sampled MPPI trajectories to induce smoothness in the sampled actions, with tuned filter coefficients. This has been found to be useful in prior works (Summers et al., 2020; Lowrey et al., 2018 8) for getting MPQ(A) to work on high dimensional control tasks. The temperature, initial covariance and step size parameters for MPPI were tuned using a grid search with true dynamics. Temperature and initial covariance were set within the range of [0.0,1.0] and step size from [0.5,1.0] with a discretization of 0.05. The number of particles were searched from [40]120] with a step size of 10 and the horizon was chosen The best performing parameters then chosen based on average
from 4 different values [16,20,32, reward over 30 episodes with a fixed seed value to ensure reproducibility. The same parameters were then used in the case of biased dynamics and MPQ(A), to clearly demonstrate that MPQ(A) can overcome sources of error in the base MPC implementation.
15


Reviews:
[{"review_id": "e1srX7cgX47", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Summary\n\nThe paper describes an algorithm to tackle model bias in the MPC. They address the question of optimal horizon length as well the model errors observed in MPC based systems. The paper is well motivated and written in clear concise manner. Experiments with cart-pole and robot models demonstrate the practical feasibility of the proposed method.\n\n### Strong Points\n\n1. Good description of the sources of errors in MPC based models\n2. The Theorems are useful though I was not able to check their algebraic\n   accuracy, the limiting constructs are intuitively correct. \n3. Choosing the horizon limit and tackling model errors are import challenges\n   for MPC and the method proposed in this paper would be a good addition to the\n   knowledge, hence recommendation for accept.\n\n### To improve\n\n1. I am not sure the MPPI is the SOTA baseline for this comparison, there are\n   other MPC methods that achieve better results than MPPI.\n2. Although citations from the machine learning community seem to be covered the\n   standard MPC literature seem to be completely ignored. At the bare minimum,\n   when speak of stability and fast horizon planning, no reference to tube based\n   MPC [see Mayne 2011 ]\n3. Major advantage of MPC is the ability to deal with constraints again see\n   [Mayne et ref 2 below]. These are not recent developments but classic\n   position papers that address lot of questions you pose and attempt to answer\n   in the paper. \n\n4. The figures need a significant improvements. In their current form the plots\n   are too thin to read them correctly. \n\n\n\n## Refs\n\n1. Mayne, D.Q., Kerrigan, E.C., van Wyk, E.J. and Falugi, P. (2011), Tube‐based robust nonlinear model predictive control. Int. J. Robust Nonlinear Control, 21: 1341-1353. doi:10.1002/rnc.1758\n2. D.Q. Mayne, J.B. Rawlings, C.V. Rao, P.O.M. Scokaert,\nConstrained model predictive control: Stability and optimality,\nAutomatica,Volume 36, Issue 6,2000,Pages 789-814, ISSN 0005-1098,", "extra_scores": {}, "extra_text": {}}, {"review_id": "dgnxoWGl6Cc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##\n**Summary:**\nThe paper proposes to combine MPC and model-free RL to overcome the possible modelling errors. Thereby the approach achieves the sample-efficiency of MPC and the control quality of model-free RL. The resulting MPQ(\\lambda) algorithm uses MPPI to obtain the actions by optimizing the blended MPC objective. The Q-targets for fitting the q function also use the blended Q-estimate.\n\n\n##\n**Quality, Originality & Significance:**\nThe idea of combining MPC and model-free RL is straight forward and not novel (the paper also does not claim this). However, the exact instantiation is novel, very well motivated and feels natural. My biggest concerns are the experiments. The cartpole experiments show the improved performance compared to MPPI on the biased model and the impact of the lambda and model bias. However, the PPO baseline is missing for the cartpole, right? Furthermore, is PPO a fair comparison for the MPQ($\\lambda$) algorithm to evaluate sample complexity? While PPO is a batched update, the MPQ($\\lambda$) uses step based updates. Wouldn't a model-free step-based update algorithm such as DDPG, SAC etc. be a better baseline to evaluate the improved sample complexity? Regarding the high-dimensional tasks, the provided evaluations do not enable an evaluation whether the task is solved or not. Could you please provide videos of the final policies, otherwise the achieved reward is just a random number. Furthermore, the paper shows confidence bounds for the MPPI baselines but not for the MPQ($\\lambda$) algorithm. Also the learning curves are cut before converging, could you train every instance until convergence? Furthermore, could you please include the asymptotic performance of your baseline in the plots. The definition of 'validation iteration' remains unclear. Given the current evaluations the stated claim of applicability to high-dim tasks cannot be made as the evaluations are not sufficient. The used modelling bias is also very limited as the paper only compares to biases of the model parameters but not against other sources of biases. Ultimately the increased performance can only be shown on the physical system.\n\n##\n**Clarity & Style:**\nThe paper is really well written and understandable! A few sections could be improved, e.g., text between Eq. 6 & Eq. 8. In this section it is a bit unclear what is expanded and how it is expanded. It would be beneficial to rethink the labeling of the Q-functions as they can be quite confusing. Maybe a table of the different subscript/superscript definitions would simplify the reading, as I had to search for the exact definitions frequently. Furthermore, there are minor styling issues:\n* Inline equations are consuming too much space to mess up line spacing, e.g, Section 2.1 argmin, Theorem 3.1, norms,\n* the min in Equation 13 needs two spaces and a subscript\n* Experiment O4 ends with two dots\n* White space around figure 2 can be optimized\n\n##\n**Conclusion:**\nAll in all the paper is nicely written with a clear and well motivated idea of combining MPC & model-free rl. Right now the main problem is the execution of the evaluations. The performance on the high-dim tasks is unclear and the baseline is missing for the cartpole. I would be happy to improve my score to\n* **weak accept** if a step based model free RL algorithm is added to cartpole, claims regarding high-dim tasks are adapted and videos of the high-dimensional tasks are released.\n* **accept** if the high-dimensional tasks are working properly with MPQ($\\lambda$)\n* **strong accept** if MPQ($\\lambda$) shows this performance on a physical system\n\n\nP.S. You might also try to get medium dimension tasks working such as hopper or cheetah. That might be a bit easier. \n\n##\n**Post Discussion Comments:** So the author did a **filibuster** and **flooded the discussions** with bloated comments. In this manner it was close to impossible to keep track of anything. **There has to be character limit for responses otherwise this is not feasible**. I looked at the videos and your physics simulators looks **really catchy**. At one point in time, the pole of the cartpole is at 10-11 o'clock and the cart starts moving right (the pole has close to no velocity and hence only a very small angular momentum). In this setting it would be natural that the pole would fall down if this state is maintained for a longer period (which it is in the video). However the pole goes upwards into the balancing position. This is really weird. And don't get me started on the pen-orientation as the pen sometimes floats mid-air. For this setting the gravitational constant really does not seem right. I also wouldn't consider the task solved as this is more an really uncoordinated movements for three specific configurations. \n\nFor the simulation studies some doubts remain, but the authors improved the paper. Therefore, I am going to increase my score to weak accept. Nevertheless, the experimental evaluation could be improved and the paper would really benefit from real experiments. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "aLPXoWfYbln", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nBlending MPC & Value Function Approximation for Efficient Reinforcement Learning\nreview:\n\nsummarization:\n\nIn this paper, the authors consider using a blending of Q value which is predicted directly from the neural network,\nand a Q value which is predicted by unrolling the learnt dynamics.\nThe empirical results suggest improved performance, \nsample efficiency and good robustness in choosing different values of blending coefficient.\n\nPros:\n1. The idea of blending the Q values is novel.\nI also think the connection to GAE is quite natural and interesting,\nwhere both algorithms consider trade off between bias and variance (in this paper’s case, bias in learned dynamics).\n\n2. The supporting experiments consider some of the interesting questions.\nFor example in section 5.1, the question of how sensitive lambda is is addressed.\n\n3. The direction of combining value estimation in model-predictive control is interesting and under-explored.\nThat being said, this paper can be inspiring and helpful towards future research.\n\nCons:\n\n1. The experiment section lacks comparison among state-of-the-art algorithms.\nWhile MPPI and PPO were generally considered state-of-the-art at the time when they are published (2017),\ntheir performance is now outperformed heavily given the fast development in the research direction.\nIt would be great if some of the strong baselines between 2019-2020 are included (SAC, TD3, MBPO etc.).\nAlso given the similarity to MCTS algorithms, it would be more convincing to include one variant of it as a baseline.\n\n2. Experiments on more environments are also appreciated. \n\nQuestions:\n\nI didn’t see study on the training time and testing time (how much time needed to generate one action during testing), but it seems to be referred to in the introduction?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A6pYK9FhVJ1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper provides and interesting analysis of and new method for model-predictive control in reinforcement learning. \nSpecifically, it proposes a new framework MPQ(lambda) for joining model-based MPC with learned value estimates in RL.\nThe authors develop an formulation to find an optimal prediction horizon and and how this works in an online reinforcement learning framework.\nThe new approach is evaluated on 3 continuous control tasks and compared to some other baselines.\n\n-----\nScore reasoning:\n\nThis paper has interesting theoretical contributions to multiple areas of machine learning: model-based RL, MPC, and value estimation, but the somewhat limited experimental evaluation make the efficacy of the method more difficult to judge. Overall, the paper is well written and I enjoyed reading it. I now will address my conceptual comments followed by more minor suggestions.\n\n-----\nRebuttal update: the authors have gone beyond the normal scope of a rebuttal phase to update their experiments and the motivation of the work, and for that reason I have improved my recommendation to be above the acceptance threshold.\n\n-----\nExperimental Validation Questions.\nI am breaking this section of the review into it's own section because it is where the majority of my questions are.\n\nE1) The authors bring up model-based RL algorithms, but do not baseline against other algorithms generally considered sample efficient. PPO is not always easy to use (authors mention it not converging, and not substantial parameter tuning), how about SAC? \nE1b) It would be very interesting to compare something similar to the PETS optimizer for MPC (cited in intro, but not really mentioned). These baseline changes could make the results much more believable.\n\nE2) \"All parameters were found using a coarse grid search\" This makes the results suspect to me. Please clarify how coarse? Is the same search space used for all algorithms? Were defaults used for algorithms with previously published results? Do the results match?\n\nE3) Was tuning of the reward functions done by hand in A.2.1, or are they referenced elsewhere? Are states like x-position and pole angle normalized in cartpole? This can have bigger effects in more complicated environments.\n\nE4) \"shaded regions represent standard deviation\" for MPPI, is this over the same 30x3 evaluations? Very important to standard dev. is the number of samples.\n\nE5) Cartpole swing up is a very similar task. It seems to be the only one where MPQ(lambda) substantially outperforms the baselines given (no error bars on MPQ too). How do the ablation studies of figure 2 reproduce on more challenging environments?\n\n-----\nComments:\n1) The authors refer to MPC as a \"simpler, more practical alternative\" to RL or a \"more pragmatic approach\" for \"simple\" policies. \nSome would argue that RL is a simpler approach because it does not require any model in the case of model-free RL. \nMPC also has many design decisions such as which optimizer to use or the planning horizon (multiple papers written on this topic). \nI would like the authors to explain this with more detail, or defend their stance.\n\n2) The authors may consider including these two other papers that relate to model-based RL, model-bias, and MPC horizon https://arxiv.org/pdf/2009.09593.pdf, https://arxiv.org/pdf/2002.04523.pdf. \n\n3) How does model bias differ from model inaccuracy? \nIn MBRL, model-bias often refers to the model being more accurate in some areas of the state-space than others, and how this impacts the downstream ranking of action choices. \nDo the authors consider this difference at all? How does model-accuracy drop when the bias terms are introduced in some basic metrics like mean-squared-error or negative-log-likelihood (metrics used in MBRL to quantify model-accuracy).\n\n4) The position of the contribution in related works could be made stronger. I was unaware that MPQ was not the proposal of this paper until section 4. The difference between the two and why this matters should be in the introduction (unless the authors decide to add a dedicated related works section).\n4b) how does entropy-regularized formulation impact the results? From my reading, that is an important part of the original MPQ paper, so I think it should be explained.\n\n5) The conclusion to this paper is weak. It re-iterates what is done, but the authors should make a case how this impacts developments in robotics & control to better match up with the experiments and introduction. What should I take away from studying this paper? \n\n6) It would be interesting to see the authors propose how to combine the MPQ framework with other forms of MPC that don't have an implicit terminal cost included. This may be for future work, but I would be interested in a comment.\n\n-----\nMinor comments:\n1) There are some typos that impede reading, but overall the paper is well written.\n- intro, paragraph 2, \"owing to its ability to\" is weird\n- section 2.2 \"since it plans...\" it is vague here\n- some missing commas in first paragraph of section 3 \"First...\"\n- Missing period at end of paragraph \"Baselines\", missing period Figure 3 end, double period before Conclusion\n- Typo in PPO A.2.2 \"The\"\n\n2) in 3.2, the authors show how to blend the model-based and model-free methods, but point to a reference that is not obviously connected to me and call the approach \"common\". \nI would suggest adding more references, or adjusting the claim.\n\n3) Why was MPPI chosen as the MPC algorithm? It is a suitable choice, but could be added.\n\n4) there is a lot of visuals in Figures 2 and 3. Maybe have fewer lines? The font could be enlarged and it is very confusing that the y-axis's are not all the same for similar data types.\n4b) Figure 4b) has strange shading from the MPPI variance - it's not readable.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors put a lot of effort in replying to questions and improving the paper (to a point that the reviewers felt overwhelmed).

Pros:
- An interesting way of dealing with model bias in MPC
- They successfully managed to address the most important concerns of the reviewers, with lots of additional experiments and insights
- R3's concerns have also been successfully addressed by the authors, the review & score were unfortunately not updated

Cons:
- The only remaining point is that the simulations seem to be everything but physically realistic (update at end of R1's review), which is probably a problem of the benchmarks and not the authors faults.
Decision: Accept (Poster)