Information Theoretic Model Predictive Q-Learning
Authors: Mohak Bhardwaj
Ankur Handa
Dieter Fox
Byron Boots
Venue:  (2020)
Keywords: entropy regularized reinforcement learning
information theoretic MPC
robotics

Abstract: Model-free Reinforcement Learning (RL) algorithms work well in sequential decision-making problems when experience can be collected cheaply and model-based RL is effective when system dynamics can be modeled accurately. However, both of these assumptions can be violated in real world problems such as robotics, where querying the system can be prohibitively expensive and real-world dynamics can be difficult to model accurately. Although sim-to-real approaches such as domain randomization attempt to mitigate the effects of biased simulation, they can still suffer from optimization challenges such as local minima and hand-designed distributions for randomization, making it difficult to learn an accurate global value function or policy that directly transfers to the real world. In contrast to RL, Model Predictive Control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effectively contend with real-world dynamics. MPC performance is usually limited by factors such as model bias and the limited horizon of optimization. In this work, we present a novel theoretical connection between information theoretic MPC and entropy regularized RL and develop a Q-learning algorithm that can leverage biased models. We validate the proposed algorithm on sim-to-sim control tasks to demonstrate the improvements over optimal control and reinforcement learning from scratch. Our approach paves the way for deploying reinforcement learning algorithms on real-robots in a systematic manner.

Under review as a conference paper at ICLR 2020
INFORMATION THEORETIC MODEL PREDICTIVE Q-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Model-free reinforcement learning (RI. algorithms work well in sequential decision-making problems when experience can be collected cheaply and model- based RL is effective when system dynamics can be modeled accurately. How- ever, both of these assumptions can be violated in real world problems such robotics, where querying the system can be prohibitively expensive and real-world dynamics be difficult to model accurately. Although sim-to-real approaches such as domain randomization attempt to mitigate the effects of biased simulation, they still suffer from optimization challenges such as local minima and hand- designed distributions for randomization, making it difficult to learn an accurate global value function or policy that directly transfers the real world. In contrast to RL, model predictive control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effec- tively contend with real-world dynamics. MPC performance is usually limited by factors such as model bias and the limited horizon of optimization. this work, we present novel theoretical connection between information theoretic MPC and entropy regularized RL and develop Q-learning algorithm that can leverage bi- ased models. We validate the proposed algorithm sim-to-sim control tasks demonstrate the improvements over optimal control and reinforcement learning from scratch. Our approach paves the way for deploying reinforcement learning algorithms on real-robots in systematic manner.
INTRODUCTION
Deep reinforcement learning algorithms have recently generated great interest due to their successful application to a range of difficult problems including Computer Go (Silver et al., 2016) and high- dimensional control tasks such as humanoid locomotion (Lillicrap et al., 2015; Schulman et al., 2015). While these methods are extremely general and can learn policies and value functions for complex tasks directly from data, they also be sample inefficient, and partially-optimmzzz lutions can be arbitrarily poor. These challenges severely restrict RL's applicability to real systems such robots due to data collection challenges and safety concerns.
One straightforward way to mitigate these issues is to learn a policy or value function entirely in a high-fidelity simulator (Shah al., 2017: Todorov al., 2012) and then deploy the optimized policy on the real system. However, this approach can fail due to model bias, external disturbances, the subtle differences between the real robot's hardware and poorly modeled phenomena such as friction and contact dynamics. Sim-to-real transfer approaches based on domain randomization (Sadeghi & Levine, 2016; Tobin et al.. 2017) and model ensembles (Kurutach et al., 2018; Shyam et al., 2019) aim to make the policy robust by training it to be invariant to varying dynamics. However, learning a globally consistent value function or policy is hard due to optimization issues such as local optima and covariate shift between the exploration policy used for learning the model and the actual control policy executed on the task (Ross Bagnell, 2012).
Model predictive control (MPC) is widely used method for generating feedback controllers that repeatedly re-optimizes a finite horizon sequence of controls using an approximate dynamics model the system. The first control in the optimized sequence
that predicts the effect of these controls executed on the real system and the optimization is performed again from the resulting next state. However, the performance of MPC can suffer due approximate simplified models and lim-

---

Under review as a conference paper at ICLR 2020
ited lookahead. Therefore the parameters of MPC, including the model and horizon H should be carefully tuned to obtain good performance. While using a longer horizon is generally preferred, real-time requirements may limit the amount of lookahead and biased model can result in com- pounding model errors.
In this work, we present an approach RL that leverages the complementary properties of model- free reinforcement learning and model-based optimal control. Our proposed method views MPC as a way to simultaneously approximate and optimize local Q function via simulation, and Q learning as a way to improve MPC using real-world data. We focus on the paradigm of entropy regularized reinforcement learning where the aim is to learn a stochastic policy that minimizes the cost-to-go as well as KL divergence with respect to a prior policy. This approach enables faster convergence by mitigating the over-commitment issue the early stages of Q-learning and better exploration (Fox et al., 2015). We discuss how this formulation of reinforcement learning has deep find control connections to information theoretic stochastic optimal control where the objective (Theodorou inputs that minimize the cost while staying close to the passive dynamics of the system & Todorov, 2012). This helps in both injecting domain knowledge into the controller as well as mitigating issues caused by over optimizing the biased estimate of the current cost due to model error and the limited horizon of optimization. We explore this connection in depth and derive an infinite horizon information theoretic model predictive control algorithm based Williams et al. (2017). We test our approach called Model Predictive Q Learning (MPQ) on simulated continuous control tasks and compare it against information theoretic MPC and soft Q-Learning (Haaroja et al., 2017), where we demonstrate faster learning with fewer system interactions and better performance as compared to MPC and soft Q-Learning even in the presence of sparse rewards. The learned Q function allows us to truncate the MPC planning horizon which provides additional computational benefits. Finally, we also compare MPQ versus domain randomization(DR) on sim-to-sin tasks. We conclude that DR approaches can be sensitive to the hand-designed distributions for randomizing parameters which causes the learned Q function to be biased and suboptimal on the true system's parameters, whereas learning from data generated on true system is able to overcome biases and adapt to the real dynamics.
2 RELATED WORK
Model predictive control has a rich history in robotics, ranging from control of mobile robots such as quadrotors (Desaraju & Michael, 2016) and aggressive autonomous vehicles (Wagener et al., 2019; Williams et al., 2017) to generating complex behaviors for high-dimensional systems such as contact-rich manipulation (Fu et al., 2016; Kumar et al., 2014) and humanoid locomotion (Erez et al., 30.3). The success of MPC can largely be attributed to online policy optimization which helps mitigate model bias. The information theoretic view of MPC aims to find a policy at every timestep that minimizes the cost over a finite horizon as well as the KL-divergence with respect to a prior policy usually specified by the system's 's passive dynamics (Theodorou & Todorov, 2012: Williams et al., 2017). This helps maintain exploratory behavior and avoid over-commitment to the current estimate of the cost function, which is biased due to modeling errors and a finite horizon. campling-based MPC algorithms (Wagener et al., 2019; Williams et al., 2017) are also highly par- allelizable enabling GPU implementations that aid with real-time control. However, efficient MPC implementations still require careful system identification and extensive amounts of manual tuning.
Deep RL methods are extremely general and can optimize neural network policies from raw sen- sory inputs with little knowledge of the system dynamics. Both value-based and paseey-based approaches (Schulman et al., 2015) have demonstrated excellent performance on complex control problems. These approaches, however, fall short on several accounts when applying them to a real robotic system. First, they have high sample complexity. potentially requiring millions of interac- tions with the environment. This can be very expensive on a real robot, not least because the initial performance of the policy can be arbitrarily bad. Using random exploration methods such a c-greedy can further aggravate this problem. Second, a value function or policy learned entirely in simulation inherits the biases of the simulator. Even if perfect simulation is available, learning a globally consistent value function or policy is an extremely hard task as noted in (Silver et al., 2016; Zhong et al., 2013). This can be attributed to local optima when using neural network representations or the inherent biases in the Q learning update rules (Fox et al., 2015; Van Hasselt et al., 2016). In fact, it can be difficult to explain why Q-learning algorithms work or fail (Schulman et al., 2017).
2

---

Under review as a conference paper at ICLR 2020
Domain randomization aims to make policies learned in simulation more robust by randomizing simulation parameters during training with the aim of making the policies invariant to potential parameter error (Peng al., 2018; Sadeghi & Levine, 2016; Tobin et al., 2017). However, these policies are not adaptive to unmodelled effects, i.e they take into account only aleoteric and not epis- temic uncertainty. Also, such approaches are highly sensitive hand-designed distributions used for randomizing simulation parameters and can be highly suboptimal on the real-systems parame- ters, for example, if a very large range of simulation parameters is used. Model-based approaches aim to use real data to improve the model of the system and then perform reinforcement learning or optimal control using the new model or ensemble of models (Kurutach et al., 2018; Ross & Bagnell, 2012: Shyam al., 2019). Although learning accurate models is a promising avenue, we argue that learning a globally consistent model an extremely hard problem and instead we should learn a policy that can rapidly adapt to experienced real-world dynamics.
The use of entropy regularization has been explored in RL and Inverse RL. for its better sample efficiency and exploration properties (Fox et al., 2015; Haarnjja et al., 2017: 2018; Schulman et al., 2017: Ziebart et al., 2008). This framework allows incorporating prior knowledge into the problem and learning multi-modai policies that can generalize across different tasks. Fox et al. (2015) analyze the theoretical properties of the update rule derived using mutual information minimization and show that this framework can overcome the over-estimation issue inherent in the vanilla Q-learning update. In the past, Todorov (2009) have shown that using KI--divergence can convert the optimal control problem into one that is linearly solvable.
Infinite horizon MPC aims to learn a terminal cost function that can add global information to the finite horizon optimization. Rosolia & Borrelli (2017) learn a terminal cost as a control Lyapunov function and safety set for the terminal state. These quantites are calculated using all previously visited states and they assume the presence of controller that can deterministically drive the any state to the goal. Tamar et al. (2017) learns cost shaping to make a short horizon MPC mimic the actions produced by long horizon MPC offine. However, since their approach to mimic longer horizon MPC, the performance of the learner is fundamentally limited by the the performance of the longer horizon MPC. On the contrary, learning an optimal value function as the terminal cost can potentially lead to close to optimal performance.
Using local optimization is an effective way of improving an imperfect value function as noted in RL literature by Anthony et al. (2017); Lowrey et al. (2018); Silver et al. (2016; 2017); Sun et al. (2018). However, these approaches assume that perfect model of the system is available. In order to make the policy work on the real system, argue that it is essential learn a value function form real data and utilize local optimization to stabilize learning.
3 PRELIMINARIES
3.1 REINFORCEMENT LEARNING WITH ENTROPY REGULARIZATION
A Markov Decision Process (MDP) is defined by tuple (S, c, 7) where S is the state space, A is the action space, is one step cost function, P is the space of transition functions and ry is a discount factor. Let P E 'P be a particular transition function. A closed-loop policy ((ass) is a distribution over actions given state. Given policy π and a prior policy म. the KL divergence between them at a state is given by KL (T [[a]s)][[[[[])) = E, [log regularized RL (Fox et al., 2015) aims to optimize the following objective
Entropy-
= arg min Eg.P
(c(st, az) XKL (TelFa)) A so s
(I)
where n. and ,, are shorthand for 7 (a. (t)) and म (aels)) respectively, is the temperature parameter that penalizes deviation of N from म. For policy we can define the soft value and action-value functions
(*((s = Er,
Σ x r-1 (c(s), + AKL +KL((T((EE) Isu
(=((, = c(s,a) BBBw IV" [s')
(2)
3

---

Under review as a conference paper at ICLR 2020
Given horizon of H timesteps, we use above definitions to write the value functions
(*( E
-- (c(s). AKL (FTel - (sH))s1
#-1
(²(s,a) = a( + -E.
Σ
7 1- (c(s,, ar) + AKL (Telat))
H (AKLLL (H))ss = s, =
(3)
is straightforward to verify that (s) = Eaws [bg(((alss/ [(),a)]- The objective
in Eq. (1) can equivalently be written
arg min (*)s)
This optimization can be performed either by policy gradient methods that aim to find the optimal policy т E II via stochastic gradient descent (Schulman et al., 2017: Williams, 1992) or value based methods that try to iteratively approximate the value function of the optimal policy. In either case, the output of solving the above optimization is a global closed-loop control policy x (a(ss.
3.2 INFORMATION THEORETIC MPC
Solving the above optimization be prohibitively expensive and hard to accomplish online. In contrast to RL, MPC performs online optimization of a simple policy class with a truncated horizon. This process effectively creates closed-loop controller. In order to do so, MPC algorithms such as MPPI (Williams et al., 2017) use an approximate dynamics model P. which This is the case when using a simulator such as MuJoCo (Todorov et al., 2012) as the dynamics model. At timestep t. starting from the current state St, an open loop sequence of actions A (at, at+1,... O[+H) is sampled from the control distribution denoted by z<A). The objective is to find an optimal sequence of actions to solve
be deterministic.
-
A" arg minEE(())
Σ 1=1
/' 1-1 cc(ss,)) AKL (TI [m] + (eg"4(( a++H) AKL (Teee|R
A
(S)
where Cy(8++ H. H+) is a terminal cost function and ((A) is the passive dynamics of the system, ie the distribution over actions produced when the control input is zero. The first action in the sequence is then executed on the system and the optimization is performed again from the result- ing next state. The re-optimization and entropy regularization helps in mitigating model-bias and inaccuracies with optimization by avoiding overcommitment to the current estimate of the cost. A shortcoming of the MPC procedure is the finite horizon. This is especially pronounced tasks with sparse rewards where short horizon can make the agent myopic to future rewards. To mitigate this, an approach known as infinite horizon MPC sets the terminal cost ey as a value function that adds global information to the problem.
In the next section, we build our approach by focussing on the MPPI algorithm and its relationship with entropy regularized reinforcement learning. Specifically, functions from Eq. (2) to derive an optimal Boltzmann distribution for H H-step actions that optimally solves the infinite horizon control problem. This helps derive the MPPI update rule from Williams et al. (2017) for the infinite horizon case, which then leads to our Model Predictive Q Learning (MPQ) algorithm, which utilizes predictive model for Q updates and stochastic optimal control as the policy. In the case where H/ = 1, the algorithm is equivalent to soft Q learning (Haarnoja et al., 2018) or G-learning (Fox et al., 2015).
the definitions of the soft value
APPROACH
4.1 OPTIMAL H-STEP BOLTZMANN DISTRIBUTION
Let (AA) and (A) be the joint control distribution and prior over H-horizon open-loop actions.
The distributions are assumed to be independent over timesteps, i.e ( -.. -an) ) I Tt where
4

---

Under review as a conference paper at ICLR 2020
Te is shorthand for x(a)). Since P is deterministic, the following equations hold (*) = Ex X log(T/R) +
(" +(5.())
(6)
For clarity, we consider
1. Substituting from Eq. (3) for (s(sa)
V (6) E B
c(s az) +
log(me/Tm) + Q am
1=1
E E
c(st, az) Alog II
(Te T.) Q as
-H-1
E
Σ
c(st, az) + Alog(a/ Q
Consider the following distribution over H-horizon
H
1
exp
Σ
((), (sHr, m((1.....
(8)
where 77 normalization constant given by
1 = (ai
exp
Σ
c(sr, at) + Q"
(9)
We show that this is the optimal control distribution as VV*(s) = 0. Substituting Eq. (8) in (7)
H Σ
H-1
(s) = E,
d(s.,a))
Alog - Σ
c(su, 0z) - (" (SHH,H)) + Q aH)
t=1
[=1
(*()) = E log vl
Since v is a constant, we have V (s) = A log 1). constant with gradient zero given by
Hence for Eq. (8), the soft value function
-1 A
)
V (s) - log E (as...a)
exp
Σ c(st, ar) Q*(sH
(10)
which is often referred to in optimal control literature as the "free energy" of the system (Theodorou & Todorov, 2012; Williams et al., 2017). For H=1, Eq. (10) takes the form of the soft value function from Fox al. (2015) and Haarnoja et al. (2018).
4.2 INFINITE HORIZON MPPI UPDATE RULE
Similar to Williams et al. (2017), we derive the MPPI update rule for online policy optimization. Since sampling actions from the optimal control distribution in Eq. (8) is intractable, we consider control policies (A) E Il which are easy to sample from. We then optimize for vector of H control inputs U. such that the resulting action distribution minimizes the KI. divergence with the optimal policy
U = arg min KL (T (A))
(11)
x(A)
The objective can be expanded out
TA(
n: (A log TA(A) ((A)
KL (T (A)][=(())
=
xt" (A)
AA
Aa
T(A)
*(A) T(A)
A
A
*((A)
7(A)
n (A) log
dA
nt" (AA log
dA
(12)
म (A)
((A)
5

---

Under review as a conference paper at ICLR 2020
Since the first term does not depend on the control input,
remove it from the optimization
/ nt (A) log
क (A) (A)
U
arg max s(A) A
dA
(13)
Consider 11 to be independent multivariate Gaussians over sequence of the H controls with constant covariance 2 each timestep. We can write the control distribution and prior follows
(A) = 1
1 II exp (-2 (w a.) -aaaa=='' - a:) 1=1
#(A) = 1 Z t=1
1 II exp (- (14)
where in and a, are the control inputs and actions timestep and Z the normalization constant. Here the prior corresponds to the passive dynamics of the system (Theodorou & Todorov, 2012; Williams et al., 2017), although other choices of prior are possible. Substituting in Eq. (13) we get
U = arg max ((A)
/ *A)
-
H
1 Juk
uu uuf as
dA
(15)
The objective can simplified to the following by integrating out the probability in the first term
H
Σ
Σー
u + u!! x² (A)EE- aa dA
(16)
Since this is a concave function with respect to every ue, we can find the maximum by setting its gradient with respect u zero to solve for optimal ui
" / (A)a,dA=
((A) TT (A) π (AA) (A) (A)
(A) कt ((A) (A)
a,dA = (T(()
Bm(a) [w(A)ad)
(17)
where the second equality comes from importance sampling to convert the optimal controls into an expectation over the control distribution instead of the optimal distribution which is impossible to sample from. The importance weight w<A) can be written as follows (substituting x¹ from Eq. (8))
wr( ())
1 (A))
exp
Σ c(sr, at) + Q ($H; @H)
#(A) x ((A)
(18)
Making change of variables Ue + E = ar for noise sequence & = (E1 sampled from inde- pendant Gaussians with zero mean and covariance D we get
1
TU + TU +
+ +Q (SH, us + ten)) )
w((E
(AT(A)
exp
c(st, u4 + 6t)
17
1 Ex(A) 17
Σ clo,, 0t + 6t) + Σ
M 1 (u+ 2es) + Q (sn, un + Et)
exp
2 u
(19)
Note that 1) is the optimal H-step free energy derived in Eq. (10) and can be estimated from N Monte-Carle samples as
N
#-1 Σ
H (u, + 2c?) + Q" (sH, un + Eii
exp
u( + + -
u
2
(20)
We can form the following iterative update rule where at every iteration the sampled control quence is updated according to
++1 ui +
(21)
where a is a step-size parameter as proposed by Wagener et al. (2019). This gives us the infinite horizon MPPI update rule. For H = this corresponds soft Q-learingg where stochastie optimiza- tion is performed to solve for the optimal action online. Now we develop soft Q-learning algorithm that utilizes infinite horizon MPPI to generate actions well as Q-targets.

---

Under review as a conference paper at ICLR 2020
4.3 THE INFORMATION THEORETIC MODEL PREDICTIVE Q-LEARNING ALGORITHM
Since we do not have access to QT we can not estimate the importance weight in Eq. (19) exactly. Hence, we consider functions parameterized by 0 denoted by Qa(s, a). Similar deep Q-learning algorithms, we maintain an replay buffer (Mnih et al., 2015), and update parameters by stochastic gradient descent on the loss L = R Ex (yi (s (()),,;)) a;))² for a batch of K experience tuples (s, a, c, 8") sampled from the buffer where targets yi are given by
H1 Σ as) + A Alog Qu(sn,an) ))
y a) Alog (ar.
exp
81 = (22)
Since the value function updates are performed offline we can utilize large amount of computa- tion (Tamar et al., 2017) to calculate n" (a1 ... ....). In case it is obtained by performing the infinite horizon MPPI update in Eq. (21) for multiple iterations starting from state s'. This allows for directed exploration at a state which leads to better approximation of the free energy (which is akin to approaches such Covariance Matrix Adaption, except MPPI does not adapt the covari- ance). This especially helps in early stages of learning by providing better quality targets than random Q function. Intuitively, this update rule leverages the biased dynamics model f for H steps and soft Q function the end learned from interactions with the real system.
At every timestep during online rollouts, a -hoorizonn sequence of actions is optimized using a single iteration of infinite horizon MPPI update rule in Eq. (21) and the first action is executed on the system. Online optimization with predictive models can lookahead to produce better actions than acting greedily with respect to the biased Q function and makes ad-hoc exploration strategies such as e-greedy unnecessary. Using predictive models for generating value targets and online pol- icy optimization helps accelerate convergence we demonstrate in our experiments in the next section. Algorithm shows the complete MPQ algorithm.
A closely related approach in literature is (Lowrey al., 2018) which also uses online MPC and offline value function learning, however they assume access to the true dynamics of the system and do not explore the connection between MPPI and entropy regularized RL and hence do use free energy targets, even though they use MPPI in their implementation.
Algorithm 1: MPQ
Input
Approximate model P., initial Q function parameters 0,, experience buffer DD Parameter: Number of episodes N, length episode T,, planning horizon H, number update episodes N Nupdate, minibatch-size K, number of minibatches M
for 1...N.N do
for t = do
(a...... O++ Infinite horizon MPPI (Eq. (21))
Execute a, on the real system obtain c(s1, 0x) and next state 81+1
D (8t, at, C, 11+)
if i% 1%/Naadate 0 then
Sample M minibatches of size K from D Generate targets using Eq. (22) and update parameters p
return dx or best 0 on validation.
EXPERIMENTS
We perform experiments to test the efficacy of MPQ in overcoming the shortcomings stochastic optimal control and model free RI. in terms of convergence rate, computational requirements and model bias. We also compare MPQ against domain randomization for learning policies that perform well systems for which accurate models are not known.
5.1 EXPERIMENTAL SETUP
We test our approach on sim-to-sim continuous control tasks based on the Mujoco simula- tor (Todorov al., 2012) to study the properties of the algorithm in controlled manner. The
7

---

Under review as a conference paper at ICLR 2020
agent is not provided with the true dynamics parameters, but uniform distribution over them with a biased mean and added noise. This serves as a reasonable approximation of model bias due to inaccurate measurements of physical quantities. Details of the tasks considered are as follows
1. PENDULUMSWINGUS the agent tries to swingup and stabilize pendulum by applying torque on the hinge. The agent is provided with a distribution over the mass and length of the pendulum. The state the system is given by (0, θ), where is the angular displacement. The cost function penalizes the deviation from the upright position and angular velocity. The initial state of the system is randomized after every episode which is 10 seconds long.
2. BALLINCUPSPARSE: a sparse version of the ball in cup task inspired from Tassa et al. (2018). Given a cup and spherical ball attached by tendon, the goal is swing and catch the ball. The agent can actuate motors on the two slide joints on the cup and is provided with a biased distribution over the mass of the ball, its moment of inertia and stiffness of the tendon. A cost of 1 is incurred at every timestep and O if the ball is in the cup. The position of the ball is randomized after every episode which is seconds long. An episode is successful if agent catches the ball in the cup.
FETCHPUSHBLOCK: proposed by Plappert et al. (2018), the agent position controls the end- effector of a simulated Fetch robot to push a block to goal location the table. The cost is the distance between the center of mass of the block and the goal. We provide the agent biased distribution over the mass, moment of inertia inertia, friction coefficients and size of the object. An episode is considered successful if the agent gets the block within 5cm of the goal in 4 seconds. The positions of both block and goal is randomized after every episode.
the agent velocity controls 7DOF Franka Panda arm open a drawer on cabinet. A simple cost function based on Euclidean distance and relative orientation of the end effector with respect to the handle and the displacement of the slide joint on the drawer is used. The agent is provided a biased distribution over damping and frictionloss of robot and drawer joints. Every episode 4 seconds long after which the agent's starting configuration randomized. Success corresponds to opening the drawer within lcm of target displacement.
4. FRANKADRAWFROOS
The parameters we selected to randomize are reasonable in real world scenarios since estimating quantities like moment inertia and friction coefficients especially error prone. All our experiments are performed on desktop with 12 Intel Core i7-3930K @ 3.20GHz CPUs and 32 GB RAM with only few hours of CPU training. Q-functions are parameterized with reed-forwrrd neural networks that take as input an observation vector and action. Refer to A. for detailed explanation of tasks.
Environment
True parameters
Biased distribution
1kg 1 = 1m
m [0.9, 1.5)
PENDULUMSWINGUP
1 - [0.9, 1.5)
m 0.058kg
m 00.0087, 0.87]
BALLIICCUPSPARSE
Tzyz = 1.47 x 10
Tays = [0.00375, 1.5]
[0.22, 22] x 10
T = 0.05
2kg
bias 0.35
1ay: 8.33c
σ = 0.45
FETCHIPSSIBLLOKK
J [1, 0.005, 10-41
1 1= 0.025m
frictionloss = 0.1
bias 0.1
FRANKADRAWFRROFEN
damping=0.1
10.0
Table I: Details of environment parameters and dynamics randomization. The last column denotes the range for the uniform distribution. Tays implies that moment of inertia the same along all three axes. T the tendon stiffness. For FErCHPUsHBLOCK, the block is assumed to be a cube with sides of length FETCHPUSHBLOCK and FRANKADRAWEKOPEN use uniform distribution for every parameter defined by: mean bias x true value and range- [-o true value, true value]
5.2 BASELINES
We compare MPQ with a fixed horizon H against three baselines: MPPI using same horizon as MPQ and terminal value function, MPPI using a longer horizon and SOFTQLEARNING. For
8

---

Under review as a conference paper at ICLR 2020
(a) PENDCLUMSWINGUP
(b) BALLINCUPSPARSE
(c) PETCHPUSHBLOCK
FRANKADRAWWRÖPEN
Figure I: Comparison MPQ against baselines during training. Baselines are MPPI and SOPTQLEARNIG. The number following
the legend corresponds horizon for MPC optimization. SOFTQ is equivalent to MPQ with H = (a) In the PENDULUMSWINGUP tasks MPQ with H 8 able to outperform MPPI with H = 32 due to the Q function adding global information and correcting for model task, SOFTQLEAR
to the sparse nature BALLINCUPSPASE to succeed consistenily. whereas the learner is able to outperfonn them after only a few episodes of training. (c) In the FEtChPUSHBLOCE (d) Similarly in PRANKADRAWEROON MPQ with H 10, considerably outperforms MPPI with H-10 and H-64 within a few episodes of training demoustrating scalability to high-dimensional problems, MPQ beats SOFTQLEARNING baseline all tasks.
bias.
MPPI with long horizon H 18 unable
MPQ
perform MPPT using
merely minutes of real system interaction.
SOFTQLEARNING we additionally use target network stabilize learning whereas MPQ does not use target networks.
5.3 ANALYSIS OF OVERALL PERFORMANCE
5.3.1 COMPARISON OF MPQ WITH MPPI AND soft Q LEARNING
We test the hypotheses that (1) using a soft Q function as the terminal cost can improve MPPI per- formance even with shorter horizon (2) learning from real-data can mitigate effects of model error by adapting to true system dynamics and (3) finite horizon optimization leads to faster convergence compared to soft Q Learning especially in sparse reward tasks. Fig. 1 shows the training curves for MPQ versus soft Q learning. Online optimization is able to improve upon the inaccuracies of the Q function and lead faster convergence. In sparse reward task such BALL INLPSPCUPSSRRRS and high-dimensional FETCHPUSHBLOCCK and FRANKADRAWEROPEN, SOFRQLEARNING is unable learn a consistent policy whereas MPQ improves very rapidly. Additionally, an MPQ agent with a short horizon consistently outperforms MPPI with much longer horizon in all tasks. This can be attributed to (1) larger model bias in longer horizon optimization (2) hardness of optimizing longer sequences and (3) global information encapsulated in the Q function. Using a shorter horizon has added computational benefits as well. Since the Q function learned using data generated from the true system parameters, it is not affected by model bias. In FETCHPUSHBII performs MPPI with H=64 within the first 30 episodes of training which corresponds to roughly minutes of experience on simulation with true parameters. In contrast, SOFTQLEARNING barely ever moves the block and MPPI with H=10 succeeds only when arm is close to initial position of block. Similarly, in TRANKADRAWEROPE than 5 times as compared to MPPI with H=10 and outperforms MPPI with H=64 as well. The consistent results across all different tasks demonstrates the robustness and scalability of MPQ.
the agent out-
the agent with H 10 achieves success rate of more
5.3.2 BENEFIT MPQ OVER DOMAIN RANDOMIZATION
Agent MPQH4REAL
Avg. success rate
Domain randomization(DR) techniques aim to make the pol- icy learned in simulation robust to modelling errors by ran- domizing the simulation parameters using manually chosen distributions. However, such policies can be far from optimal on the true system parameters as the learned Q function is in- herently biased. However, A Q function learned using rollouts from the real system can overcome model bias. We validate this hypothesis on the BALLIICCUSSPARSE task by taking a DR approach inspired by Peng et al. (2018). Simulated roll- outs are generated by sampling different parameters at every timestep from broad range of dynamics parameters in Ta- ble 1, whereas real system rollouts use the true parameters. The average success rate reported in Table 2 demonstrates that a Q function learned solely using DR is unable to generalize to the true system parameters and MPQ has more than twice the success rate when learned real system.
0.85
MPQH4DR
0.41
MPQUIREAL
0.09
MPQH4DR
0.06
Table 2:
Performance comparison between training using real system rollouts (uames cading REAL)
(names ending BALLINCUPSPARSS task in terms of average num- ber of successful attempts. Training opisodes 350, test episodes 100. H is barizon of MPC optimization in both iraining and testing, where is soft learning.
9

---

Under review as a conference paper at ICLR 2020
6 DISCUSSION
In this work we have presented theoretical connection between information theoretic MPC and soft Q learning approaches that naturally provides an algorithm to combine stochastic optimal control and model-free reinforcement learning. The theoretical insight not only ties together the different fields, but opens avenues to designing pragmatic RL algorithms that leverage the benefits both. However, some important questions extricably tied with the model error and optimization artifacts. Investigating this dependence a principled manner is important for real- wolld applications. Another interesting avenue of research characterizing the performance of a parameterized Q function and using it to adapt the horizon of MPC rollouts for smarter exploration.
yet to be answered. The optimal horizon for MPC is in-
REFERENCES
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and free search. In Advances in Neural Information Processing Systems, pp. 5360-5370, 2017.
Vishnu Desaraju and Nathan Michael. Fast nonlinear model predictive control via partial enumer- ation. In 2016 IEEE International Conference Robotics and Automation (ICRA), pp. 1243- 1248. IEFE, 2016.
Tom Erez, Kendall Lowrey, Yuval Tassa, Vikash Kumar, Svetoslav Kolev, and Emanuel Todorov. An integrated system for real-time model predictive control humanoid robots. 2013 13th IEEE- RAS International Conference on Humanoid Robots (Humanoids), pp. 292-299. IEEE, 2013.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft 2015.
updates. arXiv preprint
Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4019-4026. IEEE, 2016.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep encrgy-based policies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1352-1380 1361. JMLR. org, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01299. 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:1412.6988. 2014.
Vikash Kumar, Yuval Tassa, Tom Erez, and Emanuel Todorov. Real-time behaviour synthesis for dynamic hand-manipulation. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 6808-66 6815. IEEE, 2014.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviy Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802. 10592, 2018
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
02000,...9...... 2015.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, leam offline: Efficient learning and exploration via model-base control. arXiv preprint 1rX1v:1811.01888, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 510 518(7540):529, 2015.
10

---

Under review as a conference paper at ICLR 2020
Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
Matthias Plappert, Marein Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments request for research, 2018.
Ugo Rosolia and Francesco Borrelli. Learning model predictive control for iterative tasks. data- driven control framework. IEEE Transactions on Automatic Control, 63(7):1883-1866, 2017.
Stephane Ross and J Andrew Bagnell, Agnostic system identification for model-based reinforcement learning. arXiv preprint arXi:: 2012.
Fereshteh Sadeghi and Sergey Levine, Cad2rl: Real single-image flight without a single real image. arXiv preprint (eX.v::611.0040000 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1889-1897, 2015.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning. arXiv preprint 1oX.v:/70.004444 2017.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics, 2017. URL https: /
05065.
Pranav Shyam, Wojeiech Jaskowski, and Faustino Gomez. Model-based active exploration. In International Conference Machine Learning, pp. 5779-5788, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwisser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, I Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint 17Xiv:1712.01855. 2017.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining rein- forcement learning & imitation learning. arXiv preprint 10X.v:180..11240, 2018.
Aviv Tamar, Garrett Thomas, Tianhao Zhang, Sergey Levine, and Pieter Abbeel. Learning from the hindsight plan episodic mpe improvement. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 336-343 IEEE, 2017.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap. and Martin Ried- miller. DeepMind control suite. Technical report, DeepMind, January 2018. URL https: //aaxiv.orr// haag///ab//aa
Evangelos A Theodorou and Emanuel Todorov. Relative entropy and free energy dualities: Con- nections to path integral and kl control. In 2012 IEEE 51st IEEE Conference Decision and Control (CDC), pp. 1466- IEEE, 2012.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov. Efficient computation of optimal actions. Proceedings of the national academy of sciences, 106(28):1 1478-1 1483, 2009.

---

Under review as a conference paper at ICLR 2020
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Thirtieth AAAI conference on artificial intelligence, 2016.
Nolan Wagener, Ching-An Cheng, Jacob Sacks, and Byron Boots. An online learning approach to model predictive control. arXiv preprint arXiv: 190..09900.08900 2019.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1714-1721. IEEE, 2017.
Ronald J Williams. Simple statistical gradient-folooinng algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Mingyuan Zhong, Mikala Johnson, Yuval Tassa, Tom Erez, and Emanuel Todorov. Value function approximation and model predictive control. 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 100-107. IEEE, 2013.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008
A APPENDIX
A.1 FURTHER EXPERIMENTAL DETAILS
The learned Q function takes as input the current action and a observation vector per task:
1. PENDULUMSWINGUP. (os(()), sin(O), 6) (3 dim)
2. BALLINCUPSPARSE: [Tbolls Ttarget: Prall, 2; argget, Ptarget Øball, cos(O), sin (9)] (12 dim) where θ is angle of line joining ball and target.
3. FETCHIUSSIBL.CCK: ['grippers Tohjy Tohjj
Igrip, gripper opening, rotatj Mohyj Wabi
gripper opening vel, gripper, Agrippee, obj), Toool Liajj d'gool,, obj), Tgoot (33 dim)
4. FRANKADR&WERRPEN: [Tee Th. Th Jeer hee. &h. quat quat, drawerdisp d(ee, h), d (ee, h), d (39 dim)
For all our experiments parameterize Q functions with feedforward neural networks with two lay- ers containing 100 units each and tanh activation. We use Adam (Kingma & Ba, 2014) optimization with learning rate of 0.001. For generating value function targets in Eq. (22), we use 3 iterations of MPPI optimization except FRANKADRAWEROPER. where we use 1. The MPPI parameters used listed in Table 3.
Environment
Cost function
Samples
X
Qr
PENDULLUMSWINGUT
@ + 0
4.0
0.15
0.5
0.9
BALL.INCUPSPARSE
if ball in cup else
36
4.0
0.1.
0.55
0.9
FETCHPUSHBLOOCK
dslock gool
36
3.0
0.01
0.5
0.9
FRANKADRAWEROPEN
dec 0.u8a 1.0 derawer/ demoe
36
4.0
0.05
0.55
0.9
Table Cost function and MPPI parameters
12


Reviews:
[{"review_id": "Hye5ppju5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nIn this paper, the authors proposed the algorithm to introduce model-free reinforcement learning (RL) to model predictive control~(MPC), which is a representative algorithm in model-based RL, to overcome the finite-horizon issue in the existing MPC. The authors evaluated the algorithm on three environments and demonstrated the outperformance comparing to the model predictive path integral (MPPI) and soft Q-learning.\n\n1, The major issue of this paper is its novelty. The proposed algorithm is a straightforward extension of MPPI [1], which adds a Q-function to the finite accumulated reward to predict the future rewards to infinite horizon. The eventual algorithm ends up like a straightforward combination of MPPI and DQN. The algorithm derivation in Sec. 4 follows almost exact as [1] without appropriate reference. \n\n2, The empirical comparison is quite weak. The algorithm is only tested on three environments and only one (Pendulum) from the MuJoCo benchmark. Without more comprehensive comparison  on other MuJoCo environments, the empirical experiment is not convincing. \n\n\nMinors:\n\n1, The derivation from 15-16 is wrong. The first term should be negative.\n\nThere are many claims without justification. For example:\n\"Solving the above optimization can be prohibitively expensive and hard to accomplish online.\"\n\n- This is not true. There are already plenty of work solving the entropy-regularized MDP online [2, 3, 4] and achieving good empirical performance.\n\n\"The re-optimization and entropy regularization helps in mitigating model-bias and\ninaccuracies with optimization by avoiding overcommitment to the current estimate of the cost.\"\n- There is no evidence that the entropy-regularization will reduce the mode-bias. Actually, as discussed in [4, 5] the entropy regularization will also incur some extra bias. \n\n[1] Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1714–1721. IEEE, 2017.\n[2] Haarnoja, Tuomas, Aurick Zhou, Pieter Abbeel, and Sergey Levine. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" arXiv preprint arXiv:1801.01290 (2018).\n[3] Nachum, O., Norouzi, M., Xu, K. and Schuurmans, D., 2017. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems (pp. 2775-2785).\n[4] Dai, Bo, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. \"SBEED: Convergent reinforcement learning with nonlinear function approximation.\" arXiv preprint arXiv:1712.10285 (2017).\n[5] Geist, Matthieu, Bruno Scherrer, and Olivier Pietquin. \"A Theory of Regularized Markov Decision Processes.\" arXiv preprint arXiv:1901.11275 (2019).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgk1myAFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds a connection between information theoretical MPC and entropy-regularized RL and also develops a novel Q learning algorithm (Model Predictive Q-Learning).  Experiments show that the proposed MBQ algorithm outperforms MPPI and soft Q learning in practice. \n\nThe paper is well-written. The derivation is clean and easy to understand. Adding a terminal Q function, which makes horizon infinite, is a very natural idea. Most of the derivation is unrelated to the additional Q^pi^* (Eq 11-17) and Eq (18-20) are simply plugging in the formula of pi^*, which is derived at Eq (9), which is also quite natural if one knows the result for finite horizon horizon MPPI (i.e., without the terminal Q). For experiments, I'd like to see some results on more complex environments (e.g., continuous control tasks in OpenAI Gym) and more comparison with recent model-based RL work. \n\nQuestions:\n\n1. The experiment setup is that a uniform distribution of dynamics parameters (with biased mean and added noise) are used. Why not using a neural network? \n2. Eq (22): The estimation for eta is unbiased, However, 1/eta might be biased, so is the estimator for w(E) also biased? \n\nMinor Comments:\n1. Eq (16): should the first term be negated? \n2. Page 6, last paragraph: performved -> performed.\n3. Page 7, second paragraph: produces -> produce.\n4. Algorithm 1, L6: N % N_update -> i % N_update. \n5. Appendix A.1, second paragraph: geneerating -> generating. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gjSMattS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how a known, imperfect dynamics can be used to accelerate policy search. The main contribution of this paper is a model-based control algorithm that uses n-step lookahead planning and estimates the value of the last state with the prediction from a learned, soft Q value. The n-step planning uses the imperfect dynamics model, which can be cheaply queried offline. A second contribution of the paper is an efficient, iterative procedure for optimizing the n-step actions w.r.t. the imperfect model. The proposed algorithm is evaluated on three continuous control tasks. The proposed algorithm outperforms the two baselines on one of the three tasks. Perhaps the most impressive aspect of the paper is that the experiments only require a few minutes of \"real-world\" interaction to solve the tasks.\n\nI am leaning towards rejecting this paper. My main reservation is that the empirical results are not very compelling. In Figure 1, it seems like the proposed method (MPQ) only beats that MPPI baseline in BallInCupSparse. The paper seems remiss to not include comparisons to any recent MBRL algorithms (e.g., [Chua 2018, Kurutach 2018, Janner 2019]). Moreover, the tasks considered are all quite simple. Finally, for a paper claiming to \"pave the way for deploying reinforcement learning algorithms on real-robots,\" it seems important to show results on real robots, or at least on a very close approximation thereof. Instead, since the experiments are done in simulation and the true model is known exactly, the paper constructs an approximate model that is a noisy version of the true model. \n\nI do think that this paper is tackling an important problem, and am excited to see work in this area. I would consider increasing my review if the paper were revised to include a comparison to a state-of-the-art MBRL method, if it included experiments on more complex task, and if the proposed method were shown to consistently outperform most baselines on most tasks.\n\nMinor comments:\n* The derivation of the update rule in Section 4.2 was unclear to me. In Equation 17, why can the RHS not be computed directly? In Equation 21, where did this iterative procedure come from?\n* \"Reinforcement Learning\" -- No need to capitalize\n* \"sim-to-real approaches … suffer from optimization challenges such as local minima and hand designed distributions.\" -- Can you provide a citation for the \"local minima\" claim? The part about \"hand designed distributions\" seems to have a grammar error.\n* \"Model Predictive Control\" -- No need to capitalize.\n* \"Computer Go\" -- How is this different from the board game Go?\n* \"Entropy-regularized\" -- Missing a period before this.\n* Eq 1 -- The \"\\forall s_0\" suggests that this equation depends on s_0. Can you clarify this dependency?\n* \"stochastic gradient descent\" -- I believe that [Williams 1992] is a relevant citation here.\n\n----------------------UPDATE AFTER AUTHOR RESPONSE------------------------\nThanks for the detailed response!  The new experiments seem strong, and help convince me that the method isn't restricted to simple tasks. I'm inclined to agree that it's not overly onerous to assume access to an imperfect model of the world. Because of this, I will increase my vote to \"weak accept.\"\n\nIn the next version, I hope that the authors (1) include multiple random seeds for all experiments, and (2) student the degree to which model misspecification degrades performance (i.e., if you use the wrong model, how much worse do you do?).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors develop a novel connection between information theoretic MPC and entropy regularized RL. Using this connection, they develop Q learning algorithm that can work with biased models. They evaluate their proposed algorithm on several control tasks and demonstrate performance over the baseline methods.

Unfortunately, reviewers were not convinced that the technical contribution of this work was sufficient. They felt that this was a fairly straightforward extension of MPPI. Furthermore, I would have expected a comparison to POLO. As the authors note, their approach is more theoretically principled, so it would be nice to see them outperforming POLO as a validation of their framework.

Given the large number of high-quality submissions this year, I recommend rejection at this time.
Decision: Reject