Year: 2019

Abstract:
Model-free reinforcement learning (RL) algorithms work well in sequential decision-making problems when experience can be collected cheaply and modelbased RL is effective when system dynamics can be modeled accurately. However, both of these assumptions can be violated in real world problems such as robotics, where querying the system can be prohibitively expensive and real-world dynamics can be difficult to model accurately. Although sim-to-real approaches such as domain randomization attempt to mitigate the effects of biased simulation, they can still suffer from optimization challenges such as local minima and handdesigned distributions for randomization, making it difficult to learn an accurate global value function or policy that directly transfers to the real world. In contrast to RL, model predictive control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effectively contend with real-world dynamics. MPC performance is usually limited by factors such as model bias and the limited horizon of optimization. In this work, we present a novel theoretical connection between information theoretic MPC and entropy regularized RL and develop a Q-learning algorithm that can leverage biased models. We validate the proposed algorithm on sim-to-sim control tasks to demonstrate the improvements over optimal control and reinforcement learning from scratch. Our approach paves the way for deploying reinforcement learning algorithms on real-robots in a systematic manner.

heading: 1 INTRODUCTION
text: Deep reinforcement learning algorithms have recently generated great interest due to their successful application to a range of difficult problems including Computer Go (Silver et al., 2016) and highdimensional control tasks such as humanoid locomotion (Lillicrap et al., 2015; Schulman et al., 2015). While these methods are extremely general and can learn policies and value functions for complex tasks directly from data, they can also be sample inefficient, and partially-optimized solutions can be arbitrarily poor. These challenges severely restrict RL’s applicability to real systems such as robots due to data collection challenges and safety concerns.
One straightforward way to mitigate these issues is to learn a policy or value function entirely in a high-fidelity simulator (Shah et al., 2017; Todorov et al., 2012) and then deploy the optimized policy on the real system. However, this approach can fail due to model bias, external disturbances, the subtle differences between the real robot’s hardware and poorly modeled phenomena such as friction and contact dynamics. Sim-to-real transfer approaches based on domain randomization (Sadeghi & Levine, 2016; Tobin et al., 2017) and model ensembles (Kurutach et al., 2018; Shyam et al., 2019) aim to make the policy robust by training it to be invariant to varying dynamics. However, learning a globally consistent value function or policy is hard due to optimization issues such as local optima and covariate shift between the exploration policy used for learning the model and the actual control policy executed on the task (Ross & Bagnell, 2012).
Model predictive control (MPC) is a widely used method for generating feedback controllers that repeatedly re-optimizes a finite horizon sequence of controls using an approximate dynamics model that predicts the effect of these controls on the system. The first control in the optimized sequence is executed on the real system and the optimization is performed again from the resulting next state. However, the performance of MPC can suffer due to approximate or simplified models and lim-
ited lookahead. Therefore the parameters of MPC, including the model and horizon H should be carefully tuned to obtain good performance. While using a longer horizon is generally preferred, real-time requirements may limit the amount of lookahead and a biased model can result in compounding model errors.
In this work, we present an approach to RL that leverages the complementary properties of modelfree reinforcement learning and model-based optimal control. Our proposed method views MPC as a way to simultaneously approximate and optimize a local Q function via simulation, and Q learning as a way to improve MPC using real-world data. We focus on the paradigm of entropy regularized reinforcement learning where the aim is to learn a stochastic policy that minimizes the cost-to-go as well as KL divergence with respect to a prior policy. This approach enables faster convergence by mitigating the over-commitment issue in the early stages of Q-learning and better exploration (Fox et al., 2015). We discuss how this formulation of reinforcement learning has deep connections to information theoretic stochastic optimal control where the objective is to find control inputs that minimize the cost while staying close to the passive dynamics of the system (Theodorou & Todorov, 2012). This helps in both injecting domain knowledge into the controller as well as mitigating issues caused by over optimizing the biased estimate of the current cost due to model error and the limited horizon of optimization. We explore this connection in depth and derive an infinite horizon information theoretic model predictive control algorithm based on Williams et al. (2017). We test our approach called Model Predictive Q Learning (MPQ) on simulated continuous control tasks and compare it against information theoretic MPC and soft Q-Learning (Haarnoja et al., 2017), where we demonstrate faster learning with fewer system interactions and better performance as compared to MPC and soft Q-Learning even in the presence of sparse rewards. The learned Q function allows us to truncate the MPC planning horizon which provides additional computational benefits. Finally, we also compare MPQ versus domain randomization(DR) on sim-to-sim tasks. We conclude that DR approaches can be sensitive to the hand-designed distributions for randomizing parameters which causes the learned Q function to be biased and suboptimal on the true system’s parameters, whereas learning from data generated on true system is able to overcome biases and adapt to the real dynamics.

heading: 2 RELATED WORK
text: Model predictive control has a rich history in robotics, ranging from control of mobile robots such as quadrotors (Desaraju & Michael, 2016) and aggressive autonomous vehicles (Wagener et al., 2019; Williams et al., 2017) to generating complex behaviors for high-dimensional systems such as contact-rich manipulation (Fu et al., 2016; Kumar et al., 2014) and humanoid locomotion (Erez et al., 2013). The success of MPC can largely be attributed to online policy optimization which helps mitigate model bias. The information theoretic view of MPC aims to find a policy at every timestep that minimizes the cost over a finite horizon as well as the KL-divergence with respect to a prior policy usually specified by the system’s passive dynamics (Theodorou & Todorov, 2012; Williams et al., 2017). This helps maintain exploratory behavior and avoid over-commitment to the current estimate of the cost function, which is biased due to modeling errors and a finite horizon. Sampling-based MPC algorithms (Wagener et al., 2019; Williams et al., 2017) are also highly parallelizable enabling GPU implementations that aid with real-time control. However, efficient MPC implementations still require careful system identification and extensive amounts of manual tuning.
Deep RL methods are extremely general and can optimize neural network policies from raw sensory inputs with little knowledge of the system dynamics. Both value-based and policy-based approaches (Schulman et al., 2015) have demonstrated excellent performance on complex control problems. These approaches, however, fall short on several accounts when applying them to a real robotic system. First, they have high sample complexity, potentially requiring millions of interactions with the environment. This can be very expensive on a real robot, not least because the initial performance of the policy can be arbitrarily bad. Using random exploration methods such a -greedy can further aggravate this problem. Second, a value function or policy learned entirely in simulation inherits the biases of the simulator. Even if a perfect simulation is available, learning a globally consistent value function or policy is an extremely hard task as noted in (Silver et al., 2016; Zhong et al., 2013). This can be attributed to local optima when using neural network representations or the inherent biases in the Q learning update rules (Fox et al., 2015; Van Hasselt et al., 2016). In fact, it can be difficult to explain why Q-learning algorithms work or fail (Schulman et al., 2017).
Domain randomization aims to make policies learned in simulation more robust by randomizing simulation parameters during training with the aim of making the policies invariant to potential parameter error (Peng et al., 2018; Sadeghi & Levine, 2016; Tobin et al., 2017). However, these policies are not adaptive to unmodelled effects, i.e they take into account only aleoteric and not epistemic uncertainty. Also, such approaches are highly sensitive to hand-designed distributions used for randomizing simulation parameters and can be highly suboptimal on the real-systems parameters, for example, if a very large range of simulation parameters is used. Model-based approaches aim to use real data to improve the model of the system and then perform reinforcement learning or optimal control using the new model or ensemble of models (Kurutach et al., 2018; Ross & Bagnell, 2012; Shyam et al., 2019). Although learning accurate models is a promising avenue, we argue that learning a globally consistent model is an extremely hard problem and instead we should learn a policy that can rapidly adapt to experienced real-world dynamics.
The use of entropy regularization has been explored in RL and Inverse RL for its better sample efficiency and exploration properties (Fox et al., 2015; Haarnoja et al., 2017; 2018; Schulman et al., 2017; Ziebart et al., 2008). This framework allows incorporating prior knowledge into the problem and learning multi-modal policies that can generalize across different tasks. Fox et al. (2015) analyze the theoretical properties of the update rule derived using mutual information minimization and show that this framework can overcome the over-estimation issue inherent in the vanilla Q-learning update. In the past, Todorov (2009) have shown that using KL-divergence can convert the optimal control problem into one that is linearly solvable.
Infinite horizon MPC aims to learn a terminal cost function that can add global information to the finite horizon optimization. Rosolia & Borrelli (2017) learn a terminal cost as a control Lyapunov function and a safety set for the terminal state. These quantites are calculated using all previously visited states and they assume the presence of a controller that can deterministically drive the any state to the goal. Tamar et al. (2017) learns a cost shaping to make a short horizon MPC mimic the actions produced by long horizon MPC offline. However, since their approach is to mimic a longer horizon MPC, the performance of the learner is fundamentally limited by the the performance of the longer horizon MPC. On the contrary, learning an optimal value function as the terminal cost can potentially lead to close to optimal performance.
Using local optimization is an effective way of improving an imperfect value function as noted in RL literature by Anthony et al. (2017); Lowrey et al. (2018); Silver et al. (2016; 2017); Sun et al. (2018). However, these approaches assume that a perfect model of the system is available. In order to make the policy work on the real system, we argue that it is essential to learn a value function form real data and utilize local optimization to stabilize learning.

heading: 3 PRELIMINARIES

heading: 3.1 REINFORCEMENT LEARNING WITH ENTROPY REGULARIZATION
text: A Markov Decision Process (MDP) is defined by tuple (S,A, c,P, γ) where S is the state space, A is the action space, c is a one step cost function, P is the space of transition functions and γ is a discount factor. Let P ∈ P be a particular transition function. A closed-loop policy π(a|s) is a distribution over actions given state. Given a policy π and a prior policy π, the KL divergence between them at a state is given by KL (π(a|s)||π(a|s)) = Eπ [log (π(a|s)/π(a|s))]. Entropyregularized RL (Fox et al., 2015) aims to optimize the following objective
π∗ = arg min π Eπ,P [ ∞∑ t=1 γt−1 (c(st, at) + λKL (πt||πt)) ] ∀ s0 ∈ S (1)
where πt and πt are shorthand for π(at|st) and π(at|st) respectively, λ is the temperature parameter that penalizes deviation of π from π. For a policy π, we can define the soft value and action-value functions
V π(s) = Eπ,P [ ∞∑ t=1 γt−1 (c(st, at) + λKL (πt||πt)) |s0 = s ] Qπ(s, a) = c(s, a) + γEs′∼P (s′|s,a) [V π(s′)] (2)
Given a horizon of H timesteps, we can use above definitions to write the value functions as
V π(s) = Eπ,P [ H−1∑ t=1 γt−1(c(st, at) + λKL (πt||πt)) + γH−1V π(sH)|s1 = s ]
Qπ(s, a) = c(s, a) + Eπ,P [ H−1∑ t=2 γt−1(c(st, at) + λKL (πt||πt))
+ γH−1(λKL (πH ||πH) +Q(sH , aH))|s1 = s, a1 = a ] (3)
It is straightforward to verify that V π(s) = Ea∼π [log(π(a|s)/π(a|s)) +Q(s, a)]. The objective in Eq. (1) can equivalently be written as
π∗ = arg min π V π(s) ∀ s ∈ S (4)
This optimization can be performed either by policy gradient methods that aim to find the optimal policy π ∈ Π via stochastic gradient descent (Schulman et al., 2017; Williams, 1992) or value based methods that try to iteratively approximate the value function of the optimal policy. In either case, the output of solving the above optimization is a global closed-loop control policy π∗(a|s).

heading: 3.2 INFORMATION THEORETIC MPC
text: Solving the above optimization can be prohibitively expensive and hard to accomplish online. In contrast to RL, MPC performs online optimization of a simple policy class with a truncated horizon. This process effectively creates a closed-loop controller. In order to do so, MPC algorithms such as MPPI (Williams et al., 2017) use an approximate dynamics model P̂ , which can be deterministic. This is the case when using a simulator such as MuJoCo (Todorov et al., 2012) as the dynamics model. At timestep t, starting from the current state st, an open loop sequence of actions A = (at, at+1, . . . at+H) is sampled from the control distribution denoted by π(A). The objective is to find an optimal sequence of actions to solve

heading: A∗ = arg min
text: A Eπ(A) [ t+H−1∑ l=t γl−tc(sl, al) + λKL (πl||πl) + γH−1(cf (st+H , at+H) + λKL (πt+H ||πt+H)) ] (5) where cf (st+H , at+H) is a terminal cost function and π(A) is the passive dynamics of the system, i.e the distribution over actions produced when the control input is zero. The first action in the sequence is then executed on the system and the optimization is performed again from the resulting next state. The re-optimization and entropy regularization helps in mitigating model-bias and inaccuracies with optimization by avoiding overcommitment to the current estimate of the cost. A shortcoming of the MPC procedure is the finite horizon. This is especially pronounced in tasks with sparse rewards where a short horizon can make the agent myopic to future rewards. To mitigate this, an approach known as infinite horizon MPC sets the terminal cost cf as a value function that adds global information to the problem.
In the next section, we build our approach by focussing on the MPPI algorithm and its relationship with entropy regularized reinforcement learning. Specifically, we use the definitions of the soft value functions from Eq. (2) to derive an optimal Boltzmann distribution forH-step actions that optimally solves the infinite horizon control problem. This helps us derive the MPPI update rule from Williams et al. (2017) for the infinite horizon case, which then leads to our Model Predictive Q Learning (MPQ) algorithm, which utilizes a predictive model for Q updates and stochastic optimal control as the policy. In the case where H = 1, the algorithm is equivalent to soft Q learning (Haarnoja et al., 2018) or G-learning (Fox et al., 2015).

heading: 4 APPROACH

heading: 4.1 OPTIMAL H-STEP BOLTZMANN DISTRIBUTION
text: Let π(A) and π(A) be the joint control distribution and prior over H-horizon open-loop actions. The distributions are assumed to be independent over timesteps, i.e π(a1 . . . aH) = ∏H t=1 πt, where
πt is shorthand for π(at). Since P̂ is deterministic, the following equations hold
V π(s) = Eπ [λ log(π/π) +Qπ(s, a)] Qπ(s, a) = c(s, a) + γV π(s′) (6)
For clarity, we consider γ = 1. Substituting from Eq. (3) for Qπ(s, a)
V π(s) = Eπ1...πH [ H−1∑ t=1 c(st, at) + λ H∑ t=1 log(πt/πt) +Q π(sH , aH) ]
= Eπ1...πH [ H−1∑ t=1 c(st, at) + λ log H∏ t=1 (πt/πt) +Q π(sH , aH) ]
= Eπ [ H−1∑ t=1 c(st, at) + λ log(π/π) +Q π(sH , aH) ] (7)
Consider the following distribution over H-horizon
π = 1
η exp ( −1 λ ( H−1∑ t=1 c(st, at) +Q π(sH , aH) )) π(a1 . . . aH) (8)
where η is a normalization constant given by
η = Eπ(a1...aH)
[ exp ( −1 λ ( H−1∑ t=1 c(st, at) +Q π(sH , aH) ))] (9)
We show that this is the optimal control distribution as∇V π(s) = 0. Substituting Eq. (8) in (7)
V π(s) = Eπ [ H−1∑ t=1 c(st, at)− λ log η − H−1∑ t=1 c(st, at)−Qπ(sH , aH) +Qπ(sH , aH) ] V π(s) = Eπ [−λ log η]
Since η is a constant, we have V π(s) = −λ log η. Hence for π in Eq. (8), the soft value function is a constant with gradient zero given by
V π ∗ (s) = −λ logEπ(a1...aH)
[ exp ( −1 λ ( H−1∑ t=1 c(st, at) +Q π(sH , aH) ))] (10)
which is often referred to in optimal control literature as the “free energy” of the system (Theodorou & Todorov, 2012; Williams et al., 2017). For H=1, Eq. (10) takes the form of the soft value function from Fox et al. (2015) and Haarnoja et al. (2018).

heading: 4.2 INFINITE HORIZON MPPI UPDATE RULE
text: Similar to Williams et al. (2017), we derive the MPPI update rule for online policy optimization. Since sampling actions from the optimal control distribution in Eq. (8) is intractable, we consider control policies π(A) ∈ Π which are easy to sample from. We then optimize for a vector of H control inputs U , such that the resulting action distribution minimizes the KL divergence with the optimal policy
U∗ = arg min π(A) KL (π∗(A)||π(A)) (11)
The objective can be expanded out as KL (π∗(A)||π(A)) = ∫ A π∗(A) log π∗(A) π(A) dA = ∫ A π∗(A) log π∗(A) π(A) π(A) π(A) dA
= ∫ A π∗(A) log π∗(A) π(A) dA− ∫ A π∗(A) log π(A) π(A) dA (12)
Since the first term does not depend on the control input, we can remove it from the optimization
U∗ = arg max π(A) ∫ A π∗(A) log π(A) π(A) dA (13)
Consider Π to be independent multivariate Gaussians over sequence of the H controls with constant covariance Σ at each timestep. We can write the control distribution and prior as follows
π(A) = 1
Z H∏ t=1 exp ( −1 2 (ut − at)T Σ−1 (ut − at) ) π(A) = 1 Z H∏ t=1 exp ( −1 2 aTt Σ −1at ) (14)
where ut and at are the control inputs and actions at timestep t and Z is the normalization constant. Here the prior corresponds to the passive dynamics of the system (Theodorou & Todorov, 2012; Williams et al., 2017), although other choices of prior are possible. Substituting in Eq. (13) we get
U∗ = arg max π(A)
∫ π∗(A) ( H∑ t=1 −1 2 uTt Σ −1ut + u T t Σ −1at ) dA (15)
The objective can be simplified to the following by integrating out the probability in the first term H∑ t=1 −1 2 uTt Σ −1ut + u T t ∫ π∗(A)Σ−1at dA (16)
Since this is a concave function with respect to every ut, we can find the maximum by setting its gradient with respect to ut to zero to solve for optimal u∗t
u∗t = ∫ π∗(A)atdA = ∫ π(A) π∗(A)
π(A)
π(A) π(A) atdA = Eπ(A)
[ π∗(A)
π(A)
π(A) π(A) at
] = Eπ(A) [w(A)at]
(17) where the second equality comes from importance sampling to convert the optimal controls into an expectation over the control distribution instead of the optimal distribution which is impossible to sample from. The importance weight w(A) can be written as follows (substituting π∗ from Eq. (8))
w(A) = 1
η Eπ(A)
[ exp ( −1 λ ( H−1∑ t=1 c(st, at) +Q π∗(sH , aH) )) π(A) π(A) ] (18)
Making change of variables ut + t = at for noise sequence E = ( 1 . . . H) sampled from independant Gaussians with zero mean and covariance Σ we get
w(E) = 1 η Eπ(A)
[ exp ( −1 λ ( H−1∑ t=1 c(st, ut + t) + λ π(U + E) π(U + E) +Qπ ∗ (sH , uH + H) ))]
= 1
η Eπ(A)
[ exp ( −1 λ ( H−1∑ t=1 c(st, ut + t) + λ H∑ t=0 1 2 uTt Σ −1(ut + 2 t) +Q π∗(sH , uH + H) ))] (19)
Note that η is the optimal H-step free energy derived in Eq. (10) and can be estimated from N Monte-Carlo samples as
η = N∑ n=1 exp ( −1 λ ( H−1∑ t=1 c(st, ut + n t ) + λ H∑ t=0 1 2 uTt Σ −1(ut + 2 n t ) +Q π∗(sH , uH + n H) )) (20) We can form the following iterative update rule where at every iteration i the sampled control sequence is updated according to
ui+1t = u i t + α N∑ n=1 w(En) n (21)
where α is a step-size parameter as proposed by Wagener et al. (2019). This gives us the infinite horizon MPPI update rule. For H = 1, this corresponds soft Q-learning where stochastic optimization is performed to solve for the optimal action online. Now we develop soft Q-learning algorithm that utilizes infinite horizon MPPI to generate actions as well as Q-targets.

heading: 4.3 THE INFORMATION THEORETIC MODEL PREDICTIVE Q-LEARNING ALGORITHM
text: Since we do not have access to Qπ ∗ , we can not estimate the importance weight in Eq. (19) exactly. Hence, we consider Q functions parameterized by θ denoted byQθ(s, a). Similar to deep Q-learning algorithms, we maintain an replay buffer (Mnih et al., 2015), and update parameters by stochastic gradient descent on the loss L = 1K ∑K i=1(yi − Qθ(si, ai))2 for a batch of K experience tuples (s, a, c, s′) sampled from the buffer where targets yi are given by
y = c(s, a)−λ logEπ∗(a1...aH ) exp(−1 λ ( H−1∑ t=1 c(st, at) + λ log π∗t πt +Qθ(sH , aH) )) ∣∣∣∣∣s1 = s′  (22)
Since the value function updates are performed offline we can utilize large amount of computation (Tamar et al., 2017) to calculate π∗(a1 . . . aH). In our case it is obtained by performing the infinite horizon MPPI update in Eq. (21) for multiple iterations starting from state s′. This allows for directed exploration at a state which leads to better approximation of the free energy (which is akin to approaches such as Covariance Matrix Adaption, except MPPI does not adapt the covariance). This especially helps in early stages of learning by providing better quality targets than a random Q function. Intuitively, this update rule leverages the biased dynamics model P̂ for H steps and a soft Q function at the end learned from interactions with the real system.
At every timestep t during online rollouts, a H-horizon sequence of actions is optimized using a single iteration of infinite horizon MPPI update rule in Eq. (21) and the first action is executed on the system. Online optimization with predictive models can lookahead to produce better actions than acting greedily with respect to the biased Q function and makes ad-hoc exploration strategies such as -greedy unnecessary. Using predictive models for generating value targets and online policy optimization helps accelerate convergence as we demonstrate in our experiments in the next section. Algorithm 1 shows the complete MPQ algorithm.
A closely related approach in literature is (Lowrey et al., 2018) which also uses online MPC and offline value function learning, however they assume access to the true dynamics of the system and do not explore the connection between MPPI and entropy regularized RL and hence do not use free energy targets, even though they use MPPI in their implementation.
Algorithm 1: MPQ
Input : Approximate model P̂ , initial Q function parameters θ1, experience buffer D Parameter: Number of episodes N , length of episode T , planning horizon H , number of update
episodes Nupdate, minibatch-size K, number of minibatches M 1 for i = 1 . . . N do 2 for t = 1 . . . T do 3 (at, . . . , at+H)← Infinite horizon MPPI (Eq. (21)) 4 Execute at on the real system to obtain c(st, at) and next state st+1 5 D ← (st, at, c, st+1) 6 if i%Nupdate == 0 then 7 Sample M minibatches of size K from D 8 Generate targets using Eq. (22) and update parameters to θi+1 9 return θN or best θ on validation.

heading: 5 EXPERIMENTS
text: We perform experiments to test the efficacy of MPQ in overcoming the shortcomings of stochastic optimal control and model free RL in terms of convergence rate, computational requirements and model bias. We also compare MPQ against domain randomization for learning policies that perform well on systems for which accurate models are not known.

heading: 5.1 EXPERIMENTAL SETUP
text: We test our approach on sim-to-sim continuous control tasks based on the Mujoco simulator (Todorov et al., 2012) to study the properties of the algorithm in a controlled manner. The
agent is not provided with the true dynamics parameters, but a uniform distribution over them with a biased mean and added noise. This serves as a reasonable approximation of model bias due to inaccurate measurements of physical quantities. Details of the tasks considered are as follows
1. PENDULUMSWINGUP: the agent tries to swingup and stabilize a pendulum by applying torque on the hinge. The agent is provided with a distribution over the mass and length of the pendulum. The state of the system is given by (Θ, Θ̇), where Θ is the angular displacement. The cost function penalizes the deviation from the upright position and angular velocity. The initial state of the system is randomized after every episode which is 10 seconds long.
2. BALLINCUPSPARSE: a sparse version of the ball in cup task inspired from Tassa et al. (2018). Given a cup and spherical ball attached by a tendon, the goal is to swing and catch the ball. The agent can actuate motors on the two slide joints on the cup and is provided with a biased distribution over the mass of the ball, its moment of inertia and stiffness of the tendon. A cost of 1 is incurred at every timestep and 0 if the ball is in the cup. The position of the ball is randomized after every episode which is 4 seconds long. An episode is successful if agent catches the ball in the cup .
3. FETCHPUSHBLOCK: proposed by Plappert et al. (2018), the agent position controls the endeffector of a simulated Fetch robot to push a block to a goal location on the table. The cost is the distance between the center of mass of the block and the goal. We provide the agent a biased distribution over the mass, moment of inertia inertia, friction coefficients and size of the object. An episode is considered successful if the agent gets the block within 5cm of the goal in 4 seconds. The positions of both block and goal is randomized after every episode.
4. FRANKADRAWEROPEN: the agent velocity controls a 7DOF Franka Panda arm to open a drawer on a cabinet. A simple cost function based on Euclidean distance and relative orientation of the end effector with respect to the handle and the displacement of the slide joint on the drawer is used. The agent is provided a biased distribution over damping and frictionloss of robot and drawer joints. Every episode is 4 seconds long after which the agent’s starting configuration is randomized. Success corresponds to opening the drawer within 1cm of target displacement.
The parameters we selected to randomize are reasonable in real world scenarios since estimating quantities like moment inertia and friction coefficients is especially error prone. All our experiments are performed on a desktop with 12 Intel Core i7-3930K @ 3.20GHz CPUs and 32 GB RAM with only few hours of CPU training. Q-functions are parameterized with feed-forward neural networks that take as input an observation vector and action. Refer to A.1 for detailed explanation of tasks.

heading: 5.2 BASELINES
text: We compare MPQ with a fixed horizon H against three baselines: MPPI using same horizon as MPQ and no terminal value function, MPPI using a longer horizon and SOFTQLEARNING. For
SOFTQLEARNING we additionally use a target network to stabilize learning whereas MPQ does not use target networks.

heading: 5.3 ANALYSIS OF OVERALL PERFORMANCE
text: 5.3.1 COMPARISON OF MPQ WITH MPPI AND soft Q LEARNING
We test the hypotheses that (1) using a soft Q function as the terminal cost can improve MPPI performance even with a shorter horizon (2) learning from real-data can mitigate effects of model error by adapting to true system dynamics and (3) finite horizon optimization leads to faster convergence as compared to soft Q Learning especially in sparse reward tasks. Fig. 1 shows the training curves for MPQ versus soft Q learning. Online optimization is able to improve upon the inaccuracies of the Q function and lead to faster convergence. In sparse reward task such as BALLINCUPSPARSE and high-dimensional FETCHPUSHBLOCK and FRANKADRAWEROPEN, SOFTQLEARNING is unable to learn a consistent policy whereas MPQ improves very rapidly. Additionally, an MPQ agent with a short horizon consistently outperforms MPPI with much longer horizon in all tasks. This can be attributed to (1) larger model bias in longer horizon optimization (2) hardness of optimizing longer sequences and (3) global information encapsulated in the Q function. Using a shorter horizon has added computational benefits as well. Since the Q function is learned using data generated from the true system parameters, it is not affected by model bias. In FETCHPUSHBLOCK, the agent outperforms MPPI with H=64 within the first 30 episodes of training which corresponds to roughly 2 minutes of experience on simulation with true parameters. In contrast, SOFTQLEARNING barely ever moves the block and MPPI with H=10 succeeds only when arm is close to initial position of block. Similarly, in FRANKADRAWEROPEN, the agent with H = 10 achieves a success rate of more than 5 times as compared to MPPI with H=10 and outperforms MPPI with H=64 as well. The consistent results across all different tasks demonstrates the robustness and scalability of MPQ.

heading: 5.3.2 BENEFIT OF MPQ OVER DOMAIN RANDOMIZATION
text: Domain randomization(DR) techniques aim to make the policy learned in simulation robust to modelling errors by randomizing the simulation parameters using manually chosen distributions. However, such policies can be far from optimal on the true system parameters as the learned Q function is inherently biased. However, A Q function learned using rollouts from the real system can overcome model bias. We validate this hypothesis on the BALLINCUPSPARSE task by taking a DR approach inspired by Peng et al. (2018). Simulated rollouts are generated by sampling different parameters at every timestep from a broad range of dynamics parameters in Table 1, whereas real system rollouts use the true parameters. The average success rate reported in Table 2 demonstrates
that a Q function learned solely using DR is unable to generalize to the true system parameters and MPQ has more than twice the success rate when learned on real system.

heading: 6 DISCUSSION
text: In this work we have presented a theoretical connection between information theoretic MPC and soft Q learning approaches that naturally provides an algorithm to combine stochastic optimal control and model-free reinforcement learning. The theoretical insight not only ties together the different fields, but opens avenues to designing pragmatic RL algorithms that leverage the benefits of both. However, some important questions are yet to be answered. The optimal horizon for MPC is inextricably tied with the model error and optimization artifacts. Investigating this dependence in a principled manner is important for real-world applications. Another interesting avenue of research is characterizing the performance of a parameterized Q function and using it to adapt the horizon of MPC rollouts for smarter exploration.

heading: A APPENDIX
text: A.1 FURTHER EXPERIMENTAL DETAILS
The learned Q function takes as input the current action and a observation vector per task: 1. PENDULUMSWINGUP: [ cos(Θ), sin(Θ), Θ̇ ] (3 dim)
2. BALLINCUPSPARSE: [xball, xtarget, ˙xball, ˙xtarget, xtarget−xball, cos(Θ), sin(Θ)] (12 dim) where Θ is angle of line joining ball and target.
3. FETCHPUSHBLOCK: [xgripper, xobj , xobj − xgrip, gripper opening, rotobj , ẋobj , ωobj gripper opening vel, ẋgripper, d(gripper, obj), xgoal − xobj , d(goal, obj), xgoal] (33 dim)
4. FRANKADRAWEROPEN: [xee, xh, xh − xee, ẋee, ẋh, quatee, quath, drawerdisp d(ee, h), dquat(ee, h), dangee,h] (39 dim)
For all our experiments we parameterize Q functions with feedforward neural networks with two layers containing 100 units each and tanh activation. We use Adam (Kingma & Ba, 2014) optimization with a learning rate of 0.001. For generating value function targets in Eq. (22), we use 3 iterations of MPPI optimization except FRANKADRAWEROPEN where we use 1. The MPPI parameters used are listed in Table 3.

Review summaries:
This paper studies how a known, imperfect dynamics can be used to accelerate policy search. The main contribution of this paper is a model-based control algorithm that uses n-step lookahead planning and estimates the value of the last state with the prediction from a learned, soft Q value. The n-step planning uses the imperfect dynamics model, which can be cheaply queried offline. A second contribution of the paper is an efficient, iterative procedure for optimizing the n-step actions w.r.t. the imperfect model. The proposed algorithm is evaluated on three continuous control tasks. The proposed algorithm outperforms the two baselines on one of the three tasks. Perhaps the most impressive aspect of the paper is that the experiments only require a few minutes of "real-world" interaction to solve the tasks.