Information Theoretic Model Predictive Q-Learning

1 INTRODUCTION . Deep reinforcement learning algorithms have recently generated great interest due to their successful application to a range of difficult problems including Computer Go ( Silver et al. , 2016 ) and highdimensional control tasks such as humanoid locomotion ( Lillicrap et al. , 2015 ; Schulman et al. , 2015 ) . While these methods are extremely general and can learn policies and value functions for complex tasks directly from data , they can also be sample inefficient , and partially-optimized solutions can be arbitrarily poor . These challenges severely restrict RL ’ s applicability to real systems such as robots due to data collection challenges and safety concerns . One straightforward way to mitigate these issues is to learn a policy or value function entirely in a high-fidelity simulator ( Shah et al. , 2017 ; Todorov et al. , 2012 ) and then deploy the optimized policy on the real system . However , this approach can fail due to model bias , external disturbances , the subtle differences between the real robot ’ s hardware and poorly modeled phenomena such as friction and contact dynamics . Sim-to-real transfer approaches based on domain randomization ( Sadeghi & Levine , 2016 ; Tobin et al. , 2017 ) and model ensembles ( Kurutach et al. , 2018 ; Shyam et al. , 2019 ) aim to make the policy robust by training it to be invariant to varying dynamics . However , learning a globally consistent value function or policy is hard due to optimization issues such as local optima and covariate shift between the exploration policy used for learning the model and the actual control policy executed on the task ( Ross & Bagnell , 2012 ) . Model predictive control ( MPC ) is a widely used method for generating feedback controllers that repeatedly re-optimizes a finite horizon sequence of controls using an approximate dynamics model that predicts the effect of these controls on the system . The first control in the optimized sequence is executed on the real system and the optimization is performed again from the resulting next state . However , the performance of MPC can suffer due to approximate or simplified models and lim- ited lookahead . Therefore the parameters of MPC , including the model and horizon H should be carefully tuned to obtain good performance . While using a longer horizon is generally preferred , real-time requirements may limit the amount of lookahead and a biased model can result in compounding model errors . In this work , we present an approach to RL that leverages the complementary properties of modelfree reinforcement learning and model-based optimal control . Our proposed method views MPC as a way to simultaneously approximate and optimize a local Q function via simulation , and Q learning as a way to improve MPC using real-world data . We focus on the paradigm of entropy regularized reinforcement learning where the aim is to learn a stochastic policy that minimizes the cost-to-go as well as KL divergence with respect to a prior policy . This approach enables faster convergence by mitigating the over-commitment issue in the early stages of Q-learning and better exploration ( Fox et al. , 2015 ) . We discuss how this formulation of reinforcement learning has deep connections to information theoretic stochastic optimal control where the objective is to find control inputs that minimize the cost while staying close to the passive dynamics of the system ( Theodorou & Todorov , 2012 ) . This helps in both injecting domain knowledge into the controller as well as mitigating issues caused by over optimizing the biased estimate of the current cost due to model error and the limited horizon of optimization . We explore this connection in depth and derive an infinite horizon information theoretic model predictive control algorithm based on Williams et al . ( 2017 ) . We test our approach called Model Predictive Q Learning ( MPQ ) on simulated continuous control tasks and compare it against information theoretic MPC and soft Q-Learning ( Haarnoja et al. , 2017 ) , where we demonstrate faster learning with fewer system interactions and better performance as compared to MPC and soft Q-Learning even in the presence of sparse rewards . The learned Q function allows us to truncate the MPC planning horizon which provides additional computational benefits . Finally , we also compare MPQ versus domain randomization ( DR ) on sim-to-sim tasks . We conclude that DR approaches can be sensitive to the hand-designed distributions for randomizing parameters which causes the learned Q function to be biased and suboptimal on the true system ’ s parameters , whereas learning from data generated on true system is able to overcome biases and adapt to the real dynamics . 2 RELATED WORK . Model predictive control has a rich history in robotics , ranging from control of mobile robots such as quadrotors ( Desaraju & Michael , 2016 ) and aggressive autonomous vehicles ( Wagener et al. , 2019 ; Williams et al. , 2017 ) to generating complex behaviors for high-dimensional systems such as contact-rich manipulation ( Fu et al. , 2016 ; Kumar et al. , 2014 ) and humanoid locomotion ( Erez et al. , 2013 ) . The success of MPC can largely be attributed to online policy optimization which helps mitigate model bias . The information theoretic view of MPC aims to find a policy at every timestep that minimizes the cost over a finite horizon as well as the KL-divergence with respect to a prior policy usually specified by the system ’ s passive dynamics ( Theodorou & Todorov , 2012 ; Williams et al. , 2017 ) . This helps maintain exploratory behavior and avoid over-commitment to the current estimate of the cost function , which is biased due to modeling errors and a finite horizon . Sampling-based MPC algorithms ( Wagener et al. , 2019 ; Williams et al. , 2017 ) are also highly parallelizable enabling GPU implementations that aid with real-time control . However , efficient MPC implementations still require careful system identification and extensive amounts of manual tuning . Deep RL methods are extremely general and can optimize neural network policies from raw sensory inputs with little knowledge of the system dynamics . Both value-based and policy-based approaches ( Schulman et al. , 2015 ) have demonstrated excellent performance on complex control problems . These approaches , however , fall short on several accounts when applying them to a real robotic system . First , they have high sample complexity , potentially requiring millions of interactions with the environment . This can be very expensive on a real robot , not least because the initial performance of the policy can be arbitrarily bad . Using random exploration methods such a -greedy can further aggravate this problem . Second , a value function or policy learned entirely in simulation inherits the biases of the simulator . Even if a perfect simulation is available , learning a globally consistent value function or policy is an extremely hard task as noted in ( Silver et al. , 2016 ; Zhong et al. , 2013 ) . This can be attributed to local optima when using neural network representations or the inherent biases in the Q learning update rules ( Fox et al. , 2015 ; Van Hasselt et al. , 2016 ) . In fact , it can be difficult to explain why Q-learning algorithms work or fail ( Schulman et al. , 2017 ) . Domain randomization aims to make policies learned in simulation more robust by randomizing simulation parameters during training with the aim of making the policies invariant to potential parameter error ( Peng et al. , 2018 ; Sadeghi & Levine , 2016 ; Tobin et al. , 2017 ) . However , these policies are not adaptive to unmodelled effects , i.e they take into account only aleoteric and not epistemic uncertainty . Also , such approaches are highly sensitive to hand-designed distributions used for randomizing simulation parameters and can be highly suboptimal on the real-systems parameters , for example , if a very large range of simulation parameters is used . Model-based approaches aim to use real data to improve the model of the system and then perform reinforcement learning or optimal control using the new model or ensemble of models ( Kurutach et al. , 2018 ; Ross & Bagnell , 2012 ; Shyam et al. , 2019 ) . Although learning accurate models is a promising avenue , we argue that learning a globally consistent model is an extremely hard problem and instead we should learn a policy that can rapidly adapt to experienced real-world dynamics . The use of entropy regularization has been explored in RL and Inverse RL for its better sample efficiency and exploration properties ( Fox et al. , 2015 ; Haarnoja et al. , 2017 ; 2018 ; Schulman et al. , 2017 ; Ziebart et al. , 2008 ) . This framework allows incorporating prior knowledge into the problem and learning multi-modal policies that can generalize across different tasks . Fox et al . ( 2015 ) analyze the theoretical properties of the update rule derived using mutual information minimization and show that this framework can overcome the over-estimation issue inherent in the vanilla Q-learning update . In the past , Todorov ( 2009 ) have shown that using KL-divergence can convert the optimal control problem into one that is linearly solvable . Infinite horizon MPC aims to learn a terminal cost function that can add global information to the finite horizon optimization . Rosolia & Borrelli ( 2017 ) learn a terminal cost as a control Lyapunov function and a safety set for the terminal state . These quantites are calculated using all previously visited states and they assume the presence of a controller that can deterministically drive the any state to the goal . Tamar et al . ( 2017 ) learns a cost shaping to make a short horizon MPC mimic the actions produced by long horizon MPC offline . However , since their approach is to mimic a longer horizon MPC , the performance of the learner is fundamentally limited by the the performance of the longer horizon MPC . On the contrary , learning an optimal value function as the terminal cost can potentially lead to close to optimal performance . Using local optimization is an effective way of improving an imperfect value function as noted in RL literature by Anthony et al . ( 2017 ) ; Lowrey et al . ( 2018 ) ; Silver et al . ( 2016 ; 2017 ) ; Sun et al . ( 2018 ) . However , these approaches assume that a perfect model of the system is available . In order to make the policy work on the real system , we argue that it is essential to learn a value function form real data and utilize local optimization to stabilize learning . 3 PRELIMINARIES . 3.1 REINFORCEMENT LEARNING WITH ENTROPY REGULARIZATION . A Markov Decision Process ( MDP ) is defined by tuple ( S , A , c , P , γ ) where S is the state space , A is the action space , c is a one step cost function , P is the space of transition functions and γ is a discount factor . Let P ∈ P be a particular transition function . A closed-loop policy π ( a|s ) is a distribution over actions given state . Given a policy π and a prior policy π , the KL divergence between them at a state is given by KL ( π ( a|s ) ||π ( a|s ) ) = Eπ [ log ( π ( a|s ) /π ( a|s ) ) ] . Entropyregularized RL ( Fox et al. , 2015 ) aims to optimize the following objective π∗ = arg min π Eπ , P [ ∞∑ t=1 γt−1 ( c ( st , at ) + λKL ( πt||πt ) ) ] ∀ s0 ∈ S ( 1 ) where πt and πt are shorthand for π ( at|st ) and π ( at|st ) respectively , λ is the temperature parameter that penalizes deviation of π from π . For a policy π , we can define the soft value and action-value functions V π ( s ) = Eπ , P [ ∞∑ t=1 γt−1 ( c ( st , at ) + λKL ( πt||πt ) ) |s0 = s ] Qπ ( s , a ) = c ( s , a ) + γEs′∼P ( s′|s , a ) [ V π ( s′ ) ] ( 2 ) Given a horizon of H timesteps , we can use above definitions to write the value functions as V π ( s ) = Eπ , P [ H−1∑ t=1 γt−1 ( c ( st , at ) + λKL ( πt||πt ) ) + γH−1V π ( sH ) |s1 = s ] Qπ ( s , a ) = c ( s , a ) + Eπ , P [ H−1∑ t=2 γt−1 ( c ( st , at ) + λKL ( πt||πt ) ) + γH−1 ( λKL ( πH ||πH ) +Q ( sH , aH ) ) |s1 = s , a1 = a ] ( 3 ) It is straightforward to verify that V π ( s ) = Ea∼π [ log ( π ( a|s ) /π ( a|s ) ) +Q ( s , a ) ] . The objective in Eq . ( 1 ) can equivalently be written as π∗ = arg min π V π ( s ) ∀ s ∈ S ( 4 ) This optimization can be performed either by policy gradient methods that aim to find the optimal policy π ∈ Π via stochastic gradient descent ( Schulman et al. , 2017 ; Williams , 1992 ) or value based methods that try to iteratively approximate the value function of the optimal policy . In either case , the output of solving the above optimization is a global closed-loop control policy π∗ ( a|s ) .

Summary: This paper studies how a known, imperfect dynamics can be used to accelerate policy search. The main contribution of this paper is a model-based control algorithm that uses n-step lookahead planning and estimates the value of the last state with the prediction from a learned, soft Q value. The n-step planning uses the imperfect dynamics model, which can be cheaply queried offline. A second contribution of the paper is an efficient, iterative procedure for optimizing the n-step actions w.r.t. the imperfect model. The proposed algorithm is evaluated on three continuous control tasks. The proposed algorithm outperforms the two baselines on one of the three tasks. Perhaps the most impressive aspect of the paper is that the experiments only require a few minutes of "real-world" interaction to solve the tasks.