Bootstrapping the Expressivity with Model-based Planning

1 INTRODUCTION . Model-based deep reinforcement learning ( RL ) algorithms offer a lot of potentials in achieving significantly better sample efficiency than the model-free algorithms for continuous control tasks . We can largely categorize the model-based deep RL algorithms into two types : 1. model-based policy optimization algorithms which learn policies or Q-functions , parameterized by neural networks , on the estimated dynamics , using off-the-shelf model-free algorithms or their variants ( Luo et al. , 2019 ; Janner et al. , 2019 ; Kaiser et al. , 2019 ; Kurutach et al. , 2018 ; Feinberg et al. , 2018 ; Buckman et al. , 2018 ) , and 2. model-based planning algorithms , which plan with the estimated dynamics Nagabandi et al . ( 2018 ) ; Chua et al . ( 2018 ) ; Wang & Ba ( 2019 ) . A deeper theoretical understanding of the pros and cons of model-based and the model-free algorithms in the continuous state space case will provide guiding principles for designing and applying new sample-efficient methods . The prior work on the comparisons of model-based and model-free algorithms mostly focuses on their sample efficiency gap , in the case of tabular MDPs ( Zanette & Brunskill , 2019 ; Jin et al. , 2018 ) , linear quadratic regulator ( Tu & Recht , 2018 ) , and contextual decision process with sparse reward ( Sun et al. , 2019 ) . In this paper , we theoretically compare model-based RL and model-free RL in the continuous state space through the lens of approximability by neural networks , and then use the insight to design practical algorithms . What is the representation power of neural networks for expressing the Qfunction , the policy , and the dynamics ? How do the model-based and model-free algorithms utilize the expressivity of neural networks ? Our main finding is that even for the case of one-dimensional continuous state space , there can be a massive gap between the approximability of Q-function and the policy and that of the dynamics : The optimal Q-function and policy can be significantly more complex than the dynamics . We construct environments where the dynamics are simply piecewise linear functions with constant pieces , but the optimal Q-functions and the optimal policy require an exponential ( in the horizon ) * indicates equal contribution Figure 1 : Left : The dynamics of two randomly generated MDPs ( from the RAND , and SEMI-RAND methods outlined in Section 4.3 and detailed in Appendix D.1 ) . Right : The corresponding Q-functions which are more complex than the dynamics ( more details in Section 4.3 ) . number of linear pieces , or exponentially wide neural networks , to approximate.1 The approximability gap can also be observed empirically on ( semi- ) randomly generated piecewise linear dynamics with a decent chance . ( See Figure 1 for two examples . ) When the approximability gap occurs , any deep RL algorithms with policies parameterized by neural networks will suffer from a sub-optimal performance . These algorithms include both model-free algorithms such as DQN ( Mnih et al. , 2015 ) and SAC ( Haarnoja et al. , 2018 ) , and model-based policy optimization algorithms such as SLBO ( Luo et al. , 2019 ) and MBPO ( Janner et al. , 2019 ) . To validate the intuition , we empirically apply these algorithms to the constructed or the randomly generated MDPs . Indeed , they fail to converge to the optimal rewards even with sufficient samples , which suggests that they suffer from the lack of expressivity . However , in such cases , model-based planning algorithms should not suffer from the lack of expressivity , because they only use the learned , parameterized dynamics , which are easy to express . The policy obtained from the planning is the maximizer of the total future reward on the learned dynamics , and can have an exponential ( in the horizon ) number of pieces even if the dynamics has only a constant number of pieces . In fact , even a partial planner can help improve the expressivity of the policy . If we plan for k steps and then resort to some Q-function for estimating the total reward of the remaining steps , we can obtain a policy with 2k more pieces than what Q-function has . We hypothesize that the real-world continuous control tasks also have a more complex optimal Qfunction and a policy than the dynamics . The theoretical analysis of the synthetic dynamics suggests that a model-based few-steps planner on top of a parameterizedQ-function will outperform the original Q-function because of the addtional expressivity introduced by the planning . We empirically verify the intuition on MuJoCo benchmark tasks . We show that applying a model-based planner on top of Q-functions learned from model-based or model-free policy optimization algorithms in the test time leads to significant gains over the original Q-function or policy . In summary , our contributions are : 1 . We construct continuous state space MDPs whose Q-functions and policies are proved to be more complex than the dynamics ( Sections 4.1 and 4.2 . ) 2 . We empirically show that with a decent chance , ( semi- ) randomly generated piecewise linear MDPs also have complex Q-functions ( Section 4.3 . ) 3 . We show theoretically and empirically that the model-free RL or model-based policy optimization algorithms suffer from the lack of expressivity for the constructed MDPs ( Sections 4.3 ) , whereas model-based planning solve the problem efficiently ( Section 5.2 . ) 4 . Inspired by the theory , we propose a simple model-based bootstrapping planner ( BOOTS ) , which can be applied on top of any model-free or model-based Q-learning algorithms at 1In turn , the dynamics can also be much more complex than the Q-function . Consider the following situation : a subset of the coordinates of the state space can be arbitrarily difficult to express by neural networks , but the reward function can only depend on the rest of the coordinates and remain simple . the test time . Empirical results show that BOOTS improves the performance on MuJoCo benchmark tasks , and outperforms previous state-of-the-art on MuJoCo humanoid environment . 2 ADDITIONAL RELATED WORK . Comparisons with Prior Theoretical Work . Model-based RL has been extensively studied in the tabular case ( see ( Zanette & Brunskill , 2019 ; Azar et al. , 2017 ) and the references therein ) , but much less so in the context of deep neural networks approximators and continuous state space . ( Luo et al. , 2019 ) give sample complexity and convergence guarantees suing principle of optimism in the face of uncertainty for non-linear dynamics . Below we review several prior results regarding model-based versus model-free dichotomy in various settings . We note that our work focuses on the angle of expressivity , whereas the work below focuses on the sample efficiency . Tabular MDPs . The extensive study in tabular MDP setting leaves little gap in their sample complexity of model-based and model-free algorithms , whereas the space complexity seems to be the main difference . ( Strehl et al. , 2006 ) . The best sample complexity bounds for model-based tabular RL ( Azar et al. , 2017 ; Zanette & Brunskill , 2019 ) and model-free tabular RL ( Jin et al. , 2018 ) only differ by a poly ( H ) multiplicative factor ( where H is the horizon . ) Linear Quadratic Regulator . Dean et al . ( 2018 ) and Dean et al . ( 2017 ) provided sample complexity bound for model-based LQR . Recently , Tu & Recht ( 2018 ) analyzed sample efficiency of the modelbased and model-free problem in the setting of Linear Quadratic Regulator , and proved an O ( d ) gap in sample complexity , where d is the dimension of state space . Unlike tabular MDP case , the space complexity of model-based and model-free algorithms has little difference . The sample-efficiency gap mostly comes from that dynamics learning has d-dimensional supervisions , whereasQ-learning has only one-dimensional supervision . Contextual Decision Process ( with function approximator ) . Sun et al . ( 2019 ) prove an exponential information-theoretical gap between mode-based and model-free algorithms in the factored MDP setting . Their definition of model-free algorithms requires an exact parameterization : the value-function hypothesis class should be exactly the family of optimal value-functions induced by the MDP family . This limits the application to deep reinforcement learning where overparameterized neural networks are frequently used . Moreover , a crucial reason for the failure of the model-free algorithms is that the reward is designed to be sparse . Related Empirical Work . A large family of model-based RL algorithms uses existing model-free algorithms of its variant on the learned dynamics . MBPO ( Janner et al. , 2019 ) , STEVE ( Buckman et al. , 2018 ) , and MVE ( Feinberg et al. , 2018 ) are model-basedQ-learning-based policy optimization algorithms , which can be viewed as modern extensions and improvements of the early model-based Q-learning framework , Dyna ( Sutton , 1990 ) . SLBO ( Luo et al. , 2019 ) is a model-based policy optimization algorithm using TRPO as the algorithm in the learned environment . Another way to exploit the dynamics is to use it to perform model-based planning . Racanière et al . ( 2017 ) and Du & Narasimhan ( 2019 ) use the model to generated additional extra data to do planning implicitly . Chua et al . ( 2018 ) study how to combine an ensemble of probabilistic models and planning , which is followed by Wang & Ba ( 2019 ) , which introduces a policy network to distill knowledge from a planner and provides a prior for the planner . Piché et al . ( 2018 ) uses methods in Sequential Monte Carlo in the context of control as inference . Oh et al . ( 2017 ) trains a Q-function and then perform lookahead planning . Nagabandi et al . ( 2018 ) uses random shooting as the planning algorithm . Lowrey et al . ( 2018 ) uses the dynamics to improve the performance of model-free algorithms . Heess et al . ( 2015 ) backprops through a stochastic computation graph with a stochastic gradient to optimize the policy under the learned dynamics . Levine & Koltun ( 2013 ) distills a policy from trajectory optimization . Rajeswaran et al . ( 2016 ) trains a policy adversarially robust to the worst dynamics in the ensemble . Clavera et al . ( 2018 ) reformulates the problem as a meta-learning problem and using meta-learning algorithms . Predictron ( Silver et al. , 2017 ) learns a dynamics and value function and then use them to predict the future reward sequences . Another line of work focus on how to improve the learned dynamics model . Many of them use an ensemble of models ( Kurutach et al. , 2018 ; Rajeswaran et al. , 2016 ; Clavera et al. , 2018 ) , which are further extended to an ensemble of probabilistic models ( Chua et al. , 2018 ; Wang & Ba , 2019 ) . Luo et al . ( 2019 ) designs a discrepancy bound for learning the dynamics model . Talvitie ( 2014 ) augments the data for model training in a way that the model can output a real observation from its own prediction . Malik et al . ( 2019 ) calibrates the model ’ s uncertainty so that the model ’ s output distribution should match the frequency of predicted states . Oh et al . ( 2017 ) learns a representation of states by predicting rewards and future returns using representation .

Summary: This paper presents a mainly theoretical argument comparing the expressivity of model-free and model-based RL methods contrary to analysis in the past which usually relies on sample complexity. They construct a family of MDPs, where the true dynamics belong to a simple function class (in terms of the number of linear pieces needed to define the function), but the corresponding optimal Q-function belongs to a function class not necessarily expressible by a simple function. The paper then builds a similar case for randomly/semi-randomly generated MDPs. Finally, they propose to bootstrap the Q-function with n-step returns to boost the expressivity exponentially.