Bootstrapping the Expressivity with Model-based Planning

1 INTRODUCTION . Model-based deep reinforcement learning ( RL ) algorithms offer a lot of potentials in achieving significantly better sample efficiency than the model-free algorithms for continuous control tasks . We can largely categorize the model-based deep RL algorithms into two types : 1. model-based policy optimization algorithms which learn policies or Q-functions , parameterized by neural networks , on the estimated dynamics , using off-the-shelf model-free algorithms or their variants ( Luo et al. , 2019 ; Janner et al. , 2019 ; Kaiser et al. , 2019 ; Kurutach et al. , 2018 ; Feinberg et al. , 2018 ; Buckman et al. , 2018 ) , and 2. model-based planning algorithms , which plan with the estimated dynamics Nagabandi et al . ( 2018 ) ; Chua et al . ( 2018 ) ; Wang & Ba ( 2019 ) . A deeper theoretical understanding of the pros and cons of model-based and the model-free algorithms in the continuous state space case will provide guiding principles for designing and applying new sample-efficient methods . The prior work on the comparisons of model-based and model-free algorithms mostly focuses on their sample efficiency gap , in the case of tabular MDPs ( Zanette & Brunskill , 2019 ; Jin et al. , 2018 ) , linear quadratic regulator ( Tu & Recht , 2018 ) , and contextual decision process with sparse reward ( Sun et al. , 2019 ) . In this paper , we theoretically compare model-based RL and model-free RL in the continuous state space through the lens of approximability by neural networks , and then use the insight to design practical algorithms . What is the representation power of neural networks for expressing the Qfunction , the policy , and the dynamics ? How do the model-based and model-free algorithms utilize the expressivity of neural networks ? Our main finding is that even for the case of one-dimensional continuous state space , there can be a massive gap between the approximability of Q-function and the policy and that of the dynamics : The optimal Q-function and policy can be significantly more complex than the dynamics . We construct environments where the dynamics are simply piecewise linear functions with constant pieces , but the optimal Q-functions and the optimal policy require an exponential ( in the horizon ) * indicates equal contribution Figure 1 : Left : The dynamics of two randomly generated MDPs ( from the RAND , and SEMI-RAND methods outlined in Section 4.3 and detailed in Appendix D.1 ) . Right : The corresponding Q-functions which are more complex than the dynamics ( more details in Section 4.3 ) . number of linear pieces , or exponentially wide neural networks , to approximate.1 The approximability gap can also be observed empirically on ( semi- ) randomly generated piecewise linear dynamics with a decent chance . ( See Figure 1 for two examples . ) When the approximability gap occurs , any deep RL algorithms with policies parameterized by neural networks will suffer from a sub-optimal performance . These algorithms include both model-free algorithms such as DQN ( Mnih et al. , 2015 ) and SAC ( Haarnoja et al. , 2018 ) , and model-based policy optimization algorithms such as SLBO ( Luo et al. , 2019 ) and MBPO ( Janner et al. , 2019 ) . To validate the intuition , we empirically apply these algorithms to the constructed or the randomly generated MDPs . Indeed , they fail to converge to the optimal rewards even with sufficient samples , which suggests that they suffer from the lack of expressivity . However , in such cases , model-based planning algorithms should not suffer from the lack of expressivity , because they only use the learned , parameterized dynamics , which are easy to express . The policy obtained from the planning is the maximizer of the total future reward on the learned dynamics , and can have an exponential ( in the horizon ) number of pieces even if the dynamics has only a constant number of pieces . In fact , even a partial planner can help improve the expressivity of the policy . If we plan for k steps and then resort to some Q-function for estimating the total reward of the remaining steps , we can obtain a policy with 2k more pieces than what Q-function has . We hypothesize that the real-world continuous control tasks also have a more complex optimal Qfunction and a policy than the dynamics . The theoretical analysis of the synthetic dynamics suggests that a model-based few-steps planner on top of a parameterizedQ-function will outperform the original Q-function because of the addtional expressivity introduced by the planning . We empirically verify the intuition on MuJoCo benchmark tasks . We show that applying a model-based planner on top of Q-functions learned from model-based or model-free policy optimization algorithms in the test time leads to significant gains over the original Q-function or policy . In summary , our contributions are : 1 . We construct continuous state space MDPs whose Q-functions and policies are proved to be more complex than the dynamics ( Sections 4.1 and 4.2 . ) 2 . We empirically show that with a decent chance , ( semi- ) randomly generated piecewise linear MDPs also have complex Q-functions ( Section 4.3 . ) 3 . We show theoretically and empirically that the model-free RL or model-based policy optimization algorithms suffer from the lack of expressivity for the constructed MDPs ( Sections 4.3 ) , whereas model-based planning solve the problem efficiently ( Section 5.2 . ) 4 . Inspired by the theory , we propose a simple model-based bootstrapping planner ( BOOTS ) , which can be applied on top of any model-free or model-based Q-learning algorithms at 1In turn , the dynamics can also be much more complex than the Q-function . Consider the following situation : a subset of the coordinates of the state space can be arbitrarily difficult to express by neural networks , but the reward function can only depend on the rest of the coordinates and remain simple . the test time . Empirical results show that BOOTS improves the performance on MuJoCo benchmark tasks , and outperforms previous state-of-the-art on MuJoCo humanoid environment . 2 ADDITIONAL RELATED WORK . Comparisons with Prior Theoretical Work . Model-based RL has been extensively studied in the tabular case ( see ( Zanette & Brunskill , 2019 ; Azar et al. , 2017 ) and the references therein ) , but much less so in the context of deep neural networks approximators and continuous state space . ( Luo et al. , 2019 ) give sample complexity and convergence guarantees suing principle of optimism in the face of uncertainty for non-linear dynamics . Below we review several prior results regarding model-based versus model-free dichotomy in various settings . We note that our work focuses on the angle of expressivity , whereas the work below focuses on the sample efficiency . Tabular MDPs . The extensive study in tabular MDP setting leaves little gap in their sample complexity of model-based and model-free algorithms , whereas the space complexity seems to be the main difference . ( Strehl et al. , 2006 ) . The best sample complexity bounds for model-based tabular RL ( Azar et al. , 2017 ; Zanette & Brunskill , 2019 ) and model-free tabular RL ( Jin et al. , 2018 ) only differ by a poly ( H ) multiplicative factor ( where H is the horizon . ) Linear Quadratic Regulator . Dean et al . ( 2018 ) and Dean et al . ( 2017 ) provided sample complexity bound for model-based LQR . Recently , Tu & Recht ( 2018 ) analyzed sample efficiency of the modelbased and model-free problem in the setting of Linear Quadratic Regulator , and proved an O ( d ) gap in sample complexity , where d is the dimension of state space . Unlike tabular MDP case , the space complexity of model-based and model-free algorithms has little difference . The sample-efficiency gap mostly comes from that dynamics learning has d-dimensional supervisions , whereasQ-learning has only one-dimensional supervision . Contextual Decision Process ( with function approximator ) . Sun et al . ( 2019 ) prove an exponential information-theoretical gap between mode-based and model-free algorithms in the factored MDP setting . Their definition of model-free algorithms requires an exact parameterization : the value-function hypothesis class should be exactly the family of optimal value-functions induced by the MDP family . This limits the application to deep reinforcement learning where overparameterized neural networks are frequently used . Moreover , a crucial reason for the failure of the model-free algorithms is that the reward is designed to be sparse . Related Empirical Work . A large family of model-based RL algorithms uses existing model-free algorithms of its variant on the learned dynamics . MBPO ( Janner et al. , 2019 ) , STEVE ( Buckman et al. , 2018 ) , and MVE ( Feinberg et al. , 2018 ) are model-basedQ-learning-based policy optimization algorithms , which can be viewed as modern extensions and improvements of the early model-based Q-learning framework , Dyna ( Sutton , 1990 ) . SLBO ( Luo et al. , 2019 ) is a model-based policy optimization algorithm using TRPO as the algorithm in the learned environment . Another way to exploit the dynamics is to use it to perform model-based planning . Racanière et al . ( 2017 ) and Du & Narasimhan ( 2019 ) use the model to generated additional extra data to do planning implicitly . Chua et al . ( 2018 ) study how to combine an ensemble of probabilistic models and planning , which is followed by Wang & Ba ( 2019 ) , which introduces a policy network to distill knowledge from a planner and provides a prior for the planner . Piché et al . ( 2018 ) uses methods in Sequential Monte Carlo in the context of control as inference . Oh et al . ( 2017 ) trains a Q-function and then perform lookahead planning . Nagabandi et al . ( 2018 ) uses random shooting as the planning algorithm . Lowrey et al . ( 2018 ) uses the dynamics to improve the performance of model-free algorithms . Heess et al . ( 2015 ) backprops through a stochastic computation graph with a stochastic gradient to optimize the policy under the learned dynamics . Levine & Koltun ( 2013 ) distills a policy from trajectory optimization . Rajeswaran et al . ( 2016 ) trains a policy adversarially robust to the worst dynamics in the ensemble . Clavera et al . ( 2018 ) reformulates the problem as a meta-learning problem and using meta-learning algorithms . Predictron ( Silver et al. , 2017 ) learns a dynamics and value function and then use them to predict the future reward sequences . Another line of work focus on how to improve the learned dynamics model . Many of them use an ensemble of models ( Kurutach et al. , 2018 ; Rajeswaran et al. , 2016 ; Clavera et al. , 2018 ) , which are further extended to an ensemble of probabilistic models ( Chua et al. , 2018 ; Wang & Ba , 2019 ) . Luo et al . ( 2019 ) designs a discrepancy bound for learning the dynamics model . Talvitie ( 2014 ) augments the data for model training in a way that the model can output a real observation from its own prediction . Malik et al . ( 2019 ) calibrates the model ’ s uncertainty so that the model ’ s output distribution should match the frequency of predicted states . Oh et al . ( 2017 ) learns a representation of states by predicting rewards and future returns using representation .

Summary: The paper highlights an interesting issue regarding approximability of function approximators (neural networks). The paper provides cases where the action value function is difficult to approximate and is much more difficult than the dynamics of a model. The author conducts some experiments to claim that even with a large NN, DQN still finds a suboptimal policy. Theorems regarding the appoximability of the action value function are presented. Then the paper proposes that rollout-based search should be preferred for planning and conducts some experiments to verify this. Although the paper points out interesting issues of approximating action-value function, both the motivation and the suggestion regarding MBRL are not convincing.