Surprising Negative Results for Generative  Adversarial Tree Search 

While many recent advances in deep reinforcement learning rely on model-free methods , model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment dynamics . One prospect is to pursue hybrid approaches , as in AlphaGo , which combines Monte-Carlo Tree Search ( MCTS ) —a model-based method—with deep-Q networks ( DQNs ) —a model-free method . MCTS requires generating rollouts , which is computationally expensive . In this paper , we propose to simulate roll-outs , exploiting the latest breakthroughs in image-to-image transduction , namely Pix2Pix GANs , to predict the dynamics of the environment . Our proposed algorithm , generative adversarial tree search ( GATS ) , simulates rollouts up to a specified depth using both a GANbased dynamics model and a reward predictor . GATS employs MCTS for planning over the simulated samples and uses DQN to estimate the Q-function at the leaf states . Our theoretical analysis establishes some favorable properties of GATS vis-a-vis the bias-variance trade-off and empirical results show that on 5 popular Atari games , the dynamics and reward predictors converge quickly to accurate solutions . However , GATS fails to outperform DQNs in 4 out of 5 games . Notably , in these experiments , MCTS has only short rollouts ( up to tree depth 4 ) , while previous successes of MCTS have involved tree depth in the hundreds . We present a hypothesis for why tree search with short rollouts can fail even given perfect modeling . 1 INTRODUCTION . The earliest and best-publicized applications of deep reinforcement learning ( DRL ) involve Atari games ( Mnih et al. , 2015 ) and the board game of Go ( Silver et al. , 2016 ) , where experience is inexpensive because the environments are simulated . In such scenarios , DRL can be combined with Monte-Carlo tree search ( MCTS ) methods ( Kearns et al. , 2002 ; Kocsis & Szepesvári , 2006 ) for planning , where the agent executes roll-outs on the simulated environment ( as far as computationally feasible ) to finds suitable policies . However , for RL problems with long episodes , e.g . Go , MCTS can be very computationally expensive . In order to speed up MCTS for Go and learn an effective policy , Alpha Go ( Silver et al. , 2016 ) employs a depth-limited MCTS with the depth in the hundreds on their Go emulator and use an estimated Q-function to query the value of leaf nodes . However , in real-world applications , such as robotics ( Levine et al. , 2016 ) and dialogue systems ( Lipton et al. , 2016 ) , collecting samples often takes considerable time and effort . In such scenarios , the agent typically can not access either the environment model or a corresponding simulator . Recently , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have emerged as a popular tool for synthesizing realistic-seeming data , especially for high-dimensional domains , including images and audio . Unlike previous approaches to image generation , which typically produced blurry images due to optimizing an L1 or L2 objective , GANs produces crisp images . Since theire original conception as an unsupervised method , GANs have been extended for conditional generation , e.g. , generating an image conditioned on a label ( Mirza & Osindero , 2014 ; Odena et al. , 2016 ) or the next frame in a video given a context window ( Mathieu et al. , 2015 ) . Recently , the PIX2PIX approach has demonstrated impressive results on a range of image-to-image transduction tasks ( Isola et al. , 2017 ) . In this work , we propose and analyze generative adversarial tree search ( GATS ) , a new DRL algorithm that utilizes samples from the environment to learn both a Q-function approximator , a near-term reward predictor , and a GAN-based model of the environment ’ s dynamics ( state transitions ) . Together , the dynamics model and reward predictor constitute a learned simulator on which MCTS can be performed . GATS leverages PIX2PIX GANs to learn a generative dynamics model ( GDM ) that efficiently learns the dynamics of the environment , producing images that agree closely with the actual observed transactions and are also visually crisp . We thoroughly study various image transduction models , arriving ultimately at a GDM that converges quickly ( compared to the DQN ) , and appears from our evaluation to be reasonably robust to subtle distribution shifts , including some that destroy a DQN policy . We also train a reward predictor that converges quickly , achieving negligible error ( over 99 % accuracy ) . GATS bridges model-based and model-free reinforcement learning , using the learned dynamics and reward predictors to simulate roll-outs in combination with a DQN . Specifically , GATS deploys the MCTS method for planning over a bounded tree depth and uses the DQN algorithm to estimate the Q-function as a value for the leaf states ( Mnih et al. , 2015 ; Van Hasselt et al. , 2016 ) . One notable aspect of the GATS algorithm is its flexibility , owing to consisting of a few modular building blocks : ( i ) value learning : we deployed DQN and DDQN ( ii ) planning : we use pure Monte Carlo sampling ; ( iii ) a reward predictor : we used a simple 3-class classifier ; ( iv ) dynamics model : we propose the GDM architecture . Practically , one can swap in other methods for any among these blocks and we highlight some alternatives in the related work . Thus , GATS constitutes a general framework for studying the trade-offs between model-based and model-free reinforcement learning . 1.1 TECHNICAL CONTRIBUTIONS . Theoretical analysis We analyze the components of error in the estimation of the expected return used by GATS , and further study the trade-offs in its bias and variance . Since GATS utilizes the learned Q function of DQN/DDQN in the leaf nodes of the MCTS tree , the existing errors in the Q-estimation decays exponentially as the depth of MCTS grows . We further study the bias in the Q estimate of DQN and DDQN , where we found ( empirically ) that GATS with even one step look-ahead or rollout ( depth one ) , can help to reduce the negative effects of these biases . This leads to a reduction in the sample complexity of DQN and DDQN by a factor of 2 on the game Pong . Furthermore , we develop a heuristic optimism-based strategy for GATS using the GDM . The low computation cost of Pong allows us to do an extensive study of the bias-variance of Q for different model-based planning and exploration strategies . Experimental results For this work , we also developed a new OpenAI gym ( Brockman et al. , 2016 ) -like interface for the latest Atari Learning Environment ( ALE ) ( Machado et al. , 2017 ) , which supports different modes and difficulties for Atari games . We study the sample complexity required by GDM and RP to adapt and transfer from one domain of the game ( a mode and difficulty ) to another domain ( another mode and difficulty ) . We show that GDM and RP adapt quickly to the new mode in a few numbers of samples , while the estimated Q-function requires significantly more samples to adapt . We documented and open-sourced this wrapper on the latest ALE as well as the code for GDM , RP , and the GATS algorithm . Surprising negative results Despite learning environment models ( GDM and RP ) that converge efficiently and achieve accuracy exceeding our expectations , we are unable to improve the return of the learned policy using short rollouts ( at most depth 5 ) on any other game besides Pong . The negative result persisted across an extensive and costly study with many different hyper-parameter settings and learning strategies , including several different strategies to use the generated frames to train the Q-model inspired by Sutton ( 1990 ) . We put forth a hypothesis for why GATS , despite the good performance of its constituent models and its theoretical advantages , might fail in short rollout depths : in short , under this training regime , the problem may be that the Q-learner does not observe the outcomes of its mistakes . We also explain why in order to test our hypothesis , one might need to experiment with significantly longer rollouts , which may be prohibitively expensive ( computationally ) in this domain . Find a more detailed explanation of this hypothesis in Section 7 . Consider the fact that all known successes of MCTS have involved tree depth in the hundreds . For example , ( Guo et al. , 2014 ) shows for Atari games , when a plain MCTS is deployed , a depth of 300 with 1000 trajectories is required to learn a reasonable policy . This depth of MCTS on GDM requires massive amounts of computation beyond the scale of academic research and the scope of this paper . Considering the broader enthusiasm for both model-based RL and generative adversarial networks , we believe that this study , despite its failure to advance the leaderboard , illuminates several important considerations for future work to develop tree-search and rollout based methods to combine model-based and model-free reinforcement learning . 2 PRELIMINARIES . An infinite horizon γ-discounted MDP M is a tuple 〈X , A , T , R , P0 , γ〉 , with state space X , action space A , and P0 , the distribution over the initial states . The transition kernel T : x , a → ∆x , drives the model dynamics accompanied with [ 0 , 1 ] -bounded reward of R : x , a → ∆r , where 0 ≤ γ < 1 . The agent ’ s objective is to find a policy π : = X → A that maximizes the overall expected discounted reward η∗ : = η ( π∗ ) = maxπ limN→∞ Eπ [ ∑N t=0 γ trt|x0 ∼ P0 ] . By Qπ ( x , a ) : = limN→∞ Eπ [ ∑N t=0 γ trt|x0 = x , a0 = a ] , we denote the expected cumulative discounted reward under policy π starting from state-action x , a . In value based RL , we aim to learn the Q-function in order to derive the optimal policy . In order to learn the Q function , we might aim to minimize square loss for any given pair of state and action ( x , a ) , ( Q ( x , a ) − Eπ [ r + γQ ( x′ , a′ ) |x , a ] ) 2 ( 1 ) In order to minimize the expression in Eq . 1 , a double sampling is required to estimate the inner expectation . To avoid the cost of the double sampling , a common approach is to instead minimize the Bellman residual ( Lagoudakis & Parr , 2003 ; Antos et al. , 2008 ) : Eπ [ ( Q ( x , a ) − ( r + γQ ( x′ , a′ ) ) ) 2 ∣∣∣x , a ] = ( Q ( x , a ) − Eπ [ r + γQ ( x′ , a′ ) ∣∣∣x , a ] ) 2 + Varπ ( r + γQ ( x′ , a′ ) ∣∣∣x , a ) The Bellman residual is the sum of the expression in Eq . 1 and an additional variance term . DQN partially addresses this bias1 , by computing the target value with a separate function approximator , typically updated less frequently than the policy , L ( Q , Qtarget ) =Eπ [ ( Q ( x , a ) − r−γQtarget ( x′ , a′ ) ) 2 ] ( 2 ) Generally , in addition to this bias , there are additional statistical biases due to limited capacity of network , optimization algorithm , model mismatch , as well as bias induced by the max operator or choice of a′ . In the next section , we theoretically and empirically study this bias and show how GATS can address this undesirable effect . For the generative dynamic model , we propose a generic GDM which consists of a generator G and a discriminator D , trained adversarially w.r.t . the extended conditional Wasserstein metric between two probability measures P $ , PG conditioned on a third probability measure P ; W ( P $ , PG|P ) : = supD∈‖·‖LE $ ∼P $ | % , % ∼P [ D ( $ | % ) ] − E $ : G ( % ∼P , z∼N ( 0 , I ) ) [ D ( $ | % ) ] ( 3 ) Here , z is a mean-zero unit-variance Gaussian vector random variable and ‖ · ‖L indicates the space of all 1-Lipschitz functions . In GDM , D solves the interior sup , while G ’ s objective is to minimize this distance and learn the P $ | % for all % . We deploy our proposed GDM in GATS where P is the distribution over pairs of % : ( x , a ) in the replay buffer , and P $ | % is the distribution over the successor states $ : x′ , which is the transition kernel T ( x′|x , a ) .

Summary: The submitted paper proposes GATS, a RL model combining model-free and model-bases reinforcement learning. Estimated models of rewards and dynamics are used to perform rollouts in MTCS without actually interacting with the true environment. The authors theoretically and empirically evaluate their approach for low depths rollouts. Empirically they show improved sample complexity on the Atari game Pong.