Fast Exploration with Simplified Models and Approximately Optimistic Planning in Model Based Reinforcement Learning

Humans learn to play video games significantly faster than the state-of-the-art reinforcement learning ( RL ) algorithms . People seem to build simple models that are easy to learn to support planning and strategic exploration . Inspired by this , we investigate two issues in leveraging model-based RL for sample efficiency . First we investigate how to perform strategic exploration when exact planning is not feasible and empirically show that optimistic Monte Carlo Tree Search outperforms posterior sampling methods . Second we show how to learn simple deterministic models to support fast learning using object representation . We illustrate the benefit of these ideas by introducing a novel algorithm , Strategic Object Oriented Reinforcement Learning ( SOORL ) , that outperforms state-of-the-art algorithms in the game of Pitfall ! in less than 50 episodes . 1 INTRODUCTION . The coupling of deep neural networks and reinforcement learning has led to exciting advances , enabling reinforcement learning agents that can reach human-level performance in many Atari2600 games ( Mnih et al. , 2015 ; 2016 ; Silver et al. , 2017 ; Hessel et al. , 2017 ) . However , such agents typically require hundreds of millions of time steps to learn to play well . This is in sharp contrast to people , who typically learn to play Atari games within a few episodes ( Lake et al. , 2017 ) . Prior work on human learning for Atari suggests that people may be systematically building models of the reward and dynamics in the domain and using those to plan efficiently ( Tsividis et al. , 2017 ; Dubey et al. , 2018 ) . Given that human learners seemingly perform model-based RL very quickly , we are motivated to consider alternative approaches to current model-based RL , which often involves building complex predictive deep neural networks from scratch . Deep neural networks can require a large amount of data to accurately train , and performing perfect planning with those models is computationally expensive . Indeed , while there is a considerable amount of theoretical work done on tabular modelbased reinforcement learning that suggests model-based approaches can be provably sample efficient ( Dann et al. , 2017 ; Brafman & Tennenholtz , 2002 ; Strehl & Littman , 2008 ) , there is much less success in using model-based reinforcement learning in extremely large environments , in part due to the challenges of learning simple accurate models in these domains . Instead , in this paper , we investigate two issues in leveraging model-based RL to speed learning in large domains . First we explore how to perform approximate planning using models during model-based RL to support exploration , assuming models are given . Models can facilitate deep exploration , since one key benefit of using models is that it is often easier to quantify uncertainty in those models , and perform planning in a way to guide the agent towards exploring and reducing that uncertainty . Recent work has suggested the benefit of Thompson Sampling methods over optimism methods for reinforcement learning ( Osband & Van Roy , 2016 ) , and , indeed , empirically Thompson Sampling methods have done very well in contextual bandits and small state space MDPs where it is possible to plan exactly . However , to our knowledge , there has not been an investigation of how these popular approaches scale to larger domains where it is computationally prohibitive to perform exact planning . We introduce optimistic MCTS , a variant of the popular planning technique Monte Carlo Tree Search , and find that optimism here outperforms other approaches in several simulation experiments when planning can only be done approximately . Second , while our first investigation explores how to plan to encourage exploration given a model , we further investigate how to choose the model class to support computational tractability and learning efficiency . Here we propose to learn object-oriented deterministic models of the domain . People may leverage and test models of object interactions during video game learning ( Tsividis et al. , 2017 ; Dubey et al. , 2018 ) , and object-oriented learning has the nice benefit that data from all similar objects can be pooled when building a model . Prior work has shown that object-oriented model-based reinforcement learning can yield provably efficient learning and scale to larger domains ( Diuk et al. , 2008 ) . Deterministic models offer an additional benefit over object-oriented models–they require less data to train ( since one does not have to model a large stochastic distribution of outcomes ) and they have additional benefits for planning , reducing the branching factor of next states to 1 . Indeed past work ( Diuk et al. , 2008 ) also proposed learning deterministic object-oriented models . In contrast , we investigate how to learn the transition model and temporal abstraction to make the world appear deterministic . In other words , we assume a candidate set of possible transition model classes and temporal abstractions , and perform model selection to select a level of temporal abstraction and model that deterministically predicts the outcome of an action . We illustrate the potential benefit of these ideas by introducing an object oriented algorithm that uses prior knowledge of model classes and object representation , and show that our algorithm can learn to achieve positive reward in the notoriously difficult Atari game Pitfall ! within 50 episodes . Almost no RL methods have achieved positive reward on Pitfall ! without human demonstrations , and even with demonstrations , such approaches often take hundreds of millions of frames to learn ( Aytar et al. , 2018 ; Hester et al. , 2017 ) . In contrast to demonstrations , we assume two forms of prior knowledge–a predefined object representation and a class of potential model features . Computer vision is rapidly advancing to the state that soon we will be able to easily extract objects from even artificial scenes like the Arcade Learning Environment . The second assumption of candidate model classes is a stronger assumption , but a large set of models could be defined directly given the object classes and only incur a cost quadratic in the set of features . While encouraging , such results should be mostly viewed as a case study . We believe the key contributions of our paper are not this particular demonstration on Pitfall ! , but rather the investigation of exploration approaches when planning can only be done approximately , and the benefit of selecting among model representations to support computationally tractable planning and fast learning . The second can be viewed as a bias/variance tradeoff and in future work we plan to consider how to identify and account for model biases that could limit asymptotic performance during the model-based planning procedure . 2 PRELIMINARIES . We consider a finite horizon Markov Decision Process ( MDP ) 〈S , A , T , R , γ〉 , where S is the state space , A the action space , T : S × A → S the transition function , R : S × A → R the reward function , and γ the discount factor . The goal of the RL agent is to maximize the expected discounted reward Eπ [ ∑T t=0 γ tR ( st , at ) ] following a policy π . 3 APPROXIMATE MODEL-BASED PLANNING TO SUPPORT EXPLORATION . Computing an optimal policy for a Markov decision process in a large state space can be extremely computationally expensive , if not impossible . Therefore some form of approximate planning procedure is almost always needed for large scale domains . Especially in model-based RL algorithms when the transition and reward models are initially unknown and should be learned online , a computationally tractable planning algorithm is more critical . Since the model estimates are changing and one should plan more frequently with the new model estimates . One popular and powerful approach for scaling up planning for large Markov decision processes is Monte Carlo Tree Search ( Chaslot et al. , 2008 ; Browne et al. , 2012 ) , which has been used to achieve better than world class performance in Go ( Silver et al. , 2016 ) . When a perfect transition and reward model is known , MCTS is guaranteed to converge to the optimal value function in the limit of infinite computation ( i.e . infinite number of rollouts ) . However , when a model is estimated from the data and computational power is bounded , prior works has suggested how to adjust planning horizon to get the best computational performance ( Jiang et al. , 2015 ) . As of our knowledge there is no work that additionally investigate the performance of different exploration methods with approximate planning . We are interested in how to leverage models to support deep efficient exploration in large domains . Prior research has strong guarantees to find a near optimal policy with deep exploration when exact planning can be done , often in small state space ( Brunskill et al. , 2009 ; Brafman & Tennenholtz , 2002 ) . In contrast , in this work , we investigate how to leverage models to support deep exploration when exact planning is not possible ( large state space ) . We propose a new optimistic MCTS to support deep exploration guided by models . Particularly , we suggest to perform a MCTS algorithm ( e.g . UCT ( Kocsis & Szepesvári , 2006 ) ) , and use the learned reward R and transition model T that is optimistic toward unseen or less frequently seen part of the state-action space ( we further discuss how to learn simple models for planning in section 4 ) . Concretely , while performing rollouts using learned models at each step an optimistic reward bonus is given . Optimistic reward bonus can be given by any optimism in the face of uncertainty ( OFU ) algorithm ( e.g . MBIE-EB ( Strehl & Littman , 2008 ) or Rmax ( Brafman & Tennenholtz , 2002 ) ) . Optimistic MCTS may have some advantages over alternate ways to achieve deep exploration in large state spaces given limited computation . In particular optimistic MCTS is leveraging model uncertainty to drive deep exploration , in contrast to policy search methods with simulated models ( Sutton et al. , 2000 ) that relies on the stochasticity of the policy for exploration and it is unclear how to leverage model uncertainty . Additionally , MCTS with posterior sampling methods , like Thompson Sampling ( Thompson , 1933 ) ( by sampling a model from posterior distribution and performing rollouts ) has strong guarantees when exact planning can be done , might not be optimistic enough for efficient exploration . With limited number of rollouts , agent might not observe the optimistic part of the model , in contrast to optimistic MCTS where optimism is built into every node of the tree . To investigate the impact of approximate planning on deep exploration , we compare optimistic MCTS , with Thomson Sampling , and BAMCP ( Guez et al. , 2012 ) ( a tractable sample-based method for approximate Bayes-optimal planning ) in two toy domains , Pong Prime and mini-Pitfall ! . 3.1 EXPERIMENTS . In order to test our hypotheses on the impact of imperfect model-based planning on deep exploration we introduced two toy environments , PongPrime and mini-Pitfall ! that are similar to general Atari games but in smaller scale . Pong Prime environment is designed for a hard exploration task . Dynamics of this game is similar to Atari2600 ’ s Pong environment ( Bellemare et al. , 2013 ) with minor tweaks that make the game significantly harder . The enemy paddle is made 3 times larger than the player paddle . Additionally , the top and bottom 10 % percent of the enemy paddle hit the ball back at 1.5 times the normal speed . Similarly , the player paddle also consists of 3 regions with distinct behavior . The top region of the paddle hits the ball back at 1.5 times speed . The middle region hits the ball back at normal speed . Finally , the lower region covers 5 % of the paddle and instantly wins a point for the player . This configuration is set up so that it is difficult but not impossible for the player to score using the top region ( scoring on average around 5 % of the time the ball bounces off the top region ) . In this setting , the optimal policy is to always hit the ball with the lower region of the ball . The game is deterministic and model free methods with -greedy exploration ( e.g . DQN ) consistently loses the game with lowest possible score across 5000 episodes . Figure1 ( a ) shows this environment . mini-Pitfall ! is a small version of Atari2600 game Pitfall ! which we use as a final test bed of our algorithm . In this version we limit the agent to two rooms of the game ( the initial room , room 0 , that agents starts the game in , and the room on the left side , room -1 ) . There exist a dummy reward Rmax at the right end of the room 0 , and the left end of room -1 is a terminal state ( underground connection is also terminal state ) . Figure 2 ( a ) shows this environment . We provided the right model class for both experiments so we can separate the effect of exploration from model mismatch . Figure 1 ( b ) compares the performance of different exploration strategies to the baseline ( which uses a MLE model with UCT algorithm ) combining with UCT algorithm in Pong Prime domain . We perform 500 total tree searches for all runs in Figure 1 ( b ) . Additionally , Table 1 shows the performance of different exploration methods in achieving the reward on the right side of the room 0 in mini-Pitfall ! consistently .

Summary: The paper proposes to learn (transition) models (for MDPs) in terms of objects and their interactions. These models are effectively deterministic and are compatible with algorithms for planning with count-based exploration. The paper demonstrates the performance of one such planning method in toy tasks and in Pitfall, as well as a comparison with other planning methods in the toy tasks. The proposed model-based method, called SOORL, yields agents that perform better on Pitfall with a small amount of data.