Learning Abstract Models for Long-Horizon Exploration

1 INTRODUCTION . Exploration is a key bottleneck in high-dimensional , sparse-reward reinforcement learning tasks . Random exploration ( e.g. , via epsilon-greedy ) suffices when rewards are abundant ( Mnih et al. , 2015 ) , but when rewards are sparse , it can be difficult for an agent starting out to even find any positive reward needed to bootstrap learning . For example , the infamously difficult game MONTEZUMA ’ S REVENGE from the Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) contains over 10100 states and requires the agent to go thousands of timesteps without receiving reward . Performing effective exploration in this setting is thus an open problem ; without demonstrations , even state-of-the-art intrinsically-motivated RL agents ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Tang et al. , 2017 ) achieve only about one-tenth the score of an expert human ( Hester et al. , 2018 ) . In this paper , we investigate model-based reinforcement learning ( Kearns & Singh , 2002 ) as a potential solution to the exploration problem . The hope is that with a model of the state transitions and rewards , one can perform planning under the model to obtain a more informed exploration strategy . However , as the model being learned is imperfect , errors in the model compound ( Talvitie , 2014 ; 2015 ) when planning over many time steps . Furthermore , even if a perfect model were known , in high-dimensional state spaces ( e.g . over 10100 states ) , planning—computing the optimal policy ( e.g . via value iteration ) —is intractable . As a result , model-based RL has had limited success in high-dimensional settings ( Talvitie , 2014 ) . To address this , some prior work has focused on learning more accurate models by using more expressive function approximators ( Nagabandi et al. , 2018 ) , and learning local models ( Levine & Koltun , 2013 ; Zhang et al. , 2018 ) . Others have attempted to robustly use imperfect models by conditioning on , instead of directly following , model-based rollouts ( Weber et al. , 2017 ) , frequently replanning , and combining model-based with model-free approaches ( Abbeel et al. , 2006 ; Sutton , 1990 ) . However , none of these techniques offer a fundamental solution . Instead of directly learning a model over the concrete state space , we propose an approach inspired by hierarchical reinforcement learning ( HRL ) ( Sutton et al. , 1999 ; Vezhnevets et al. , 2017 ; Oh et al. , 1 Videos of our trained agent : https : //sites.google.com/view/abstract-models/home 2017 ) , and learn a model over a much smaller abstract state space . Specifically , we assume we have a state abstraction function ( Li et al. , 2006 ; Singh et al. , 1995 ; Dietterich , 1998 ) , which maps a highdimensional concrete state ( e.g . all pixels on the screen ) to a low-dimensional abstract state ( e.g . the position of the agent ) . We then aim to learn an ( abstract ) Markov Decision Process ( MDP ) over this abstract state space as follows : A manager maintains an ( abstract ) MDP over a subset of all possible abstract states which we call the known set , which is grown over time . The crucial property we enforce is that this abstract MDP is highly accurate and near deterministic on the known set , so we can perform planning without suffering from compounding errors , and do it efficiently since we are working with a much smaller number of abstract states . Concurrently , we learn a worker policy that the manager uses to transition between abstract states . The worker policy has access to the concrete states ; its goal is to hide the messy details of the real world from the manager ( e.g. , jumping over monsters ) so that the manager has a much simpler planning problem ( e.g. , traversing between two locations ) . In our implementation , the worker keeps an inventory of skills ( i.e. , options ( Sutton et al. , 1999 ) ) , each of which is driven by a deep neural network ; the worker assigns an appropriate skill for each transition between abstract states . In this way , the worker does not “ forget ” ( Kirkpatrick et al. , 2017 ) , and we ensure monotonic progress in learning the abstract MDP . This abstract MDP , which enables us to efficiently explore via planning , is a key difference between our work and previous HRL work ( e.g. , ( Bacon et al. , 2017 ; Vezhnevets et al. , 2017 ) ) , which also learn skills and operate on latent abstract state spaces but without forming an MDP . We evaluate our approach on three of the most challenging games from the ALE ( Bellemare et al. , 2013 ) : MONTEZUMA ’ S REVENGE , PITFALL ! , and PRIVATE EYE . In all three domains , our approach achieves more than 2x the reward of prior non-demonstration state-of-the-art approaches . In PITFALL ! , we are the first to achieve superhuman performance without demonstrations , surpassing the prior state-of-the-art by over 100x . Additionally , since our approach is model-based , we can generalize to new rewards without re-training , as long as the reward function is a function of the abstract states . When evaluated on a new reward function never seen during training , our approach achieves over 3x the reward of prior state-of-the-art methods explicitly trained on the new rewards . 2 APPROACH OVERVIEW . We assume the world is an unknown episodic finite-horizon MDP with ( concrete ) states x ∈ X and actions a ∈ A . We further assume we have a simple predefined state abstraction function mapping concrete states x to abstract states s = φ ( x ) . In MONTEZUMA ’ S REVENGE , for instance , a concrete state contains the pixels on the screen , while the corresponding abstract state contains the agent ’ s position and inventory ( Figure 1 ) . We assume that reward only depends on the abstract states : taking any action transitioning from concrete state x to x′ leads to reward R ( φ ( x ) , φ ( x′ ) ) . Model-based approaches promise better exploration via planning , but struggle in high-dimensional state spaces due to compounding errors and computational intractability . To avoid these problems , we propose to construct and operate on a low-dimensional representation of the world consisting of abstract states , which we call the abstract MDP ( we refer to the original MDP , the world , as the concrete MDP ) . Then we plan in the abstract MDP . Concretely , the abstract MDP consists of : • The state space is a subset of all abstract states , which we call the known set S , consisting of abstract states that can be reliably reached from the initial abstract state via actions in the action set . Over time , we monotonically grow the known set , which initially only contains the starting abstract state . • The action set comprises of calling the worker policy πw ( a|x , ( s , s′ ) ) on transitions ( s , s′ ) from the current abstract state s to a nearby abstract state s′ . When called on a transition ( s , s′ ) , the worker navigates from s to s′ by taking concrete actions a conditioned on the current concrete state x . The worker abstracts away the messiness of the underlying concrete MDP so that all other parts of the system can operate on the abstract MDP by calling the worker . We denote calling the worker on transition ( s , s′ ) as the action go ( s , s′ ) . • The transition dynamics of calling action go ( s , s′ ) at abstract state s are defined by the original MDP : i.e. , if the worker takes a concrete trajectory x0 , a0 , x1 , a1 , · · · , xT , the resulting abstract state is φ ( xT ) . The rewards for transitioning from s to s′ are the rewards in the concrete MDP R ( s , s′ ) , which only depend on the abstract states by assumption . The core idea behind our approach is to construct the abstract MDP ( Section 3 ) by growing the action set ( training the worker ) , which in turn grows the known set . At each point in time , the manager maintains the known set and ( accurate , to avoid compounding errors ) estimates of the reward and transition dynamics of the abstract MDP . With these dynamics estimates , the manager can solve the abstract MDP at all times ( e.g. , by value iteration ) , since the abstract MDP is small . As the abstract MDP grows , it captures more and more of the concrete MDP , enabling the manager to recover a better and better policy via planning . Ultimately , the known set of the abstract MDP contains all abstract states , enabling the manager to recover a high-reward policy . As the manager constructs the abstract MDP , the abstract MDP maintains two key properties : • The action set of the abstract MDP consists of only reliable actions , actions go ( s , s′ ) that transition from abstract state s to abstract state s′ with probability at least 1 − δ for some small δ to avoid compounding uncertainty . This enables the manager to reach any abstract state in the known set with high probability . To simplify notation , the manager estimates the success rate P ( s , s′ ) of action go ( s , s′ ) instead of the full dynamics P ( •|go ( s , s′ ) , s ) , treating the ( vanishingly small ) fraction of failures equally . • The action set and known set of the abstract MDP grow monotonically . Since the action set only contains reliable transitions , a key danger is if learning new reliable transitions ( adding new actions ) causes the worker to forget already learned transitions ( removing actions ) , stalling progress . We opt for a non-parametric approach , where the worker learns a skill ( neural subpolicy ) for each transition , reusing skills when possible . When a worker learns to reliably traverse a transition , it freezes the corresponding skill ’ s parameters . 3 CONSTRUCTING THE ABSTRACT MDP . The manager ’ s goal is to fully construct the abstract MDP so that the known set contains all abstract states . Then , it can compute a high-reward policy on the concrete MDP via planning on the abstract MDP . To construct the abstract MDP , the manager adds new actions to the abstract MDP : training the worker to reliably traverse new transitions ( driving the transition success rates toward 1 ) . Concretely , the manager discovers new transitions , trains the worker on these transitions , and updates its dynamics estimates using Algorithm 1 . On each episode , the manager either chooses to discover new transitions via randomized exploration ( Section 3.1 ) or trains the worker . This is done by constructing a prioritized list of exploration goals , where each goal is either a transition to learn Algorithm 1 MANAGER 1 : while abstract MDP not fully constructed do 2 : Compute a set of candidate exploration goals C 3 : ( c ∈ C is either a transition ( s , s′ ) or an abstract state s ) 4 : Score all candidates and select highest priority candidate c 5 : Compute a plan ( s0 , s1 ) , ( s1 , s2 ) , · · · , ( sT−1 , sT = s ) with model 6 : for t = 1 to T do 7 : Call worker to navigate transition ( st−1 , st ) 8 : if c is a transition ( s , s′ ) to learn then 9 : LEARNWORKER ( s , s′ ) 10 : else c is an abstract state s to explore 11 : DISCOVERTRANSITIONS ( ) or an abstract state to find nearby transitions ( Section 3.2 ) . Upon selecting the highest-priority goal ( e.g. , the transition ( s , s′ ) ) , the manager navigates to the relevant abstract state ( e.g. , s ) by planning with its dynamics models ( e.g. , the plan go ( s0 , s1 ) , go ( s1 , s2 ) , · · · , go ( sT−1 , sT = s ) ) , executing the plan , and then calling the worker on or randomly exploring from the selected goal . Finally , the manager updates its dynamics models . It uses a sliding window estimate of the past Ntransition worker attempts of traversing transition ( s , s′ ) for the transition dynamics , and updates its reward estimate of a transition ( s , s′ ) as the reward accumulated by the first successful traversal of ( s , s′ ) . When a transition ( s , s′ ) becomes reliable ( i.e. , the dynamics T ( s , s′ ) exceeds the threshold 1− δ ) , the manager adds go ( s , s′ ) to the action set of the abstract MDP and adds s′ to the known set .

Summary: This paper considers reinforcement learning tasks that have high-dimensional space, long-horizon time, sparse-rewards. In this setting, current reinforcement learning algorithms struggle to train agents so that they can achieve high rewards. To address this problem, the authors propose an abstract MDP algorithm. The algorithm consists of three parts: manager, worker, and discoverer. The manager controls the exploration scheduling, the worker updates the policy, and the discoverer purely explores the abstract states. Since there are too many state, the abstract MDP utilize the RAM state as the corresponding abstract state for each situation.