Learning Abstract Models for Long-Horizon Exploration

1 INTRODUCTION . Exploration is a key bottleneck in high-dimensional , sparse-reward reinforcement learning tasks . Random exploration ( e.g. , via epsilon-greedy ) suffices when rewards are abundant ( Mnih et al. , 2015 ) , but when rewards are sparse , it can be difficult for an agent starting out to even find any positive reward needed to bootstrap learning . For example , the infamously difficult game MONTEZUMA ’ S REVENGE from the Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ) contains over 10100 states and requires the agent to go thousands of timesteps without receiving reward . Performing effective exploration in this setting is thus an open problem ; without demonstrations , even state-of-the-art intrinsically-motivated RL agents ( Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Tang et al. , 2017 ) achieve only about one-tenth the score of an expert human ( Hester et al. , 2018 ) . In this paper , we investigate model-based reinforcement learning ( Kearns & Singh , 2002 ) as a potential solution to the exploration problem . The hope is that with a model of the state transitions and rewards , one can perform planning under the model to obtain a more informed exploration strategy . However , as the model being learned is imperfect , errors in the model compound ( Talvitie , 2014 ; 2015 ) when planning over many time steps . Furthermore , even if a perfect model were known , in high-dimensional state spaces ( e.g . over 10100 states ) , planning—computing the optimal policy ( e.g . via value iteration ) —is intractable . As a result , model-based RL has had limited success in high-dimensional settings ( Talvitie , 2014 ) . To address this , some prior work has focused on learning more accurate models by using more expressive function approximators ( Nagabandi et al. , 2018 ) , and learning local models ( Levine & Koltun , 2013 ; Zhang et al. , 2018 ) . Others have attempted to robustly use imperfect models by conditioning on , instead of directly following , model-based rollouts ( Weber et al. , 2017 ) , frequently replanning , and combining model-based with model-free approaches ( Abbeel et al. , 2006 ; Sutton , 1990 ) . However , none of these techniques offer a fundamental solution . Instead of directly learning a model over the concrete state space , we propose an approach inspired by hierarchical reinforcement learning ( HRL ) ( Sutton et al. , 1999 ; Vezhnevets et al. , 2017 ; Oh et al. , 1 Videos of our trained agent : https : //sites.google.com/view/abstract-models/home 2017 ) , and learn a model over a much smaller abstract state space . Specifically , we assume we have a state abstraction function ( Li et al. , 2006 ; Singh et al. , 1995 ; Dietterich , 1998 ) , which maps a highdimensional concrete state ( e.g . all pixels on the screen ) to a low-dimensional abstract state ( e.g . the position of the agent ) . We then aim to learn an ( abstract ) Markov Decision Process ( MDP ) over this abstract state space as follows : A manager maintains an ( abstract ) MDP over a subset of all possible abstract states which we call the known set , which is grown over time . The crucial property we enforce is that this abstract MDP is highly accurate and near deterministic on the known set , so we can perform planning without suffering from compounding errors , and do it efficiently since we are working with a much smaller number of abstract states . Concurrently , we learn a worker policy that the manager uses to transition between abstract states . The worker policy has access to the concrete states ; its goal is to hide the messy details of the real world from the manager ( e.g. , jumping over monsters ) so that the manager has a much simpler planning problem ( e.g. , traversing between two locations ) . In our implementation , the worker keeps an inventory of skills ( i.e. , options ( Sutton et al. , 1999 ) ) , each of which is driven by a deep neural network ; the worker assigns an appropriate skill for each transition between abstract states . In this way , the worker does not “ forget ” ( Kirkpatrick et al. , 2017 ) , and we ensure monotonic progress in learning the abstract MDP . This abstract MDP , which enables us to efficiently explore via planning , is a key difference between our work and previous HRL work ( e.g. , ( Bacon et al. , 2017 ; Vezhnevets et al. , 2017 ) ) , which also learn skills and operate on latent abstract state spaces but without forming an MDP . We evaluate our approach on three of the most challenging games from the ALE ( Bellemare et al. , 2013 ) : MONTEZUMA ’ S REVENGE , PITFALL ! , and PRIVATE EYE . In all three domains , our approach achieves more than 2x the reward of prior non-demonstration state-of-the-art approaches . In PITFALL ! , we are the first to achieve superhuman performance without demonstrations , surpassing the prior state-of-the-art by over 100x . Additionally , since our approach is model-based , we can generalize to new rewards without re-training , as long as the reward function is a function of the abstract states . When evaluated on a new reward function never seen during training , our approach achieves over 3x the reward of prior state-of-the-art methods explicitly trained on the new rewards . 2 APPROACH OVERVIEW . We assume the world is an unknown episodic finite-horizon MDP with ( concrete ) states x ∈ X and actions a ∈ A . We further assume we have a simple predefined state abstraction function mapping concrete states x to abstract states s = φ ( x ) . In MONTEZUMA ’ S REVENGE , for instance , a concrete state contains the pixels on the screen , while the corresponding abstract state contains the agent ’ s position and inventory ( Figure 1 ) . We assume that reward only depends on the abstract states : taking any action transitioning from concrete state x to x′ leads to reward R ( φ ( x ) , φ ( x′ ) ) . Model-based approaches promise better exploration via planning , but struggle in high-dimensional state spaces due to compounding errors and computational intractability . To avoid these problems , we propose to construct and operate on a low-dimensional representation of the world consisting of abstract states , which we call the abstract MDP ( we refer to the original MDP , the world , as the concrete MDP ) . Then we plan in the abstract MDP . Concretely , the abstract MDP consists of : • The state space is a subset of all abstract states , which we call the known set S , consisting of abstract states that can be reliably reached from the initial abstract state via actions in the action set . Over time , we monotonically grow the known set , which initially only contains the starting abstract state . • The action set comprises of calling the worker policy πw ( a|x , ( s , s′ ) ) on transitions ( s , s′ ) from the current abstract state s to a nearby abstract state s′ . When called on a transition ( s , s′ ) , the worker navigates from s to s′ by taking concrete actions a conditioned on the current concrete state x . The worker abstracts away the messiness of the underlying concrete MDP so that all other parts of the system can operate on the abstract MDP by calling the worker . We denote calling the worker on transition ( s , s′ ) as the action go ( s , s′ ) . • The transition dynamics of calling action go ( s , s′ ) at abstract state s are defined by the original MDP : i.e. , if the worker takes a concrete trajectory x0 , a0 , x1 , a1 , · · · , xT , the resulting abstract state is φ ( xT ) . The rewards for transitioning from s to s′ are the rewards in the concrete MDP R ( s , s′ ) , which only depend on the abstract states by assumption . The core idea behind our approach is to construct the abstract MDP ( Section 3 ) by growing the action set ( training the worker ) , which in turn grows the known set . At each point in time , the manager maintains the known set and ( accurate , to avoid compounding errors ) estimates of the reward and transition dynamics of the abstract MDP . With these dynamics estimates , the manager can solve the abstract MDP at all times ( e.g. , by value iteration ) , since the abstract MDP is small . As the abstract MDP grows , it captures more and more of the concrete MDP , enabling the manager to recover a better and better policy via planning . Ultimately , the known set of the abstract MDP contains all abstract states , enabling the manager to recover a high-reward policy . As the manager constructs the abstract MDP , the abstract MDP maintains two key properties : • The action set of the abstract MDP consists of only reliable actions , actions go ( s , s′ ) that transition from abstract state s to abstract state s′ with probability at least 1 − δ for some small δ to avoid compounding uncertainty . This enables the manager to reach any abstract state in the known set with high probability . To simplify notation , the manager estimates the success rate P ( s , s′ ) of action go ( s , s′ ) instead of the full dynamics P ( •|go ( s , s′ ) , s ) , treating the ( vanishingly small ) fraction of failures equally . • The action set and known set of the abstract MDP grow monotonically . Since the action set only contains reliable transitions , a key danger is if learning new reliable transitions ( adding new actions ) causes the worker to forget already learned transitions ( removing actions ) , stalling progress . We opt for a non-parametric approach , where the worker learns a skill ( neural subpolicy ) for each transition , reusing skills when possible . When a worker learns to reliably traverse a transition , it freezes the corresponding skill ’ s parameters . 3 CONSTRUCTING THE ABSTRACT MDP . The manager ’ s goal is to fully construct the abstract MDP so that the known set contains all abstract states . Then , it can compute a high-reward policy on the concrete MDP via planning on the abstract MDP . To construct the abstract MDP , the manager adds new actions to the abstract MDP : training the worker to reliably traverse new transitions ( driving the transition success rates toward 1 ) . Concretely , the manager discovers new transitions , trains the worker on these transitions , and updates its dynamics estimates using Algorithm 1 . On each episode , the manager either chooses to discover new transitions via randomized exploration ( Section 3.1 ) or trains the worker . This is done by constructing a prioritized list of exploration goals , where each goal is either a transition to learn Algorithm 1 MANAGER 1 : while abstract MDP not fully constructed do 2 : Compute a set of candidate exploration goals C 3 : ( c ∈ C is either a transition ( s , s′ ) or an abstract state s ) 4 : Score all candidates and select highest priority candidate c 5 : Compute a plan ( s0 , s1 ) , ( s1 , s2 ) , · · · , ( sT−1 , sT = s ) with model 6 : for t = 1 to T do 7 : Call worker to navigate transition ( st−1 , st ) 8 : if c is a transition ( s , s′ ) to learn then 9 : LEARNWORKER ( s , s′ ) 10 : else c is an abstract state s to explore 11 : DISCOVERTRANSITIONS ( ) or an abstract state to find nearby transitions ( Section 3.2 ) . Upon selecting the highest-priority goal ( e.g. , the transition ( s , s′ ) ) , the manager navigates to the relevant abstract state ( e.g. , s ) by planning with its dynamics models ( e.g. , the plan go ( s0 , s1 ) , go ( s1 , s2 ) , · · · , go ( sT−1 , sT = s ) ) , executing the plan , and then calling the worker on or randomly exploring from the selected goal . Finally , the manager updates its dynamics models . It uses a sliding window estimate of the past Ntransition worker attempts of traversing transition ( s , s′ ) for the transition dynamics , and updates its reward estimate of a transition ( s , s′ ) as the reward accumulated by the first successful traversal of ( s , s′ ) . When a transition ( s , s′ ) becomes reliable ( i.e. , the dynamics T ( s , s′ ) exceeds the threshold 1− δ ) , the manager adds go ( s , s′ ) to the action set of the abstract MDP and adds s′ to the known set .

Summary: This paper considers how to effectively perform exploration in the setting where a difficult, high-dimensional MDP can be mapped to a simpler, lower-dimensional MDP. They propose a hierarchical approach where a model of the abstract MDP is incrementally learned, and then used to train sub-policies to transition between abstract states. These sub-policies are trained using intrinsic rewards for transitioning to the correct state, and the transition probabilities in the abstract MDP reflect how well a sub-policy can perform the transition.