Hierarchical Reinforcement Learning using Spatio-Temporal Abstractions and Deep Neural Networks

This paper introduces an automated skill acquisition framework in reinforcement learning which involves identifying a hierarchical description of the given task in terms of abstract states and extended actions between abstract states. Identifying such structures present in the task provides ways to simplify and speed up reinforcement learning learning algorithms. These structures also help to generalize such algorithms over multiple tasks without relearning policies from scratch. We use ideas from dynamical systems to find metastable regions in the state space and associate them with abstract states. The spectral clustering algorithm PCCA+ is used to identify suitable abstractions aligned to the underlying structure. Skills are defined in terms of the transitions between such abstract states. The connectivity information from PCCA+ is used to generate these skills or options. The skills are independent of the learning task and can be efficiently reused across a variety of tasks defined over a common state space. Another major advantage of the approach is that it does not need a prior model of the MDP and can work well even when the MDPs are constructed from sampled trajectories. Finally, we present our attempts to extend the automated skills acquisition framework to complex tasks such as learning to play video games where we use deep learning techniques for representation learning to aid our spatio-temporal abstraction framework. Proceedings of the 33 rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48. Copyright 2016 by the author(s). 1. Motivation and Introduction The core idea of hierarchical reinforcement learning is to break down the reinforcement learning problem into subtasks through a hierarchy of abstractions. Typically, in the full reinforcement learning problem, the agent is assumed to be in one state of the Markov Decision Process at every time step. The agent then performs one of several possible primitive actions. Based on the agent’s state at time t, and the action it takes from that state, the agent’s state at time t + 1 is determined. For large problems, however, this can lead to too much granularity: when the agent has to decide on each and every primitive action at every granular state, it can often lose sight of the bigger picture. However, if a series of actions can be abstracted out as an abstract action, the agent can just remember the series of actions that was useful in getting it to a temporally distant useful state from the initial state. This is typically referred to as an option or a skill in the reinforcement learning literature. A good analogy is a human planning his movement for a traversal from current location A to a destination B. We identify intermediate destinations Ci to lead us fromA toB when planning fromA, instead of worrying about the exact mechanisms of immediate movement at A which are abstracted over. Options are a convenient way of formalising this abstraction. In keeping with the general philosophy of reinforcement learning, we want to build agents that can automatically discover options with no prior knowledge, purely by exploring the environment. Thus, our approach falls into the broad category of automated discovery of skills. In order to exploit task structure, hierarchical decomposition introduces models defined by stand-alone policies (also known as temporally-extended actions, options, or skills) that can take multiple time steps to execute. Skills can exploit representing structure by representing subroutines that are executed multiple times during execution of a task. Such skills which are learnt in one task can be reused ar X iv :1 60 5. 05 35 9v 1 [ cs .L G ] 1 7 M ay 2 01 6 Submission and Formatting Instructions for ICML 2016 in a different task as long as it requires execution of the same subroutine. Options also make exploration more efficient by providing the decision maker with a high-level behaviour to look ahead to the completion of the corresponding subroutine. Automated discovery of skills or options has been an active area of research and several approaches have been proposed for the same. The current methods could be broadly classified into sample trajectory based and partition based methods. Some of them are: • Identifying bottlenecks in the state space, where the state space is partitioned into sets and the transitions between two sets of states that are rare can be seen as introducing bottleneck sets at the respective points of such rare transitions. Policies to reach such states are cached as options (McGovern & Barto, 2001). • Using the structure present in a factored state representation to identify sequences of actions that cause what are otherwise infrequent changes in the state variables: these sequences are cached away as options (Hengst, 2004). • Obtaining a graphical representation of an agent’s interaction with its environment and using betweenness centrality measures to identify subtasks (Simsek & Barto, 2008). • Using clustering methods (spectral or otherwise) to separate out different strongly connected components of the Markov Decision Process (MDP) and identifying access-states that connect different clusters (Menache et al., 2002). While these methods have had varying amounts of success, they have certain deficiencies. Bottleneck based approaches don’t have a natural way of identifying the part of the state space where options are applicable without external knowledge about the problem domain. Spectral methods need some form of regularization in order to prevent unequal splits that might lead to arbitrary splitting of the state space. We present a framework that detects well-connected or meta stable regions of the state space from a MDP model estimated from trajectories. We use PCCA+, a spectral clustering algorithm from conformal dynamics (Weber et al., 2004) that not only partitions the MDP but also returns the connectivity information between the regions. We then propose a very effective way of composing options using the same framework to take us from one metastable region to another, giving us the policy for free. Once we have these options, we can use standard reinforcement learning algorithms to learn a policy over subtasks to solve the given task. Specifically, we show results using SMDP Qlearning on the 2-room domain. For our attempt at extending it to higher dimensional state space tasks such as Atari 2600 video games, we append the learnt options to the set of primitive actions using Intra-Option Value learning to learn a policy solving the given task. One major advantage of the approach is that we get the policy for the options for free while doing the partitioning by exploiting the membership functions returned by PCCA+. Our approach is able to learn reasonably good skills even with limited sampling which makes it useful in situations where exploration is limited by the environment costs. It also provides a way to refine the abstractions in an online fashion without explicitly reconstructing the entire MDP. More importantly, we extend it to the case where the state space is so large that exact modeling is not possible. In this case, we take inspiration from the recent work on forward prediction to learn the model (Oh et al., 2015) to use Deep Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) to learn spatio-temporal representations of the state space. Using the learnt representation, we perform state-aggregation using clustering techniques and estimate our transitional model for the abstract space on these aggregated states. We list the advantages of this approach below: • Skills are acquired online, from sampled trajectories instead of requiring a prior model of the MDP. • Instead of looking for bottleneck states, we look for well connected regions and hence, the discovered options are better aligned to the structure of the state space. • The approach returns connectivity information between the metastable regions which can be used to construct an abstract graph of the state space, combining spatial and temporal information meaningfully. • The clustering algorithm provides a fuzzy membership for every state in belonging to a particular metastable region, which provides a powerful way to compose options naturally. We organize the rest of the paper as follows: We first explain the Option Generation Framework that we propose, by delving on the important aspects of the spectral clustering algorithm PCCA+ and how PCCA+ can be used to generate options. We show results on the 2-room domain for this framework. The next part of the paper focuses on our attempt to extend this framework for more complex tasks such as playing video games like Seaquest on the Atari 2600 domain with options. We explain the motivation and usage of deep networks, the clustering algorithms used for state-aggregation, followed by the model used and Submission and Formatting Instructions for ICML 2016 initial results. We then conclude with a discussion on the challenges in this approach and also comparison with other recent attempts at Hierarchical reinforcement learning for higher dimensional tasks. 2. Option Generation Framework We divide this section into two parts: We first explain the spectral clustering algorithm PCCA+ and motivate its usage for spatial abstraction. The second part discusses the option generation using PCCA+. 2.1. Spatial Abstraction using PCCA+ Given an algebraic representation of the graph representing a MDP we want to find suitable abstractions aligned to the underlying structure. We use a spectral clustering algorithm to do this. Central to the idea of spectral clustering is the graph Laplacian which is obtained from the similarity graph. There are many tight connections between the topological properties of graphs and the graph Laplacian matrices, which spectral clustering methods exploit to partition the data into clusters. However, although the spectra